A system for monitoring cargo

Abstract

To provide a system and method for tracking cargo. [Solution] The tracking system 15 includes a wireless system and a vision system, each comprising a tag connected to the cargo and emitting identification data, and a plurality of locators 25 connected to the vehicle and receiving the identification data emitted from the tag. The vision system includes a plurality of cameras 41 positioned inside the vehicle and capturing images of the cargo. The control unit 50 comprises a processor circuit 51, which identifies the cargo and its location based on signals transmitted from the tag and received by the plurality of locators, and tracks the cargo's location based on images captured by the vision system.

Description

Technical Field

[0001]

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 684,939, filed August 20, 2024, which is hereby incorporated by reference in its entirety.

[0002]

[0002] The present disclosure broadly relates to the field of cargo handling, and more particularly to systems that use both wireless and visual systems to identify cargo and monitor its movement.

Background Art

[0003]

[0003] A wide variety of vehicles are used to transport cargo. For example, but not limited to, airplanes, ocean liners, and trucks. The transportation process generally includes loading cargo onto a vehicle, placing the cargo within the vehicle, transporting the cargo from a first location to a second location, and then unloading the cargo. During the transportation process, there is a need to identify and monitor the cargo.

[0004]

[0004] Existing systems provide various ways to identify the cargo loaded onto a vehicle. However, these systems are often inaccurate. This is because these systems cannot identify the cargo during handling, cannot accurately monitor the position of the cargo during handling, and / or cannot identify the position of the cargo on the vehicle. This can result in the cargo being misidentified and / or improperly loaded onto the vehicle. In some cases, such as with airplanes, improperly loaded airplanes may need to have the cargo unloaded and then reloaded in the correct position prior to flight to ensure that the weight is properly balanced.

[0005]

[0005] Some existing systems require operators to visually inspect and identify cargo. However, visual identification of cargo has been found to be inaccurate, because operators are often unable to accurately identify cargo or properly enter the cargo identification information into the monitoring software. Furthermore, this can be expensive, as it requires one or more operators to identify and enter the identification information into the monitoring software. This process is also time-consuming, which can slow down the loading process and lead to delays in transportation. [Overview of the Initiative]

[0006]

[0006] One embodiment relates to a method for tracking cargo. The method includes receiving a radio signal from a tag on the cargo, identifying the cargo based on the radio signal, locating the cargo based on the radio signal, and capturing images of the cargo over a period of time and monitoring the cargo's location based on the images as the cargo moves through an area.

[0007]

[0007] In another embodiment, the method further includes attaching a tag to the cargo before receiving a radio signal from the cargo tag.

[0008]

[0008] In another embodiment, the method further includes capturing an image of the cargo after locating the cargo based on a radio signal.

[0009]

[0009] In another embodiment, the method further includes identifying the cargo based on identification data contained in the radio signal.

[0010]

[0010] In another embodiment, the method further includes monitoring the location of the cargo based on a radio signal and an image simultaneously.

[0011]

[0011] In another embodiment, the method further includes determining, based on the image, that the cargo has stopped moving within the area, and identifying the final position of the cargo as the point where the cargo was located when it stopped moving.

[0012]

[0012] In another embodiment, the method further includes identifying the cargo and locating the cargo while it is being loaded onto the aircraft.

[0013]

[0013] In another embodiment, the method further includes receiving radio signals at a plurality of locators and determining the location of the cargo based on the signal strength of the radio signals received at the plurality of locators.

[0014]

[0014] One embodiment relates to a method for tracking cargo loaded onto a vehicle. The method includes receiving a radio signal at one or more of a plurality of locators while the radio signal is being emitted from a tag attached to the cargo; identifying the cargo based on the radio signal; receiving the radio signal at the plurality of locators and tracking the location of the cargo based on the radio signal as the cargo is moving within the vehicle; capturing an image of the cargo as the cargo is moving within the vehicle; and tracking the location of the cargo within the vehicle based on the image.

[0015]

[0015] In another embodiment, the method further includes receiving radio signals at a plurality of locators and determining the location of the cargo based on the signal strength of the radio signals received at the plurality of locators.

[0016]

[0016] In another embodiment, the method further includes receiving radio signals in a plurality of locators mounted in fixed positions relative to the vehicle.

[0017]

[0017] In another embodiment, receiving a radio signal in one or more of a plurality of locators includes receiving a BLE signal transmitted from a Bluetooth Low Energy (BLE) tag attached to the cargo.

[0018]

[0018] In another embodiment, the method further includes identifying points on the cargo based on an image and tracking the location of the cargo based on the points identified in the image.

[0019]

[0019] In another embodiment, the method further includes identifying the leading edge of the cargo based on an image and tracking the leading edge of the cargo as the cargo moves within a vehicle.

[0020]

[0020] One embodiment relates to a cargo tracking system. The system comprises a wireless system comprising a tag configured to be connected to cargo and configured to emit identification data, and a plurality of locators configured to be connected to a vehicle and configured to receive the identification data emitted from the tag. A vision system comprises a plurality of cameras positioned inside the vehicle and configured to capture images of the cargo. A control unit comprises a processing circuit configured to identify the cargo and track its location based on signals transmitted from the tag and received by the plurality of locators.

[0021]

[0021] In another embodiment, the wireless system is a Bluetooth low-energy system.

[0022]

[0022] In another embodiment, the control unit is configured to first identify the cargo and its location based on identification data received by one or more of the locators, and then track the location of the cargo based on images captured by the vision system.

[0023]

[0023] The above-described features, functions, and advantages can be realized individually in various embodiments or in combination in yet another set of embodiments, and these details can be found by referring to the following description and accompanying drawings. [Brief explanation of the drawing]

[0024] [Figure 1]

[0024] This is a schematic diagram of a cargo tracking system including a wireless system and a vision system. [Figure 2]

[0025] This is an isometric view of an aircraft equipped with a cargo tracking system. [Figure 3]

[0026] This is an isometric view of the cargo being loaded into the cargo compartment of the vehicle through the opening. [Figure 4]

[0027] This is a schematic diagram of a cargo compartment including an alignment area and a lane extending along the length. [Figure 5]

[0028] This is a schematic diagram of a cargo compartment including a lane extending along the length. [Figure 6]

[0029] This is a schematic diagram of a control unit configured to receive signals from locators and tags and output the calculated information. [Figure 7]

[0030] This is a flowchart of a method for identifying and tracking cargo. [Figure 8]

[0031] This is a flowchart of a method for identifying and tracking cargo. [Figure 9]

[0032] This is a flowchart of a method for identifying and tracking cargo. [Figure 10]

[0033] This is a schematic diagram of a control unit.

Embodiments for Carrying Out the Invention

[0025]

[0034] This disclosure relates to a cargo tracking system that identifies and tracks the location of cargo, including determining the final location of the cargo on a vehicle. In some embodiments, the final location may be a location in the cargo compartment of a vehicle or a location within a warehouse. As shown in Figure 1, the cargo tracking system 15 includes a radio system 20 and a visual system 40. The radio system 20 is configured to identify cargo 90 based on a tag 21 configured to be attached to the cargo 90. The tag 21 also enables the radio system 20 to track the location of the cargo 90. The visual system 40 visually tracks the location of the cargo 90, such as during handling during loading and unloading. The visual system 40 identifies the exact location where the cargo 90 is located, such as the location where the cargo 90 is located in the cargo compartment during transport. A control unit 50 identifies and locates the cargo 90 based on identification data and / or images. In some embodiments, the radio system 20 identifies the cargo 90 and locates the approximate location of the cargo 90. Once the approximate location is known, the visual system 40 tracks the location of the cargo 90 more accurately. The wireless system 20 and the visual system 40 can operate sequentially and / or in parallel.

[0026]

[0035] Figure 2 illustrates one application of a cargo tracking system 15 for cargo transported by a vehicle 100. In this embodiment, the vehicle 100 is an aircraft configured to transport cargo 90. The vehicle 100 generally includes a fuselage 101 having one or more doors 102. One or more doors 102 lead to a cargo compartment 103 inside the fuselage 101. The cargo compartment 103 includes a floor, ceiling, and side walls and is configured to hold the cargo 90 during transport. The cargo tracking system 15 is integrated into the vehicle 100 and is configured to locate and track the cargo 90 loaded on the vehicle 100.

[0027]

[0036] Figure 3 shows cargo 90 positioned on a platform 110 for loading into vehicle 100. A door 102 inside the vehicle's body 101 is in the open position for the cargo 90 to be moved into the cargo compartment 103 through an opening 104. The cargo 90 can be unloaded in a similar manner, and the cargo 90 from the cargo compartment 103 is moved through the opening 104 and loaded onto the platform 110.

[0028]

[0037] The cargo 90 can have various shapes and sizes. In one embodiment, the cargo 90 includes a unit load device (ULD). The ULD can include various configurations, several examples of which include, but are not limited to, pallets supporting smaller packages, as well as containers used to hold contents on aircraft with wide bodies or certain narrow bodies, and which are shaped and sized according to the dimensions of the cargo compartment 103. In another embodiment, the cargo 90 includes smaller containers (e.g., boxes, wooden crates). The smaller containers are placed on a pallet and held together with packaging material (e.g., mesh, plastic, packaging material). Figure 3 shows one embodiment in which a tag 21 is connected to a pallet and to one or more of the individual packages placed on the pallet.

[0029]

[0038] During loading, the cargo is moved into the cargo compartment 103 through door 102. Figure 4 shows the cargo compartment 103 and opening 104 inside the fuselage 101 where the cargo 90 is loaded onto the vehicle 100. The cargo compartment 103 includes an inner alignment area 106 from opening 104. The alignment area 106 is a place for aligning the cargo 90 along one of the lanes 105 that extend along the length of the cargo compartment 103. During loading, the cargo 90 is moved to the alignment area 106, aligned with one of the lanes 105, and then moved into the selected lane 105. The lane 105 is divided into bays 109 along its length, and each bay 109 is sized to hold one or more pieces of the cargo 90. The cargo 90 is moved along the lane 105 into one of the bays 109. In some embodiments, this includes moving the cargo 90 along lane 105 until it reaches the end of lane 105 or another cargo 90 already loaded in lane 105. Figure 4 includes one embodiment in which the cargo compartment 103 includes six lanes 105, and Figure 5 includes one embodiment having two lanes 105. The layout may include various numbers of lanes 105 extending along the cargo compartment 103.

[0030]

[0039] In some embodiments, the cargo 90 has an assigned location within the cargo compartment 103, such as a specific bay 109 on a specific lane 105. In some embodiments, the cargo 90 is loaded onto the vehicle 100 according to a Loading Instruction Report (LIR). The LIR is used by the operator loading the vehicle 100 to comply with weight and balance limits and provides instructions on where in the vehicle 100 the cargo 90 should be loaded and positioned. In some embodiments, the assigned location is determined to distribute the weight of the cargo 90. When the vehicle 100 is an aircraft, weight distribution is important for balancing the aircraft to ensure safe flight. In another embodiment where the vehicle is a ship, weight distribution helps keep the vehicle 100 stable on the water and reduces the risk of capsizing or uncontrollable swaying. The assigned location also facilitates the loading and unloading of the cargo 90, as the lane arrangement in the cargo compartment 103 employs a first-in, first-out (FILO) loading and unloading order. Accessing a specific piece of cargo 90 requires moving other pieces of cargo 90 that are positioned inward along the corresponding lane 105 (i.e., positioned between the desired cargo 90 and the opening 104). The assigned location of the cargo 90 is also important for monitoring exposure during transport. One or more environmental factors (e.g., temperature, humidity) may be monitored within the cargo compartment 103 and used to identify the exposure of the cargo 90.

[0031]

[0040] The wireless system 20 includes an identification tag 21 and a locator 25. The tag 21 emits unique identification data configured to be picked up by the locator 25. In some embodiments, the tag 21 is powered by a battery to emit the identification information to be detected by the locator 25.

[0032]

[0041] The tags 21 are configured to be attached to the cargo 90 such that each piece of cargo 90 has an individual tag 21. The tags 21 can be attached to the cargo 90 in a variety of ways, including, but not limited to, one or more fasteners, adhesives, and wires. In one embodiment, the cargo 90 is equipped with a container for receiving the tags 21. The tags 21 include identification data that identifies the cargo 90. In some embodiments, the data includes an alphanumeric code, such as a serial number, that identifies the cargo 90. Further or alternatively, the data 90 includes, but not limited to, a written description, the owner of the cargo, the destination, and the identification of the cargo (e.g., a cargo ID code), and other identification information and / or information.

[0033]

[0042] The locator 25 is configured to receive identification data from the tag 21. The locator 25 is positioned in the vicinity of the cargo 90 to allow the identification data to be read as the cargo 90 passes the locator 25. In some embodiments, the locator 25 is configured to be mounted on the vehicle 100. Further or alternatively, the locator 25 is configured to be mounted in close proximity to the vehicle 100, such as on a stand or mount for positioning near the opening 104 where the cargo 90 is loaded and unloaded.

[0034]

[0043] In some embodiments, as shown in Figures 3 and 4, one or more locators 25 are positioned in the opening 104 of the cargo compartment 103. This arrangement allows the radio system 20 to identify and locate the cargo 90 before and / or when the cargo 90 is loaded onto the vehicle 100. The locators 25 are also positioned along the cargo compartment 103. As shown in Figures 4 and 5, the locators 25 are spaced apart along the length of the lane 105.

[0035]

[0044] The control unit 50 receives data from the locator 25 and identifies and locates the position of the cargo 90. The control unit 50 can be located in various positions, including the cargo compartment 103 inside the vehicle 100 or a remote location outside the vehicle 100. In some embodiments, the control unit 50 is dedicated to the cargo tracking system 15. In several other embodiments, the control unit 50 is a component of another data processing system of the vehicle 100.

[0036]

[0045] The cargo tracking system 15 may use various technologies to identify and track the cargo 90. In some embodiments, the cargo tracking system 15 uses Bluetooth Low Energy (BLE). A tag 21 is a hardware transmitter that broadcasts identification data. A locator 25 is configured to receive identification data from the tag 21. In some embodiments, multiple locators 25 are positioned in the vehicle 100 to receive identification data. With multiple locators 25 positioned in the cargo compartment 103, trilateration or multilateration is used to locate the cargo 90. The locator 25 receives a signal from the tag 21 and identifies a Received Signal Strength Indicator (RSSI). The RSSI is determined based on a known signal strength at a known distance and the strength of the signal received from the tag 21. The RSSI is transmitted to the control unit 50. The control unit 50 uses intensity values ​​from multiple different locators 25 to determine the location of the tag 21 (for example, trilateration for three different locators, or multilateration for four or more locators).

[0037]

[0046] In several other embodiments, the control unit 50 uses a stigmazi approach that uses an intensity map to estimate the position of the tag 21.

[0038]

[0047] Other networking protocols may be used by the wireless system 20 to locate and track the cargo. Several examples of wireless networking protocols include, but are not limited to, ZIGBEE and Wi-Fi. Each of these protocols enables communication between the tag 21 and the locator 25 to transfer identification data. To locate the cargo 90, calculations using signal strength are used. Several other embodiments use an RFID wireless system. In this case, the locator 25 emits radio waves and receives signals back from the RFID tag 21.

[0039]

[0048] The cargo tracking system 15 includes a vision system 40 to track the location of the cargo 90. The vision system 40 includes an electro-optic sensor 41 positioned on the vehicle 100 to capture an image of the cargo 90. The following disclosure includes an electro-optic sensor 41 which is a camera, but other types of electro-optic sensors may be used to capture an image of the cargo 90.

[0040]

[0049] Camera 41 is configured to capture individual discrete images and / or video of cargo 90. Camera 41 is configured to capture two-dimensional and / or three-dimensional images. In one embodiment, camera 41 includes a fixed field of view, which provides a sequence of images to be captured, including cargo 90 moving from edge to edge of the field of view. For example, a first image in the sequence captures cargo 90 on a first side of the image, a second image captures cargo 90 in the center of the image, and a third image captures cargo 90 on a second side, which is the opposite side of the field of view.

[0041]

[0050] In some embodiments, each camera 41 has an independent field of view distinct from any other camera 41. In several other embodiments, the cameras 41 are arranged to have overlapping fields of view. This makes it easier to track the movement of the cargo 90 as it moves through the different fields of view of different cameras 41 within the cargo compartment 103.

[0042]

[0051] The camera 41 is mounted on the vehicle 100 in a variety of known locations, including one or more of the doors 102, the body wall of the opening 104, and inside the cargo compartment 103. In some embodiments, the camera 41 is positioned high up, particularly inside the cargo compartment 103, to prevent and / or reduce the cargo 90 from blocking the camera 41. Certain embodiments include positioning the camera 41 along a side wall that is spaced upward from the ceiling or floor of the cargo compartment 103 (e.g., 75 inches above the floor).

[0043]

[0052] Figure 5 shows one network of cameras 41 positioned between the front end 107 and the rear end 108 of the cargo compartment 103. In this embodiment, six cameras 41 are positioned along the length of the cargo compartment 103. The cameras 41 are spaced apart along the lane 105 and in different bays 109 along the lane 105. In some embodiments, the cameras 41 are spaced apart along the lane 105, with the largest gap between the cameras 41 being approximately five bays 109. This allows different cameras 41 to capture images of the same area in the event that one or more of the cameras 41 are blocked by the cargo 90.

[0044]

[0053] The camera 41 may face in various directions to cover the cargo compartment 103. In some embodiments, the camera 41, spaced away from the front end 107, faces forward. In another embodiment, the camera 41, spaced away from the rear end 108, faces rearward.

[0045]

[0054] In some embodiments, camera 41 streams at a minimum frame rate of 2 Hz or higher. This rate ensures that transition events with cargo 90 are not missed in the image. In several other embodiments, the rate is one image per second when the Real-Time Streaming Protocol (RTSP) is used to emulate a live streaming scenario. This setting eliminates any large frame delays between different camera streams and therefore reduces the likelihood of missing transition events with cargo 90.

[0046]

[0055] The control unit 50 uses images to track the position of the cargo 90. In some embodiments, the control unit 50 identifies specific points on the cargo 90 that are used to track its position. These points result in the cargo 90 being tracked in different images, for example, as the cargo 90 moves through the fields of view of multiple cameras 41. For example, the control unit 50 identifies and tracks the position of the centroid of the cargo 90. Centroid tracking can use a variety of methods, non-limited to K-means clustering. Several other embodiments include, non-limited to, selecting different points on the cargo 90, such as corners, upper edge points, and lower edge points (e.g., the center of the lower edge).

[0047]

[0056] In some embodiments, the control unit 50 uses background subtraction to locate the cargo 90. One process involves static background subtraction, which uses a background image as a reference to detect changes in pixel values. Another process uses dynamic background subtraction, which uses a dynamically selected background image as a reference. The current image is compared to this reference for changes in pixel values.

[0048]

[0057] One embodiment of tracking the location of cargo 90 involves constructing a motion detector using dynamic background subtraction to area-exclude small moving objects (e.g., people) and then tracking the remaining pixels where detected motion exists. When the sum of detected pixels is sufficiently large, the location of cargo 90 is identified using the output of background subtraction to cluster the detections as objects. The K-means clustering algorithm is then used to find the centroid of cargo 90. When the centroid enters a particular region of interest that correlates with the cargo's location, cargo 90 is marked as being in that location from the viewpoint of a camera viewing cargo 90 at that location. In some embodiments, the process includes a camera voting system, in which images from multiple cameras 41 are analyzed to determine whether they contain cargo 90 at a particular location. Generally, cameras 41 located in the lane 105 opposite to where cargo 90 is moving have a better view of the cargo.

[0049]

[0058] Another embodiment of tracking cargo 90 involves leading edge detection. This approach uses a combination of static and dynamic computer vision techniques to identify the active area of ​​cargo compartment 103 from a given video feed. In some embodiments, this approach uses a bird's-eye view transformation to convert a 3D image to a 2D image. This approach uses both static and dynamic approaches and then identifies one of them. The static approach uses subtraction from static background subtraction to identify ground shift. If the amount of pixel difference exceeds a threshold, this approach assumes that cargo 90 is present. The dynamic approach uses dynamic background subtraction to extract only the moving parts of the video and performs Canny edge detection logic to identify the contours of moving objects. This approach then detects the presence of cargo 90 by searching for the leading edge of the cargo within the contours. This approach also distinguishes between static and dynamic approaches and determines that cargo is present when the leading edge is detected within the area within a short period before static background subtraction also flags the presence of the object. This algorithm also clears previous detections if static background subtraction does not flag an object while the leading edge is not detected within the preceding short window. In some embodiments, heuristic / debounce logic is applied to handle hysteresis states.

[0050]

[0059] Another approach to tracking cargo 90 is static background subtraction in the Hue, Saturation, and Value (HSV) space. This approach uses static computer vision techniques to identify the active area of ​​cargo compartment 103 from a pre-recorded video feed. This approach uses a bird's-eye view transformation to convert the 3D image into a 2D top-down view. This has several advantages, including simplifying the selected region of interest and allowing for the application of region of interest cropping. In some embodiments, this approach uses static background subtraction to identify ground shift. This approach operates in the HSV color space to handle differences in illumination intensity. The static region of interest is mapped to the floor, and the number of pixels that have changed compared to a reference background image is counted to determine whether an object occupies the area. Objects that are too small to be cargo are excluded. Kernel erasure techniques were used to reduce the noise floor of static background subtraction to account for slight pixel-level differences caused by vibrations and the resulting changes in light reflection. Furthermore, the arbitrator logic was updated to track state transitions to accurately estimate the cargo's position within the area when it is not directly within the camera's field of view. Each state is recorded in JSON, and the camera and locator data are fused.

[0051]

[0060] Figure 6 shows a schematic diagram of the cargo tracking system 15. The cargo tracking system 15 includes a control unit 50 that receives data from a locator 25 and a camera 41. Different components may communicate with the control unit 50 individually or via one or more data buses 59. The locator 25 and camera 41 may be powered in a variety of ways, including, but not limited to, via Ethernet, batteries, and various other wireless and wired structures.

[0052]

[0061] In one embodiment, the cargo tracking system 15 is integrated with the vehicle 100. The control unit 50 may be standalone, solely providing to monitor the cargo 90, or it may be part of another system of the vehicle 100, such as a flight control computer that monitors the vehicle's movements. In some embodiments, the control unit 50 is located remotely from the vehicle 100. One embodiment includes a control unit 50 that is a remote server that receives signal information from a locator 25, receives images from a camera 41, and processes the visual data.

[0053]

[0062] The control unit 50 is further configured to transmit cargo information to a remote node 99. The remote node 99 is located on the ground or on an airborne vehicle and is interested in the cargo 90. Multiple embodiments include, but are not limited to, an airline operating the vehicle 100, a transport company responsible for transporting the cargo 90, and the owner of the cargo 90. In some embodiments, the control unit 50 maintains a record 70 of the location of the cargo 90 within the vehicle 100. The record 70 includes lanes 105 and bays 109 where the cargo 90 is located within the cargo compartment 103. In some embodiments, the entire record 70 is communicated to the remote node 99. In other embodiments, discrete information from the record 70 is communicated to the remote node 99. Communication with the remote node 99 may be initiated from the control unit 50 or via a communication system mounted on the vehicle 100. Communication can take place over a wide variety of networks, including, but not limited to, public networks (e.g., the Internet) or private networks, and packet data networks such as mobile communication networks (e.g., WCDMA, LTE, WiMAX networks).

[0054]

[0063] The cargo tracking system 15 uses both the radio system 20 and the visual system 40 to locate and monitor the cargo 90 during loading and / or unloading. Figure 7 shows how the cargo 90 is located and tracked. The cargo 90 is first located via the radio system 20 (block 200). The location is based on identification data emitted from the tag 21, which is received by one or more of the locators 25. The location of the cargo 90 is also tracked via the radio system 20 (block 202). Location tracking is again performed via data received from the tag 21, such as through RSSI calculation from signals received by multiple locators 25. The cargo tracking system 15 also allows the visual system 40 to track the location of the cargo 90 (block 204).

[0055]

[0064] In some embodiments, the visual system 40 provides more accurate tracking than the wireless system 20, which enables identification and coarse localization. Once the cargo 90 is identified and its coarse location is known, more precise movement and positioning of the cargo 90 is possible via the visual system 40. In some embodiments, the visual system 40 is used to determine the final location within the cargo compartment 103. The final location is identified as the position where the cargo 90 is located when it is determined that the cargo 90 has stopped moving.

[0056]

[0065] In some embodiments, tracking of the cargo 90 is performed simultaneously by both the wireless system 20 and the visual system 40. The wireless system 20 first identifies the cargo 90 and allows for the determination of a relatively rough location. The control unit 50 can then analyze images received from one or more cameras 41 to track the further movement of the cargo 90.

[0057]

[0066] Figure 8 illustrates a method in which the location of cargo 90 is tracked simultaneously by both a wireless system 20 and a visual system 40. The process begins (block 209) and the cargo 90 is identified based on data transmitted from a tag 21 (block 210). The location of cargo 90 is also identified based on the location of the tag 21, which is received by one or more locators 25 (block 212). Simultaneously with the wireless system 20, the visual system 40 tracks the cargo 90. The cargo is detected in the image (block 220), and the location of the cargo is identified based on the image (block 222). In some embodiments, a control unit 50 first identifies the location of cargo 90 via the wireless system 20. Once the approximate location is known, the control unit 50 analyzes images from a camera 41. The camera 41 has a field of view that includes the approximate location. The control unit 50 can then more specifically identify the location of cargo 90 based on the images. In some embodiments, the images are not analyzed until cargo 90 has been identified via the wireless system 20. In several other embodiments, the image is analyzed before being transmitted via the wireless system 20.

[0058]

[0067] The method continues to base the location of the cargo 90 on the location identified by the wireless system 20 and the visual system 40 (block 225). In some embodiments, the location is monitored by both systems 20 and 40 in events where one of the systems cannot identify the location. This can occur in a variety of situations, not limited to, when the locator 25 cannot receive a signal from the tag 21, or when one or more of the cameras 41 are blocked, resulting in an image that cannot include the cargo 90. In some embodiments, the location of the cargo (block 225) is identified based on the visual system 40, because the visual system 40 is generally a more accurate system. In some embodiments, the location of the cargo 90 is based on a combination of locations identified by both systems 20 and 40 (e.g., an average location).

[0059]

[0068] In some embodiments, the wireless system 20 and the visual system 40 work sequentially. One method in Figure 9 involves using the wireless system 20 to locate the cargo 90 (block 210). The location of the cargo 90 is also located using the wireless system 20. Once the cargo 90 is located and initially located, its location is tracked using the visual system 40 (block 212). The initial location located via the wireless system 20 allows the control unit 50 to locate the cargo 90 in the image. Subsequently, the control unit 50 tracks the location using only the visual system 40.

[0060]

[0069] Figure 10 shows one embodiment of the control unit 50. The control unit 50 includes a processing circuit 51, a memory circuit 52, a camera interface circuit 53, a locator interface circuit 58, and a communication circuit 54. The processing circuit 51 controls the overall operation of the cargo tracking system 15 according to program instructions stored in the memory circuit 52. The processing circuit 51 may include one or more circuits, a microcontroller, a microprocessor, hardware, or a combination thereof. The processing circuit 51 may include varying amounts of computing power to provide the required functions.

[0061]

[0070] The memory circuit 52 includes a non-transient computer-readable storage medium that stores program instructions, such as a computer program product. This configures the processing circuit 51 to perform one or more of the techniques described herein. The memory circuit 52 may include various memory devices, such as read-only memory and flash memory. The memory circuit 52 may be a separate component, as shown in Figure 10, or it may be combined with the processing circuit 51. Alternatively, the processing circuit 51 may omit the memory circuit 52, for example, according to at least some embodiments in which the processing circuit 51 is dedicated and non-programmable. The memory circuit 52 is configured to support the loading of images into runtime memory for real-time processing and storage. In one embodiment, the memory circuit 52 includes a solid-state device (SSD).

[0062]

[0071] The control unit 50 includes a graphics processing unit (GPU) 55. The GPU 55 is a special electronic circuit designed to operate and modify the memory circuit 52 to accelerate the creation of images in a frame buffer intended for output. The GPU 55 may include varying amounts of computing power to provide the required functionality. In one embodiment, the GPU 55 has a computing power greater than 1 teraflop. This processing power provides large-scale machine learning. In one embodiment, the computing device 50 includes a separate GPU 55. In another embodiment, this processing is performed by a processing circuit 51.

[0063]

[0072] The memory circuit 52 is configured to store a record 70 of the cargo 90. The record 70 includes, but is not limited to, identification data about the cargo 90, including an alphanumeric identification code, name, owner, volume, contents, origin, destination, and loading location on the vehicle 100.

[0064]

[0073] The camera interface circuit 53 receives images from the camera 41. The camera interface circuit 53 can provide one-way communication with the camera 41 or two-way communication with the camera 41. The camera interface circuit 58 receives identification data from the locator 25. The locator interface circuit 58 can be configured for one-way or two-way communication.

[0065]

[0074] The communication circuit 54 provides communication with the control unit 50. This communication may include communication with other circuits on the transporter 100 (e.g., the transporter control system) and / or with the remote node 99. The communication circuit 54 provides data transmission and reception at the remote node 99.

[0066]

[0075] The user interface 60 allows the user to access data about the cargo 90. The user interface 60 includes, but is not limited to, one or more input devices 62, such as a keypad, touchpad, rollerball, and joystick. The user interface 60 also includes one or more displays 61 for displaying information about the cargo 90 and / or for the operator to input commands to the processing circuit 51.

[0067]

[0076] In some embodiments, the control unit 50 operates autonomously to process identification data and images. Autonomous capability minimizes and / or eliminates operator intervention that could slow down the process and / or introduce errors.

[0068]

[0077] The cargo tracking system 15 can be used in a variety of vehicles 100, including, but not limited to, trucks, trains, ships, and aircraft. The cargo tracking system 15 can also be used in other contexts, including, but not limited to, warehouses, airport loading facilities, and distribution centers.

[0069]

[0078] In some embodiments, the image includes a timestamp indicating the time the image was captured. This timestamp may be provided by the camera 41 or the control unit 50. This timestamp may be used by the control unit 50 to track the movement of the cargo 90 and different images captured by the camera 41.

[0070]

[0079] In some embodiments, tracking the location of cargo 90 uses a heatmap. This function includes a tag 21 attached to the cargo 90, configured to emit a signal. The signal is received by one or more locators 25 to form a heatmap of the cargo 90's approximate location. The heatmap is used to narrow down the location of cargo 90. Once the approximate location is known via the heatmap, one or more other systems (e.g., a visual system 40, perception, hearing) are used to pinpoint the location more precisely.

[0071]

[0080] Furthermore, information regarding the cargo tracking system 15 is disclosed in Appendix A.

[0072]

[0081] The phrase "substantially" in relation to quantities or measurements means that it is not necessary to achieve the listed characteristics, parameters, or values ​​exactly. Rather, deviations or variations (including, for example, tolerances, measurement errors, measurement accuracy limits, and other factors known to those skilled in the art) may occur, to the extent that they do not negate the effect intended to be produced by the features.

[0073]

[0082] The present invention can be carried out in ways other than those specifically described herein, without departing from the essential characteristics of the invention. This embodiment should be considered in all respects as illustrative and non-limiting, and all modifications that fall within the meaning and scope of the claims are intended to be encompassed within the claims.

Claims

1. A method for tracking cargo (90), Receiving a radio signal from the tag (21) of the cargo (90), Identifying the cargo (90) based on the aforementioned radio signal, Identifying the location of the cargo (90) based on the aforementioned radio signal, and A method comprising capturing images of the cargo (90) over a period of time and monitoring the position of the cargo (90) based on the images as the cargo (90) moves through an area.

2. The method according to claim 1, further comprising attaching the tag (21) to the cargo (90) before receiving the radio signal from the tag (21) of the cargo (90).

3. The method according to claim 1, further comprising capturing an image of the cargo (90) after determining the location of the cargo (90) based on the wireless signal.

4. The method according to claim 1, further comprising identifying the cargo (90) based on identification data contained in the radio signal.

5. The method according to claim 1, further comprising simultaneously monitoring the position of the cargo (90) based on the wireless signal and the image.

6. Based on the aforementioned image, it is determined that the cargo (90) has stopped moving within the area, and The method according to claim 1, further comprising identifying the final position of the cargo (90) as the point where the cargo (90) is located when the cargo (90) stops moving.

7. The method according to claim 1, further comprising identifying the cargo (90) and identifying the location of the cargo (90) while the cargo (90) is being loaded onto an aircraft.

8. The method according to claim 1, further comprising receiving the radio signal at a plurality of locators (25) and determining the location of the cargo (90) based on the signal strength of the radio signal received at the plurality of locators (25).

9. A method for tracking cargo (90) loaded onto a vehicle, With a radio signal being emitted from a tag (21) attached to the cargo (90), one or more of the multiple locators (25) receive the radio signal. Identifying the cargo (90) based on the aforementioned radio signal, When the cargo (90) is moving within the vehicle, the plurality of locators (25) receive the radio signal and track the position of the cargo (90) based on the radio signal. Capture an image of the cargo (90) while it is moving within the vehicle, and A method comprising tracking the position of the cargo (90) within the vehicle based on the aforementioned image.

10. The method according to claim 9, further comprising receiving the radio signal in the plurality of locators (25) and determining the location of the cargo (90) based on the signal strength of the radio signal received in the plurality of locators (25).

11. The method according to claim 9, further comprising receiving the radio signal in the plurality of locators (25) that are mounted in a fixed position relative to the vehicle.

12. The method according to claim 9, wherein receiving the wireless signal in one or more of the plurality of locators (25) includes receiving a BLE signal transmitted from a Bluetooth Low Energy (BLE) tag (21) attached to the cargo (90).

13. The method according to claim 9, further comprising identifying a point on the cargo (90) based on the image, and tracking the position of the cargo (90) based on the point identified in the image.

14. Identifying the leading edge of the cargo (90) based on the aforementioned image, and The method according to claim 9, further comprising tracking the leading edge of the cargo (90) as the cargo (90) moves within the vehicle.

15. It is a cargo tracking system, A wireless system (20), A tag (21) configured to be attached to cargo (90) and configured to emit identification data, and A wireless system (20) comprising a plurality of locators (25) configured to connect to a vehicle and to receive the identification data transmitted from the tag (21), A vision system (40) comprising a plurality of cameras positioned within the vehicle and configured to capture images of the cargo (90), and The system includes a control unit (50) equipped with a processing circuit (51), and the processing circuit (51) is Based on signals transmitted from the tag (21) and received by the plurality of locators (25), the cargo (90) is identified (90), and the location of the cargo (90) is tracked, and A cargo tracking system configured to perform the task of tracking the location of the cargo (90) based on the image captured by the visual system (40).

16. The cargo tracking system according to claim 15, wherein the wireless system (20) is a Bluetooth low-energy system.

17. The cargo tracking system according to claim 15, wherein the control unit (50) is configured to first identify the cargo (90) and the location of the cargo (90) based on the identification data received by one or more of the plurality of locators (25), and then track the location of the cargo (90) based on the image captured by the vision system (40).

18. The cargo tracking system according to claim 15, wherein the control unit (50) is configured to start capturing the image of the cargo (90) after the location of the cargo (90) has been identified by the wireless system (20).

19. The cargo tracking system according to claim 15, wherein the camera is positioned in a fixed location within the vehicle.

20. The cargo tracking system according to claim 15, wherein the wireless system (20) is configured to determine the location of the cargo (90) based on the signal strength of the wireless signals received by the plurality of locators (25).

Images