System and method for monitoring goods

Abstract

The invention relates to a system and a method for monitoring goods. A monitoring system for monitoring cargo on a vehicle. The monitoring system includes a camera configured to take an image of the cargo while the cargo is located on the vehicle, where the camera is aligned to take an image of the cargo from different perspectives. The computing device includes processing circuitry configured to process an image received from the camera. The computing device is configured to: identify a base of a cargo; tracking the location of the cargo within the vehicle based on the location of the base within the image; determining that the goods are pallets; and determining a volume of the tray.

Description

Technical Field

[0001] The present disclosure generally relates to the field of cargo monitoring, and more particularly to monitoring systems and methods for identifying cargo moving within an area, such as within the interior of a vehicle. Background Art

[0002] A variety of vehicles are used to transport cargo. Examples include but are not limited to aircraft, ocean vessels, and trucks. The transportation process typically includes loading the cargo onto the vehicle, positioning the cargo within the vehicle, transporting the cargo from a first location to a second location, and then unloading the cargo. There is a need to identify and monitor the cargo during the transportation process.

[0003] Existing systems provide various ways to identify the cargo loaded onto a vehicle. However, these systems are unable to accurately monitor the cargo, including failing to track the location of the cargo and / or failing to accurately identify the cargo. This can result in the cargo being inappropriately loaded onto the vehicle. This also results in the location of the cargo on the vehicle being unknown during transportation.

[0004] Some existing systems require an operator to visually inspect and identify the cargo. However, it has been found that visual identification of the cargo is inaccurate because the operator typically cannot accurately identify the cargo or cannot correctly enter the cargo identification into the monitoring software. Further, this can be expensive as it requires one or more operators to identify the identification and enter it into the monitoring software. This process can also be time-consuming, which slows down the loading process and may result in transportation delays. Summary of the Invention

[0005] In one aspect, there is provided a monitoring system for monitoring cargo on a vehicle. The monitoring system includes a camera configured to capture an image of the cargo when the cargo is located on the vehicle, wherein the camera is aligned to capture images of the cargo from different perspectives. A computing device includes processing circuitry configured to process the images received from the camera, and the computing device is configured to: perform a rough detection and a fine detection on the images and detect the cargo within the images; identify features of the cargo from the fine detection; and determine the type of the cargo based on the features.

[0006] In another aspect, the rough detection includes identifying the foreground portion of the image and removing the background portion of the image, and the fine detection includes detecting features based on a generalized Hough transform.

[0007] In another aspect, the computing device is further configured to identify the feature as a mesh and determine that the cargo is a pallet.

[0008] In another aspect, the computing device is configured to perform a fine detection on a limited portion of the image, the limited portion including the base of the goods and a limited number of vertical rows of pixels extending upward from the base.

[0009] In another aspect, the computing device is further configured to determine a confidence value of the image based on a match between the goods captured in the image and an image template.

[0010] In another aspect, the computing device is configured to determine that the image captures the goods when the confidence value is higher than a predetermined threshold.

[0011] In another aspect, the computing device is configured to determine the direction of movement of the goods within the vehicle based on a change in the confidence value of the image.

[0012] In another aspect, the computing device is configured to identify a feature as not having one or more of a net and a strap.

[0013] In another aspect, the computing device is configured to determine the dimensions of the goods based on the image, determine the scale of the image, and determine the volume of the goods based on the dimensions and the scale.

[0014] In another aspect, the computing device determines the volume of the goods only when the type of the goods is a pallet.

[0015] In another aspect, the computing device is further configured to identify reference points on the goods and track the position of the goods within the vehicle based on the positions of the reference points within the image.

[0016] One aspect relates to a monitoring system for monitoring goods on a vehicle. The monitoring system includes a camera configured to capture an image of the goods when the goods are in the vehicle, wherein the camera is aligned to capture images of the goods from different perspectives. A computing device includes a processing circuit and is configured to: receive the image from the camera; remove the background portion of the image from the foreground portion of the image; perform a generalized Hough transform and detect the goods in the foreground portion of the image; and determine the type of the goods.

[0017] In another aspect, the computing device is further configured to identify features of the goods and determine the type of the goods based on the features.

[0018] In another aspect, the computing device is further configured to determine a confidence score for each image and track the position of the goods within the vehicle based on the confidence score of the image and the positions of the reference points of the goods in the image.

[0019] In another aspect, the computing device is configured to perform a generalized Hough transform on a lower portion of the foreground portion that includes the base of the goods.

[0020] On the one hand, it relates to a method for monitoring goods in a vehicle, the method comprising: receiving an image of the goods when the goods are located within the vehicle; performing a rough detection and a fine detection for each image; identifying features of the goods after performing the fine detection; and identifying the goods based on the features.

[0021] In another aspect, identifying the features includes identifying a net extending over the packaging of the goods.

[0022] In another aspect, identifying the features includes being unable to locate a specific visible item in the image, where the specific visible item includes a net, a strap, and a specific shape of the goods.

[0023] In another aspect, the method further comprises performing a fine detection and identifying the edges of the goods.

[0024] In another aspect, the method further comprises determining an image of the goods filling a bounding box in the image and determining the center of the goods based on that one image.

[0025] The features, functions, and advantages discussed above can be implemented independently in various aspects or can be combined in other aspects, and further details thereof can be seen with reference to the following description and drawings. Description of the Drawings

[0026] Figure 1 is an isometric view of an aircraft having a vision system configured to monitor goods in an interior space.

[0027] Figure 2 is an isometric view of goods being loaded through a door in the fuselage into the interior space of a vehicle.

[0028] Figure 3 is a schematic diagram of a container.

[0029] Figure 4 is a schematic diagram of a pallet.

[0030] Figure 5 is a schematic diagram of a monitoring system configured to monitor goods in the interior space of a vehicle.

[0031] Figure 6 is a flowchart of a method for monitoring goods in a vehicle.

[0032] Figure 7 is a flowchart of a method for monitoring goods in a vehicle.

[0033] Figure 8 is an image of goods located along a path in the interior space of a vehicle.

[0034] Figure 9 is a view showing an output screen of a graph illustrating the confidence value of an image.

[0035] Figure 10 Schematic diagram of a monitoring system operatively connected to one or more remote nodes.

[0036] Figure 11 Schematic diagram of a computing device. Detailed implementation

[0037] Figure 1 Shows a vehicle 100 equipped with an autonomous monitoring system 20 for monitoring goods. In this example, the vehicle 100 is an aircraft having a fuselage 101, and the fuselage 101 includes an internal space 103 configured to accommodate goods. One or more doors 102 are provided to load goods into the internal space 103 and unload goods from the internal space 103.

[0038] Figure 2 Shows goods 200 positioned on a platform for loading into the vehicle 100. The door 102 in the fuselage 101 of the vehicle is in an open position. This allows the goods 200 to move through an opening 104 in the fuselage 101 and into the internal space 103. A variety of different types of goods 200 can be transported by the vehicle 100. The goods 200 can include, for example Figure 3 as shown in the container 202. The container 202 includes an outer wall that extends around and encloses an interior. These walls are typically rigid and have a fixed shape that conforms to the internal space 103 of the vehicle 100. In some examples, the container 202 includes an aluminum frame with Lexan or fiberglass walls. The goods 200 can also include, for example Figure 4 as shown in the pallet 203. The pallet 203 includes a base 207, which is a rough rigid sheet of a material such as aluminum or plastic having a fixed size. The base 207 is typically rectangular, but also periodically includes other shapes that conform to the shape of the internal space 103. The base 207 is configured to support individual packages 204 stacked on top of each other. A net 205 extends over the packages 204 to secure the packages 204 to the base 207. The net 205 can be formed from various materials including nylon or polyester and forms a mesh with diamond-shaped openings 206. The goods 200 can also include various other configurations, each of which includes a base.

[0039] Figure 5 Schematically shows a monitoring system 20 for monitoring goods 200. The monitoring system 20 includes a camera 70 positioned within the internal space 103. The camera 70 is positioned within the internal space 103, which provides protection against weather and other elements in the external environment that may cause damage. The camera 70 captures images of the goods that are processed by a computing device 50. Figure 5It includes a camera 70 positioned at an alignment area 105 adjacent to the opening 104.

[0040] The alignment area 105 is configured to allow the goods 200 to enter the internal space 103 and align with a path 106 that further extends into the internal space 103. The path 106 extends away from the alignment area 105 in the front - to - rear direction and is provided to load the goods 200 in an efficient manner. During loading, the goods 200 move through the door 102 and into the alignment area 105. The goods 200 align with one of the paths 106 and then move along the path 106 until reaching the end of the path 106 or abutting against other goods 200 that have previously been loaded into the path 106. In some examples, the goods 200 are loaded according to a Load Instruction Report (LIR). The LIR is used by an operator to load the vehicle 100 and provides instructions on where to position the goods 200 on the vehicle 100 to comply with weight and balance restrictions. The alignment area 105 and the path 106 are configured to provide loading and unloading in a Last - In - First - Out (LIFO) manner.

[0041] The camera 70 is positioned on the vehicle 100 to capture images of the goods 200 when the goods 200 are in the internal space 103. The camera 70 is configured to capture images of the goods 200 as individual discrete images and / or video images. Figure 5 The camera 70 positioned at the alignment area 105 is shown. A first camera 70a is positioned at the alignment area 105, opposite the cargo door 102. The first camera 70a faces the door 102 to capture the goods 200 when the goods 200 move through the opening 104 and within the alignment area 105. A second camera 70b is located in front of the alignment area 105. The second camera 70b faces rearward and captures the goods 200 within the alignment area 105 and the rear section of the internal space 103 (i.e., the section of the internal space 103 behind the alignment area 105). A third camera 70c is positioned behind the alignment area 105. The third camera 70c faces forward and captures the goods 200 within the alignment area 105 and the front section of the internal space 103 (i.e., the section in front of the alignment area 105). In some examples, the camera 70 has a fixed position and a fixed field of view. The camera 70 can record images at different frequencies.

[0042] In some examples, cargo monitoring is performed using the images 71 from a single camera 70. In other examples, cargo monitoring is performed using the images 71 from two or more different cameras 70.

[0043] The computing device 50 receives the images 71 from one or more cameras 70 and processes the images to monitor the goods 200 within the internal space 103. Figure 6An example of a method for monitoring a cargo 200 is shown. Image processing detects the cargo 200 in an image 71 (block 300). A computing device 50 processes the image 71 and identifies features 201 of the cargo 200 (block 302). Different features 201 can be identified, including but not limited to the shape of the cargo, the base of the cargo, the net, and the strap. The computing device 50 then identifies the type of the cargo 200 based on the feature (block 304). The type of the cargo 200 can vary with examples including but not limited to a container 200, a pallet 203, and individual items (e.g., machinery, vehicles).

[0044] In some examples, the processing further includes tracking the position of the cargo 200 as the cargo 200 moves within the internal space 103.

[0045] Figure 7 Another example of a method for monitoring a cargo 200 is shown. An image 71 is received from one or more cameras 70. The image processing initially includes a rough detection (block 310). The rough detection identifies a first portion of the image and removes the remaining portion of the image. A fine detection is performed on the first portion (block 312). In some examples, the fine detection includes a generalized Hough transform to identify the cargo 200. Based on the fine detection, the image processing identifies the features of the cargo 200 (block 314), and then identifies the cargo type (block 316).

[0046] In some examples, the processing determines the volume of the cargo 200 based on one or more of the dimensions of the cargo 200 and the scale of the image 71.

[0047] Cargo monitoring detects the cargo 200 based on one or more images 71 captured by one or more cameras 70. The cameras 70 have a fixed field of view such that the position of the cargo 200 within multiple consecutive images changes as the cargo 200 moves along a path 106b. For a cargo 200 moving away from the camera, the cargo 200 initially appears larger visually and may not be fully visible in a first image 71. Once the cargo 200 moves a certain distance away from the camera within the internal space 103, the cargo 200 is fully captured in one or more images 71. As the cargo 200 continues to move away, the cargo 200 appears smaller visually. Similarly, a cargo 200 moving towards the camera 70 at a certain distance initially appears small. As the cargo 200 moves closer to the camera 70, the cargo 200 appears larger in subsequent images 71. Eventually, as the cargo 200 moves through the field of view of the camera 70, a portion of the cargo 200 will become invisible.

[0048] Image processing performs a rough detection on image 71. The rough detection identifies the goods 200 in the foreground of the image. Then, the image processing subtracts or otherwise removes the other parts of the image. In some examples, the background of the image is removed. The rough detection removes the parts of the image that are unnecessary for identifying the goods 200.

[0049] Figure 8 An image 71 is shown that captures a first path 106a and a second path 106b within the interior space 103. The image 71 also includes goods 200 positioned within the second path 106b. In some examples, the rough detection includes a bounding box 80 that outlines the boundaries of a portion within the image 71. In some examples, the bounding box 80 is statically positioned at a fixed reference point in each of the images 71. The bounding box 80 has a statically defined shape with fixed dimensions. The fixed dimensions of the bounding box 80 are a function of the size of the vehicle 100 and the field of view of the camera 70.

[0050] Figure 8 An image 71 is shown where the goods 200 substantially fill the bounding box 80. This includes the base of the goods 200 being positioned at the bottom of the bounding box 80 and the top of the goods 200 being positioned at the top of the bounding box 80. The rough detection removes the parts of the image 71 that are outside of the bounding box 80. In some examples, the rough detection only retains the parts of the image 71 that are within the bounding box 80. The remaining parts of the image 71 are deleted because such data is not needed to determine the characteristics of the goods 200.

[0051] After the rough detection, the image processing includes a fine detection of the image 71. In some examples, the fine detection uses a Generalized Hough Transform (GHT) to identify the goods 200. In some examples, the GHT provides characteristics for identifying the goods, which are subsequently used to identify the goods 200.

[0052] In some examples, the image processing converts the image to grayscale. This conversion helps to identify the edges of the goods 200 and / or features 201. The conversion to grayscale also reduces the amount of data, which simplifies the image processing algorithm and reduces the computational requirements, such that the computational requirements can be completed with less processing power. After the conversion, a Canny edge detection algorithm is used to produce an edge image. The edge image includes the edges of the goods 200 and / or features 201 without identifying unwanted details. A reference point of the goods 200 is determined. In some examples, the reference point is the centroid of the goods 200. In another example, the reference point is the center point along the bottom edge of the goods 200.

[0053] The GHT uses an edge image to generate a template (i.e., a general shape model). Image processing compares the edge points detected in the image with the template to determine the identity and location of the goods. This processing includes determining the matching probability between the feature 201 in the image and the corresponding element in the template. In some examples, this is achieved by determining boundary points based on a vector representation related to a reference point.

[0054] The goods 200 are identified based on the recognition of one or more features 201. Various features 201 of the goods 200 can be recognized. An example of a feature is the overall shape of the goods 200. In a specific example, the container 202 is identified based on the overall shape. Features can also include the shape of a section of the goods 200, such as the shape of the bottom. In some examples, the feature 201 is a net 205 extending over the package 204 on the pallet 203, as Figure 4 and Figure 8 shown. The net 205 is identified by one or more diamond-shaped portions of the net within the image. In some examples, a single diamond-shaped portion is provided to identify the net 205 and classify the goods 200 as a pallet 203. In other examples, two or more diamond-shaped portions are identified to classify the pallet 203. In another example, the feature 201 is a strap 208 located on the exterior of the goods 200. Figure 8 The strap 208 that secures the package 204 to the base 207 of the pallet 203 is shown. In another example, the feature 201 does not recognize other known features. For example, when the image 71 does not include the net 205, the strap 208 and has an unexpected shape, the image processing determines that the goods type is a miscellaneous goods covering random goods.

[0055] In some examples, the fine detection uses the entire cropped image resulting from the coarse detection. In other examples, the fine detection uses a limited portion of the cropped image. In a specific example, the fine detection crops the image 71 to include only the bottom portion. This processing identifies the bottom of the goods 200 and additional pixel rows upward from the bottom. The image processing uses the bottom 73 as a reference because it is known that the bottom 73 is on the floor of the internal space 103.

[0056] In some examples, the image processing determines one or more dimensions of the goods 200 from the image 71. The width is determined using points upward from the bottom 73 on the image 71. For example, the width is determined by a predetermined number of pixel rows upward from the bottom 73. This interval upward from the bottom ensures that the width is obtained at points other than the bottom side of the goods, which includes the bottom side of the container or the base of the pallet. The bottom 73 of the image is not used to determine the width because the goods bottom has a fixed width for both the container 202 and the pallet 203. However, the width upward from the bottom has a variable width based on the number of packages 204 and the stacked pallets 203.

[0057] In some examples, width determination uses the image at the bounding box 80. Since the dimensions are known, the size corresponding to each pixel is known (i.e., x pixels in the image = y inches of the goods). This enables the determination of the dimensions of the goods 200. One or more other dimensions (e.g., height) of the goods 200 can be determined in a similar manner.

[0058] Image processing analyzes multiple images 71 captured by the camera 70. Fine detection includes determining a confidence score using GHT processing. A confidence score is determined for each image 71 based on the degree of match between the goods 200 and the identified template. A higher score indicates a higher confidence that the image 71 includes the goods 200, while a lower score has a lower confidence of capturing the goods 200 in the image 71.

[0059] Figure 9 An example of a score graph with confidence scores of the captured and analyzed images 71 is shown. In this example, the images 71 capture the goods 200 as the goods 200 moves towards the camera 70. The first image includes the goods 200 that is farthest from the camera 70, and the last image includes the goods 200 that is closest to the camera 70. In this example, the first and second images have the highest confidence scores because the goods 200 is fully visible within the corresponding images. The first image includes the goods 200 that is far from the camera 70, and when the goods 200 is closer, the first image has a lower score than the second image.

[0060] As further shown in Figure 9 Images 3 - 10 include lower confidence scores than images 1 and 2. This occurs because as the goods 200 moves towards the camera 70, the goods 200 starts to move out of the image 71. This results in a lower confidence score because not all of the goods 200 is visible in the image 71. Image processing can match the image to the template, but the match is incomplete, thus resulting in a lower score. At some points as shown in Figure 9 the goods 200 completely moves out of the field of view of the camera and does not appear in the image 71. Image processing does match the image to the template, resulting in a score that is relatively higher than that of the partially cropped goods (e.g., images 3 - 10). This score is considered noise.

[0061] In some examples, the detection process includes one or more of the images 71 having a confidence value higher than the threshold 81. The threshold is set to ensure that the confidence score is high enough for the image to be used to track the goods 200. In as shown in Figure 9In some of the examples shown, multiple thresholds, such as upper threshold 81a and lower threshold 81b, are used to analyze the confidence score. An image 71 with a confidence score higher than the upper threshold 81a is determined to include the goods 200. An image 71 with a confidence score lower than the lower threshold 81b is determined not to include the goods 200. An image 71 with a confidence score between the upper threshold 81a and the lower threshold 81b requires additional processing to determine whether the goods 200 are included in the image 71. The number and positioning of the thresholds 81 can vary.

[0062] In some examples, only the image 71 with the highest confidence score is used for comparison with the threshold 81. Other images 71 are not used as part of the confidence determination. In other examples, two or more of the images 71 are used to determine the confidence value.

[0063] The image processing is configured to determine the volume of the goods 200. In some examples, the volume of each piece of goods 200 is determined. In other examples, the volume is determined only for certain types of goods. In one specific example, since each pallet 203 has a unique shape and / or size due to different packages 204, the volume of the pallet 203 is determined and thus the volume is needed, such as for shipping information.

[0064] The volume calculation uses the dimensions of the goods 200, such as height, width, and surface area. In some examples, the surface area is determined by using a contour mapping algorithm. The volume determination also uses the image scale. The scale is determined based on the known field of view of the camera 70 that captured the image 71. In one example, the image 71 used for scaling is captured at a point with a known reference in the field of view of the camera 70. For example, a point where the number of pixels per unit is equal to a known length of the goods 200 (e.g., x pixels in the image = x feet of the goods; 477 pixels = 10 ft.; 1023 pixels = 12 ft.). In some examples, multiple cameras 70 are used to capture images and scaling is performed for each of the different cameras. For example, the first camera 70a takes a side view of the goods 200 at a location with a first scale. The second camera 70b takes a rear view of the goods 200 at a point with a second scale. The scaling ratios from the two separate cameras 70a, 70b are used to determine two dimensions of the goods 200. The volume of the pallet 203 is determined based on the surface area, width, and scale (block 408).

[0065] The monitoring system 20 tracks the position of the goods 200 as the goods 200 move within the internal space 103. Tracking the position of the goods 200 can be performed through a single-camera tracking mode and / or a multi-camera tracking mode. In the single-camera tracking mode, one of the cameras 70 facing forward or backward is used to track the position. In using Figure 5In a specific example, camera 70b is used to track the position of the cargo 200 moving behind the alignment area 105, and camera 70c is used to track the position of the cargo 200 moving in front of the alignment area 105. The images captured by the camera 70 are processed to obtain a confidence score. The change in the confidence score used to determine whether the cargo 200 is moving towards or away from the camera 70 is utilized to process multiple images. Additionally, the positions of reference points in different images 71 are compared to determine the direction of movement. In a specific example, the position of the reference point is measured along the y-axis that is aligned with the camera 70 and extends directly away from the camera 70.

[0066] For multi-camera tracking, two or more cameras 70 track the position of the cargo 200 within the internal space 103. In one example as Figure 5 shown, camera 70a is used to track the position of the cargo 200 within the alignment area 105, camera 70b tracks movement within the rear section of the internal space 103, and camera 70c tracks movement within the front section. In some examples, the images captured by each of the cameras 70 are processed, where reference points are monitored in each image to detect changes in position, and where a confidence score is calculated for each image and the confidence scores are compared to determine the direction of movement. In some examples, once it is determined that the cargo 200 has left the field of view of a particular camera 70, image processing is no longer performed on the images, or is performed at a lower frequency.

[0067] Figure 10 A schematic diagram of a monitoring system 20 including a computing device 50 and cameras 70 is shown. In one example, the cameras 70 communicate with the computing device 50 via a data bus 29. In some examples, the system utilizes Ethernet transmission to enable the transfer of camera data from the cameras 70 to the computing device 50. The cameras 70 can send images to the computing device 50 via various other wireless and wired architectures.

[0068] In one example, the monitoring system 20 is integrated with the vehicle 100. The computing device 50 can be a stand-alone device solely for monitoring the cargo 200. In another example, the computing device 50 performs one or more additional functions. For example, the computing device 50 can be part of a flight control computer that monitors the operation of the vehicle 100. In another example, the computing device 50 is part of an overall vision system that includes cameras 70 located throughout the vehicle 100 and is used to monitor passengers and / or cargo. In yet another example, the computing device 50 is located remotely from the vehicle 100. One example includes the computing device 50 being a remote server that receives images from the cameras 70 and processes the image data.

[0069] The computing device 50 receives images from the camera 70. The computing device 50 is configured to automatically process and transmit data in real time using a combination of machine learning perception, photogrammetry, and automation software modules.

[0070] In some examples, the image 71 includes a timestamp indicating the time the image was captured. The timestamp can be applied by the camera 70 or the computing device 50. The computing device 50 can use the timestamp to further track the movement of the goods 200 in different images 71 captured by the camera 70.

[0071] Data determined by the computing device 50 is transmitted to one or more remote nodes 99. The monitoring system 20 is configured to detect the connectivity and bandwidth of the communication capabilities with the remote nodes 99. The vehicle 100 is configured to communicate with the remote nodes 99 through one or more different communication channels, such as through a wireless communication network or a wired connection.

[0072] As Figure 11 shown, the computing device 50 includes a processor circuit 51, a memory circuit 52, a camera interface circuit 53, and a communication circuit 54. The processor circuit 51 controls the overall operation of the monitoring system 20 according to program instructions stored in the memory circuit 52. The processor circuit 51 can include one or more circuits, microcontrollers, microprocessors, hardware, or a combination thereof. The processor circuit 51 can include different amounts of computing power to provide the required functionality.

[0073] The memory circuit 52 includes a non-transitory computer-readable storage medium storing program instructions, such as a computer program product, the program instructions configuring the processor circuit 51 to implement one or more of the techniques discussed herein. The memory circuit 52 can include various memory devices, such as, for example, read-only memory, and flash memory. The memory circuit 52 can be a separate component as Figure 11 shown, or can be integrated with the processor circuit 51. Alternatively, for example, according to at least some examples where the processor circuit 51 is dedicated and non-programmable, the processor circuit 51 can omit the memory circuit 52. The memory circuit 52 is configured to support loading images into runtime memory for real-time processing and storage. In one example, the memory circuit 52 includes a solid-state device (SSD).

[0074] In some examples, the memory circuit 52 is configured to store a record 90 of the goods 200. The record 90 includes the calculated volume of the goods 200 (if the goods 200 is a pallet 203) and the location of the goods 200 on the vehicle 100. The record 90 can include additional information about the goods 200, such as, but not limited to, weight, contents, specific shipping instructions, origin, and destination locations.

[0075] The camera interface circuit 53 is provided to receive an image 71 from a camera 70. The camera interface circuit 53 can provide for unidirectional communication from the camera 70 or bidirectional communication to and from the camera 70.

[0076] The communication circuit 54 is provided for communication to and from the computing device 50. The communication can include communication with other circuits on the vehicle 100 (e.g., the vehicle control system) and / or communication with a remote node 99. The communication circuit 54 is provided to send and receive data with the remote node 99. The computing device 50 automatically detects available connectivity and bandwidth.

[0077] The user interface 58 is provided for a user to access data regarding the cargo 200. The user interface 58 includes one or more input devices 57, such as but not limited to a keyboard, a touchpad, a scroll ball, and a joystick. The user interface 58 also includes one or more displays 56 for displaying information regarding the cargo 200 and / or for an operator to input commands to the processor circuit 51.

[0078] In some examples, the computing device 50 outputs on a graphical user interface (GUI) displayed on the display 56 or outputs to the remote node 99. The GUI can include one or more graphical components (e.g., icons, menus, etc.) that are configured to provide the user with the ability to interact with the GUI to convey the desired information to the user.

[0079] In some examples, the computing device 50 operates autonomously to process the image 71. This autonomous capability minimizes and / or eliminates operator intervention, which may slow down the process and / or introduce errors.

[0080] In one example, the monitoring system 20 is integrated into the vehicle 100. The monitoring system 20 can be used on various vehicles 100. Vehicles 100 include but are not limited to trucks, trains, ships, and aircraft.

[0081] The vision system 15 can also be used in other contexts. Examples include but are not limited to warehouses, airport loading facilities, and distribution centers.

[0082] The examples disclosed above use cargo detection as a step in monitoring the position of the cargo 200 within the vehicle 100. In another application, cargo detection can be used to determine whether an object is present at a predetermined location. Cargo detection is used to determine whether the cargo is at a location and to calculate a corresponding confidence value. In this application, other steps of the monitoring process may or may not be used.

[0083] In some examples, image processing uses coarse detection to initially identify the goods 200. Coarse detection can also be used for other applications, such as determining whether an item is in a specific location. Coarse detection is used to determine whether an item is in a specific location. In some applications, coarse detection does not specifically identify the item. These other applications can be used for the goods 200, or for various other objects that can be detected.

[0084] Regarding a quantity or a measured value, the term "substantially" means that the characteristic, parameter, or value need not be precisely achieved. Instead, deviations or variations (including, for example, tolerances, measurement errors, measurement precision limitations, and other factors known to those skilled in the art) can occur in amounts that do not preclude the effect that the characteristic is intended to provide.

[0085] Without departing from the basic characteristics of the present invention, the present invention may be implemented in other ways different from those specifically set forth herein. This embodiment is considered illustrative rather than restrictive in all respects, and all changes falling within the meaning and scope of equivalents of the appended claims are intended to be included therein.

Claims

1. A monitoring system for monitoring cargo on a vehicle, the monitoring system comprising: a camera configured to capture images of the cargo while the cargo is on the vehicle, the camera being aligned to capture images of the cargo from different viewing angles; A computing device comprising a processing circuit configured to process the image received from the camera, the computing device configured to: performing coarse detection and fine detection on the image and detecting the goods within the image; identifying characteristics of the cargo from the detailed detection; as well as The type of the cargo is determined based on the characteristics.

2. The monitoring system according to claim 1, wherein: The rough detection includes identifying a foreground portion of the image and removing a background portion of the image; as well as The refined detection includes detecting the feature based on a generalized Hough transform.

3. The monitoring system according to claim 1, wherein: The computing device is further configured to identify the feature as a net and determine that the freight shipment is a pallet.

4. The monitoring system according to claim 1, wherein: The computing device is configured to perform the fine detection on a limited portion of the image, the limited portion including a base of the cargo and a limited number of vertical pixel rows upward from the base.

5. The monitoring system according to claim 1, wherein: The computing device is further configured to determine a confidence value for the image based on a match between the cargo captured in the image and an image template.

6. The monitoring system according to claim 5, in, The computing device is configured to determine that the image captures the cargo when the confidence value is above a predetermined threshold, and / or, Wherein, the computing device is configured to determine a moving direction of the cargo within the vehicle based on a change in the confidence value of the image.

7. The monitoring system according to claim 1, wherein: The computing device is configured to identify the feature as an absence of one or more of a web and a strap.

8. The monitoring system according to claim 1, wherein: The computing device is configured to: determining a size of the cargo based on the image; determining a scale of the image; as well as A volume of the cargo is determined based on the size and the ratio.

9. The monitoring system according to claim 1, wherein: The computing device determines the volume of the cargo only when the type of the cargo is a pallet.

10. The monitoring system according to claim 1, wherein: The computing device is further configured to: Identify reference points on the goods; and The location of the cargo within the vehicle is tracked based on the location of the reference point within the image.

11. A monitoring system for monitoring cargo on a vehicle, the monitoring system comprising: a camera configured to capture images of the cargo while the cargo is in the vehicle, the camera being aligned to capture images of the cargo from different viewing angles; A computing device, comprising a processing circuit, the computing device being configured to: receiving the image from the camera; removing a background portion of the image from a foreground portion of the image; performing a generalized Hough transform and detecting the cargo in the foreground portion of the image; as well as Identify the type of cargo in question.

12. The monitoring system according to claim 11, wherein: The computing device is further configured to: determining a confidence score for each of the images; and The location of the cargo within the vehicle is tracked based on the confidence score of the image and the location of a reference point of the cargo in the image.

13. The monitoring system according to claim 11, wherein: The computing device is configured to perform the generalized Hough transform on a lower portion of the foreground portion including a base of the cargo.

14. A method for monitoring cargo in a vehicle, the method comprising: receiving an image of the cargo while the cargo is within the vehicle; For each of the images, performing coarse detection and fine detection; identifying characteristics of the cargo after performing the detailed inspection; as well as The cargo is identified based on the characteristics.

15. The method according to claim 14, wherein: Identifying the feature includes identifying a web extending over a wrapper of the cargo.

16. The method according to claim 14, wherein: Identifying the features includes failing to locate specific visible items in the image, wherein the specific visible items include a web, a strap, and a specific shape of the cargo.

17. The method of claim 14, further comprising performing said fine detection and identifying edges of said cargo.

18. The method according to claim 14, further comprising: Determining an image of the cargo filling a bounding box in the image; as well as A center of the cargo is determined based on the one image.

Images