Machine learning real estate object detection and analysis device, system, and method

Abstract

The physical and logical components of an apparatus, system, and method for detecting, identifying, and classifying objects on a work site and other objects on real property ("Objects"), Artificial intelligence machine learning analysis of object sensor data to identify objects in the object sensor data, Determine the category of the object, Determine at least one of site map, hazardous conditions, theft, and / or object behavior, and output warning and usage reports for equipment, vehicles, and personnel.

Description

[Technical Field]

[0001] The present disclosure relates to improved apparatus, systems, and methods for detecting, identifying, and classifying construction site objects and other real estate site objects ("objects") through visual analysis of images, associating such objects with object sensor data, and determining and responding to object behavior, wherein the visual analysis, determination, and response are performed at least in part by an artificial intelligence machine learning system, and further for outputting site images, including stitched images of the site, for visually identifying objects in such images, and associating objects in such images with object sensor data. [Background technology]

[0002] Construction sites and other real property (hereafter "sites") often contain people, vehicles, machinery, materials, and other objects, including objects that will be or are being incorporated into buildings and landscaping. Site walkways, roads, and other areas allow people, vehicles, and machinery to travel. Some of these areas may be designated for use by authorities, while others may be used without authorization. Hazardous conditions can be created intentionally or accidentally, such as areas under, near, or in the path of objects with high potential or kinetic energy, under cranes or near buildings under construction, and in areas where vehicles or machinery are moving.

[0003] Property owners, property managers, construction site managers, construction workers, government and private authorities, and others want to understand and monitor site usage and activity for a variety of reasons, including identifying hazardous conditions; reducing accidents; monitoring the entry and exit of people, vehicles, materials, and surface water from and to a site; monitoring wind speeds; monitoring the status of construction projects; monitoring the utilization (utilization rate) of site resources (cranes, machinery, workers, etc.); determining the profile of loads suspended below cranes (size, shape, and type of suspended load); monitoring variances between planned and actual construction; verifying compliance with zoning and property lines; and monitoring the condition and use of buildings.

[0004] However, even if some monitoring services can be provided remotely, on-site human monitoring is costly. Furthermore, human assessments can be inconsistent. Systems have been developed to monitor and control various aspects of real estate and construction sites with reduced human involvement, including motion sensors, infrared sensors, lasers, light detection and ranging sensors ("LIDAR sensors"), radar sensors, acoustic sensors, smoke sensors, carbon monoxide sensors, radio frequency identifiers ("RFID"), crane load sensors, cameras, and access control systems. However, such systems can be constructed with physical and logical configurations and can be "fragile," in the sense that the physical and logical configurations must be planned, designed, and implemented to address the specific conditions being monitored. For example, an RFID system may require that a user possess an RFID chip, a mobile phone, or other radio frequency transmitter that transmits an identifier. Such an RFID system can determine whether a user with a functioning transmitter has entered a monitored area if the monitored area is equipped with a functioning receiver (or transceiver).

[0005] An artificial intelligence system coupled with a camera may be able to distinguish between different objects, e.g., dogs and humans, based on analysis of images, but may not be able to determine the physical properties of the objects, may not be able to determine the behavior of the objects, may not see elevations of the site, may not have views from multiple camera positions, may not be able to perform time series analysis of objects, may not be able to identify dangerous conditions, and may further include an architecture that is computationally expensive to train and / or has computationally expensive execution times.

[0006] As used herein, a "carrier" may refer to a crane, helicopter, drone, fixed-wing aircraft, lighter-than-air craft (aerostat), satellite, or other mobile vehicle, or a fixed building, pole, tower, tree, cliff, or other object or structure that is generally immobile. All of these may have or provide a view of a site, which may be at ground level or above the site. They may also carry cameras or other sensors ("object sensors") that collect object sensor data for objects within the site.

[0007] In some cases, carrier operators may use equipment that allows them to exercise some degree of control over a load ("load") suspended below the carrier by a hoisting cable. This equipment may use winches to hoist or lower the load relative to the carrier, powered fans (or other actuators) to propel a thrust fluid to generate thrust, reaction wheels, or the like (all of which are referred to herein as "thrusters") to control the load at or near the load. Such equipment is referred to herein as a slung load control system (SLCS). When including thrusters, SLCSs are known to be able to control the yaw angle of the load, and, if the SLCS has a sufficiently high thrust-to-mass ratio, may also be able to control the pendulum motion of the load (e.g., away from the lowest load position below the carrier) or move the load horizontally.

[0008] What is needed are systems, methods, and apparatus for training a machine learning neural network to identify and classify objects found at a site through analysis of data from an object sensor, such as analysis of images of the site, and output a run-time object detection neural network that identifies and classifies objects based on analysis of data from the object sensor, such as analysis of images of the site.

[0009] What is further needed are systems, methods, and apparatus for training a neural network to perform object time series analysis and output a run-time object time series analysis neural network that determines at least one of a site map or the behavior of objects within the site. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 shows a site with a crane, a site monitor machine learning computer, a load control system, a load, a hazardous situation, a warning beacon, vehicles, safety equipment, people on site, site construction materials, and a site building. [Figure 2] FIG. 2 is a network and device diagram illustrating an example of a site monitor machine learning computer, a site monitor machine learning computer data store, a load control system, a site monitor runtime computer system, and a network incorporating the teachings of the present disclosure, according to some embodiments. [Figure 3] FIG. 3 is a functional block diagram illustrating an example of the site monitor machine learning computer of FIG. 2 incorporating teachings of the present disclosure, according to some embodiments. [Figure 4] FIG. 4 is a functional block diagram illustrating an example site monitor machine learning computer data store incorporating teachings of the present disclosure, according to an embodiment of the present disclosure. [Figure 5]FIG. 5 is a flow diagram illustrating an example of a method performed by an object identification neural training module, according to some embodiments. [Figure 6] FIG. 6 is a flow diagram illustrating an example of a method performed by a time series neural training module, according to some embodiments. [Figure 7] FIG. 7 is a flow diagram illustrating an example of a method performed by a runtime generation module, according to some embodiments. [Figure 8] FIG. 8 is a flow diagram illustrating an example of a method performed by a runtime object identification module, according to some embodiments. [Figure 9] FIG. 9 is a flow diagram illustrating an example of a method performed by a runtime object time series analysis module according to some embodiments. [Figure 10] FIG. 10 is a schematic diagram of a site viewed by a site monitor machine learning computer, according to some embodiments. [Figure 11] FIG. 11 is a schematic diagram of a user interface of a visualization module according to some embodiments. [Figure 12] FIG. 12 is a flow diagram illustrating an example of a method performed by an image stitching module, according to some embodiments. [Figure 13] FIG. 13 is a perspective view of a load control system according to some embodiments. [Figure 14] FIG. 14 is a perspective view of a load control system and carrier, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] The following numbers refer to the drawing numbers in Figures 1 through 14. These numbers identify elements of the figures, which elements of the figures represent examples. When a number is followed by a letter, it is understood that elements having a common number, e.g., 1305A and 1305B, represent generally equivalent units.

[0012] In overview, a site monitor machine learning computer (such as site monitor machine learning computer 150 or site monitor machine learning computer 300) has physical and logical components. The physical components may include a housing for the site monitor machine learning computer, a camera or other object sensor (e.g., object sensor 106A or object sensor 106B), a computer processor 315, computer device memory 350, a network interface 330, and input and output interfaces 345 and 340. The object sensors may be located within or part of the housing of the site monitor machine learning computer; for example, object sensor 106A may be integrated into the housing of load control system 125, or may be remote therefrom, such as object sensor 106B.

[0013] Records relating to the object sensors for identifying the object sensors may be stored, for example, as one or more object sensor 460 records, and records relating to the object sensor data may be stored, for example, as one or more object sensor data 405 records, in the site monitor machine learning computer data store 400 in the computer device memory 350.

[0014] The logical components of the site monitor machine learning computer 300 include an object identification neural training module 500, a time series neural training module 600, a runtime generation module 700, a runtime object identification module 800, a runtime object time series analysis module 900, an image stitching module 1200, and a visualization module 210, which may be stored in the site monitor machine learning computer memory 350.

[0015] One or more physical components, such as cameras or other object sensors (e.g., object sensor 106A and object sensor 106B), can be present at one or more locations (hereinafter referred to in the singular as "sensor locations") that have a view of the site 100. A sensor location may be at a relatively high elevation relative to the site and have a high line of sight to the site. A sensor location may be above or below a carrier, such as a construction crane 110, tower, pole, or building 115. The sensor location may be suspended on a suspension cable below the carrier, and the carrier may be moving construction materials. A sensor location may also be a person, drone, or the like that may be moving on or near the site.

[0016] The logic component may train an object detection neural network ("ODNN") of a site monitor machine learning computer, such as object identification neural training module 500, to identify objects in object sensor data 405 recorded by object sensors 460. For example, object identification neural training module 500 may be trained to identify objects based on analysis of images from a camera or based on analysis of other object sensor data from another object sensor. The identified objects may be stored, for example, as one or more object 410 records. The object 410 records may include a weight, strength, or other indicator of confidence ("confidence") in the ODNN's identification of the object 410. Multiple (including multiple overlapping) objects may be identified in the object sensor data processed by the ODNN.

[0017] The ODNN may be further trained to categorize or describe the identified objects, including, for example, carriers, types of carriers, such as cranes, loads carried by cranes, people, authorized people, unauthorized people, safety equipment (e.g., helmets, vests, fences, safety ropes, etc.), site machinery and types of site machinery, such as SLCS, vehicles, saws, pneumatic equipment, hammers, wrenches, etc., site buildings, site materials, contact objects, and other items found at sites over time that the ODNN is trained to recognize and classify.

[0018] The ODNN may be trained to identify and output object 410 records along with the object's category, object label, and the like (hereinafter "object category"). The object category may be stored, for example, in one or more object category 450 records. One or more object categories may be output for each object 410. The object category may further include a confidence level associated with the object category.

[0019] The output of the ODNN may further include an identification of which portions of the object sensor data triggered the identification of the object 410, such as the portions of the image that prompted the ODNN to identify the object and object category. Such portions of the object sensor data may be stored, for example, as one or more object-triggered object sensor data 475 records. Such output may include or enable the identification of the object within the original source object sensor data. For example, such output may include the original image in which the identified object is highlighted or otherwise identified within the image.

[0020] The output of the trained ODNN may further include or be in the form of one or more vectors, tensors, etc. (hereinafter "tensors") that encode at least one of the object sensor data (e.g., images), objects in the object sensor data, object categories (classifications), and / or confidence levels for the objects and / or object categories. Such tensors are stored as one or more object tensor 465 records in the site monitor machine learning computer data store 400.

[0021] Unless otherwise clear from the context, references herein to processing images, categories (classification), sensor data, and / or confidence should be understood to refer to either or both of: i) processing images, categories, object sensor data, and / or confidence; and / or ii) processing tensors encoding images, categories, object sensor data, and / or confidence.

[0022] The trained ODNN is output as an executable runtime object detection neural network 425 record, which can be incorporated into a runtime application, such as a runtime object identification module 800, for example, by a runtime generation module 700.

[0023] The executable ODNN may be executed by a runtime object identification module 800, which may be used, for example, to identify and classify objects at a site such as a construction site. During execution of the runtime object identification module 800, any corrections from a human or other process may be stored and fed back as training data for the ODNN in the object identification neural training module 500.

[0024] The output of the executable ODNN, such as an object tensor encoding the objects identified by the ODNN, the classification (category) of such objects, the object sensor data that triggered the object identification, and / or the confidence level of the object and / or object category, can be input to a time series neural net ("TSNN"), such as in time series neural training (learning) module 600.

[0025] The TSNN may be trained to identify a site map of a site and object behavior within the site, which may include, for example, at least one of object movement, velocity, object movement prior to contact, object contact, instances of a crane picking up an object (hereinafter referred to as a "crane pick"), unsafe conditions, theft, and accidents.

[0026] The trained TSNN is output as an executable runtime sequence analysis neural network 430 record, which can be incorporated into a runtime application, such as a runtime object time series analysis module 900, for example, by the runtime generation module 700.

[0027] The executable TSNN may be executed, for example, by the runtime object time series analysis module 900, which may be used to determine the site map of a site and the behavior of objects on the site. During execution of the runtime object time series analysis module 900, any modifications from humans or other processes may be stored and fed back as training data to the TSNN of the time series neural learning module 600.

[0028] The output of an executable TSNN may include a site map of the site and the behavior of objects on the site. The output of an executable TSNN may further include object sensor data related to object categories. The output of an executable TSNN may further include utilization of equipment, vehicles, and personnel, as well as other outputs such as alarms and notifications in response to hazardous conditions.

[0029] In embodiments, the object sensors discussed herein may include sensors that acquire or generate data or information about objects identified by an ODNN. Object sensors may include, for example, cameras, microphones, accelerometers, voltmeters, ammeters, scales, global positioning systems (GPS), radar systems, sonar systems, depth sensors, LIDAR systems, fluid level sensors, pH sensors, speed sensors, compasses, pressure sensors, magnetic field sensors, electric field sensors, temperature sensors, wind speed sensors, etc., and the sensors may measure information of or related to an object, such as electromagnetic waves reflected or emitted by the object, sound or decibel levels, acceleration, voltage, current, mass or weight, position or location, size, relative position, density, distance, fluid level, pH, speed, orientation, atmospheric pressure, pressure of or pressure on a component, magnetic field, electric field, temperature, wind speed, etc.

[0030] In an embodiment, the object sensor may be part of a system that acquires and processes information and reports the results, which are referred to herein as object sensor data.

[0031] For example, in an embodiment, the object sensor may be an SLCS, such as an SLCS 1300. For example, the SLCS may include a sensor suite (sensor set), actuators such as thrusters, a computer processor, and computer memory, which may include a load control module within the computer memory.

[0032] When executed by the SLCS's computer processor, the load control module can acquire sensor data from sensors in the SLCS and process the SLCS sensor data in or according to a state estimation module, which may include a system model. Some of the information determined by the system model can be described as "state information" or "state," and some can be described as "parameter information" or "parameters." For example, parameters may include elements that can be actively changed by the SLCS, such as the length or motion control settings of the carrier's suspension cables (if the SLCS can control the carrier or obtain information from the carrier) or the thrust output of the SLCS's thrusters (thruster power). For example, "state information" may include elements that cannot be actively changed by the SLCS and / or elements that respond to changes in parameters, such as the mass of the SLCS or load, the moment of inertia of the SLCS or load, the position or movement of the SLCS or load, the position or movement of the carrier, power usage, battery status, and even disturbances such as wind. Importantly, parameter information, state information, and disturbance forces are not "hardwired" into the SLCS as fixed values, but can be dynamically determined by its logical components according to the system model.

[0033] The sensor suite of the SLCS may include a camera, a vector navigation unit, and a remote interface unit. The vector navigation unit of the SLCS may include an inertial measurement unit ("IMU"), also known as an orientation measurement system. The IMU can provide inertial navigation data to the load control module, such as a three-degree-of-freedom (3DoF) accelerometer, a magnetometer or magnetometer such as a gyroscope, a compass, an inclinometer, a direction encoder, a radio frequency relative orientation system, and a gravity or acceleration sensor, which may include a microelectromechanical system (MEMS) sensor. The IMU may include an integrated processor that provides onboard state estimation that fuses data from the sensors within the IMU; in this case, the IMU may be referred to as an inertial navigation system ("INS"). The SLCS may include or be communicatively coupled to one or more sensors in addition to the IMU. Such additional sensors may include, for example, absolute position measurement systems, proximity sensors, LIDAR sensors and systems (e.g., point, sweep, rotation, radial, distance, or linear), ultrasonic sensors, optical sensors such as one or more cameras or infrared (IR) sensors, and thrust sensors equipped to determine thrust output by actuators of the SLCS, such as thrusters. Proximity sensors may include ground height sensors and object proximity sensors. Absolute position measurement systems may include global positioning system (GPS) sensors.

[0034] The remote interface unit of the SLCS provides input to the SLCS and load control module, and may provide, for example, a function mode or command state, or may provide further input such as a location (or position) and / or orientation. The function mode or command state may instruct the SLCS to idle, maintain a position or orientation relative to the carrier, move to or towards a location, hold position, or respond to direct user input. The location or orientation may be the location or orientation of the remote interface unit, or a location or orientation entered into the remote interface unit by a user or other process.

[0035] When executed by the processor of the SLCS, the load control module and the system model of the SLCS can, for example, determine the current state of the SLCS and the load and estimate the future state of the SLCS and the load according to the system model. The current and future states may include the center of track of the carrier, the center of track of the SLCS, the target position of the load, the mass of the SLCS and the load, the length of the lifting cable (if this is a state and not a parameter that can be changed), the moment of inertia of the SLCS and the load, the translation and rotation of the SLCS and the load, the height of the SLCS above the ground, the translation and rotation of the carrier, the height of the carrier above the ground, and estimated wind disturbances acting on the SLCS and the load, and the relative motion between the SLCS and the carrier.

[0036] Based on the current state, estimated future state, and feedback from the functional mode or command state, the load control module of the SLCS may modify parameters of the SLCS to achieve the objectives of the functional mode or command state.

[0037] The sensors of the SLCS may act as object sensors for the site monitor machine learning computer. Condition and parameter information sensed or determined by the SLCS or another object sensor may be stored in the site monitor machine learning computer data store 400 as one or more object sensor data 460 records.

[0038] In embodiments where the object sensor is a camera, an image stitching module, such as image stitching module 1200, may stitch multiple images of the site into one or more composite site images. Through the output of the ODNN and TSNN, site maps, objects, object categories, object behaviors, and object sensor data associated with the object categories may be identified within the composite site images.

[0039] In this manner, the system, method, and / or apparatus can train a neural network to identify and categorize objects at a site through analysis of object sensor data, such as through analysis of images of the site, and output a runtime ODNN, where the runtime ODNN identifies and categorizes objects, the classifications including at least one of carriers, people, authorized people, unauthorized people, safety equipment (hard hats, vests, fences), on-site machinery, on-site buildings, on-site building materials, etc.

[0040] Further, in this manner, the system, method, and / or apparatus can train a neural network to perform time series analysis of objects and output a runtime TSNN, which determines at least one of a site map and / or behavior of the objects. Object behavior can include at least one of object movement, velocity, object movement before contact, object contact, crane pick, unsafe condition, theft, accident, etc. An accident can include, for example, contact between objects, contact between objects where at least one object has high potential or kinetic energy, unexpected contact between objects, abnormal contact between objects, etc.

[0041] The system, method, and / or apparatus may be installed on a construction crane, e.g., a lifting cable. The system, method, and / or apparatus may be a component of or have access to an object sensor, such as an SLCS. The SLCS or other object sensor may provide state or parameter information of objects monitored by the SLCS or object sensor, which state or parameter information may further be used to train the TSNN to determine the behavior of objects detected in images recorded at the site. Additionally, the system, method, and apparatus may output equipment, vehicle, and personnel utilization information. Additionally, the system, method, and apparatus may output alarms or other outputs in response to hazardous conditions, etc.

[0042] This allows the disclosed systems, methods, and / or apparatus to monitor sites and activity on the sites with a machine learning artificial intelligence system.

[0043] A detailed description of the illustrated embodiments follows. While the embodiments are described in connection with the drawings and associated description, there is no intent to limit the scope to the embodiments disclosed herein. On the contrary, the intent is to cover all alternatives, modifications, and equivalents. In alternative embodiments, additional devices or combinations of the illustrated devices may be added or combined without limiting the scope to the embodiments disclosed herein. The embodiments are exemplary and are not intended to limit the disclosed technology to any particular application or platform.

[0044] The phrases "in one embodiment," "various embodiments," "in some embodiments," and the like are used repeatedly. Such phrases do not necessarily refer to the same embodiment. The terms "comprising," "having," and "including" are synonymous unless the context dictates otherwise. As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the content clearly dictates otherwise. It should also be noted that the term "or" is generally synonymous with "and / or" or "and-or" unless the content clearly dictates otherwise.

[0045] 1 illustrates a construction site 100, including a carrier, such as a crane 110, a site monitor runtime computer 105, a camera or object sensor 106A and a camera or object sensor 106B, a suspended load control system 125 ("SLCS 125"), a suspended load 140, a hazardous condition 145, a warning beacon 135, a vehicle 130, a hard hat safety device 123, a visibility safety device 122, people on site 120, including authorized personnel 121 and unauthorized personnel 124, site construction materials 126, and a site building 115, as well as a site monitor machine learning computer 300 and a network 250. The site building 115 may also function as a carrier to hold or transport object sensor 106B.

[0046] Both the site monitor runtime computer 105 and the site monitor machine learning computer 300 are shown in FIG. 1 to convey that the site monitor machine learning computers described herein may be located at the site (on-site) and integrated with object sensors, such as the site monitor runtime computer 105, or may be remote from the site and acquire data from object sensors on the site, such as the site monitor machine learning computer 300.

[0047] As discussed herein, the site monitor machine learning computer (site monitor runtime computer 105 and / or site monitor machine learning computer 300) may execute or operate one or more of the object identification neural training module 500, the time series neural training module 600, the runtime generation module 700, the runtime object identification module 800, the runtime object time series analysis module 900, the image stitching module 1200, and the visualization module 210. In the examples described herein, the site monitor runtime computer 105 may execute a subset of these modules, such as the runtime object identification module 800 and the runtime object time series analysis module 900.

[0048] The physical and logical components of the site monitor machine learning computer may be remotely located, provided by a network or other cloud-based resources, or may be distributed.

[0049] The site monitor machine learning computer may execute an object identification neural training module 500 that trains an object detection neural network ("ODNN") to identify objects at a site, such as the construction site 100. The object identification neural training module 500 may be trained to identify objects, such as, for example, a carrier, e.g., a crane 110, an authorized person (e.g., an authorized person 121), an unauthorized person (e.g., an unauthorized person 124), a person performing a task (e.g., a task person 132), safety equipment (e.g., visibility safety equipment 122 (e.g., vest, color, etc.), personal protective safety equipment 123 (e.g., hard hat, pads, gloves, etc.), site machinery, a vehicle (e.g., a vehicle 130), a site building (e.g., a site building 115), and site building materials (e.g., site building materials 126 (e.g., boards, pipes, glass sheets, drywall, sandbags, etc.)).

[0050] The site monitor runtime computer 105 can obtain a pre-trained ODNN from or via the site monitor machine learning computer 300. The site monitor machine learning computer 300 can aggregate multiple images from multiple sites from multiple instances of the site monitor runtime computer 105 and train an ODNN for distribution to multiple instances of the site monitor runtime computer 105. In this way, a central or collector instance of the object identification neural training module 500 within the site monitor machine learning computer 300 can train an ODNN across images of multiple sites.

[0051] The site monitor runtime computer 105 may execute a runtime object identification module 800 including an ODNN to identify objects within a site (worksite) such as the construction site 100. Objects identified by the ODNN in the runtime object identification module 800 include, for example, carriers, types of carriers (e.g., cranes), suspended loads, authorized personnel, unauthorized personnel, safety equipment, such as helmets, vests, fences, safety ropes, etc., site machinery and / or types of site machinery, such as SLCS, vehicles, saws, pneumatic equipment, hammers, wrenches, etc., site buildings, site building materials, etc.

[0052] The output of the runtime object identification module 800 may include, for example, a labeled or classified image, e.g., a still or video image with a highlighted area around the identified object and a classification associated with the highlighted object, and a confidence level for the object and / or classification.

[0053] The runtime object identification module 800 may further obtain user feedback, e.g., corrections, related to the identified objects and / or their classifications. The runtime object identification module 800 may update the classifications according to the user feedback and may store the updated and confirmed object classifications as training data for use by the object identification neural training module 500.

[0054] The output of the runtime object identification module 800 may further include a non-human readable output that encodes the object, such as a tensor suitable for feeding into another neural network.

[0055] The objects and classifications identified by the runtime object identification module 800, or encoded therefrom into tensors, may be provided to the runtime object time series analysis module 900. The runtime object time series analysis module 900 implements a time series neural network ("TSNN"). The TSNN processes the tensors or objects from the runtime object identification module 800 and is trained to identify aspects of the objects that require time series analysis (e.g., a site map of a site). The site map identifies the locations and boundaries of various types of objects.

[0056] The TSNN may identify object movements and behaviors including object movements that lead to or precede contact between objects, injury or accident, etc. For example, the TSNN may identify danger zones 145 below or adjacent to the carrier 110 and the suspended load 140, vehicles 130 within danger zones 145, authorized personnel 121 within danger zones 145, areas ahead of the vehicle 130 when the vehicle 130 is moving, etc.

[0057] The TSNN may further identify object behavior. The object behavior may include, for example, at least one of object movement, velocity, object movement prior to contact, object contact, crane pick, unsafe condition, theft, and accident. The accident may include, for example, contact between objects, contact between objects where at least one object has high potential or kinetic energy, unexpected contact between objects, abnormal contact between objects, etc.

[0058] The TSNN may be trained to provide the aforementioned types of output by the time series neural learning module 600. Similar to the ODNN, one or both of the site monitor runtime computer 105 and the site monitor machine learning computer 300 may execute the time series neural training module 600 to train the TSNN. The site monitor machine learning computer 300 may aggregate large training datasets from multiple objects from multiple sites and multiple instances of the site monitor runtime computer 105 to train the TSNN for distribution to multiple instances of the site monitor runtime computer 105. The site monitor runtime computer 105 may then obtain pre-trained TSNNs from or via the site monitor machine learning computer 300. In this manner, a central or collector instance of the time series neural training module 600 can train a TSNN across imagery or other sensor data from multiple sites. During execution of the runtime object time series analysis module 900, human corrections may be saved and fed back as training data to the TSNN in the time series neural training module 600.

[0059] The runtime object time series analysis module 900 may also determine utilization of equipment, vehicles, and personnel (e.g., utilization of cranes 110), utilization or consumption rates of materials (e.g., on-site construction materials), construction rates of buildings (e.g., on-site construction, etc.), etc.

[0060] The runtime object time series analysis module 900 may identify behaviors such as hazardous conditions in the danger zone 145 below the load 140 suspended by the crane 110, in the area traversed by the vehicle 130, and the like. The runtime object time series analysis module 900 may identify hazardous conditions based on training and / or based on object motion preceding an accident. In this case, the accident may be identified to the TSNN and the accident and preceding motion fed back into the training data (as described herein), or the accident may be identified by the TSNN and the accident and preceding motion fed back in the training data. An accident may include, for example, contact between objects, contact between objects where at least one object has high speed, high potential energy, or high kinetic energy, unexpected contact between objects, unusual contact between objects, etc.

[0061] As described herein, various of the outputs of the runtime object time series analysis module 900 may trigger alarms, such as warning outputs by the warning beacon 135 or other outputs.

[0062] The site monitor machine learning computer may further execute an image stitching module 1200, for example, stitching multiple images of the site 100 to create a composite site image 455. The output of the runtime object identification module 800 and / or the runtime object time series analysis module 900, and the sensor data associated with the categories, may be combined with or output in association with an image of the site 100, for example, a composite image of the site 100. The composite image and output are further discussed in connection with the image stitching module 1200, the visualization module 210, and the examples in FIGS. 10, 11, and 12.

[0063] FIG. 2 is a network and device diagram illustrating an example of a site monitor machine learning computer 300, a site monitor machine learning computer data store 400, a suspended load control system 125 (“SLCS 125”), a site monitor runtime computer system 105, and a network 250 incorporating the teachings of the present disclosure, according to some embodiments. The SLCS 125 should be understood as an example of an object sensor.

[0064] The site monitor machine learning computer 300 is shown connected to a site monitor machine learning computer data store 400. The site monitor machine learning computer data store 400 is described further herein, but should generally be understood as a data store used by the site monitor machine learning computer 300.

[0065] Network 250 includes computers, network connections between the computers, and software routines that enable communication between the computers over the network connections. Examples of network 250 include an Ethernet network, the Internet, and / or a wireless network, such as GSM, TDMA, CDMA, EDGE, HSPA, LTE, a satellite network, or other network provided by a wireless service provider. Connection to network 250 may be via a Wi-Fi connection. Multiple networks may be involved in a communication session between the illustrated devices. Connection to network 250 may require the computers to execute software routines that enable, for example, the seven layers of the OSI model of computer networking or an equivalent in wireless telephone networks.

[0066] 3 is a functional block diagram illustrating an example of a site monitor machine learning computer 300 incorporating teachings of the present disclosure, according to some embodiments. The site monitor machine learning computer 300 may include a chipset 355. The chipset 355 may include a processor 315, input / output (I / O) port(s), peripheral devices such as outputs 340 and inputs 345, a network interface 330, and computer device memory 350, all interconnected via a bus 320. The network interface 330 may be utilized to connect to the network 250, to the site monitor machine learning computer data store 400, or to form device-to-device connections with other computers.

[0067] Chipset 355 may include communication components and / or paths, such as bus 320, that connect processor 315 to peripheral devices, such as outputs 340 and inputs 345, which may be connected, for example, via I / O ports. The processor 315 may include one or more execution cores (CPUs). For example, the chipset 355 may also include a peripheral controller hub (PCH) (not shown). In another example, the chipset 355 may also include a sensor hub (not shown). The input 345 and output 340 may include, for example, user interface device(s) including displays, touchscreen displays, printers, keypads, keyboards, etc.; sensor(s) including accelerometers, global positioning systems (GPS), gyroscopes, etc.; wired and / or wireless communication logic; and storage device(s) including hard disk drives, solid state drives, removable storage media, etc. The I / O ports for input 345 and output 340 may be configured to transmit and / or receive commands and / or data according to one or more communication protocols. For example, one or more I / O ports may conform to and / or be compatible with the Universal Serial Bus (USB) protocol, a Peripheral Component Interconnect (PCI) protocol (e.g., PCI Express (PCIe)), etc.

[0068] The hardware acceleration module 310 can provide hardware acceleration of various functions performed by the object identification neural training module 500, the time series neural training module 600, the runtime generation module 700, the runtime object identification module 800, the runtime object time series analysis module 900, the image stitching module 1200, and the visualization module 210. The hardware acceleration module may be provided, for example, by Intel Corporation's Integrated Performance Primitives software library, may be executed by an Intel (or other compatible) chip, and may implement, for example, a library of programming functions involved in real-time computer vision and machine learning systems. Such libraries include, for example, OpenCV. OpenCV includes application areas such as 2D and 3D feature toolkits, egomotion estimation, face recognition, gesture recognition, human-computer interaction, mobile robotics, action understanding, object identification, segmentation and recognition, stereoscopic stereo vision (including depth perception from two cameras), structure from motion, motion tracking, and augmented reality. OpenCV also includes a statistical machine learning library that includes boosting, decision tree learning, gradient boosted trees, expectation maximization algorithm, k-nearest neighbor algorithm, naive Bayes classifier, artificial neural networks, random forests, and support vector machines.

[0069] The hardware acceleration module may be provided, for example, by NVIDIA® CUDA-X libraries, tools, and technologies built on NVIDIA® CUDA technology. Such libraries may include, for example, mathematical libraries, parallel algorithm libraries, image and video libraries, communication libraries, deep learning libraries, and partner libraries. The mathematical libraries may include, for example, a GPU-accelerated Basic Linear Algebra (BLAS) library, a GPU-accelerated library for fast Fourier transforms, a GPU-accelerated standard mathematical function library, a GPU-accelerated random number generation (RNG), GPU-accelerated dense and sparse direct solvers, a GPU-accelerated BLAS for sparse matrices, a GPU-accelerated tensor linear algebra library, a GPU-accelerated linear solver for simulation and implicit unstructured methods. The parallel algorithm library may include, for example, a GPU-accelerated library of C++ parallel algorithms and data structures. The image and video libraries may include, for example, a GPU-accelerated library for JPEG decoding, GPU-accelerated image, video, and signal processing functions, a set of APIs, examples, and documentation for hardware-accelerated video encoding and decoding on various operating systems, and a software developer kit that exposes the hardware capabilities of NVIDIA TURING™ GPUs specialized for calculating relative pixel motion between images. Communication libraries may include standards for GPU memory with extensions to improve performance on GPUs, and open-source libraries for fast multi-GPU, multi-node communication that maximizes bandwidth while maintaining low latency. Deep learning libraries may include, for example, GPU-accelerated libraries of primitives for deep neural networks, deep learning inference optimizers and runtimes for production deployment, real-time streaming analytics toolkits for AI-based video understanding and multi-sensor processing, and open-source libraries for image and video decoding and augmentation to accelerate deep learning applications. Partner libraries may include, for example, OpenCV, FFmpeg, ArrayFire, Magma, IMSL Fortan Numerical Library, Gunrock, Cholmod, Triton Ocean SDK, CUVIlib, etc.

[0070] In an embodiment, hardware acceleration module 310 may be or include a programmed field programmable gate array ("FPGA"), i.e., an FPGA including a gate array configured with a bitstream to embody the logic of a hardware acceleration function (equivalent to the logic provided by executable instructions of a software embodiment of the function). In an embodiment, hardware acceleration module 310 may include or support a component of computer device memory 350.

[0071] The computing device memory 350 may generally include random access memory ("RAM"), read-only memory ("ROM"), and mass storage devices such as disk drives or synchronous dynamic random-access memory (SDRAM). The computing device memory 350 may store program code for modules and / or software routines, such as the hardware acceleration module 310, the object identification neural training module 500, the time series neural training module 600, the runtime generation module 700, the runtime object identification module 800, the runtime object time series analysis module 900, the image stitching module 1200, and the visualization module 210.

[0072] The computer device memory 350 may also store an operating system 380. These software components may be loaded into the computer device memory 350 from a non-transitory computer readable storage medium 396 using a drive mechanism associated with the non-transitory computer readable storage medium 396, such as a floppy disk, tape, DVD / CD-ROM drive, memory card, or other similar storage medium. In some embodiments, software components may also be loaded via or instead of a drive mechanism and mechanism other than the computer-readable storage medium 396 (eg, the network interface 330).

[0073] Additionally, computer device memory 350 is illustrated as including kernel 385, kernel space 395, user space 390, user protected address space 360, and site monitor machine learning computer data store 400 (shown in FIG. 4 and further described in connection therewith).

[0074] The computing device memory 350 can store one or more processes 365 (i.e., running software applications). The processes 365 may be stored in user space 390. The processes 365 may include one or more other processes 365a...365n. The one or more processes 365 may generally be executed in parallel, i.e., as multiple processes and / or multiple threads.

[0075] Computer device memory 350 is further shown as storing operating system 380 and / or kernel 385. Operating system 380 and / or kernel 385 may be stored in kernel space 395. In some embodiments, operating system 380 may include kernel 385. Operating system 380 and / or kernel 385 may attempt to protect kernel space 395 and prevent access by certain processes 365a...365n.

[0076] Kernel 385 may be configured to provide an interface between user processes and circuitry associated with site monitor machine learning computer 300. In other words, kernel 385 may be configured to manage access by processes 365 to processor 315, chipset 355, I / O ports, and peripheral devices. Kernel 385 may include one or more drivers configured to manage and / or communicate with elements of site monitor machine learning computer 300 (i.e., processor 315, chipset 355, I / O ports, and peripheral devices).

[0077] Site monitor machine learning computer 300 may also communicate with or include site monitor machine learning computer data store 400, shown in and further described in connection with FIG. 4, via bus 320 and / or network interface 330. In various embodiments, bus 320 may include a high-speed serial bus, and network interface 330 may be coupled to a storage area network (“SAN”), a high-speed wired or wireless network, and / or via other suitable communication technologies. Site monitor machine learning computer 300 may, in some embodiments, include more components than are shown. However, not all components need be shown to disclose example embodiments.

[0078] Figure 4 is a functional block diagram of a site monitor machine learning computer data store 400 depicted in the site monitor machine learning computer 300 of Figure 3, according to some embodiments. Components of the site monitor machine learning computer data store 400 may include data used by modules and / or routines, such as object sensor data 405 (e.g., images, state and parameter information from the SLCS, etc.), objects 410, physical properties 415, behaviors 420, runtime object detection neural network 425, runtime time series analysis neural network 430, object identification training data 435, time series training data 440, site map 445, object categories 450, synthetic site images 455, object sensors 460, object tensors 465, and object triggering object sensor data 475. The data groups used by the modules or routines shown in FIG. 4 may be represented by cells of columns or values ​​separated from other values ​​in a defined structure within the digital document or file containing the neural network. Although referred to herein as individual records or entries, a record may have multiple database entries, which may be, represent, or encode numbers, numeric operators, binary values, logical values, text, string operators, references to other database entries, joins, conditional logic, tests, and the like.

[0079] The components of the site monitor machine learning computer data store 400 are further described in the descriptions of other figures herein.

[0080] 5 is a flow diagram illustrating an example of a method performed by an object identification neural training module 500 incorporating teachings of the present disclosure, according to some embodiments. This module may be performed by or with the assistance of a hardware accelerator, such as the hardware acceleration module 310. This module may be performed by, for example, one or both of the site monitor runtime computer 105 and / or the site monitor machine learning computer 300.

[0081] Opening (start) loop block 505 through closing (end) loop block 530 may iterate through one or more new object detection neural networks ("ODNNs"), such as the first time an ODNN is trained.

[0082] At block 510, the object identification neural training module 500 may obtain a training data set. The training data set may include object sensor data, e.g., images, in which objects are identified and labeled or categorized. The training data set may be obtained, for example, from one or more object identification training data 435 records. The training data set may be prepared, for example, from user feedback from block 825.

[0083] In block 515, the object identification neural training module 500 may select a mapping function for an object detection neural network (“ODNN”), such as a convolutional neural network (“CNN”), such as a region-based CNN.

[0084] At block 520, the object identification neural training module 500 may set the scale of the ODNN, where the scale may include depth, width, and resolution. The depth may include, for example, the number of convolutional layers of the ODNN. The width may include, for example, the number of channels in each convolutional layer of the ODNN. The resolution may include, for example, the resolution of the images passed to the ODNN. The width, depth, and / or resolution may be set based on computer resources, such as resources within the site monitor runtime computer system 105.

[0085] In block 525, the object discrimination neural training module 500 may initialize the optimizer weights used in the loss function of the ODNN. The weights may be initialized, for example, with small random values. The loss function measures the discrepancy between the target output and the computed output. For classification purposes, a loss function such as categorical cross-entropy can be used.

[0086] At return block 530, the object identification neural training module 500 returns to the opening loop block 505 to iterate over any new ODNNs.

[0087] Opening loop block 535 through closing loop block 570 may iteratively process a new or existing ODNN.

[0088] Opening loop block 536 through closing loop block 541 may be processed on the new ODNN.

[0089] In block 540, the object identification neural training module 500 may provide the ODNN with a portion of the training dataset of block 510 to train the ODNN to identify objects in images and classify or label the objects. During this block, the object identification neural training module 500 may optimize filter weights applied to a loss function, such as through backpropagation.

[0090] In block 545, the object identification neural training module 500 may test the ODNN on untrained training data (e.g., a portion of the training data set in block 510 that was not previously provided during training in block 540) to determine whether the ODNN returns acceptable results.

[0091] At decision block 550, the object identification neural training module 500 can determine whether the ODNN produces an acceptable error rate when rejecting objects that it was not trained to classify (categorize) or label, and when identifying and classifying objects that it was trained to identify and classify. An acceptable error rate is, for example, less than 10%.

[0092] If decision block 550 returns no or equivalent, then in block 555, the object identification neural training module 500 may adjust the weights of the optimizer used in the loss function. The weights may be adjusted, for example, proportional to the derivative of the error. The scale of the CNN may also be adjusted if necessary or desirable.

[0093] In block 560 , the object identification neural training module 500 may prepare or obtain additional training set data and return to block 535 .

[0094] In block 565, which may follow decision block 550 after a positive or equivalent determination, object identification neural training module 500 may output a runtime object detection neural network. The output runtime object detection neural network may be stored, for example, as one or more runtime object detection neural network 425 records.

[0095] In block 599, the object identification neural training module 500 may return to the module that called it and / or another process and / or exit.

[0096] 6 is a flow diagram illustrating an example of a method performed by a time-series neural training module 600 incorporating teachings of the present disclosure, according to some embodiments. This module may be performed by or with the assistance of a hardware accelerator, such as the hardware acceleration module 310. This module may be performed by, for example, one or both of the site monitor runtime computer 105 and / or the site monitor machine learning computer 300.

[0097] One or more new TSNNs can be processed iteratively from opening loop block 605 to closing loop block 630 .

[0098] At block 610, the time series neural training module 600 may obtain a first training data set including object sensor data, e.g., an image of a site, a site map of the site, objects identified in the object sensor data, e.g., objects identified in the image of the site, categories of objects identified in the object sensor data, and object behaviors. Some or all of the training data may be encoded into tensors.

[0099] The training dataset may be obtained from one or more time-series training data 440 records, which may consist of tensors of size (s x b x t), where "s" is the number of parameters used for training, "b" is the batch size that determines the number of time steps included as labeled data chunks, and "t" is the number of batch instances included in the training matrix. The label data may contain a vector of length “t”, where every training batch is a (sxb) matrix labeled as one or more of a single sitemap, object, object category, behavior, etc.

[0100] In block 615, the time series neural training module 600 may select a mapping function for a neural network that performs well in analyzing time series data, such as a recurrent neural network, e.g., a long short-term memory architecture recurrent neural network ("LSTM RNN"), which may include, for example, a cell, an input gate, an output gate, and a forget gate.

[0101] In block 620, the time series neural training module 600 may scale the LSTM RNN. The scaling may include depth and width. The depth may be the number of layers. The width may be the number of channels in each layer. The scaling may be at least in part according to runtime execution resources, for example, in the site monitor runtime computer system 105. The scaling may also be according to the computational requirements presented by the tensors input by the runtime object identification module 800 to the runtime object time series analysis module 900.

[0102] In block 625, the time-series neural training module 600 may initialize weights for an optimizer, such as a gradient-based optimizer, used in the loss function of the LSTM RNN. The weights may be initialized, for example, with small random values.

[0103] At closing loop block 630, the time series neural training module 600 may return to opening loop block 605 and iterate with any other new TSNNs.

[0104] The opening loop block 635 through the closing loop block 670 may be repeated for a new or existing TSNN instance.

[0105] In block 640, the time series neural training module 600 may feed a portion of the training dataset of block 610 to an LSTM RNN to train the LSTM RNN to create a site map of the site and identify object behavior over time.

[0106] In block 645, the time series neural training module 600 may test the LSTM RNN on untrained training data, e.g., a portion of the training data set of block 610 that was not provided during training in block 640, to determine whether the LSTM RNN returns acceptable results. During this testing, the previously unprovided portion of the training data of block 610 may be provided without the site map and behaviors that may have been used during training in block 640 or present in the training data. The site map and behaviors may be used to test the results of the LSTM RNN on the untrained training data.

[0107] At decision block 650, the time series neural training module 600 may determine whether the LSTM RNN produces an acceptable error rate in labeling the time series object data relative to the untrained training data of block 645. For example, does the LSTM RNN produce site maps and behaviors within an acceptable error rate or margin relative to the site maps and behaviors in the original training data of block 610?

[0108] If decision block 650 returns no or equivalent, then time series neural training module 600 may adjust the weights of the optimizer used in the loss function at block 655. The weight adjustment may be proportional to the derivative of the error, for example. If necessary or desirable, the scaling of the LSTM RNN may also be adjusted.

[0109] At block 660, the time series neural training module 600 may prepare or acquire additional training set data and return to block 635.

[0110] Following decision block 650 after a positive or equivalent determination, in block 665, the time series neural training module 600 may output the runtime time series neural network. The output runtime time series neural network may be stored, for example, as one or more runtime time series analysis neural network 430 records. The runtime time series analysis neural network 430 records may be used, for example, by the runtime object time series analysis module 900.

[0111] At block 699, the time series neural training module 600 may exit and / or return to the module that called it and / or another process.

[0112] 7 is a flow diagram illustrating an example of a method performed by a runtime generation module 700 incorporating teachings of the present disclosure, according to some embodiments. This module may be performed by or with the assistance of a hardware accelerator, such as the hardware acceleration module 310. This module may be performed by, for example, one or both of the site monitor runtime computer 105 and / or the site monitor machine learning computer 300.

[0113] Opening loop block 705 through closing loop block 755 may iterate through the runtime application generated by runtime generation module 700 , such as runtime object identification module 800 and / or runtime object time series analysis module 900 .

[0114] At block 710, the runtime generation module 700 may receive, obtain, or generate code for hardware interfaces or input / output. Hardware interfaces may enable the runtime execution computer's hardware to interface with one or more humans, to interface with processes, and to interface with hardware input / output devices.

[0115] The hardware interfaces prepared or obtained by the runtime generation module 700 may enable the site monitor runtime computer system 105 and modules executed by the site monitor runtime computer system 105, such as the runtime object identification module 800 and the runtime object time series analysis module 900, to interface with one or more humans, with other processes, and with hardware input / output devices.

[0116] For example, the hardware interface prepared or acquired by the runtime generation module 700 may allow the runtime object identification module 800 to acquire sensor data, such as an image (digital photograph), or a series of images, such as a video, from a camera in or on the site monitor runtime computer system 105, and the runtime object identification module 800 to process the images using an ODNN to identify or classify objects in the images and output the objects, object categories, and confidence levels in a computer-readable format, such as an object tensor 465 record, and in a human-readable format, such as an object sensor data 405 record, including, for example, the images input to the runtime object identification module 800. The human readable format may further include object trigger object sensor data 475 records, object category 450 records, etc. that identify the portion of the object sensor data that triggered the object identification and / or classification (category). The hardware interface may further enable the runtime object identification module 800 to receive human modifications related to object identification, object category, etc. in a human interface such as may be generated by the visualization module 210.

[0117] For example, the hardware interface prepared or obtained by the runtime generation module 700 may enable the runtime object time series analysis module 900 to obtain time series object data from the runtime object identification module 800, including, for example, in the form of object tensor 465 records, and the runtime object time series analysis module 900 may process such object tensor 465 records with the runtime time series analysis neural network 430 to determine site maps and object behaviors and output the results to other processes and to human users, for example, for human review or use. For example, the output may enable the visualization module 210 to generate the user interface 1100 and an alert system, such as the alert beacon 135, to issue an alert in response to a behavior that triggers an alert.

[0118] The hardware interface may be for a human-computer interface such as a tablet computer, laptop, etc. The hardware interface for the human-computer interface may include audio, visual, keyboard, and tactile human input and output. The hardware interface for the human-computer interface may also allow human input to modules of the site monitor runtime computer system 105, such as the visualization module 210, to allow review and interpretation of the output of the time series analysis of object data for site maps, behavior, etc.

[0119] At block 715, the runtime generation module 700 may obtain or receive a runtime sub-module to be used in the current module being generated. For example, the runtime generation module 700 may obtain an executable neural network to be used in the current runtime application being prepared. For example, when preparing the runtime object identification module 800 , the runtime generation module 700 may obtain or receive the runtime object detection neural network 425 . For example, when preparing the runtime object time series analysis module 900, the runtime generation module 700 may obtain or receive the runtime time series analysis neural network 430. For example, the runtime generation module 700 may obtain executable code for the visualization module 210, which may output the user interface 1100, for example.

[0120] At block 716, the runtime generation module 700 may associate the object sensor data with a category. This may be done for the visualization module 210, for example, if not done during preparation of the visualization module 210, so that the object sensor data 405 record is associated with a category whose value may be output in the user interface when an object having the associated category is displayed in the user interface.

[0121] An example of this is described in relation to user interface 1100. As discussed herein, in embodiments, object sensor data may measure information about or related to an object, such as electromagnetic radiation reflected or emitted by the object, sound or decibel level, acceleration, voltage, current, mass or weight, position or location, size, relative position, density, distance, fluid level, pH, speed, orientation, atmospheric pressure, pressure on or of a component, magnetic field, electric field, temperature, wind speed, etc. As discussed herein, in an embodiment, the object sensor data may be state and parameter information from the SLCS. The object sensor data may be collected at the same time as the data that leads to or underlies the generation of the object tensor is collected. If not already done, object sensor data 405 records may be associated with one or more object categories by runtime generation module 700. For example, state and parameter information from an SLCS may be associated with one or more categories, such as SLCS, load, carrier, etc. As described in connection with user interface 1100 and visualization module 210, if an SLCS is displayed in a video in user interface 1100, and if such an SLCS is also labeled by the ODNN with an “SLCS” category, object sensor data from the SLCS may be output to the user interface so that the user can view the object sensor data from the SLCS when the SLCS is displayed in a video in the user interface. This association between object sensor data and categories differs from the association between object 410 records and object category 450 records generated by the ODNN, because the former is an association made by human judgment and ultimately expressed as a programmed association without a confidence level, whereas the latter is made by the ODNN with a confidence level.

[0122] In block 720, the runtime generation module 700 may configure a hardware interface to receive and output appropriate data structure(s), for example, for the site monitor runtime computer system 105 or the site monitor runtime computer system 300, and the hardware execution computer system environment for the runtime block 715.

[0123] At block 725, the runtime generation module 700 may output the current runtime application.

[0124] In decision block 730, the runtime generation module 700 may test the current runtime application in a runtime hardware emulator, such as an emulator of the site monitor runtime computer system 105, to determine whether an error occurred.

[0125] If decision block 730 returns yes or equivalent, then in decision block 735, runtime generation module 700 may determine whether the error was an error in the hardware I / O or an error in the executable, such as in the neural network.

[0126] If at decision block 735, the runtime generation module 700 debugs or can cause the hardware I / O to be debugged, at block 740. Following block 740, the runtime generation module 700 may return to block 710.

[0127] If so, at decision block 735, the runtime generation module 700 may retrain the neural network, such as by calling the time series neural training module 600, or may send the other executable to be debugged, at block 745. Following block 745, the runtime generation module 700 may return to block 715, for example.

[0128] If the answer at decision block 730 is no or equivalent, at block 750 the runtime generation module 700 may output the current runtime application to the runtime object identification module 800, the runtime object time series analysis module 900, or the visualization module 210, etc. In an embodiment, the runtime generation module 700 may output the runtime application with the neural network as a separate module, such as one or more runtime object detection neural network 425 records or runtime time series analysis neural network 430 records, which may be updated or upgraded separately from the runtime application.

[0129] At block 799, the runtime generation module 700 may terminate and / or return to the module that called it and / or another process.

[0130] 8 is a flow diagram illustrating an example of a method performed by a runtime object identification module 800 incorporating teachings of the present disclosure, according to some embodiments. The runtime object identification module 800 may be performed by or with the assistance of a hardware accelerator, such as the hardware acceleration module 310. This module may be performed, for example, by the site monitor runtime computer system 105, which may be similar to the site monitor machine learning computer 300, as described above.

[0131] In block 805, the runtime object identification module 800 may, for example, initialize the hardware I / O of the site monitor runtime computer system 105 and connect the computer system's modules to the computer system's inputs and outputs.

[0132] In block 810, the runtime object identification module 800 may receive one or more object sensor data, which may include images of the site captured by a camera of the site monitor runtime computer system 105, which may include objects such as load control systems, loads, vehicles, buildings, carriers, cranes, construction materials, people, identification indicia such as helmets or vests having colors or codes, etc.

[0133] At block 815, the runtime object identification module 800 may identify objects in the object sensor data using an object detection neural network, such as the runtime object detection neural network 425. Further, at block 815, the object identification module 800 may determine one or more categories associated with the identified objects. Further, at block 815, the object identification module 800 may determine a confidence level for one or both of the identified objects and / or categories. One or more objects may be stored by the runtime object identification module 800, for example, in one or more object 410 records. One or more categories may be stored by the runtime object identification module 800, for example, in one or more object category 450 records.

[0134] At block 820, the runtime object identification module 800 may output the object, object category, and sensor data portion associated with the object, e.g., an image portion, possibly along with a timestamp, to a user or other process. As discussed herein, the identified objects may be stored, for example, as one or more object 410 records. The object 410 records may include a weight, strength, or other indicator of confidence (“confidence”) for the identification of the object 410. Multiple (including multiple duplicates) objects may be identified in the object sensor data processed by the ODNN executed by the runtime object identification module 800 (e.g., runtime object detection neural network 425 records). The object categories may be stored, for example, in one or more object category 450 records. One or more object categories may be output for each object 410. The object categories may further include a confidence associated with the object category. Examples of object categories include carriers, types of carriers (e.g., cranes), loads carried by cranes, people, authorized people, unauthorized people, safety equipment (e.g., helmets, vests, fences, safety ropes, etc.), site machinery and / or types of site machinery (e.g., SLCS, vehicles, saws, pneumatic equipment, hammers, wrenches, etc.), site buildings, site construction materials, contact objects, and other objects found at the site over time that the ODNN may be trained to recognize and classify. The output of block 820 may further include an identification of which portion of the input object sensor data triggered the identification of the object 410, such as the portion of the image that caused the ODNN to identify the object and object category. Such portions of the object sensor data may be stored, for example, as one or more object trigger object sensor data 475 records. Such output may include or allow for the identification of objects within the original source object sensor data. For example, such output may include an original image with identified objects highlighted or otherwise identified within the image.

[0135] The output of block 820 may be provided to a user of the site monitor machine learning computer for review or correction, or may be provided to another human for review or correction, such as a human user of a "captcha" system, or via AMAZON MECHANICAL TURK or MTURK, operated by Amazon Mechanical Turk, Inc.

[0136] At decision block 825, the runtime object identification module 800 may determine whether a user, human, or process has provided a modification to the object's identification or its classification.

[0137] If decision block 825 returns yes or equivalent, then runtime object identification module 800 may update the object and / or object's category based on the user's modifications at block 830. For example, the user may identify that an image portion associated with the object is incorrect, e.g., not an object, and / or the user may be given the opportunity to deselect and / or select portions of the underlying image for identification as the object. For example, the user may identify that one or more categories assigned to the identified object are incorrect and provide a preferred category.

[0138] In block 835, the runtime object identification module 800 may store the updated object sensor data, sensor data portions, and object categories in one or more object identification training data 435 records. For example, the training data may be used to train an ODNN used in the runtime object identification module 800 to generate new runtime object detection neural network 425 records.

[0139] In block 840, the runtime object identification module 800 may output one or more object tensors, for example as one or more object tensor 465 records, that encode the object sensor data, for example, the image, the identified object, the identified category, the object sensor data portion associated with the object, for example, the image portion or the object sensor data 475 record that triggers the object, and the confidence level, for use by another process, for example, the runtime object time series analysis module 900.

[0140] In block 845, the runtime object identification module 800 may output the confirmed or unmodified object sensor data, e.g., images, confirmed objects, confirmed categories, and object sensor data portions associated with the objects, e.g., image portions and categories, in one or more object identification training data 435 records. For example, the training data can be used to train an ODNN. This output may be in the form of one or both of the original images or original object sensor data 405, and may be in the form of object tensor 465 records associated with the original images or original object sensor data 405.

[0141] In block 899, the runtime object identification module 800 may terminate and / or return to the module that called it and / or another process.

[0142] 9 is a flow diagram illustrating an example of a method performed by a runtime object time-series analysis module 900 incorporating teachings of the present disclosure, according to some embodiments. The runtime object time-series analysis module 900 may be performed by or with the assistance of a hardware accelerator, such as the hardware acceleration module 310. This module may be performed, for example, by the site monitor runtime computer system 105, which may be similar to the site monitor machine learning computer 300, as described above.

[0143] In block 905, the runtime object time series analysis module 900 can initialize the hardware I / O of the site monitor runtime computer system 105 and connect the inputs and outputs of the computer system with the modules of the computer system.

[0144] In block 910, the runtime object time series analysis module 900 can obtain a time series of object tensors from the object tensor 465 records, e.g., from the runtime object identification module 800. The object tensors may encode object sensor data, e.g., images, objects identified within such object sensor data, categories of such objects, object sensor data portions, e.g., image portions, associated with the objects, categories, and confidence levels of such object identifications and categorizations.

[0145] In block 915, the runtime object time series analysis module 900 may load the object tensor 465 records of block 910 into a TSNN, such as the runtime time series analysis neural network 430, and run the TSNN.

[0146] In block 920, the runtime object time series analysis module 900 can obtain analysis output from the TSNN, for example, from the runtime time series analysis neural network 430. The analysis may include, for example, site maps, runtime object behaviors (e.g., object movement, speed, object movement prior to contact, object contact, crane pick, hazardous conditions, theft, accidents, etc.) generated by the time series analysis neural network 430 and what it is trained to identify. The analysis may be stored, for example, as one or more site map 445 or behavior 420 records.

[0147] In block 925, the runtime object time series analysis module 900 may output the analysis of block 920 to a user interface, e.g., a tablet computer, a laptop computer, a smart phone, a dedicated monitor, another process, etc. The user interface may be operated, for example, by the user interface 210 module in the user interface 1100. The analysis may be output to identify the objects identified by the runtime object identification module 800, a site map, and the behavior of the objects identified in block 920. The output may further output object sensor data associated with categories of such objects.

[0148] As an example, if the object sensor data processed by the ODNN in the runtime object identification module 800 is image data, such as a photograph or video of a site, the image data may be output in a human-readable format, such as an object sensor data 405 record, along with highlights of the object sensor data 475 that trigger the object. Such highlights identify the object in the image data along with a category associated with the object, such as a category assigned by the ODNN. The user interface may also output highlighted or identified objects within the site map 445 and / or composite site image 455 along with information from the behavior 420 records associated with the identified corresponding related objects.

[0149] The user interface may further output object sensor data associated with the object in association with the object within the user interface, such as by block 716 of the runtime generation module 700. For example, if the object sensor includes an SLCS, and the SLCS provides object sensor data including state and parameter information for the SLCS, and an SLCS object or object is displayed in a graphical output within the user interface, the object sensor data may be output in association with some or all of the state and parameter information provided by the SLCS.

[0150] At decision block 930, the runtime object time series analysis module 900 may determine whether any corrections, updates, or additional information has been received from a user or another process related to the site map or object behavior generated by analysis of the TSNN and output at block 925.

[0151] If decision block 930 returns yes or equivalent, then runtime object time series analysis module 900 may update the analysis with corrections, updates, or additional information at block 935. For example, a user may identify that a behavior did or did not occur, e.g., that two objects did or did not touch each other, that a dangerous condition did or did not occur, that an accident did or did not occur, that a portion of an image is an area of ​​a site map, etc. For example, a user may identify that a behavior was an accident and identify the movements that preceded the accident.

[0152] In block 940, the runtime object time series analysis module 900 may store the updated records, for example, as one or more time series training data 440 records, which may be used to train the TSNN.

[0153] In block 945, the runtime object time series analysis module 900 may determine the utilization of objects, such as personnel, field equipment, or vehicles, cranes, etc. Utilization may be determined based on the availability of such objects, based on time period, weather, or other conditions that may affect utilization or availability. This may be output by the user interface module 210 to a user or another process, such as the user interface 1100.

[0154] In block 955, the runtime object time series analysis module 900 may determine whether to output an alarm regarding the unsafe condition to the warning beacon 135, to the operator of the crane 110, to the site manager, to the operator of the SLCS 125, etc., or whether to send another output in response to the output condition.

[0155] If decision block 960 returns yes or equivalent, then in block 965, the runtime object time series analysis module 900 may output an alarm or other output.

[0156] In completion block 999, the runtime object time series analysis module 900 may terminate and / or return to the module that called it and / or another process.

[0157] FIG. 10 is a schematic diagram of a set of stitched images of a site viewed by multiple object sensors, e.g., cameras. As discussed herein, multiple object sensors may be present at or have a view of a site, e.g., site 1000. As discussed herein, one or more such object sensors may be SLCSs. In this example, the first image 1060, the second image 1061, the third image 1062, the fourth image 1063, the fifth image 1064, and the sixth image 1065 may be captured by different object sensors at the same or similar times, or in one embodiment, one or more images may be captured by the same object sensor at different times, e.g., an SLCS moved over site 1000 by a carrier, such as carrier 1005. The images may overlap. In this example, the first image 1060, the second image 1061, the third image 1062, the fourth image 1063, the fifth image 1064, and the sixth image 1065 may be stitched together, e.g., by image stitching module 1200. The output may form, for example, site image 1120 of FIG.

[0158] As discussed herein, an ODNN, such as runtime object identification module 800 and runtime object detection neural network 425, may process images and identify objects to be classified, such as, for example, building 1040, building 1045, and building 1050, construction material 1010, construction material 1015, and construction material 1017, group of people 1025, group of people 1020, individual person 1055, crane 1005, gate 1065, and gate 1066. Group of people 1025, group of people 1020, and individual person 1055 may be identified as having safety equipment, such as a hard hat or high-visibility clothing, such as a vest, as described in connection with FIG.

[0159] As discussed herein, the runtime object time series analysis module 900 and a TSNN, such as the runtime time series analysis neural network 430, may process the object tensor 465 records and object sensor tensors from the ODNN to determine a site map of a site and the behavior of objects imaged within the site. For example, the site map may include human paths 1030 and vehicle paths 1031. Such elements may be identified based on time series object sensor data collected from objects along such paths and the training data of the TSNN, and such objects may be associated with the identification of human or vehicle paths. For example, an object behavior that the TSNN may identify is an individual 1055 being in a hazardous condition 1056, which the TSNN may identify because the TSNN has been trained to identify hazardous conditions based on labeled training data, which may include hazardous conditions labeled or classified as occurring below a crane pick. For example, an object behavior that the TSNN may identify may be a crane pick 1011 (which may be a behavior category also assigned to construction material 1010).

[0160] FIG. 11 is a schematic diagram of a user interface 1100 that may be generated by a visualization module, such as visualization module 210, to output information from, for example, runtime object identification module 800 and runtime object time series analysis module 900.

[0161] For example, the user interface 1100 may include a site image 1120. The site image 1120 may be stitched, for example, by the image stitching module 1200. The site image 1120 may be a still image or a video.

[0162] For example, the user interface 1100 may include object sensor data 1125. The object sensor data 1125 may include object sensor data associated with an object category, for example, by the runtime generation module 700 or another process. For example, the object sensor data 1125 may include data from the SLCS, such as state or parameter information of the SLCS and the load. For example, such state or parameter information may include the position, orientation, or movement of the SLCS and the load, and the parameters may include the length of the suspension cable, the moment of inertia of the SLCS and / or the load, the mass of the SLCS and / or the load, the battery status of the SLCS, the wind load or disturbance force on the SLCS and / or the load, the thruster setting or thrust output of the SLCS, and whether the load control module of the SLCS is active in attempting to control the thrusters of the SLCS to affect the impending state or parameters of the SLCS and / or the load.

[0163] For example, site image 1120, which includes one or more images or videos, may include objects identified by runtime object identification module 800, behaviors of those objects identified by runtime object time series analysis module 900, and may include highlighted or visually distinguished objects. For example, a person in region 1155 may be highlighted or visually distinguished because such person was identified as a "person" object by runtime object identification module 800 and because the person object exhibited unsafe behavior as identified by runtime object time series analysis module 900. Object sensor behaviors associated with the "person" category, such as personnel records or identifiers, may also be output.

[0164] For example, graph 1105 may present the number of violations and violators of personal safety requirements such as helmets, safety vests, etc. Selecting a graph bar or a portion thereof may generate an image or video corresponding to site image 1120 from the time the data for the graph bar was generated.

[0165] For example, graph 1110 may show wind speed ranges per day. For example, graph 1115 may present a crane pick count, reflecting crane usage over the past few weeks, for example. For example, the graphical display of information may follow a time series of video images. For example, the graphical display of information may include remaining battery charge 1130, battery health, average duty cycle, etc. For example, a user may be able to rewind, fast forward, pause, etc., both with respect to the video and other object sensor data and associated output.

[0166] 12 is a flow diagram illustrating an example of a method performed by an image stitching module 1200 incorporating teachings of the present disclosure, according to some embodiments. The image stitching module 1200 may be performed by or with the assistance of a hardware accelerator, such as the hardware acceleration module 310. This module may be performed, for example, by the site monitor runtime computer system 105, which may be similar to the site monitor machine learning computer 300, as described above.

[0167] In block 1205, the image stitching module 1200 may obtain images, for example, from one or more object sensor data 405 records.

[0168] The opening loop block 1210 through the closing loop block 1275 may be repeated for sets or groups of images at the same or similar times.

[0169] The opening loop block 1215 through the closing loop block 1270 may iterate over adjacent images, for example, two adjacent images.

[0170] In block 1220, the image stitching module 1200 may determine keypoints for the current image, such as using a difference of gaussians keypoint detector.

[0171] At block 1225, the image stitching module 1200 may extract a locally invariant descriptor, which may include, for example, a feature vector such as that obtained by or with a SIFT feature extractor.

[0172] In block 1230, the image stitching module 1200 may match local invariant descriptors, eg, feature vectors, between adjacent images at that time.

[0173] In block 1235, if there are multiple matches, the image stitching module 1200 may prune the multiple invariant descriptor matches based on a false positive test, such as the David Law ratio test, to identify higher quality matches.

[0174] At decision block 1240, the image stitching module 1200 may determine whether the match threshold is, for example, 4 or greater.

[0175] If the answer at decision block 1240 is no or equivalent, the image stitching module 1200 may return to block 1230 .

[0176] If decision block 1240 returns yes or equivalent, then at block 1245 the image stitching module 1200 may use the matched feature vectors to calculate a homography matrix.

[0177] At block 1250, the image stitching module 1200 may determine a warping matrix based on the homography matrix.

[0178] At block 1255, the image stitching module 1200 may apply the determined warping matrix to the current adjacent image.

[0179] At block 1260, when completed for all images for the site, the image stitching module 1200 may output a stitched image including the adjacent images.

[0180] At completion block 1299, the image stitching module 1200 may exit and / or return to the module and / or another process that called it.

[0181] FIG. 13 is a perspective view of a suspended load control system (“SLCS”) 1300, according to some embodiments. The physical components of the SLCS 1300 include a fan unit 1305A and a fan unit 1305B, each of which may include two asymmetric unidirectional fans driven by one or more motors under the control of a suspended load control module, as discussed herein. A housing 1310 may include physical and logical components. The physical components of the housing 1310 may include an object sensor 1315, which may be a video camera, including a hemispherical video camera. One or more similar object sensors, such as a video camera, may have a downward field of view on the bottom of the housing 1310. The physical components of or within the housing 1310 may include a power source, e.g., a battery, and hardware for the logical components, e.g., computer hardware and computer memory. The suspended load control module may reside in-memory in the computer memory and may be executed by the computer hardware, as discussed herein.

[0182] 14 is a perspective view of an SLCS 1300 and a carrier 1405 at a site 1400, according to some embodiments. In this example, the SLCS 1300 performs a picking operation on a suspended load 1410. Object sensors of the SLCS 1300, such as object sensor 1315 and the like on the bottom of the SLCS 1300, and additional object sensors, such as object sensor 1415, may image the site 1400. The SLCS 1300 may perform all or part of the object identification neural training module 500, the time series neural training module 600, the runtime generation module 700, the runtime object identification module 800, the runtime object time series analysis module 900, the image stitching module 1200, and the visualization module 210, as discussed herein.

[0183] Upon execution of the aforementioned modules, the SLCS 1300 may identify that the person 1420 is exhibiting behavior that constitutes an unsafe condition and trigger an alert. This identification may occur because the time series training data 440 includes video imagery of a person under a suspended load or moving into a position under the potential path of the suspended load, as discussed herein. For example, such video imagery may be processed by the runtime object identification module 800 to identify the SLCS 1300 as a “load control system” and the suspended load 1410 as a “suspended load.” The object tensor output by the runtime object identification module 800 may be processed by the runtime object time series analysis module 900 to identify that the person 1420 is exhibiting “unsafe condition” behavior, for example, because the person 1420 is moving under the suspended load 1410.

[0184] Aspects of the system may be embodied in a specialized or special purpose computing device or data processor that is specially programmed, configured, or constructed to execute one or more of the computer-executable instructions described in detail herein. Aspects of the system may also be practiced in distributed computing environments where tasks or modules are performed by remote processing devices linked through a communications network, such as a local area network (LAN), a wide area network (WAN), the Internet, or any radio frequency communications technology. Data from deployable devices may be very low bandwidth and not limited by frequency or communications protocol. In a distributed computing environment, modules may be located in both local and remote memory storage devices. Logic or circuits embodying logic may be discussed as including a particular order or structure for achieving an effect, and the order or structure may be rearranged so long as the effect is achieved.

[0185] Embodiments of the operations described herein may be implemented in a computer-readable storage device having stored thereon instructions that, when executed by one or more processors, may include, for example, processing units and / or programmable circuitry, to perform the methods. The storage device may be a machine-readable storage device including any type of tangible, non-transitory storage device, including, for example, a floppy disk, an optical disk, any type of disk including a read-only compact disk memory (CD-ROM), a re-writable compact disk (CD-RW), and a magneto-optical disk, a semiconductor device such as a read-only memory (ROM), a random access memory (RAM) such as dynamic RAM or static RAM, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic card, an optical card, or any type of storage device suitable for storing electronic instructions. The USB (Universal Serial Bus) may conform to or be compatible with the Universal Serial Bus Specification, Revision 2.0, published by the Universal Serial Bus Organization on April 27, 2000, and / or later versions of this specification (e.g., the Universal Serial Bus Specification, Revision 3.1, published on July 26, 2013). PCIe may conform to or be compatible with the PCI Express 3.0 Base Specification, Revision 3.0, published by the Peripheral Component Interconnect Special Interest Group (PCISIG) in November 2010, and / or any subsequent and / or related versions of this specification.

[0186] As used in any embodiment herein, the term “logic” may refer to the logic of app, software, and / or firmware instructions and / or logic embedded in a programmable circuit by a configuration bitstream to perform any of the foregoing operations. Software may be embodied as a software package, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, instruction sets, and / or data hard-coded (e.g., non-volatile) in a storage device.

[0187] "Circuitry," as used in any embodiment herein, may include, for example, hardwired circuitry, programmable circuitry such as an FPGA, alone or in any combination. Logic may be embodied as circuitry that collectively or individually forms part of a larger system, such as an integrated circuit (IC), an application specific integrated circuit (ASIC), a system on a chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, etc.

[0188] In some embodiments, a hardware description language (HDL) may be used to specify the circuit and / or logic implementation(s) for the various logic and / or circuits described herein. For example, in one embodiment, the hardware description language may conform to or be compatible with the Very High Speed ​​Integrated Circuit (VHDL) Hardware Description Language, which may enable semiconductor manufacturing of one or more of the circuits and / or logic described herein. VHDL may conform to or be compatible with IEEE Standard 1076-1987, IEEE Standard 1076.2, IEEE 1076.1, IEEE Draft 3.0 of VHDL-2006, IEEE Draft 4.0 of VHDL-2008, and / or other standards of the IEEE VHDL standard and / or other hardware description standards.

[0189] As used herein, the term "module" (or "logic") may refer to, be part of, or include a device or another computer hardware component that executes one or more software or firmware programs, including an application-specific integrated circuit (ASIC), a system-on-chip (SoC), electronic circuitry, programmed programmable circuitry (such as a field-programmable gate array (FPGA)), processors (shared, dedicated, or groups) and / or memory (shared, dedicated, or groups), executable machine instructions (generated from an assembler and / or compiler) or combinations thereof, combinatorial logic circuitry, and / or other suitable components with logic that provide the described functionality. The modules may be separate and independent components integrated by sharing or passing data, or the modules may be subcomponents of a single module, or may be split into multiple modules. A component may be a process executed or implemented on a single compute node, or may be a process distributed across multiple computer nodes executing in parallel, simultaneously, sequentially, or a combination thereof, as will be described in more detail in conjunction with the flow diagrams in the figures.

[0190] As used herein, a process corresponds to an instance of a program, such as an application program, that is executing on a processor, and a thread corresponds to a portion of a process. A processor may include one or more execution cores. A processor may be configured as one or more sockets, each containing one or more execution cores.

[0191] The following are non-limiting examples of embodiments of the disclosure herein:

[0192] Example 1. A system for identifying objects and determining object categories through visual analysis of a site, comprising: a computer processor and a memory, the computer processor having a runtime object identification module in the memory; To identify objects through visual analysis of the site and determine the object's category; The computer processor executes a runtime object identification module to obtain object sensor data for a site, identify objects in the object sensor data, and determine categories of the objects.

[0193] Example 2. The system of example 1 or any other example or example herein, wherein the site is a construction site.

[0194] Example 3. The system according to example 1-2 or any other example or at least one of the examples herein, wherein the object sensor data includes an image of the site.

[0195] Example 4. A system according to example 1 to example 3 or any other example or example herein, comprising: The runtime object identification module processes the image using an object detection neural network to identify objects in the image, determine categories of the objects, and determine a confidence level for at least one of the objects or categories of objects.

[0196] Example 5. The system according to example 1 through example 4 or any other example or example herein, wherein the object detection neural network includes a convolutional neural network.

[0197] Example 6. A system according to at least one of Examples 1 through 5 or any other example or example herein, wherein the runtime object identification module determines a category of an object in the image as at least one of a carrier, a crane, a load, a person, an authorized person, an unauthorized person, a safety device, a hard hat, a visibility vest, a fence, a site machinery equipment, a load control system, a site building, a site building material, or a contacting object.

[0198] Example 7. A system according to at least one of Examples 1 to 6 or any other example or example herein, wherein the system is suspended above a site by a cable of a construction crane.

[0199] Example 8. The system according to example 1 through example 7 or any other example or example herein, wherein the system comprises a load control system.

[0200] Example 9. The system according to at least one of Examples 1 to 8 or any other example or example herein, wherein the runtime object identification module is configured to output an object tensor, the object tensor encoding at least one of an image portion corresponding to the object, the object, and a categorization of the object.

[0201] Example 10. A system according to at least one of Examples 1 to 9 or any other example or example herein, wherein the runtime object identification module outputs at least one of an image portion corresponding to the object, an object, or a category of the object.

[0202] Example 11. A system according to at least one of Examples 1 to 9 or any other example or example herein, wherein the runtime object identification module outputs at least one of an image portion corresponding to the object, the object, or a category of the object to a human for human confirmation.

[0203] Example 12. The system according to example 1 through example 10 or any other example or example herein, wherein the runtime object identification module stores human feedback regarding at least one of an image portion corresponding to an object, an object, or a category of objects as object identification training data; The object identification training data trains the object detection neural network to identify at least one of an image portion corresponding to an object, an object, or a category of objects.

[0204] Example 13. A system according to at least one of Examples 1 to 12 or any other example or example herein, wherein the system is further configured to determine at least one of a site map or object behavior, and wherein the system further comprises a runtime object time series analysis module in memory to determine at least one of the site map or object behavior, and wherein the computer processor executes the runtime object time series analysis module and processes the object tensors in the runtime object time series analysis module, thereby obtaining at least one of the site map or object behavior from the runtime object time series analysis module.

[0205] Example 14. A system according to at least one of Examples 1 to 13 or any other example or example herein, wherein the behavior includes at least one of object movement, velocity, object movement prior to contact, object contact, crane pick, unsafe condition, theft, or accident.

[0206] Example 15. The system according to example 1 through example 13 or any other example or example herein, wherein the crane pick includes a crane that picks up the load.

[0207] Example 16. The system according to example 1 through example 14 or any other example or example herein, wherein the runtime object time series analysis module includes a time series neural network.

[0208] Example 17. The system according to example 1 through example 16 or any other example or example herein, wherein the time-series neural network comprises a long short-term memory recurrent neural network.

[0209] Example 18. A system according to at least one of Examples 1 to 17 or any other example or example herein, wherein the runtime object time series analysis module outputs at least one of a site map or object behavior.

[0210] Example 19. A system according to at least one of Examples 1 to 17 or any other example or example herein, wherein the runtime object time series analysis module outputs at least one of a site map or object behavior to a human for human review.

[0211] Example 20. A system according to at least one of Examples 1 to 18 or any other example or example herein, wherein the runtime object time series analysis module stores human feedback regarding at least one of the site map or object behavior as time series training data, and the time series training data trains the time series neural network.

[0212] Example 21. The system according to example 1 through example 20 or any other example or example herein, wherein the system trains an object detection neural network to identify objects and determine categories of the objects; further comprising an object discrimination neural training module in the memory; Train an object detection neural network to identify objects and determine object categories; The object identification neural training module acquires an object identification training dataset and inputs a first portion of the object identification training dataset into the object detection neural network; Training an object detection neural network to identify objects and determine object categories; Additionally, the object detection neural network is tested using a second portion of the object identification training dataset.

[0213] Example 22. A system according to at least one of Examples 1 to 21 or any other example or example herein, wherein the object identification training dataset includes images, objects identified in the images, and categories of objects identified in the images.

[0214] Example 23. A system according to example 1 to example 22 or any other example or example herein, comprising: The system further trains a time-series neural network to identify the site map and the behavior, and further includes a time-series neural network training module in memory; To train the time-series neural network to identify site maps and behaviors, the time-series neural network training module acquires a time-series training dataset, inputs a first portion of the time-series training dataset into the system, trains the time-series neural network to identify site maps and behaviors according to the first portion of the time-series training dataset, and tests the time-series neural network using a second portion of the time-series training dataset.

[0215] Example 24. A system according to at least one of Examples 1 to 23 or any other example or example herein, wherein the time-series training dataset includes images, objects identified in the images, categories of objects identified in the images, a site map, and object behaviors.

[0216] Example 25. A system according to at least one of Examples 1 to 24 or any other example or example herein, wherein the time series training data set includes an object tensor, the object tensor encoding at least one of an image, an object identified in the image, or a category of objects identified in the image, and the time series training data set further includes at least one of a site map and an object behavior derived from the object tensor.

[0217] Example 26. The system according to at least one of Examples 1 to 25 or any other example or example herein, wherein the object sensor is a first object sensor and further includes a second object sensor, wherein the second object sensor includes at least one of a wind speed sensor or a sensor suite of a suspended load control system.

[0218] Example 27. A system according to at least one of Examples 1 to 26 or other Examples or Examples herein, wherein the sensor data is first sensor data, and the system further associates second sensor data with a category of an object, and the system further outputs a processed image (processed image) to a human, determines an object present in the processed image (processed image) for the human according to the category of the object, and associates the second sensor data in the output image with the object due to the category of the object and outputs it to the human.

[0219] Example 28. The system according to at least one of Examples 1 to 27 or any other example or example herein, wherein the second sensor data includes at least one of wind speed, a state or parameter of a load control device.

[0220] Example 29. A system according to at least one of Examples 1 to 28 or any other example or example herein, wherein the system further determines and outputs at least one of equipment utilization, vehicle utilization, personnel utilization, or an alert related to identifying behaviors that constitute an unsafe condition.

[0221] Example 30. A system according to at least one of Examples 1 to 29 or other examples or examples herein, wherein the system further stitches multiple images of the site into one composite image of the site, and further comprises an image stitching module in the memory, wherein to stitch the multiple images of the site into the composite image of the site, the computer processor executes the image stitching module to obtain the multiple images of the site and combine the multiple images to form the composite image of the site.

[0222] Example 31. A system according to at least one of Examples 1 to 30 or any other example or example herein, wherein the system further outputs the composite image of the site in association with at least one of visually distinct objects, object categories, site maps, or behaviors in the composite image of the site.

[0223] Example 32. A system according to at least one of Examples 1 to 31 or any other example or example herein, wherein, to stitch multiple images to form a composite image of a site, an image stitching module determines locally invariant descriptors of keypoints of adjacent images in the multiple images of the site, determines groups of matching locally invariant descriptors of the keypoints, calculates a homography matrix using the matching locally invariant descriptors of the keypoints, determines a warping matrix based on the homography matrix, and applies the warping matrix to adjacent images in the multiple images of the site to form the composite image of the site.

[0224] Example 33. A system according to at least one of Examples 1 to 32 or other examples or examples herein, wherein the system further outputs the processed image to a human, and outputs at least one of the object's behavior or the second sensor data in the processed image to the human.

[0225] Example 34. A system according to at least one of Examples 1 to 33 or any other example or example herein, wherein the behavior includes at least one of a person wearing a safety device, a person not wearing a safety device, object movement, velocity, object movement prior to contact, object contact, crane pick, a dangerous condition, theft, an accident, or a crane pick.

[0226] Example 35. The system according to at least one of Examples 1 through 34 or any other example or example herein, wherein the second sensor data includes at least one of wind speed, a battery status of site equipment, or a state or parameter of a suspended load control system.

[0227] Example 36. The system according to at least one of Examples 1 through 35 or any other example or example herein, wherein the hazardous condition includes a person being under the suspended load.

[0228] Example 37. A system according to at least one of Examples 1 to 36 or any other example or example herein, wherein the state of the load control system includes at least one of a position, orientation, or movement of the load control system and the load, and the parameters include at least one of a length of the lifting cable, a moment of inertia of the load control system and / or the load, a mass of the load control system and / or the load, a battery status of the load control system, a wind load or disturbance force of the load control system and / or the load, a thruster setting or thrust output of the load control system, and whether a load control module of the load control system is active to control the thrusters of the load control system to affect an impending state or parameter of the load control system and the load.

[0229] Example 38. A system according to at least one of Examples 1 to 37 or any other example or example herein, wherein the object sensor comprises a load control system, the load control system having a thruster, a sensor suite, and a load control module, the load control module being executed by a processor of the object sensor and configured to estimate or predict a state or parameter of the object based on sensor data from the sensor suite.

[0230] Example 39. A method for identifying objects and determining categories of objects through visual analysis of a site, comprising a computer processor, a memory, and a runtime object identification module in the memory, the method for identifying objects and determining categories of objects through visual analysis of a site includes: acquiring object sensor data about the site, identifying objects in the object sensor data, and determining categories of objects by the computer processor executing the runtime object identification module.

[0231] Example 40. The method according to example 39 or any other example or example herein, wherein the site is a construction site.

[0232] Example 41. The method according to at least one of Examples 39 to 40 or any other example or example herein, wherein the object sensor data includes an image of the site.

[0233] Example 42. The method according to at least one of Example 39 to Example 41 or any other example or example herein, wherein determining a category of an object in the image includes processing the image with an object detection neural network to thereby identify an object in the image, determining a category of the object, and determining a confidence level for at least one of the object or the category of the object.

[0234] Example 43. The method according to at least one of Examples 39 to 42 or any other example or example herein, wherein the object detection neural network is a convolutional neural network.

[0235] Example 44. The method according to at least one of Examples 39 to 43 or any other example or example herein, further comprising determining a category of an object in the image as at least one of a carrier, a crane, a suspended load, a person, an authorized person, an unauthorized person, a safety device, a helmet, a visibility vest, a fence, a site machinery equipment, a suspended load control system, a site building, a site building material, or a contacting object.

[0236] Example 45. The method according to example 39 through example 44 or any other example or example herein, wherein the processor and memory are suspended by a sling of a construction crane above the site.

[0237] Example 46. The method according to at least one of Example 39 to Example 46 or any other example or example herein, wherein the processor and memory are of a suspended load control system.

[0238] Example 47. The method according to at least one of Examples 39 to 46 or any other example or example herein, further comprising the runtime object identification module preparing and outputting an object tensor, the object tensor encoding at least one of an image portion corresponding to the object, the object, and a category of the object.

[0239] Example 48. A method according to at least one of Examples 39 to 47 or any other example or example herein, further comprising the runtime object identification module outputting at least one of an image portion corresponding to the object, the object, and a category of the object.

[0240] Example 49. The method according to at least one of Examples 39 to 48 or any other example or example herein, further comprising the runtime object identification module outputting at least one of the image portion corresponding to the object, the object, or the category of the object to a human for human confirmation.

[0241] Example 50. The method according to at least one of Examples 39 to 49 or any other example or example herein, further comprising: the runtime object identification module storing human feedback regarding at least one of the image portion corresponding to the object, the object, or the category of the object as object identification training data; and wherein the object identification training data is used to train an object detection neural network to identify at least one of the image portion corresponding to the object, the object, or the category of the object.

[0242] Example 51. A method according to at least one of Examples 39 to 50 or any other example or example herein, for determining at least one of a site map or object behavior, further comprising a runtime object time series analysis module in the memory, wherein the computer processor executes the runtime object time series analysis module to determine at least one of the site map or object behavior, thereby processing the object tensor in the runtime object time series analysis module and obtaining at least one of the site map or object behavior from the runtime object time series analysis module.

[0243] Example 52. The method according to at least one of Examples 39 to 51 or any other example or example herein, further comprising: the runtime object time series analysis module determining the behavior to include at least one of object movement, velocity, object movement prior to contact, object contact, crane pick, unsafe condition, theft, or accident.

[0244] Example 53. The method according to example 39 through example 52 or any other example or example herein, wherein the crane pick includes a crane picking up the suspended load.

[0245] Example 54. The method according to at least one of Example 39 to Example 53 or any other example or example herein, wherein the runtime object time series analysis module includes a time series neural network.

[0246] Example 55 The method according to at least one of Example 39 to Example 54 or any other example or example herein, wherein the time-series neural network comprises a long short-term memory recurrent neural network.

[0247] Example 56. A method according to at least one of Examples 39 to 55 or other examples or examples herein, further comprising the runtime object time series analysis module outputting at least one of a site map or object behavior.

[0248] Example 57. A method according to at least one of Examples 39 to 56 or any other example or example herein, further comprising the runtime object time series analysis module outputting at least one of the site map or the object behavior to a human for human review.

[0249] Example 58. A method according to at least one of Examples 39 to 57 or other examples or examples herein, wherein the runtime object time series analysis module further includes storing human feedback regarding at least one of the site map or the object's behavior as time series training data, the time series training data being for training the time series neural network.

[0250] Example 59. The method according to at least one of Examples 39 to 58 or any other example or example herein, further comprising training an object detection neural network to identify objects and determine categories of the objects, comprising an object identification neural network training module in the memory, and further comprising: a processor executing the object identification neural training module to train the object detection neural network to identify objects and determine categories of the objects, thereby obtaining an object identification training dataset, inputting a first portion of the object identification training dataset into the object detection neural network, training the object detection neural network to identify objects and determine categories of the objects, and testing the object detection neural network using a second portion of the object identification training dataset.

[0251] Example 60. A method according to at least one of Examples 39 to 59 or any other example or example herein, wherein the object identification training dataset includes images, objects identified in the images, and categories of objects identified in the images.

[0252] Example 61. The method according to at least one of Examples 39 to 60 or other Examples or Examples herein, further comprising training a time series neural network to identify site maps and behaviors, further comprising a time series neural network training module in the memory, wherein training the time series neural network to identify site maps and behaviors includes the time series neural network training module obtaining a time series training dataset, inputting a first portion of the time series training dataset into the time series neural network, training the time series neural network to identify site maps and behaviors according to the first portion of the time series training dataset, and further testing the time series neural network using a second portion of the time series training dataset.

[0253] Example 62. A method according to at least one of Examples 39 to 61 or any other example or example herein, wherein the time-series training dataset includes images, objects identified in the images, categories of objects identified in the images, a site map, and object behaviors.

[0254] Example 63. A method according to at least one of Examples 39 to 62 or any other example or example herein, wherein the time series training data set includes an object tensor, the object tensor encoding at least one of an image, an object identified in the image, or a categorization of the object identified in the image, and the time series training data set further includes at least one of a site map and an object behavior derived from the object tensor.

[0255] Example 64. The method according to at least one of Example 39 to Example 64 or any other example or example herein, wherein the object sensor is a first object sensor and further includes a second object sensor, wherein the second object sensor includes at least one of a wind speed sensor or a sensor suite of a suspended load control system.

[0256] Example 65. The method according to at least one of Examples 39 to 64 or any other example or example herein, wherein the sensor data is first sensor data, and the method further includes associating second sensor data with a category of object and outputting the processed image to a human; determining an object present in the processed image to the human according to the category of object; and associating the second sensor data with the object in the output image according to the category of object and outputting the second sensor data to the human.

[0257] Example 66. The method according to at least one of Examples 39 to 65 or any other example or example herein, wherein the second sensor data includes at least one of wind speed, a state or parameter of a load control device.

[0258] Example 67. The method according to at least one of Example 39 to Example 66 or any other example or example herein, wherein the method further includes determining and outputting at least one of equipment utilization, vehicle utilization, personnel utilization, or an alert related to identifying a behavior including an unsafe condition.

[0259] Example 68. A method according to at least one of Examples 39 to 67 or other Examples or Examples herein, further comprising stitching a plurality of images of the site into a single composite image of the site, further comprising an image stitching module in the memory, wherein the computer processor executes the image stitching module to stitch the plurality of images of the site into the composite image of the site, thereby obtaining the plurality of images of the site and combining the plurality of images to form the composite image of the site.

[0260] Example 69. The method according to at least one of Examples 39 to 68 or any other example or example herein, wherein the method further includes outputting the composite image of the site in association with at least one of visually distinct objects, object categories, site maps, or behaviors in the composite image of the site.

[0261] Example 70. The method according to at least one of Example 39 to Example 69 or any other example or example herein, wherein stitching a plurality of images to form a composite image of the site includes: an image stitching module determining locally invariant descriptors of keypoints of adjacent images in the plurality of images of the site; determining groups of matching locally invariant descriptors of the keypoints; calculating a homography matrix using the matching locally invariant descriptors of the keypoints; determining a warping matrix based on the homography matrix; and applying the warping matrix to adjacent images in the plurality of images of the site to form the composite image of the site.

[0262] Example 71. A method according to at least one of Examples 39 to 70 or any other example or example herein, further comprising outputting the processed image to a human, and outputting at least one of the object's behavior or the second sensor data in the processed image to the human.

[0263] Example 72. The method according to at least one of Example 39 to Example 71 or any other example or example herein, wherein the behavior includes at least one of a human wearing a safety device, a human not wearing a safety device, object movement, speed, object movement prior to contact, object contact, crane pick, an unsafe condition, theft, an accident, or a crane pick.

[0264] Example 73. The method according to at least one of Examples 39 to 72 or any other example or example herein, wherein the second sensor data includes at least one of wind speed, a battery status of the site equipment, or a state or parameter of the lifted load control system.

[0265] Example 74 The method according to at least one of Example 39 through Example 73 or any other example or example herein, wherein the unsafe condition includes a person being under the suspended load.

[0266] Example 75. The method according to at least one of Examples 39 to 74 or any other example or example herein, wherein the state of the load control system includes at least one of a position, orientation, or movement of the load control system and the load, and the parameters include a length of the suspension cable, a moment of inertia of the load control system and / or the load, a mass of the load control system and / or the load, a battery status of the load control system, a wind load or disturbance force of the load control system and / or the load, a thruster setting or thrust output of the load control system, and whether a load control module of the load control system is active in an attempt to control the thrusters of the load control system to affect an upcoming state or parameter of the load control system and / or the load.

[0267] Example 76. The method according to at least one of Examples 39 to 75 or any other example or example herein, wherein the object sensor includes a load control system, the load control system including a thruster, a sensor suite, and a load control module, and further including a processor of the load control system executing the load control module to thereby estimate or predict a state or parameter of the object based on sensor data from the sensor suite.

[0268] Example 77. In response to execution of instructions by a processor of a computing device, the computing device: causing a processor of the computing device to execute a runtime object identification module retrieved from the memory of the computing device, thereby acquiring object sensor data related to the site, identifying objects in the object sensor data, and determining a category of the objects; One or more computer-readable media containing instructions for identifying objects and determining categories of objects through visual analysis of a site.

[0269] Example 78. The computer readable medium of Example 77 or any other example or at least one of the examples herein, wherein the site is a construction site.

[0270] Example 79. A computer-readable medium according to at least one of Examples 77 to 78 or any other example or example herein, wherein the object sensor data includes an image of the site.

[0271] Example 80. A computer-readable medium according to at least one of Examples 77 to 79 or any other example or example herein, wherein determining categories of objects in the image comprises processing the image with an object detection neural network, thereby identifying objects in the image, determining categories of the objects, and determining a confidence level for at least one of the objects or categories of objects.

[0272] Example 81. A computer-readable medium according to at least one of Examples 77 to 80 or any other example or example herein, wherein the object detection neural network comprises a convolutional neural network.

[0273] Example 82. A computer-readable medium according to at least one of Examples 77 to 81 or other examples or examples herein, wherein the instructions further cause the computer device to determine that a category of an object in the image is at least one of a carrier, a crane, a suspended load, a person, an authorized person, an unauthorized person, a safety device, a hard hat, a visibility vest, a fence, a site machinery and equipment, a suspended load control system, a site building, a site building material, or a contacting object.

[0274] Example 83. A computer-readable medium according to at least one of Examples 77 to 82 or any other example or example herein, wherein the computing device is suspended by a sling of a construction crane above a site.

[0275] Example 84. A computer-readable medium according to at least one of Examples 77 to 83 or any other example or example herein, wherein the computing device is of a suspended load control system.

[0276] Example 85. A computer-readable medium according to at least one of Examples 77 to 84 or any other example or example herein, wherein the instructions further cause the runtime object identification module to prepare and output an object tensor, the object tensor encoding at least one of an image portion corresponding to the object, the object, and a category of the object.

[0277] Example 86. A computer-readable medium according to at least one of Examples 77 to 85 or any other example or example herein, wherein the instructions further cause the runtime object identification module to output at least one of an image portion, an object, or a category of objects corresponding to the object.

[0278] Example 87. A computer-readable medium according to at least one of Examples 77 to 86 or any other example or example herein, wherein the instructions further cause the runtime object identification module to output at least one of an image portion corresponding to the object, the object, or a category of the object to a human for human confirmation.

[0279] Example 88. A computer-readable medium according to at least one of Examples 77 to 87 or any other example or example herein, wherein the instructions further cause the runtime object identification module to store human feedback regarding at least one of the image portion, object, or category of object corresponding to the object as object identification training data, and further cause the object detection neural network to train using the object identification training data to identify at least one of the image portion, object, or category of object corresponding to the object.

[0280] Example 89. A computer-readable medium according to at least one of Examples 77 to 88 or any other example or example herein, wherein the instructions cause a computing device to determine at least one of a site map or an object behavior, and further having a runtime object time series analysis module in the memory, and to determine at least one of the site map or the object behavior, the instructions further cause the runtime object time series analysis module to process the object tensor and obtain at least one of the site map or the object behavior from the runtime object time series analysis module.

[0281] Example 90. A computer-readable medium according to at least one of Examples 77 to 89 or any other example or example herein, wherein the instructions further cause a runtime object time series analysis module to determine behavior including at least one of object movement, velocity, object movement prior to contact, object contact, crane pick, unsafe condition, theft, or accident.

[0282] Example 91. The computer-readable medium according to example 77 through example 90 or any other example or example herein, wherein the crane pick includes a crane that picks up a suspended load.

[0283] Example 92. The computer-readable medium according to example 77 through example 91 or any other example or example herein, wherein the runtime object time series analysis module includes a time series neural network.

[0284] Example 93. The computer-readable medium according to example 77 through example 92 or any other example or example herein, wherein the time-series neural network comprises a long short-term memory recurrent neural network.

[0285] Example 94. A computer-readable medium according to at least one of Examples 77 to 93 or other examples or examples herein, wherein the instructions further cause the runtime object time series analysis module to output at least one of a site map or object behavior.

[0286] Example 95. A computer-readable medium according to at least one of Examples 77 to 94 or other examples or examples herein, wherein the instructions further cause the runtime object time series analysis module to output at least one of a site map or object behavior to a human for human confirmation.

[0287] Example 96. A computer-readable medium according to at least one of Examples 77 to 95 or other examples or examples herein, wherein the instructions further cause the runtime object time series analysis module to store human feedback regarding at least one of the site map or the object's behavior as time series training data, the time series training data being for training the time series neural network.

[0288] Example 97. A computer-readable medium according to at least one of Examples 77 to 96 or any other example or example herein, further comprising an object identification neural training module in the memory, wherein instructions execute the computer device to: execute the object identification neural training module to thereby obtain an object identification training dataset, input a first portion of the object identification training dataset into an object detection neural network, train the object detection neural network to identify the object and determine a category of the object, and further test the object detection neural network using a second portion of the object identification training dataset.

[0289] Example 98. A computer-readable medium according to at least one of Examples 77 to 97 or any other example or example herein, wherein the object identification training dataset includes images, objects identified in the images, and categories of objects identified in the images.

[0290] Example 99. A computer-readable medium according to at least one of Examples 77 to 98 or other examples or examples herein, further comprising a time series neural training module in the memory, wherein instructions cause the computer device to execute the time series neural training module, thereby obtaining a time series training dataset, inputting a first portion of the time series training dataset into a time series neural network, training the time series neural network to identify site maps and behaviors according to the first portion of the time series training dataset, and further testing the time series neural network using a second portion of the time series training dataset, thereby training the time series neural network to identify site maps and behaviors.

[0291] Example 100. A computer-readable medium according to at least one of Examples 77 to 99 or other examples or examples herein, wherein the time-series training dataset includes images, objects identified in the images, categories of objects identified in the images, a site map, and object behavior.

[0292] Example 101. A computer-readable medium according to at least one of Examples 77 to 100 or any other example or example herein, wherein the time series training data set includes an object tensor, the object tensor encoding at least one of an image, an object identified in the image, or a category of objects identified in the image, and the time series training data set further includes at least one of a site map and an object behavior derived from the object tensor.

[0293] Example 102. The computer-readable medium according to example 77 through example 101 or any other example or example herein, wherein the object sensor is a first object sensor and further includes a second object sensor, the second object sensor including at least one of a wind speed sensor or a sensor suite of a suspended load control system.

[0294] Example 103: A computer-readable medium according to at least one of Examples 77 to 102 or other examples or examples herein, wherein the sensor data is first sensor data, and the instructions further cause the computer device to associate second sensor data with a category of object, output a processed image to a human, determine an object present in the processed image to the human according to the category of object, and associate the second sensor data with the object in the output image according to the category of object and output it to the human.

[0295] Example 104. A computer-readable medium according to at least one of Examples 77 to 103 or any other example or example herein, wherein the second sensor data includes at least one of wind speed, a state or parameter of a load control device.

[0296] Example 105. A computer-readable medium according to at least one of Examples 77 to 104 or any other example or example herein, wherein the instructions further cause the computing device to determine and output at least one of an alert regarding equipment utilization, vehicle utilization, personnel utilization, or identification of behaviors constituting an unsafe condition.

[0297] Example 106. A computer-readable medium according to at least one of Examples 77 to 105 or other examples or examples herein, further comprising an image stitching module in the memory, wherein the instructions cause the computing device to execute the image stitching module, thereby acquiring multiple images of the site and combining the multiple images to form a composite image of the site.

[0298] Example 107. A computer-readable medium according to at least one of Examples 77 to 106 or any other example or example herein, wherein the instructions further cause the computer device to output a composite image of the site in association with at least one of visually distinguished objects, object categories, site maps, or behaviors in the composite image of the site.

[0299] Example 108. A computer-readable medium according to at least one of Examples 77 to 107 or any other example or example herein, wherein stitching a plurality of images to form a composite image of the site includes causing an image stitching module to determine local invariant descriptors of keypoints of adjacent images in the plurality of images of the site, determining groups of matching local invariant descriptors of the keypoints, calculating a homography matrix using the matching local invariant descriptors of the keypoints, determining a warping matrix based on the homography matrix, and applying the warping matrix to adjacent images in the plurality of images of the site to form the composite image of the site.

[0300] Example 109. A computer-readable medium according to at least one of Examples 77 to 108 or any other example or example herein, wherein the instructions further cause the computer device to output the processed image to a human, and output at least one of the object's behavior or the second sensor data in the processed image to the human.

[0301] Example 110. A computer-readable medium according to at least one of Examples 77 to 109 or any other example or example herein, wherein the instructions further identify the behavior as at least one of a human wearing a safety device, a human not wearing a safety device, object movement, speed, object movement prior to contact, object contact, crane pick, an unsafe condition, theft, an accident, or a crane pick.

[0302] Example 111. The computer-readable medium according to example 77 through example 110 or any other example or example herein, wherein the second sensor data includes at least one of wind speed, a battery status of site equipment, or a state or parameter of a suspended load control system.

[0303] Example 112. The computer-readable medium according to example 77 through example 111 or any other example or example herein, wherein the hazardous condition includes a person under a suspended load.

[0304] Example 113. A computer-readable medium according to at least one of Examples 77 to 112 or any other example or example herein, wherein the state of the load control system includes at least one of a position, orientation, or movement of the load control system and the load, and the parameters include at least one of a length of the suspension cable, a moment of inertia of the load control system and / or the load, a mass of the load control system and / or the load, a battery status of the load control system, a wind load or disturbance force of the load control system and / or the load, a thruster setting or thrust output of the load control system, and whether a load control module of the load control system is active in attempting to control the thrusters of the load control system to affect an impending state or parameter of the load control system and / or the load.

[0305] Example 114. A computer-readable medium according to at least one of Examples 77 to 113 or any other example or example herein, wherein the object sensor comprises a load control system, the load control system having a thruster, a sensor suite(s), and a load control module, and wherein a processor of the load control system executes the load control module, thereby estimating or predicting a state or parameter of the object based on sensor data from the sensor suite.

[0306] Example 115. An apparatus for identifying objects and determining categories of objects through visual analysis of a site, comprising: a runtime object identification module in a memory of the apparatus; means for causing a processor of the apparatus to execute the runtime object identification module; and means for thereby obtaining object sensor data relating to the site, identifying objects in the object sensor data, and determining categories of objects.

[0307] Example 116. The apparatus of example 115 or any other example or example herein, wherein the site is a construction site.

[0308] Example 117. The device of Example 115 to Example 116 or any other example or example herein, wherein the object sensor data includes an image of the site.

[0309] Example 118. The apparatus according to example 115 to example 117 or any other example or example herein, wherein the means for determining a category of an object in an image comprises means for processing the image with an object detection neural network, means for identifying an object in the image thereby, means for determining a category of the object, and means for determining a confidence level for at least one of the object or the category of the object.

[0310] Example 119. The apparatus according to at least one of Example 115 to Example 118 or any other example or example herein, wherein the object detection neural network comprises a convolutional neural network.

[0311] Example 120. The apparatus according to at least one of Example 115 to Example 119 or any other example or example herein, further comprising means for determining a category of an object in the image as at least one of a carrier, a crane, a suspended load, a person, an authorized person, an unauthorized person, a safety device, a hard hat, a visibility vest, a fence, on-site machinery and equipment, a suspended load control system, a building on-site, a building material on-site, or a contacting object.

[0312] Example 121. The apparatus according to example 115 through example 120 or any other example or example herein, wherein the apparatus is suspended by a sling of a construction crane above a site.

[0313] Example 122. The device according to example 115 to example 121 or any other example or example herein, wherein the device is communicatively coupled to a load control system.

[0314] Example 123. The apparatus according to at least one of Example 115 to Example 122 or any other example or example herein, wherein the runtime object identification module further comprises means for preparing and outputting an object tensor, the object tensor encoding at least one of an image portion corresponding to the object, the object, and a category of the object.

[0315] Example 124. The apparatus according to at least one of Examples 115 to 123 or any other example or example herein, further comprising means for causing the runtime object identification module to output at least one of an image portion, an object, or a category of the object corresponding to the object.

[0316] Example 125. The apparatus according to at least one of Examples 115 to 124 or any other example or example herein, further comprising means for causing the runtime object identification module to output at least one of an image portion corresponding to the object, the object, or a category of the object to a human for human confirmation.

[0317] Example 126. The apparatus according to at least one of Examples 115 to 125 or any other example or example herein, further comprising means for causing the runtime object identification module to store human feedback regarding at least one of an image portion corresponding to an object, an object, or a category of objects as object identification training data, and further comprising means for causing the runtime object identification module to use the object identification training data to train an object detection neural network to identify at least one of an image portion corresponding to an object, an object, or a category of objects.

[0318] Example 127. The apparatus according to at least one of Examples 115 to 126 or any other example or example herein, further comprising a runtime object time series analysis module in the memory, and further comprising means for causing the runtime object time series analysis module to process the object tensor and obtain at least one of a site map or object behavior from the runtime object time series analysis module.

[0319] Example 128. The apparatus according to at least one of Example 115 to Example 127 or any other example or example herein, further comprising means for causing a runtime object time series analysis module to determine behavior including at least one of object movement, velocity, object movement prior to contact, object contact, crane pick, hazardous condition, theft, or accident.

[0320] Example 129 The apparatus according to at least one of Example 115 to Example 128 or any other example or example herein, wherein the crane pick comprises a crane that picks up a suspended load.

[0321] Example 130. The apparatus according to at least one of Example 115 to Example 129 or any other example or example herein, further comprising means for the runtime object time series analysis module to include a time series neural network.

[0322] Example 131 . The apparatus according to example 115 through example 130 or any other example or example herein, wherein the time-series neural network comprises a long short-term memory recurrent neural network.

[0323] Example 132. An apparatus according to at least one of Examples 115 to 131 or other examples or examples herein, wherein the runtime object time series analysis module further comprises means for outputting at least one of a site map or object behavior.

[0324] Example 133. An apparatus according to at least one of Examples 115 to 132 or any other example or example herein, further comprising means for causing the runtime object time series analysis module to output at least one of a site map or object behavior to a human for human review.

[0325] Example 134. An apparatus according to at least one of Examples 115 to 133 or any other example or example herein, further comprising means for causing the runtime object time series analysis module to store human feedback regarding at least one of the site map or object behavior as time series training data, and further comprising means for using the time series training data to train the time series neural network.

[0326] Example 135. The apparatus according to at least one of Example 115 to Example 134 or any other example or example herein, further comprising an object identification neural training module in the memory, and means for causing the object identification neural training module to obtain an object identification training dataset, input a first portion of the object identification training dataset to an object detection neural network, train the object detection neural network to identify objects, determine a category of the objects, and further test the object detection neural network using a second portion of the object identification training dataset.

[0327] Example 136. An apparatus according to at least one of Examples 115 to 135 or any other example or example herein, wherein the object identification training dataset includes images, objects identified in the images, and categories of objects identified in the images.

[0328] Example 137. The apparatus according to at least one of Example 115 to Example 136 or any other example or example herein, further comprising: a time series neural training module in the memory; and means for causing the time series neural training module to obtain a time series training dataset, input a first portion of the time series training dataset to the time series neural network, and train the time series neural network to identify site maps and behaviors according to the first portion of the time series training dataset, and further comprising means for testing the time series neural network with a second portion of the time series training dataset, thereby training the time series neural network to identify site maps and behaviors.

[0329] Example 138. An apparatus according to at least one of Examples 115 to 137 or any other example or example herein, wherein the time-series training dataset includes images, objects identified in the images, categories of objects identified in the images, a site map, and object behaviors.

[0330] Example 139. An apparatus according to at least one of Examples 115 to 138 or any other example or example herein, wherein the time series training data set includes an object tensor, the object tensor encodes at least one of an image, an object identified in the image, or a category of objects identified in the image, and the time series training data set further includes at least one of a site map and an object behavior derived from the object tensor.

[0331] Example 140. The apparatus according to example 115 through example 139 or any other example or example herein, wherein the object sensor is a first object sensor and further includes a second object sensor, the second object sensor including at least one of a wind speed sensor or a sensor suite of a suspended load control system.

[0332] Example 141. The device according to example 115 to example 140 or any other example or example herein, wherein the sensor data is first sensor data, and further comprising means for causing the device to associate second sensor data with a category of an object, output a processed image to a human, determine an object present in the processed image to the human according to the category of the object, and associate the second sensor data with the object in the output image to the human according to the category of the object.

[0333] Example 142. The device according to at least one of Examples 115 to 141 or any other example or example herein, wherein the second sensor data includes at least one of wind speed, a state or parameter of a load control device.

[0334] Example 143: An apparatus according to at least one of Examples 115 to 142 or other examples or examples herein, further comprising means for determining and outputting at least one of an alert regarding equipment utilization status, vehicle utilization status, personnel utilization status, or identification of a behavior including a dangerous condition.

[0335] Example 144. The device according to Example 115 to Example 143 or any other example or example herein, further comprising an image stitching module in the memory, wherein the image stitching module further comprises means for acquiring a plurality of images of the site and combining the plurality of images to form a composite image of the site.

[0336] Example 145. The apparatus according to at least one of Examples 115 to 144 or any other example or example herein, further comprising means for outputting the composite image of the site in association with at least one of visually distinct objects, object categories, site maps, or behaviors in the composite image of the site.

[0337] Example 146. The apparatus according to at least one of Example 115 to Example 145 or any other example or example herein, further comprising means for causing the image stitching module to determine locally invariant descriptors of keypoints of adjacent images in the plurality of images of the site, determining groups of matched locally invariant descriptors of the keypoints, calculating a homography matrix using the locally invariant descriptors of the matched keypoints, determining a warping matrix based on the homography matrix, and applying the warping matrix to the adjacent images in the plurality of images of the site to form a composite image of the site.

[0338] Example 147 An apparatus according to at least one of Examples 115 to 146 or other examples or examples in this specification, further comprising means for outputting the processed image to a human and outputting at least one of the object's behavior or the second sensor data in the processed image to the human.

[0339] Example 148. The device according to at least one of Example 115 to Example 147 or any other example or example herein, further comprising means for identifying the behavior as at least one of a person wearing a safety device, a person not wearing a safety device, object movement, speed, object movement prior to contact, object contact, crane pick, unsafe condition, theft, accident, or crane pick.

[0340] Example 149. The apparatus according to example 115 through example 148 or any other example or example herein, wherein the second sensor data includes at least one of wind speed, a battery status of the site equipment, or a state or parameter of the lifted load control system.

[0341] Example 150. The apparatus according to example 115 through example 149 or any other example or example herein, wherein the hazardous condition includes a person under the suspended load.

[0342] Example 151. An apparatus according to at least one of Examples 115 to 150 or any other example or example herein, further comprising: means for causing the state of the load control system to include at least one of a position, orientation, or movement of the load control system and the load; and means for causing the parameters to include at least one of a length of the suspension cable, a moment of inertia of the load control system and / or the load, a mass of the load control system and / or the load, a status of the battery of the load control system, a wind load or disturbance force on the load control system and / or the load, a thruster setting or thrust output of the load control system, and whether a suspended load control module of the load control system is active to control the thrusters of the load control system to affect an upcoming state or parameter of the load control system and / or the load.

[0343] Example 152. An apparatus according to example 115 to example 151 or any other example or example herein, wherein the object sensor includes a load control system, the load control system comprising a thruster, a sensor suite, and a load control module, and wherein a processor of the load control system comprises means for executing the load control module and thereby estimating or predicting a state or parameter of the object based on sensor data from the sensor suite.

Claims

1. 1. A system for identifying objects through visual analysis of a construction site and determining a category of said objects, comprising: a computer processor and a memory; a runtime object identification module and a runtime object time series analysis module in the memory; the computer processor executes the runtime object identification module to identify the object through visual analysis of the construction site and determine the category of the object, acquiring object sensor data related to the construction site, identifying the object in the object sensor data, and determining the category of the object; the object sensor data includes images of the construction site; To determine the category of the object in the image, the runtime object identification module processes the image with an object detection neural network, thereby identifying the object in the image, determining the category of the object, and determining a confidence level for at least one of the object or the category of the object; the runtime object identification module outputs an object tensor; the object tensor encodes at least one of an image portion corresponding to the object, the object, and the category of the object; The system further determines at least one of a site map or a behavior of the object; The system further includes a runtime object time series analysis module in the memory; To determine at least one of the site map or the behavior of the object, the computer processor executes the runtime object time series analysis module and processes the object tensor with a time series neural network of the runtime object time series analysis module, thereby obtaining at least one of the site map or the behavior of the object from the runtime object time series analysis module.

2. 10. The system of claim 1, the runtime object identification module determines the category of the object in the image to include at least one of a carrier, a crane, a load, a person, an authorized person, an unauthorized person, a safety equipment, a hard hat, a visibility vest, a fence, a site machinery device, a load control system, a site building, a site building material, or a contacting object; The system, wherein the runtime object time series analysis module determines the behavior including at least one of object motion, velocity, object motion prior to contact, object contact, crane pick, unsafe condition, theft, or accident.

3. 10. The system of claim 1, the runtime object identification module outputs at least one of the image portion corresponding to the object, the object, or the category of the object to a human for human review; the runtime object identification module stores human feedback regarding at least one of the image portion corresponding to the object, the object, or the category of the object as object identification training data; the object identification training data trains the object detection neural network to identify at least one of the image portion corresponding to the object, the object, or the category of the object; A system characterized by:

4. 10. The system of claim 1, the runtime object time series analysis module outputs at least one of the site map or the behavior of the object to a human for human review; the runtime object time series analysis module stores human feedback regarding at least one of the behaviors of the site map or the object as time series training data; The system, wherein the time series training data trains a time series neural network of the runtime object time series analysis module.

5. 10. The system of claim 1, the system trains the object detection neural network to identify the object and determine the category of the object, and further includes an object identification neural network training module in the memory; training the object detection neural network to identify the object and determine the category of the object; the object identification neural network training module obtains an object identification training data set and inputs a first portion of the object identification training data set into the object detection neural network; training the object detection neural network to identify the object and determine the category of the object; The system further comprises testing the object detection neural network using a second portion of the object identification training data set.

6. 10. The system of claim 1, the system further comprising a time series neural training module in the memory for training the time series neural network to identify the site map and the behavior; To train the time-series neural network to identify the site map and the behavior, the time-series neural training module acquires a time-series training dataset, inputs a first portion of the time-series training dataset into the time-series neural network, trains the time-series neural network to identify the site map and the behavior according to the first portion of the time-series training dataset, and tests the time-series neural network with a second portion of the time-series training dataset.

7. 10. The system of claim 1, the object sensor data is first sensor data; The system further associates second sensor data with the category of the object; The system further outputs the processed image to a human, determines the object present in the processed image to the human according to the category of the object, and associates the second sensor data of the image output to the human with the object according to the category of the object.

8. 10. The system of claim 1, The system further comprises determining and outputting at least one of an alert regarding equipment usage, vehicle usage, personnel usage, or identification of a behavior including an unsafe condition.

9. 10. The system of claim 1, The system further comprises an image stitching module in memory for stitching together a plurality of images of the construction site to create a composite image of the construction site; to stitch together the plurality of images of the construction site to create the composite image of the construction site, the computer processor executes the image stitching module to obtain the plurality of images of the construction site and combine the plurality of images to form the composite image of the construction site; The system further comprises outputting the composite image of the construction site in association with at least one of the objects visually distinguished in the composite image of the construction site, the categories of the objects, the site map, or the behavior.

10. 10. The system of claim 1, further including a second object sensor, the second object sensor including a load control system; the load control system includes a thruster, a sensor suite, and a load control module; The load control module is executed by a processor of the load control system to estimate or predict a state or parameter of the load control system based on sensor data from the sensor suite, and to output the state or parameter of the load control system as data from the second object sensor.

11. 1. A method for identifying objects and determining categories of said objects through visual analysis of a construction site, comprising: a computer processor, a memory, and a runtime object identification module in said memory; The method includes the computer processor executing the runtime object identification module to acquire object sensor data about the construction site, identify the object in the object sensor data, and determine the category of the object to identify the object through visual analysis of the construction site and determine the category of the object; the object sensor data comprises an image of the construction site; determining the categories of the objects in the images of the construction site includes processing the images of the construction site with an object detection neural network of the runtime object identification module to identify the objects in the images, determine the categories of the objects, and determine a confidence level for at least one of the objects or the categories of the objects; the runtime object identification module further comprising preparing and outputting an object tensor; the object tensor encodes at least one of an image portion corresponding to the object, the object, and the category of the object; The method further comprises a runtime object time series analysis module in the memory for determining at least one of a site map or a behavior of the object; To determine at least one of the site map or the object behavior, the computer processor executes the runtime object time series analysis module to process the object tensors with a time series neural network of the runtime object time series analysis module, and obtains at least one of the site map or the behavior of the object from the runtime object time series analysis module.

12. 12. The method of claim 11, the runtime object identification module further outputs at least one of the image portion corresponding to the object, the object, or the category of the object to a human for human review; the runtime object identification module further storing human feedback regarding at least one of the image portion corresponding to the object, the object, or the category of the object as object identification training data; the object identification training data is used to train the object detection neural network to identify at least one of the image portion corresponding to the object, the object, or the category of the object.

13. 12. The method of claim 11, the runtime object time series analysis module further outputs at least one of the site map or the behavior of the object to a human for human review; the runtime object time series analysis module further storing human feedback regarding at least one of the behaviors of the site map or the object as time series training data; The method, wherein the time series training data is for training the time series neural network.

14. 12. The method of claim 11, The method further includes training the time-series neural network to identify the site map and the behavior; further comprising a time series neural training module in said memory; training the time series neural network to identify the site map and the behavior further comprises: the time series neural training module obtaining a time series training dataset; inputting a first portion of the time series training dataset into the time series neural network; training the time series neural network to identify the site map and the behavior according to the first portion of the time series training dataset; and testing the time series neural network with a second portion of the time series training dataset.

15. 12. The method of claim 11, The object sensor data is first sensor data, and the method further comprises associating second sensor data with the category of the object, outputting a processed image to a human, determining the object present in the processed image to the human according to the category of the object, and associating the second sensor data of the image output to the human with the object according to the category of the object.

16. 12. The method of claim 11, The method further comprising determining and outputting at least one of an alert regarding equipment utilization, vehicle utilization, personnel utilization, or identification of a behavior including an unsafe condition.

17. 12. The method of claim 11, The method further includes outputting the processed image to a human; and outputting at least one of the behavior of the object or second sensor data in the processed image to the human; The method, wherein the second sensor data includes at least one of wind speed, site equipment battery status, or load control system status or parameters.

18. 1. An apparatus for identifying objects and determining categories of said objects through visual analysis of a construction site, comprising: a runtime object identification module in a memory of the device; means for causing a processor of the device to execute the runtime object identification module; and means for acquiring object sensor data relating to the construction site, identifying the object in the object sensor data, and determining the category of the object; the object sensor data includes images of the construction site; the means for determining the category of the object in the image comprises means for processing the image with an object detection neural network, and means for identifying the object in the image, determining the category of the object, and determining a confidence level for at least one of the object or the category of the object; the runtime object identification module further comprising means for preparing and outputting an object tensor; the object tensor encodes at least one of an image portion corresponding to the object, the object, and the category of the object; further comprising a runtime object time series analysis module in said memory; means for causing the runtime object time series analysis module to process the object tensors with a time series neural network of the runtime object time series analysis module, thereby obtaining at least one of a site map or a behavior of the object from the runtime object time series analysis module.

19. 19. The apparatus of claim 18, The object detection neural network comprises a convolutional neural network.

20. 19. The apparatus of claim 18, The apparatus, wherein the time-series neural network includes a long short-term memory recurrent neural network.

Images