SYSTEM AND METHOD FOR COMPREHENSIVE PRODUCTION LINE MONITORING AND RAPID RESPONSE SIMPLIFICATION

Abstract

A vehicle, system, medium, and method comprise the operation, by at least one processor, of at least one autonomous vehicle equipped with multiple cameras and positioned to move near a production line arranged to supply a product. The operation includes the automatic movement of the at least one autonomous vehicle near the production line, free from a predetermined fixed route, and the capture, by the multiple cameras, of at least one sequence of images of an area encompassing the production line. The method comprises the automatic detection, by at least one processor, of at least one operation associated with the production line, in which at least one sequence of images is captured, and the provision, by at least one processor, of instructions to perform a response in reaction to the actions of the detected operations.The reaction is assigned to the production line.

Description

introduction

[0001] The present description concerns production lines that assemble a product in a manufacturing plant, and in particular the automatic monitoring, analysis, reporting and adjustment of production line operations.

[0002] Manufacturing facilities monitor production or assembly lines to better ensure efficient product assembly. This involves using camera array systems at each stage along the production line. The cameras are fixed and positioned in locations such as overhead to avoid interference with other moving objects, such as robots or human operators at the lineside performing assembly, piles of materials to be used during assembly, or the path and space for mobile vehicles delivering payloads to the lineside, including autonomous mobile robots (AMRs) and / or automated guided vehicles (AGVs).Improved camera positioning and greater integration between camera systems at different stages are desired to capture more informative perspectives of production for more accurate efficiency analysis.

[0003] Accordingly, it is desirable to provide systems and procedures that enable more effective and efficient production line monitoring without significantly disrupting other production operations. Furthermore, other desirable features and characteristics of the present description will become apparent from the detailed description below and the attached claims, in conjunction with the accompanying drawings and the preceding introduction. Brief description

[0004] In an exemplary embodiment, a method comprises the operation, by at least one processor, of at least one autonomous vehicle equipped with multiple cameras and arranged to move in the vicinity of an object to be monitored. The vicinity is defined as a three-dimensional space containing overhead equipment that is to be avoided. The operation includes the automatic movement of the at least one autonomous vehicle in the vicinity of the object, while free from a predetermined fixed route, and the acquisition, by the multiple cameras, of at least one sequence of images of an area encompassing at least a portion of the object.The method also includes automatic detection, by at least one processor, of at least one change associated with the object, in which at least one sequence of images is captured, and the provision of instructions, by at least one processor, to perform a response in response to the detected change.

[0005] In another exemplary embodiment, the object is a production line arranged to provide a product, and the change is a movement of an operation associated with the production line. The reaction involves a response to considered actions of the recognized operation.

[0006] According to another exemplary embodiment, the method also involves operating several autonomous vehicles, each capturing a different field of view of the production line, in order to track an object moving along the production line and passing through several of the fields of view.

[0007] According to another exemplary embodiment, the method also involves arranging a first autonomous vehicle to be monitored by one or more cameras of a second autonomous vehicle. The instructions include modifications to one or more actions to be performed by the first autonomous vehicle, depending on the actions detected by the one or more cameras of the second autonomous vehicle.

[0008] According to another exemplary embodiment, the method includes carrying a loaded payload on the at least one autonomous vehicle, automatically detecting when at least part of the loaded payload is removed from the at least one autonomous vehicle or is used in conjunction with the production line or both, using the at least one sequence of images, and instructing the at least one autonomous vehicle to retrieve more payload when the detection indicates that at least part of the loaded payload has been removed.

[0009] According to another exemplary embodiment, the method also includes generating a 3D map of the production line that changes over time and determining and tracking the position and movement of the at least one autonomous vehicle on the 3D map using the at least one sequence of images.

[0010] According to another exemplary embodiment, the method also involves the use of a visual simultaneous localization and mapping (VSLAM) algorithm to generate the 3D map of the production line and to track the movement of the at least one autonomous vehicle.

[0011] According to another exemplary embodiment, the multiple autonomous vehicles provide several sequences of images. The method involves registering images together from different autonomous vehicles to generate a 3D map of the production line.

[0012] According to another exemplary embodiment, the method also involves comparing data from images of predetermined desired expected actions or previously recorded undesired actions, or both, with the actions under consideration, and determining the instructions depending on the results of the comparison.

[0013] In another exemplary embodiment, a system comprises a memory and a processor circuit, forming at least one processor that is communicatively coupled to the memory and arranged to operate by: operating at least one autonomous vehicle, which has multiple cameras and is arranged to move near a production line arranged to supply a product. The operation comprises automatically moving the at least one autonomous vehicle near the production line while free from a predetermined fixed route, and capturing, by the multiple cameras, at least a sequence of images of an area comprising at least a portion of the production line.The at least one processor is configured to operate by automatically recognizing at least one operation assigned to the production line, in which at least one sequence of images is captured, and by providing instructions to execute a response in response to the actions considered in the recognized operations. The response is assigned to the production line.

[0014] According to another exemplary embodiment, the at least one processor is arranged to operate by determining one or more differences between the actions under consideration and corresponding previously stored images of an expected action or undesired action, or both, by using the at least one sequence of images, informing a user of the differences on a display device located away from the at least one autonomous vehicle, and receiving instructions from the user to modify a planned movement of the at least one autonomous vehicle depending on the differences.

[0015] According to another exemplary embodiment, the at least one processor is arranged to operate by generating real-time 3D maps and using data from the production line and positions of multiple autonomous vehicles that change over time, and by using the at least one sequence of images.

[0016] According to another exemplary embodiment, the at least one processor is arranged to operate by: comparing the actions under consideration with images of expected actions or undesired actions, or both, to determine action differences, and generating response content depending on the differences, including providing all alternative options of (1) providing a signal to send to at least one of the autonomous vehicles, (2) providing a signal to send to at least one non-AMR device near the production line, and (3) providing data to inform a person about the action differences.

[0017] According to another exemplary embodiment, the at least one autonomous vehicle also comprises a body connected to a motion mechanism, a base, and a payload area arranged above the base. The base has side walls located below the payload area, each with multiple cameras pointing outwards from the base.

[0018] According to another exemplary embodiment, the multiple cameras are positioned on individual autonomous vehicles to capture images in 360 degrees horizontally around the individual autonomous vehicle.

[0019] In another exemplary embodiment, at least one non-volatile, computer-readable medium contains instructions which, when executed by a computing device, cause the computing device to operate by: operating at least one autonomous vehicle having multiple cameras and arranged to move near a production line arranged to supply a product.The operation involves: automatically moving at least one autonomous vehicle near the production line, free from a predetermined fixed route; capturing, through multiple cameras, at least one sequence of images of an area encompassing at least part of the production line; automatically recognizing at least one operation associated with the production line, capturing at least one sequence of images; and providing instructions to execute a response in response to the actions of the recognized operations. The response is associated with the production line.

[0020] According to another exemplary embodiment, the instructions cause the computing device to operate by moving one of the autonomous vehicles along the production line to keep one or more objects moving along the production line in view of the multiple cameras on the one autonomous vehicle.

[0021] According to another exemplary embodiment, the instructions cause the computing device to operate by having several autonomous vehicles moving along the production line each follow images of a different object and capture it as it moves along the production line.

[0022] According to another exemplary embodiment, the at least one autonomous vehicle is arranged to monitor objects associated with the production line, which includes at least one of: human operators, other autonomous vehicles of the at least one autonomous vehicle, objects moving in connection with the production line, objects being added to the product, tools being used to assemble the product, objects being moved to or from the at least one autonomous vehicle, and movement of robots.

[0023] According to another exemplary embodiment, the at least one autonomous vehicle is arranged to track objects that are assigned to at least one stage of the production line, including: material preparation, manufacturing of product components, sub-assembly of product parts, product assembly, product quality control inspection, product surface treatment, product packaging, product storage, and product shipping. Brief description of the drawings

[0024] The present description is further elaborated below in conjunction with the following figures. The figures are not to scale, and numbers in the figures denote identical elements, where: Fig. 1A is a schematic representation of a top view of an exemplary system of an autonomous mobile robot (AMR) set up in a manufacturing plant according to at least one of the embodiments described herein; Fig. 1B is a schematic representation of a top view of another exemplary autonomous mobile robot (AMR) system installed in a manufacturing facility according to at least one of the embodiments described herein; Fig. 2 is a schematic representation of a perspective view of an exemplary autonomous mobile robot (AMR) that monitors operation with an object for a production line according to at least one of the embodiments described herein. Fig. 3 a schematic representation of a perspective view of an exemplary autonomous mobile robot (AMR) on a line side of a production line according to at least one of the embodiments described herein; Fig. 4 a schematic representation of an exemplary manufacturing plant system according to at least one of the embodiments described here; Fig. 5 a schematic representation of an exemplary autonomous mobile robot (AMR) according to at least one of the embodiments described herein; Fig. 6 a schematic representation of an exemplary image processing unit for the autonomous mobile robot (AMR) or the plant system of the Fig. 1-5 and according to at least one of the embodiments described herein; Fig. 7 a flowchart of an exemplary procedure for operating an exemplary autonomous mobile robot (AMR) according to at least one of the embodiments described herein; and Fig. 8 is a flowchart of an exemplary procedure for operating an exemplary autonomous mobile robot system (AMR) according to at least one of the embodiments described herein. Detailed description

[0025] The following detailed description merely presents exemplary embodiments and is not intended to limit the description, application, or uses thereof. Furthermore, there is no intention to be bound by any theory presented in the preceding background or in the following detailed description.

[0026] The system and method disclosed here provide highly adaptable, integrated, autonomous vehicles, such as autonomous mobile robots (AMRs), each equipped with sensors and camera arrays. The AMRs monitor one or more stages of a production line from start to finish to track operations along the line and / or, for example, to monitor part of the assembly of a single product, while a single system analyzes image data from multiple AMRs along the production line. This AMR system operates despite any obstacles along the production line and without restricting the AMRs' movement to predetermined paths or routes that remain fixed during operation.

[0027] The use of AMRs enables dynamic camera array positioning and, by extension, dynamic view-based data analysis. This is achieved by allowing the sensors and cameras of an AMR to "follow" and synchronize with the production line, enabling real-time, location-based data analysis. This analysis can then be used by an AMR system dashboard or other plant systems (such as programmable logic circuits, PLCs) to increase production line efficiency, send digital inputs to plant floor systems, and trigger responses to the captured images in real or near real time. This can involve the use of a single AMR capturing images over time that can be recorded (or stitched together), or multiple AMRs along a production line that can simultaneously provide images for stitching.In both cases, a 3D map (or a digital or virtual model) of the production line can be generated, and the AMRs can be located on the map. Further image analysis can be used to detect production line operations and determine modifications to reduce delays and increase efficiency and performance.

[0028] In some configurations, AMRs carry payloads to the lineside of the production line, containing components to be added to the products being assembled and / or tools to be used by operators or robots on the production line. Thus, the disclosed vision-based AMR system with mobile AMRs can reduce or eliminate reliance on other visual monitoring systems, as the payload is carried by the AMR itself, rather than occupying space that the AMR must avoid. This also enables just-in-time placement of components and materials on the production line, while simultaneously increasing the coverage of the camera arrays.

[0029] Such AMRs can provide real-time location reporting, optional line tracking, and maneuverability, while the vision-based data analytics solutions presented here offer real-time location tracking, task time analysis, visual fault detection, and payload element counting. Combining the AMRs with such vision-based analytics also enables real-time repositioning (or position refinement) of the vision systems on the AMRs to better ensure consistent visual coverage along the entire production line or where desired. The AMR system also integrates remotely collected data inputs with plant-internal data systems to enable real-time manufacturing improvements. The data inputs can be used and analyzed to enable real-time line rebalancing, trend analysis, parts ordering, and AMR reassignment.

[0030] As described below, the camera arrays can be placed on a base or platform of the AMRs, with the payload positioned above the base. With this arrangement, overhead objects such as cranes, robot arms, and so on do not obstruct the cameras' field of view.

[0031] This arrangement of procedures for operating an AMR system also enables manufacturing plant personnel to analyze the integration and interactions of various systems across different sections of a single production line. This further allows plant personnel to modify production line systems to improve overall performance. The AMR system can also perform autonomous, real-time decision-making for line rebalancing, fleet management, material delivery, and so on, thereby increasing manufacturing plant efficiency. The AMR system also allows manufacturing plant personnel to record human operator actions directly alongside the line, providing an unbiased and unobstructed view of actual interactions between personnel and machines and the work being performed.The AMR system can also provide plant personnel with data analytics that reduces value-added versus non-value-added labor per location, allowing the plant to modify the operations of a monitored location to increase value-added effort and overall performance. The AMR system also enables the simultaneous use of AMR technology and vision-based data analytics solutions in a single package.

[0032] With reference to Fig. 1A A manufacturing plant 100 can have a plant system 101 with a production line 102 and products 104 and 106 that are assembled along the production line 102, while the products (or a sub-assembly of the products) move along the production line, as shown by the arrows. The products 104 and 106 are not limited to a specific product and are, for example, automobiles. Human operators 114 and 116 can perform tasks assigned to the production line 102 and may include performing tasks to assemble the products 104 and 106.

[0033] Production line 102 can perform many different stages of a manufacturing process, depending on the product being assembled. This can include any stage of manufacturing, such as material preparation, raw material handling, material cutting and forming (including any machining or shaping), component manufacturing, subassembly manufacturing, part assembly, final product assembly, intermediate and / or final quality control, inspection and testing, surface treatment, finishing or coating (such as painting, powder coating, anodizing or other surface treatments), drying, curing, final or fine-tuning of the product, labeling operations (including adding logos, seals or peel-off strips), performance testing, packaging, storage, and shipping. Thus, any single stage or combination of stages can constitute production line 102 and is not particularly limited.For example, one or more parts of production line 102 may be located inside a building, such as a factory or plant, especially one with overhead equipment. It is often difficult, if not impossible, to maneuver overhead monitoring robots to avoid the overhead equipment.

[0034] The products 104 and 106, which are being assembled, can be moved from stage to stage along the production line 102 using any suitable device depending on the product, including conveyor belts, cranes, daggers, trolleys or other vehicles, etc.

[0035] The Plant System 101 can also include an AMR System 120 with one or more autonomous vehicles, such as AMRs 108, including AMRs 110, 112, and 118. The AMR System 120 is particularly well-suited for operation under overhead equipment within a building or factory, but alternative configurations allow for outdoor or other environments. The AMR System can be used to monitor a wide variety of objects within a building or factory, such as infrastructure (building condition), other systems (fluid delivery system), or internal factory traffic, such as at internal factory intersections, and so on, to monitor pedestrians, operators, and moving equipment.In one exemplary configuration, any object within a building, factory, plant, or other area, such as a three-dimensional space with overhead equipment that should be avoided, can be monitored, as long as an autonomous vehicle is free to automatically set its own routes to perform the monitoring. In another configuration, the autonomous vehicles move on the ground using wheels or another mobility mechanism that touches the ground.

[0036] More precisely, each AMR 108 can have one or more cameras 130, forming a local or AMR camera array 132 on each AMR 108. Here, at least eight cameras 130 are provided on each AMR 108, with two cameras on each of the four sides of the AMR 108, although any desired number of cameras and camera arrays can be used. Together, the camera arrays 132 on all AMRs 108 interact to form a global or AMR system camera array 140. The cameras 130 are outward-facing, so each AMR camera array 132 defines an aggregated (or AMR or local) field of view 109, 111, or 113. By design, each AMR field of view 109, 111, and 113 is 360 degrees horizontally and can be 180 degrees vertically, depending on the cameras and their positions.By means of a shape, downward and / or upward pointing cameras (not shown) can also be located on each or individual AMRs 108 to define the 180 degree or 360 degree vertical range of the fields of view 109, 111 and 113, if relevant.

[0037] As shown in this example, the AMRs 108 can be positioned on the line side of production line 102 and can either remain relatively stationary to monitor an assigned area of ​​the production line or move along the production line or any other desired path to or from the production line. In the first case, the AMRs 108 can be assigned to a specific task or operation performed on the production line that does not move along production line 102, while the subassembled products 104 and 106 move along the production line. In this case, the fields of view 109, 111, and 113 overlap, allowing the onboard cameras 130 to maintain a view of successive production line operations as a product or other objects move through the fields of view 109, 111, and 113.

[0038] The camera arrangement 140 can then be used to assign an AMR 108 or a local camera arrangement 132 to monitor specific operations or tasks and, in turn, more precise visible actions, including the movement of objects. The term "objects" herein may refer to human operators unless the context is clear, or it is explained that plant or factory floor personnel are not included. The monitored operations may be referred to herein as considered or actual actions as opposed to predetermined expected actions, which are described below.Here, the AMRs 108 can monitor observable or visible actions in any of the production stages mentioned above, or any others, and can capture operators following a specified work plan, operators installing a part, new parts being delivered, safety concerns being identified, sequences of operations being tracked, use of personal protective equipment (PPE), tool use, task timing, and so on. In particular, such camera monitoring can encompass at least two major categories: human operator (or personnel) actions and autonomous robot operations.The AMR System 120 can monitor personnel tasks, such as loading materials onto conveyors, assembling components, quality-checking products, or adjusting machine settings, to ensure proper ergonomic practices, avoid bottlenecks, and comply with safety protocols, such as maintaining a safe distance from moving parts or operating machinery. Additionally, the AMR System 120 can monitor proper handling during assembly, triggering corrective actions in real time. It should be noted that the term "real time," as used here, refers to near real-time and relates to the timing as perceived by a person, which can have a processing delay of less than one second or up to several seconds.

[0039] Alternatively, the AMR System 120 can monitor autonomous robot actions, such as picking and placing components, welding, packaging, or palletizing, to name a few examples. The AMR System 120 can monitor these robot movements to ensure the robots are functioning correctly, tracking movements such as arm extensions, gripping actions, and material transfers for proper component placement, collision avoidance, and up-to-dateness. Monitoring robot actions also enables predictive maintenance, as the cameras can detect signs of wear or irregular movements that indicate a need for maintenance before malfunctions occur.

[0040] Throughout production line 102, the AMR system 120 can monitor various stages, such as material handling, the movement of raw materials from storage to the assembly area, checking for inventory issues, and identifying equipment malfunctions. During assembly, the camera array 132 can better detect when components are properly aligned and that workers are following assembly instructions, thus preventing defects. In the final stages, the AMR system 120 can check the quality of the finished products before they are packaged and shipped, as well as monitor the quality control and testing procedures performed. This ensures that only products meeting the required standards move on to the next stage, ultimately contributing to improved product quality, faster production times, and enhanced workplace safety.Many other examples of operations monitored by the AMR 120 system can be used.

[0041] As another exemplary alternative, the AMRs 108 (as shown by the dashed arrows) can move along production line 102 in conjunction with the movement of objects or products 104 and 106. In this exemplary case, each AMR 108 can be assigned to a specific product 104 to monitor the assembly of that particular product. Alternatively, each AMR 108 can be assigned to a different product or a different part of a product, with one AMR being assigned to monitor a front side of the product while another AMR being assigned to a back side. Many variations are considered. In this case, a single AMR 108 can monitor a sequence of varying operations and actions that change along with the stage of production line 102 as the AMR 108 moves along the production line.

[0042] With reference to Fig. 1B For yet another exemplary arrangement, plant system 103 is similar to plant system 101 and has similar elements with the same numbering, which do not need to be described again. In this example, however, an AMR system 121 shows a single AMR 150 that tracks a product 104 or another object or operator as the product 104 moves along the production line 102. As mentioned above, the AMR 150 may have cameras (not shown) to monitor a sequence of operations performed on the product 104. Furthermore, in this case, the AMR 150 can autonomously change its linear path to a path 154 to avoid an obstacle 152 along an initial path. The path adjustment can be made relatively quickly once the AMR 150 detects the obstacle 152.Once it has passed the obstacle, the AMR 150 can resume the linear part of path 154 to continue tracking the product 104.

[0043] With reference to Fig. Figure 2 shows a plant system 200 with a production line 202, which is monitored by an AMR system 220 similar to the AMR system 120, to show exemplary camera positions and demonstrate the tracking of a human operator (or plant personnel). Thus, in this example, an AMR 204 has eight cameras 206, each with a field of view (FOV) shown in dashed lines, which overlap to form a continuous 360-degree overall AMR FOV around the AMR 204. The AMR 204 can have a body 222 with side walls 208, where the cameras are mounted on the side walls 208, or have windows on the side walls 208. As with the AMR 108, each other side (front, rear, right, left side) of the side walls 208 has two cameras 206, although many other configurations can be used instead.

[0044] Alternatively, the individual AMRs can have camera mounts, which can be adjustable mounts or robotic arms, to optimally position the cameras for the greatest field of view coverage, especially when the AMR 500 is not carrying a payload, and the positioning of the cameras when it is less restricted.

[0045] Also as part of this example, a human operator 212 can move an object 214, be it a subcomponent of a product being assembled, a tool, or another object. The operator 212 and the object 214 are within the field of view 210 of one of the cameras 206 on the AMR 204, so that the object 214 can be detected and identified by the plant system 200 (in detail with Fig. 4 explained). The movement of the operator 212 and the object 214 is recorded over time, so that the operator's performance in moving the object can be analyzed.

[0046] With reference to Fig. 3 For another exemplary form, a plant system 300 has a production line 302, which is monitored by an AMR system 320, with AMRs 308 also being used to carry payloads 314 to the line side of the production line 302. The plant system 300 has many of the same features as the plant systems in the Fig. 1A - 2 and does not need to be described again. In this case, production line 302 produces products 304, such as automobiles, and a human operator 306 performs an operation on product 304. The AMR system 320 has an AMR 308 with eight cameras 310 on side walls 312 of the AMR 308, as in example cameras 130 and 206, and each camera has a field of view (FOV) as described above. In this example, a fixed object 318, which here is a plant column or other object, can be used as an anchor point for creating the 3D map of the production line using the cameras 310, as well as for locating the AMR 308 and detecting and recognizing the operator 306. The AMR 308 in this case has a top surface 316 under a payload 314 that the AMR 308 carries.The upper surface 316 may or may not be in direct contact with the payload 314 and can have many different shapes and mechanisms for receiving, holding, and providing elements of the payload 316. Many variations exist, and the AMR systems 120, 121, and 320 are not limited to any arbitrary arrangement for holding or supporting the payload 314.

[0047] With reference to Fig. 4. A plant system 400 according to at least one of the embodiments described herein can include any of the production lines as described above and can include an AMR system 401, which here may include AMRs 1 to N (110, 112 to 402 as shown) located on a plant or factory floor and similar to or identical to the AMR systems 120, 220 and 320 described above. The AMR system 401 can alternatively be considered separate from the plant system 400, since many units of the plant system 400, such as a control center, may be located away from the AMRs on a factory or plant floor.

[0048] The AMR system 401 can include an AMR controller 404 with at least one image processing unit 406, an action difference unit 408, a motion unit 430, a camera unit 432, and an optional payload unit 434. An AMR dashboard unit 414 may or may not be considered part of the AMR controller 404 and may be considered part of other systems on the plant system 400. The remainder of the plant system 400, which may be located remotely from the AMRs, can each include at least one communication unit 410, a plant data center (or plant data unit) 412, a display device 416, and / or a user interface 418.The plant system 400 may also include plant operating systems 420, which may include at least one material unit 422, an E-stop unit 424, an alarm unit 426, other programmable logic circuits (PLCs), or units 428 that operate various systems and control machines and computing devices within the plant and on the plant floor, or have mechanisms to initiate reports to personnel who manage or perform operations on the plant floor. It is understood that the term "plant floor" or "factory floor" herein refers to any location used for operations associated with a production line.

[0049] The plant system 400 may also include one or more processors 440 and a memory 442 to operate any of the units or systems of the plant system 400, and the processors 440 and the memory 442 may be considered part of any of the units of the plant system 400 described herein, including any of the programmable logic controllers (PLCs) 428. One or more of the processors 440 and the memory 442 may be provided for any of these units or systems that are remote from any of the other units or systems of the plant system 400.

[0050] In more detail, the processor(s) 440 is / are provided to perform the computing and control functions of any of the units and systems of the Plant System 400 and the AMR System 401, including the AMR Controller 404, and may be referred to as a controller, control unit, plant system, computing device, computer, and so forth. The processor 440 may have a circuit that may be part of, or constitute, the AMR Controller 404. The processor 440 may comprise a circuit or circuits that constitute any type of processor or multiple processors, including central processing units (CPUs), digital signal processors (DSPs), individual integrated circuits such as a microprocessor, or any suitable number of integrated circuit devices and / or printed circuit boards working together to achieve the functions of a processing unit.This can be a system-on-a-chip (SoC) with one or more processor cores. The 440 processor(s) can also include image processing technology, including graphics processing units (GPUs), image signal processors (ISPs), application-specific integrated circuits (AISCs), field-programmable gate arrays (FPGAs), neural processing units (NPUs), vision processors (VPs), video processing units (VPUs), and deep learning accelerators (DLAs). These processors can be shared or dedicated hardware.Dedicated or specific functional processors may also be provided that operate neural networks, machine learning, and other structures for image processing, such as graphics processing units (GPUs) or image signal processors (ISPs). During operation, the processor(s) 440 executes any of the units or systems of the plant system 400, and the units and systems of the plant system 400 may exist in any combination of hardware, firmware, and / or software. The software components of the units or systems may be stored in the memory 442 and, as such, control the general operation of the plant system 400 and the AMR system 401 when executing the plant system processes described herein, such as the processes and embodiments in any of the Fig. 1-3 and Fig. 7-8 and as described below in connection therewith.

[0051] The memory 442 is meant in a general sense and can be any storage device and can include any type of suitable memory. For example, the memory 442 can include various types of dynamic random access memory (DRAM) such as SDRAM, various types of static RAM (SRAM), and cache, while the memory 442 can also include various types of non-volatile memory (PROM, EPROM, and Flash). In certain examples, the memory 442 is located on the same computer chip as the processor 440 and / or is located together on the same computer chip as the processor 440. In the embodiment shown, the memory 442 stores the aforementioned plant system and AMR system units together with one or more databases to store map and image processing data and other stored values ​​156.Otherwise, the memory 442 can comprise various different types of random-access memory and / or other storage devices. In one exemplary embodiment, the memory 442 comprises a program product (or unit or system) from which the memory 442 can receive a program that includes one or more embodiments of the processes and embodiments of the. Fig. 8 and as further described below in connection therewith. In another exemplary embodiment, the program product can be stored directly in memory 442 and / or a secondary storage device (e.g., disk) and / or accessed in another way. During operation, the program and the accompanying data are stored in memory 442 and the program is executed by processor 440.

[0052] Thus, the memory 442 can, for example, store data from both the action difference unit 408 and action records in an action database 444 of both expected actions and corresponding undesired, previously recorded actions (also referred to as the historical or "unexpected" action record) associated with one of the production lines described herein. The processors 440 can be used to operate the action difference unit 408 to determine differences between considered actions and expected actions (and / or similarities between considered actions and undesired (or unexpected) actions). Such differences and similarities can be compared to image processing thresholds, such as pixel distances or sum of absolute differences (SAD) type comparisons, and so on, determined by experimentation.

[0053] The plant system 101 can include hardware that forms or supports the processor 440, the memory 442, and any other unit or system on the plant system 400, and the hardware can include at least one bus for transmitting programs (e.g., units or systems), data, status, and other information or signals between the various components of the plant system 400. The bus can be any suitable physical or logical arrangement for connecting computer systems and components. This includes, but is not limited to, direct hard-wired connections, fiber optic, infrared, and wireless bus technologies.

[0054] It is understood that, although this exemplary embodiment is described in the context of a fully functioning computer system, the person skilled in the art will recognize that the mechanisms of the present description are capable of being distributed as a program product with one or more types of non-volatile, computer-readable signal-carrying media used to store the program and its instructions and to carry out its distribution, such as a non-volatile, computer-readable medium carrying the program and containing computer instructions stored therein to cause a computer processor (such as the 440 processor) to execute and carry out the program.Such a program product can take a variety of forms within Memory 442, and the present description applies equally regardless of the specific type of computer-readable signal-carrying media used to perform the distribution. Examples of signal-carrying media that constitute Memory 442 include writable media such as floppy disks, hard disks, memory cards, and optical discs, and transmission media such as digital and analog communication links. It is understood that cloud-based storage and / or other techniques may also be used in certain embodiments.As mentioned above, it is equally understood that the 440 processors of the plant system 400 can include a multitude of processors in a multitude of locations, performing a multitude of functions for the plant system 400, for example, by the 440 processor being coupled to or otherwise utilizing one or more remote computer systems and / or other control systems. Thus, each unit and system of the plant system 400 has its own processor(s) as part of the 440 processors.

[0055] The exemplary communication unit 410 can include a transceiver and antennas for remote communication with the AMRs, other remote systems, servers, devices, modules, or units. It should be noted that any parts or components (or units) of the plant system 101 can be remotely used for any of the operations described herein relating to monitoring the AMR production line, analyzing captured images, and implementing responses as needed.For example, the Communication Unit 410 can receive inputs (such as a signal or data packet of image data and AMR and object position data) via application programming interfaces (APIs), messaging queuing telemetry transport (MQTT) for real-time tasks, representational state transfer (REST), and / or a Common Industrial Protocol (CIP) bridge to plant floor systems through wireless communication such as Wi-Fi, Bluetooth, radio frequency (RF), near field communication (NFC), ultra-wideband (UWB), and so on. Thus, the Communication Unit 410 can communicate over networks that include one or more computer networks such as the internet and communication networks (e.g.,Satellite-based, cellular, and / or any number of other different types of wireless communication networks). Through a process, the Communication Unit 410 collects transmitted data (such as images or signals from the AMRs) via the internet using a web address with a Uniform Resource Locator (URL), which is an API.

[0056] One or more displays 416, such as computer monitors, smartphones, tablets, and so on, can be used to show AMR monitoring data and images from the AMR dashboard unit 414 to a user, to inform a user of significant action differences and relevant warnings, and to (1) receive input from a user as to whether to set up AMR or other actions in the first instance, and / or (2) modify AMR or other object actions in response to or in reaction to the action differences, and by communicating with the PLCs or other units. By form, the display 416 can be any device that can provide a screen on a computer, smartphone, tablet, or other computing device to view the images on the display 416.Such a Display 416 can be a digital display, a graphical user interface (GUI), an LED display, a plasma display, an LCD display, an organic light-emitting diode (OLED) display, a thin-film transistor (TFT) display, a head-up display (HUD), 3D displays, holographic displays, virtual or augmented displays, and so on. The details of the use of Display 416 are explained below.

[0057] The user interface 418 enables communication between the user and the other plant system and AMR system units, as mentioned above. Thus, the interface 418 can include a touchscreen on the display device 416 or any other display, keyboard and / or mouse operation, an audio system, and / or other suitable interface devices and architecture. The user interface 418 can be mobile relative to the other units of the plant system 400 and can include smartphones or tablets held by or with personnel at the site (or on the plant floor or production line).

[0058] The plant data center (or plant data unit) 412 may manage or have the memory 442 and / or the action database 444 to store AMR data and provide inputs received from the AMRs, along with differences, from the action difference unit 408 to the PLCs 428 to determine which response to implement based on the differences. Alternatively, the plant data center 412 may transfer AMR report data to the display device 416 and / or the user interface 418 for display. The plant data center 412, either alone or in conjunction with the PLCs or other units, may also determine a production line recalibration, trend analysis, parts ordering, AMR reassignment, and so on, based on the action differences.The plant data center 412 can also manage historical archives placed in the plant operations memory 442, including the storage of considered actions and corresponding expected actions, undesired actions, action differences, and the resulting reaction that is implemented as a response.

[0059] Alternatively, the 412 plant data center can also serve as a technological hub and supervisor for collecting, storing, and analyzing data generated by manufacturing operations. It can integrate data from sensors, AMRs, other machinery, and control systems to monitor real-time performance, track production metrics, and optimize processes that may or may not be related to the AMRs. The 412 data center can support predictive maintenance, help prevent equipment failures, and enable automation for improved efficiency, while also managing energy consumption, inventory, and supply chain data and ensuring security and regulatory compliance.

[0060] The AMR controller 404 receives at least the image data from image sequences from each camera and AMR position data, but can optionally also receive local AMR maps or 3D maps. Individual AMRs can also have their own AMR controllers to generate action differences, either on a global scale, including all AMR cameras, or on a local AMR scale at each AMR. In these cases, the action differences can also be provided to a central AMR controller 404. Many variations are available.

[0061] In the present example, the AMRs do not have their own action differentiation capability, and this is centralized at the AMR controller 404, located remotely from the AMRs, although, as mentioned, other approaches can be used. Here, the inputs are received by the communication unit 410. The image processing unit 406 then analyzes the various images by registering images from different cameras and creating a 3D model or map of the production line. For example, different local AMR 3D maps from several AMRs are merged, or several local 3D maps from a single AMR, which is moved to capture different perspectives, are merged.In an exemplary manner, this is accomplished by using feature matching algorithms, such as visual simultaneous localization and mapping (VSLAM) and / or other algorithms described below. The same or other object recognition algorithms can then be used to identify operations and movements in the image sequences in order to generate detected viewed actions for comparison with predetermined expected actions and / or predetermined undesired actions from the action database 444. Subsequently, the action difference unit 408 determines and reports the differences between viewed and expected actions (and similarities between viewed and undesired actions, if used). The details of this procedure are described below with reference to Procedure 800 (. Fig. 8) provided.

[0062] The Motion Unit 430 provides motion instructions to the AMRs and modifies instructions when needed due to differences in action. Thus, the Motion Unit 430 can provide broad instructions for an AMR to move to a specific location, while the AMR itself determines the precise path to that location. The Motion Unit 430 can override the AMRs' path-generation controls if the difference in action indicates that a specific path must be followed by the AMR. Many variations are considered.

[0063] The Camera Unit 432 can control camera operations on all AMRs by monitoring when the cameras are activated or deactivated, ensuring that the cameras are pointed in the desired directions, have the correct focus levels, and adjust other camera settings. The Camera Unit 432 can also provide modifications and updates for image quality and other functions.

[0064] The optional Payload Unit 434 is deployed when at least one AMR is carrying a payload. The Payload Unit 434 can monitor payload quantities on one or more AMRs and provide instructions to move an AMR to a payload loading dock or area to retrieve more payload and to a desired location on the production line.

[0065] The AMR dashboard unit 414 can display action differences, for example, on the display device 416, as well as other data related to AMR performance. The AMR dashboard can also be used to provide a list and view of a variety of available reports, create user-requested custom reports, display historical data, or provide other reports for the user to view. The AMR dashboard unit 414 can also receive user input requesting information or providing instructions to generate a response to action differences. This can include updating the programming of various AMR or plant systems by the AMR dashboard unit 414, although other interfaces can be used instead.

[0066] The Plant Operating Systems 420 manage the various systems that operate the production line and other areas of a manufacturing plant. For example, the PLCs 428 can receive data associated with an action difference and then decide what the appropriate response is, generate instructions for the AMR or other device that implements the response, and initiate reports associated with the action difference, including providing data or instructions to the other Plant Operating Systems 420 units, as well as reports that can be viewed by a user.

[0067] Alternatively, in some configurations, exemplary PLCs 428 can also be provided to process inputs from sensors, switches, and other devices; control outputs such as motors, actuators, and valves; and manage a wide range of operations. This can include sequential controls, such as controlling stages in a production line; process controls, such as monitoring and regulating variables like temperature, pressure, or flow; motion controls, which coordinate the movement of machines or robot arms; safety systems, which ensure emergency shutdowns or safety interlocks; material handling, such as with AMRs, conveyor systems, or robot pick-and-place operations; and communication, which is integrated with other systems, such as Supervisory Control and Data Acquisition (SCADA) or Manufacturing Execution Systems (MES) for plant-wide data exchange.

[0068] The material unit 422 can receive instructions to retrieve more material from the PLC 428, and then instruct the payload unit 434 to control an AMR to retrieve more material, and can provide the retrieval instructions to another device or personnel on the production line (or material delivery point). Many variations exist.

[0069] The E-Stop Unit 424 can receive instructions from the PLC 428 to stop operations on all or part of the production line. These instructions are then implemented by the E-Stop Unit 424 by sending signals to devices or computer systems that control the production line. Many variations exist.

[0070] The warning unit 426 can receive instructions from the PLC 428 to provide a warning to users and personnel working on or otherwise associated with the production line, to warn of significant action differences (or any action differences). Data from such warnings can be provided via the plant data center 412 and to the AMR dashboard unit 414 to display the action difference (or anomalies) and other related data on the AMR dashboard unit 414 for a user on the display device 416. Further details of the operation of the plant system are described below with reference to Procedure 800 ( Fig. 8) provided.

[0071] With reference to Fig. 5 is an exemplary autonomous vehicle, such as an autonomous mobile robot (AMR) 500, the same as or similar to the AMRs of the AMR systems 120, 220 and 401 described above. In an exemplary embodiment, the AMR 500 may have a controller 501, cameras 504, wheels 526 or other transport mechanism, motor(s) 528, a steering mechanism 530, sensors 520, a local user input device 516 which may have a display device or screen 517 (or additionally or instead, speakers and / or microphones of an audio system), and a data storage device 518 which may store the programs or code for any of the units of the AMR 500.Alternatively, the data storage unit 518 can also store the programs or code of an image processing unit 522 and / or any other unit described herein, which is used to analyze image data from the cameras 504 of any of the AMRs 500, in addition to or instead of the units that analyze the image data at the plant system 400. Thus, such image processing units 522 can be provided on at least one of the AMRs, some of the AMRs, or none of them, and can analyze image data only from the AMR's own camera array, or can receive images from other AMRs in the global camera array, including all AMRs. Alternatively, the data storage unit 518 can have any of the hardware structures mentioned above for the memory 442.

[0072] By way of example, the AMR can have wheels 526 for movement, but it can have any other suitable mechanical transport device or system, such as skate blades, rollers, fan propellers, whether for vertical rotation as in fan boats or horizontal rotation as in drones or helicopters, and so on. The wheels 526 can be differential-drive, omnidirectional, or all-terrain wheels, providing flexibility in movement without being restricted to tracks. The wheels 526 can be operationally connected to axles or other mechanisms connected to one or more motors 528, including both drive motors and steering motors. The motors 528 can be electric, fuel-powered, or of another type.A steering mechanism 530 can also be connected to the wheels 526 to steer one or more of the wheels that have a steering mechanism, such as tie rods and a rack, which rotate one or more wheels 526 to control the direction of movement of the AMR 500. Many other configurations and arrangements can be used as long as the AMR can be steered and driven autonomously.

[0073] The Sensor Unit 520 can include the sensors themselves and any processing equipment needed to collect sensor data and provide it in an expected format. This can include sensors such as light detection and ranging (LiDAR), other cameras, ultrasonic sensors, and infrared sensors that help the robot perceive its environment and detect obstacles. Other sensors that can be provided include detection sensors (e.g., radar, sonar, or the like) and / or other sensors (e.g., vehicle position sensors, speed sensors, accelerometers, gyroscopes, inertial measurement unit (IMU) sensors, brake sensors, steering sensors, and so on). In various embodiments, the Sensor Unit 520 receives additional information regarding the production line environment and / or the operation of the AMR 500 itself (e.g.,Position, speed, deceleration and / or acceleration thereof, and so on) for use in operating the AMR 500, for example according to the autonomous operation of the AMR 500 and / or certain components thereof.

[0074] In some implementations, the 504 cameras used to capture images of observational or perceptual data from the production line or its environment can be front, rear, side, top, bottom, and / or surround-view cameras on the AMR 500, including wide-angle, 360-degree, and / or fisheye lens cameras, as well as monocular, stereo, infrared, time-of-flight, thermal, LiDAR, and high-resolution cameras (e.g., RGB, depth, thermal) cameras, and so on. These 504 cameras can capture images that are then processed as described herein. In various implementations, video camera images are obtained. Additionally or alternatively, still camera images can be obtained.

[0075] The local user input devices (or interfaces) 516 and the display device 517 are as described above with the user interface 418 and the display device 416. The interface 516 can be used to view output data from the AMR 500 or input data into the AMR 500, regardless of whether the status of the AMR 500 is to be determined or whether any feature of the AMR 500 is to be controlled otherwise.

[0076] In an exemplary embodiment, the AMR controller 501 comprises one or more processors 502, a power unit 506, a mobility unit 508, a navigation unit 510, an optional local payload unit 512, a communication unit 514 and a local AMR control unit 524.

[0077] The 502 processor(s) can be formed by a processor circuit that operates or forms any of the units or systems on the AMR 500. Thus, the 502 processors are derived from the above for the 440 processors ( Fig. 4) mentioned architectural alternatives. Likewise, the communication unit 514 is similar to the communication unit 410 of the plant system 400 and can include a transceiver and antennas to transmit AMR image data as well as any AMR-related data generated by the local image processing unit 522 or the sensors 520, such as position and orientation data, and any other AMR data, including status data, performance data, and so on. The communication unit 514 also receives any instructions, requests, or other data to control the AMR.

[0078] The 506 power unit can control and / or have the power source of the AMR 500 and can be any suitable power source, including electrical power sources, whether battery, AC or DC, and especially lithium-ion batteries. Some AMRs can use any one or more of the following: lead-acid batteries, fuel cells, hydrogen-powered fuel cells, and supercapacitors, with or without other batteries. Additionally, the AMR 500 can be wirelessly charged or inductively charged.

[0079] The mobility unit 508 controls the motors 528, which in turn control the steering mechanism 530 and the wheels 526 (or other transport mechanism) that are used.

[0080] The local AMR controller (or unit) 524 controls the actions of the AMR, which include receiving and analyzing instructions and commands, whether from its own onboard protocols or from received messages or signals that instruct or command the AMR to perform specific tasks. In one form and in various embodiments, the local AMR controller 524 is coupled to the sensor unit 520, as well as to the mobility unit 508, the navigation unit 510, and the payload unit 512, when deployed. In various embodiments, the local AMR controller 524 is also coupled to the data storage unit 518, the display device 517, and the communication unit 514.The instructions can include software protocols for data transmission and coordination with other systems, as well as manage image processing by the image processing unit 522 or navigation unit 510 to locate itself on a map of the production line, move the AMR from the protocols or received instructions to target locations, and avoid obstacles, etc. Thus, the local AMR control unit 524 can operate operating systems and algorithms that manage tasks such as path planning, decision-making, and obstacle avoidance, including the operation of the image processing unit 522. The local AMR control unit 524 can also control the operation and settings of the camera array on the AMR, as well as the transmission of image and position data to the plant system 400 when generated.

[0081] The navigation unit 510 can perform local image processing and / or sensor data processing, as instructed by the AMR control unit 524, to generate a local 3D map at the AMR level when it does not rely on the image processing unit 522 or image processing unit 406 of the plant system 400 ( Fig. 4) By design, the navigation unit 510 focuses on creating a 3D map and localization, while the image processing unit 522 focuses on localizing and recognizing other objects and processes captured by the cameras 504 to provide locally determined actions for the AMR 500, rather than the remote image processing unit 406 determining the actions. It is understood that the plant system 400 can distribute the image processing tasks among the image processing unit 522, navigation unit 510, and image processing unit 406 of the plant system 400, choosing the delegation of tasks that proves most efficient, timely, or performs best. Alternatively, 2D maps can be used instead of 3D maps if sufficient.

[0082] The optional payload unit 512 can control payload carrying operations when the AMR 500 is carrying a payload. This can include receiving data such as a count of the payload elements being used and captured in images by one of the image processing units 522 or 406, determining a target time or point in time to retrieve more payload, initiating the retrieval in coordination with the local AMR control unit 524 to move the AMR 500 as needed to retrieve a payload without interrupting production line monitoring. Further details of the operation of the AMR 500 are provided below with Procedure 700.

[0083] It is understood that, in one exemplary configuration, all units of the AMR controller 501 and data storage unit 518 can be mounted on or inside a body of each or individual AMR 500. Alternatively, any parts or components (or units) of the controller 501 and data storage unit 518, which perform the processing for any of the operations described herein relating to AMR monitoring, can be removed as required. In particular, in various embodiments, the controller 501 is arranged inside the body of the AMR 500. In certain embodiments, the controller 501 and / or the local AMR controller 524 and / or one or more components thereof can be located outside the body of the AMR 500 (such as the body 222 of the AMR 504). Fig. 2)) be located, for example, on a remote server, in the cloud, or on another device where image processing is performed remotely. It is understood that the controller 501 and / or the local AMR controller 524 will otherwise differ from the one in Fig. The embodiment shown in section 5 may differ. For example, the controller 501 may be coupled to one or more remote computer systems and / or other control systems, or may otherwise use them, for example as part of one or more of the devices and systems of the AMR 500 identified above.

[0084] With reference to Fig. 6. An image processing unit 600 may be the same as or similar to an image processing unit 406 or 522. The image processing unit 600 may include a separate registration or merging unit 602, which is used when the registration itself is not performed by an imaging unit 604. The image processing unit 600 may also include a triangulation unit 606 and an action detection / verification unit 608. One or more of these units may be considered separate from the image processing units or may be considered part of another unit of the AMR 500 ( Fig. 5) or the plant system 400 ( Fig. 4) are considered.

[0085] In an exemplary first approach, the image processing unit 522 and the local AMR controller 524 on the AMR 500 perform the initial operations to register images from various cameras together in order to form an AMR or local 3D map, to locate the AMR, and to perform object detection to avoid obstacles. Meanwhile, the image processing unit 406 on the AMR controller 404 of the plant system 400 can perform object detection and operational identification to recognize the actions under consideration. In one case, the AMRs transmit inputs containing the image sequences and AMR position data to the plant system, although the local 3D maps are also sent. The action differences are then determined by the action difference unit 408, located remotely from the AMRs.Alternatively, these tasks can be shared or balanced between the two image processing units 406 and 522 in other ways, such as when practical, to increase efficiency and performance and reduce latency, to achieve real-time analysis, and to implement rapid responses on the production line as needed or desired. Thus, in the second alternative, the AMRs perform more of the image processing on board, with one or more of the individual AMRs featuring the AMR controller 404 to generate global 3D maps, localization of multiple AMRs and databases, and the action differential unit 408 to detect viewed actions, which are then sent to the level of the plant system 400.A third option allows the AMRs to perform minimal onboard image processing, whereby a remote AMR controller 404 can receive image sequences from the AMR cameras and perform the image processing to generate any desired 3D map for locating the AMRs. Any desired combination or variation of this can be used.

[0086] More precisely, when the image registration unit 602 is separate from the imaging unit 604 and located locally on the individual AMRs, it receives at least the AMR camera array images from its own onboard AMR cameras. However, it can also receive images from other AMRs to obtain images of the global camera array from all or multiple AMRs used on or associated with the same production line. This image registration unit 602 then performs registration or merging, which can be carried out using many different feature recognition and feature matching algorithms.Such techniques may include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), and Kernel Aligned Zero-Mean Extended (AKAZE or accelerated KAZE) to identify distinguishable points in the image that are invariant for scaling, rotation, and, in some cases, affine transformations. Other feature-matching techniques, such as Fast Library for Approximate Nearest Neighbors (FLANN) or BRUTEFORCE (simple exhaustive search), can be used to find matches between feature descriptors. Additionally, geometric verification procedures, such as RANSAC (Random Sample Consensus), can refine matches by rejecting outliers and ensuring robust matches between image pairs.

[0087] The imaging unit 604 then creates a 3D map of at least a portion of the production line when cameras from a single AMR are used. This can cover the entire production line or at least the portion of the production line covered by the global camera array when images from multiple AMRs are used. One form uses visual simultaneous localization and mapping (VSLAM) or, alternatively, SLAM only, and can perform both image registration and the mapping itself, which may involve the use of one or more of the registration techniques mentioned above.

[0088] The VSLAM performs simultaneous mapping, localization, and tracking of AMRs over time. Thus, the VSLAM can continuously update both the AMR's position and the map in real time as the AMR moves. This can be done using only visual data (the captured images), but can also be enhanced by using sensor data to confirm and refine the AMR's position and, if desired, the position of other objects. Mapping involves identifying and tracking key features (such as corners, edges, or textures) in the visual data, which are then used to construct a 3D map of the space. This map is continuously updated to track the AMR's movement in real time. Localization can also determine the AMR's orientation within the map.

[0089] Thus, VSLAM can perform a form of feature recognition, feature matching, and camera motion estimation to determine camera positions and orientations (or poses) relative to the environment. It can also perform map construction, which can be 3D mapping or modeling, and localization, where mapping and localization can be confirmed, populated, and refined by using other sensors (such as LiDAR and / or GPS) for more accurate localization and mapping. VSLAM can also perform feedback correction, which corrects for "scale drift" over time, where the estimated map gradually grows or shrinks in size due to errors in depth estimation. The AMR 502 (and / or 440) processors can incorporate graphics SoCs with GPUs and deep learning capabilities to power neural networks that handle the image data for the tasks mentioned above.

[0090] Alternatives to VSLAM, or algorithms that can be used in addition to or as part of VLSAM, include any combination of the techniques mentioned above with feature registration, or any combination of object detection and recognition algorithms, such as those based on one or more of the following: machine learning, neural networks, convolutional neural networks (CNNs), region-based convolutional neural networks (R-CNNs), recurrent neural networks (RNNs), masked R-CNNs, you only look once (YOLO), single-shot multibox detectors (SSDs), semantic segmentation such as fully convolutional networks (FCNs) and U-networks, for example, Haar cascades (Viola-Jones (VJ) detectors).Histogram of Oriented Gradients (HOG), MOG (Mixture of Gaussians) background subtraction, template matching, DPM (Deformable Parts Model), GMM (Gaussian Mixture Model) background subtraction, LDA (Linear Discriminant Analysis) and / or many others.

[0091] Once localization and 3D mapping are established, the 606 triangulation unit can be used to detect, recognize, and identify other objects associated with or located on the production line. Techniques such as Kalman filters, optical flow, and others mentioned here can be used to track moving objects and maintain their identities over time. Thus, the 606 triangulation unit can also perform, or establish, a non-AMR object detection and recognition operation if it is not already being performed by the VSLAM in the continuous 3D mapping and AMR localization operations.As mentioned above in an alternative scenario, non-AMR triangulation and object detection can be performed entirely by the image processing unit 406 when generating a global 3D map by registering images from all or multiple cameras of all or multiple AMRs. Alternatively, this can be performed entirely or partially by the image processing unit 522 on one or more of the AMRs. The object detection and recognition techniques mentioned above can be used, and then triangulation can be applied to track the positions of objects, where the objects can be any object within the cameras' fields of view, be it the product being assembled, robots or machines used to perform the assembly or other tasks associated with the production line, and human operators.This is done continuously to track the movement of objects recorded in the global 3D map over time, in order to create a highly accurate and precise model of the objects. A digital or virtual production line can then be generated from the 3D mapping and object tracking.

[0092] The image processing units 406 and / or 522 can then incorporate the action detection and verification unit 608. At this point, each time step (or stamp) of the AMR monitoring can have a 3D map of the camera images, creating a sequence of 3D maps to capture production line operations in data from that sequence. The sequences of identified objects are then analyzed to determine separate actions under consideration using a number of the object detection techniques mentioned above. The identification is then verified by comparing the actions under consideration with expected actions from database 444 or another database to identify separate operations, such as movement by a single robot or a single human operator.Verification is performed by inputting the actions in question into a neural network that is trained to compare them to the expected actions. If this initial comparison is sufficiently close, the identified operation can be flagged or marked.

[0093] The subsequent operations for comparing considered and expected actions (and undesired actions), generating responses, reporting, and historical archiving, as performed at the level of Plant System 400 or at the AMRs, are described below in Procedures 700 and 800.

[0094] With reference to Fig. 7. A method 700 for operating an autonomous vehicle, such as an autonomous mobile robot (AMR) used in the examples described herein, is provided according to at least one of the embodiments described herein. The method 700 is described by operations 702-734, which are generally numbered evenly. The systems, methods, vehicles, devices, vehicle displays, and components of Fig. Reference can be made to sections 1-6 where relevant.

[0095] Procedure 700 can include "Activating the AMR System" 702. Through a specific process, the AMR systems (such as 120, 220, and 401) are automatically activated as soon as power is supplied to the AMRs 500 and the units of the AMR system 401 are started. This can include initializing AMR hardware, such as motors, sensors, cameras, and so on, as well as calibrating the cameras and sensors. The AMR 500 can start from an initial location, such as a base or charging location, or it can otherwise start from a payload loading dock or any other desired location associated with deploying the AMR 500 for production line operations.

[0096] Once the AMRs 500 and the AMR System 401 are activated and running, the AMRs 500 can perform continuous surveillance, for example, by using high-speed cameras that provide 30-120 frames per second (fps), but can also use 1-30 fps, 15-30 fps, 30 fps, 60 fps, or other desired frame rates. Preprocessing of the raw image data, such as demosaicing, noise reduction, scaling, and so on, can be performed on the cameras themselves or by the ASR's 522 image processing unit.

[0097] For the AMRs 500, the procedure 700 "Detect an Area Map" 704 can be used and, as mentioned above, can involve the use of VSLAM or other techniques and provide images in real time. The result can be a 2D or 3D map of the production line area, which can be used to create 3D models of the production line. The VSLAM or other algorithm can be run continuously to generate a map with each image or at desired intervals, such as once every 10 images, to produce a map at any desired mapping time interval. The 3D maps can then be used to track changes to the production line over time.As mentioned, each AMR 500 can generate its own local 3D map at the AMR level, or a global map can be generated at each AMR or at one or more AMRs if the image data is shared among the AMRs.

[0098] Method 700 can feature “locating AMRs on the map” 706, where VSLAM also detects the AMRs and simultaneously tracks the movement of the AMRs over time. Camera-based visual mapping and localization can utilize features and objects in the production line environment as fixed anchor points, such as the object or column 318 ( Fig. 3) to anchor the map and determine AMR positions relative to the anchor.

[0099] Procedure 700 can optionally include "Retrieve Payload" 708 if one or more of the AMRs 500 are to carry a payload. In this case, the payloads are configured for loading onto the AMRs 500. The AMR 500 uses mapping and localization from the VLSAM to generate a path to a payload loading dock or area, or the AMR can determine that it is already at a target payload loading location. Payload loading can be performed automatically by autonomous equipment on or separate from the AMR 500, such as grabs, cranes, conveyors, and / or autonomous forklifts, but otherwise manually by human operators, either by hand or using equipment such as daggers or manually powered forklifts, and so on. There are many different examples.

[0100] Depending on whether or not a payload is being carried, the procedure 700 “Receiving Position Instructions” 710 can include autonomous path planning to a target location on or around the production line to monitor the production line and, if assigned, deliver a payload to the production line. The target instructions can be obtained, for example, from onboard logs or received from other units on the plant system 400. The planning can be performed before, during, or after the AMR 500 is at the payload dock.

[0101] Through a form, operation 710 “Receive / Determine Initial Production Line Position” 712 can have, and this is provided when the AMR 500 is to generally hold at least a single position on the production line, typically together with other AMRs, so that each AMR is assigned a different production line section or production line zone to monitor, as with the AMR system 120 ( Fig. 1) The AMR 500 can still move back and forth from this position to receive payloads or for other tasks, such as to gain a better view of an object or process being carried out, for example, within an assigned section of a production line. Such field-of-view adjustments to the AMR can be performed autonomously or manually and relatively quickly, if desired.

[0102] Alternatively, operation 710 “Receive Operation / Object Assignment” can have 714, where the AMR 500 is not assigned a specific section of the production line, but instead a product being assembled or another object moving along the production line. In this case, the target position is an initial position of the AMR. Once positioned on the production line, the AMR then continuously identifies or recognizes the assigned object or product and moves along the production line while keeping the assigned object or product within the local field of view of the local camera array on the AMR. In this case, a single AMR can be used instead of multiple AMRs or multiple AMRs used in parallel, each with a moving object assigned to monitor.

[0103] Other variations can be used, such as assigning an object or product to an AMR only within a specific section of the production line, or moving all AMRs in a fixed sequence along the production line regardless of the presence or location of objects and products on the production line. Many other variations exist.

[0104] Procedure 700 can include “Planning the path from the current position to the target” 715. The most up-to-date 3D (or other) map and localization data can then be used to generate a path from a current (or other significant and relevant) position of the AMR to the target position.Path generation can take into account actual and potential obstacles and traffic on the map, such as other AMRs, objects and human operators detected on the map, other mapped or known obstacles or paths, such as doors, ramps, elevators, and other factors, such as time consumption, just-in-time scheduling, speed limits, energy consumption, wear and tear of physical components on the AMR, safety protocols, prohibited AMR travel zones, other designated zones, task prioritization and spacing, payload size (if overload payloads increase the required AMR distance), sensitivity of physical impacts to the payload or other equipment on the AMR, such as shocks while the AMR is traveling, and environmental conditions (lighting, temperature, etc.).), which could affect computer vision during the journey, temporary or maintenance obstacles, backup or secondary paths if needed, and multi-AMR path planning sharing and cooperation to avoid paths generated by other AMRs. Many other factors can be considered, which are also not listed here.

[0105] Procedure 700 can include "Driving to Initial Lineside Position" 716, where the AMR 500 then travels along the autonomously generated path, taking into account factors as mentioned above, and while free from predetermined fixed routes that cannot be modified by the AMR. The AMR can adjust its position using VSLAM and the other techniques mentioned above, as well as sensor data, to stay on course and refine its positioning. The AMR travels to the instructed destination, which in this case can be on the lineside of a production line.This process of autonomously generating paths between a current position and a destination can be repeated as needed and can use paths to any desired destination provided to an AMR, whether on the line side or another destination associated with the production line, typically within or on a factory or plant site, campus, or building. Other options allow the AMR to be moved as desired for other production lines (multiple production lines) or for other reasons.

[0106] Through a form, the navigation unit 510 can perform path planning, and the local AMR controller 524 can communicate with the AMR controller 404 and other plant operating systems 420 or users via the AMR dashboard 414 to coordinate actions (e.g., sending commands to pause or reroute an AMR). The local AMR controller 524 can also adjust the path of the AMR based on real-time data received from the cameras, updating any of the factors mentioned above.

[0107] If the objective is payload delivery to or on the production line, other operations can be performed autonomously or manually, such as verifying the precise docking or unloading point position and orientation of the AMR to ensure proper payload unloading. This can be accomplished using the AMR's own cameras, cameras from another AMR that has the delivery AMR in sight, and / or other onboard sensors (e.g., proximity sensors).

[0108] The procedure 700 can include “monitoring production line operations” 718 and “performing continuous camera capture” 720, which refers to the AMR cameras 504 performing continuous image or video capture as described above while keeping a specific section of the production line in view, or specific products, objects, or operations in view, also as described above. The captured sequences of images (or video sequences) from each camera are then stored in memory 518 for analysis by the local AMR controller 524, navigation unit 510, and / or payload unit 512 to determine when a new payload should be retrieved.

[0109] Optionally, the procedure 700 can include a “movement along the production line with assigned object” 722, whereby the navigation unit 510 and the local AMR controller 524 perform path generation as described above. This can include route monitoring and obstacle avoidance by continuously scanning the environment around the AMRs using cameras, lidar, or other sensors as mentioned above to detect obstacles in a planned path. The path can be dynamically adjusted, such as when obstacles are detected, and the VSLAM 3D map can be checked for consistency by verifying that the AMR is correctly located (for example, factorization drift correction). The 3D map can then be updated in real time as needed.

[0110] Alternatively, the method 700 can optionally include “performing local image processing” 724, wherein one or more of the AMRs collect the image sequences or 3D maps at the AMR level and localization data from several AMRs 500 to generate a global 3D map. If the capacity of the onboard AMR systems is available, the local AMR controller and other units can also perform operational monitoring of the AMR controller 404, including detecting operations under consideration, comparing them with expected and undesired operations, and generating a difference that can then be provided at the plant system level or analyzed locally to determine responses in response to the difference, such as changing the path or orientation of an AMR.Thus, the individual AMRs 500 can themselves generate reports on performance, efficiency, and anomalies in the production process. Through a form factor, the local user input device 516 and the display device 517 can then provide an onboard AMR dashboard to display real-time data, trends, and warnings related to the assembly process and then receive input from a user or human operator on the plant floor.

[0111] Alternatively, the procedure 700 can include a “transmission of image data to the AMR controller” 726, wherein the AMR provides the plant system 400 with camera inputs in the form of captured images and time-stamped positions of the AMRs via onboard communications 514. In this example, the input can include the AMR positions and either captured image sequences or the generated 3D maps at the AMR level, or both, which are then transmitted from the AMRs 500 to the remote plant system 400 and, in particular, the AMR controller 404. This can include the transmission of raw sensor data and analyzed and calculated AMR and / or production line metrics, such as assembly speed, product counts, defect rates, and so on. Such analysis can be performed using the image processing unit 522 and the local AMR controller 524.

[0112] Continuing with this example, after the camera input and other AMR data have been transmitted to the remote plant system 400, the procedure 700 can include a “receiving of response instructions” 728, whereby instructions are received at the AMR 500 to modify the operation of one or more AMRs 500 when a sufficiently significant difference between the action under consideration and the expected action (or similarity between the action under consideration and the undesired action) has been determined. The analysis of the differences is discussed below with regard to the procedure 800 and at the plant system 400 level.The received instruction can include a change to the planned path of an AMR, a change to the position or orientation of an AMR, a change to the object, production, or operation assigned to an AMR, or an instruction relating to any other action of the AMR, including an emergency stop of the AMR and other moving objects assigned to the production line. The instructions can also include signals to other computing devices, such as robots that assemble a part onto a product, or other machines involved in the assembly or other stages of forming and delivering the finished product.In another form, the instructions may include the provision of warnings or other data to be read or heard by a human operator or personnel, whether on a computing device, on a display, or on machinery or equipment on or associated with the production line, including displays or audio speakers on the AMRs themselves, and this may include a printout on paper or other media.

[0113] In one of the forms mentioned above, instructions can be generated based on an action by one AMR observed by another. For example, one AMR might observe that another payload AMR is positioned too far from the production line, causing a human operator manually unloading a payload to spend extra time moving back and forth. The instructions can then direct the payload AMR to move to a position closer to the line and the human operator, thereby reducing delays and increasing efficiency.

[0114] An option allows a payload AMR to receive a "Payload Instruction Received" (730) when it detects that the payload has been fully or sufficiently unloaded and more payload should be retrieved. This can be triggered locally by the AMR's own cameras alone, or via instructions from the plant system level and observation of images from another AMR viewing the payload AMR. The instructions can be a simple code indicating that more payload should be retrieved.

[0115] The procedure 700 can include "Performing Actions According to Instructions" 732, whereby the AMR path, position, orientation, targets, tasks, and so on can be modified and executed by the AMR according to the instructions. This operation 732 can include "Receiving Next Payload" 734 for the payload AMR, which then determines a path back to the payload loading dock, leaves its current lineside (or other) position, and returns to the payload loading dock along the autonomously generated path.

[0116] With reference to Fig. Section 8 provides a method 800 for operating an autonomous mobile robot (AMR) system used in the examples described herein, according to at least one of the embodiments described herein. Method 800 is described by operations 802-836, which are generally numbered evenly. The systems, methods, vehicles, devices, displays, and components of Fig. Reference can be made to sections 1-7 where relevant.

[0117] Procedure 800 can include "Activate AMR System" 802, which has some of the same operations as operation 702 of procedure 704 for the AMRs. Here, the AMR system units on the plant system 400, as well as other systems and units, can be started if they are not already in an "always-on" mode.

[0118] Method 800 can include “receiving image data from one or more AMRs” 804, and this can include receiving at least one version of the input described by the transfer operation 726, which includes at least image sequences and time-stamped positions of the AMRs and optionally the 3D maps generated by the AMRs, sensor data and calculated AMR and production line performance parameters, such as production line assembly speeds, accuracy and so on.

[0119] The procedure 800 can include “preprocessing of image data” 806, and this can include any noise reduction, any other quality improvement, scaling, and so on, so that the received images are in a format expected by the image processing unit 406 to be performed for viewing action detection and action difference determination.

[0120] The procedure 800 can include “generating a 3D map of the production line and localizing AMR positions over time” 808, and by form this can include “using VSLAM” 810 and also on the side or level of the plant system. Alternatively or additionally, algorithms other than VLSM can be used. This operation 808 can include “registering images relative to each other from multiple cameras and / or multiple AMRs” 812. Thus, when the AMR-local 3D maps are received from the AMRs at the image processing unit 406, the maps can be combined to form a global 3D map or a global 3D model of the entire production line or a part of the production line covered by all AMRs monitoring the production line.Otherwise, if the AMRs simply transmit the separate image or video sequences from the individual cameras, the 406 image processing unit can combine the individual video sequences to form the global 3D map. This can be done by combining all of these images with the same or substantially the same timestamp to create a 3D map at each timestamp. In either case, VSLAM and / or other algorithms can be used to generate the global 3D maps and locate the AMRs on the global 3D maps. This can be done continuously at each timestamp or each frame of the video sequences, or it may be sufficient to create a map at an interval, such as once every 10 frames, depending on the frame rate. Through a specific process, the 3D maps can be converted into models, such as meshes, point clouds, and voxel grids, to name a few examples.

[0121] Procedure 800 may include “for initialization, providing monitoring assignment to AMR(s)” 814, and this may include “providing object for tracking” 816, as with the AMR system 121 ( Fig. 1B) shown. In this operation, the data center and PLCs 428, with or without user input, such as through the AMR control panel 414 or another plant system interface, can assign a product being assembled or another object to be monitored by one or more AMRs. This may initially involve the assignment to a base, tray, crane, or other moving structure or body that will hold or receive a product to be monitored. For payload AMRs, this operation 816 may also include instructions for a backup AMR to monitor objects initially assigned to a payload AMR while the payload AMR retrieves more payload. In this case, the system 400 may assign a backup target and a subsequent return target to a backup AMR as part of these instructions.

[0122] Otherwise, operation 814 “providing area for monitoring” 818 may feature, as shown with the AMR system 120, in which an AMR is first assigned a section of a production line for monitoring and does not necessarily move along the production line out of the assigned section unless the AMR has other tasks to perform, such as carrying a payload. In this case, a backup AMR may be used as described above. Many variations are considered. Procedure 800 then returns to operation 804 to perform continuous monitoring.

[0123] Method 800 can include “Identifying Actions Under Consideration” 820, and this can include triangulation by the image processing unit 522 to determine the positions of other objects, such as human operators, machines, devices, product subcomponents, tools, and so on, associated with a production line. The triangulation determines the positions of the objects by determining distances between the objects, AMRs, and any fixed anchor points, as above with anchor 318 ( Fig. 3) described. Object recognition algorithms can also be applied, such as VSLAM, to track all objects (AMRs and non-AMR objects and personnel) on the global 3D map over time.

[0124] The object recognition algorithms can then also be used in initial action recognition operations (if they are not already being performed) to identify the same objects moving over time on the global 3D map or model that are potentially separate actions under consideration. This is repeated for each potential separate action under consideration. Each action can be tagged or annotated. This can include semantic object recognition and the provision of tags and / or annotations for the recognized objects under consideration.

[0125] Once sequences of moving objects are identified as initial or potential objects of consideration within the image sequences, a considered operation recognition can be verified by comparing the moving object sequences with datasets representing operations in both desired (expected) and undesired (unexpected) states. Database 444 can contain a library of both such datasets, as well as a matching database of expected and undesired sequences for the same objects. Neural networks can be used to input the image data from the image sequences and are trained to compare the moving object sequences with the datasets. The neural networks can be trained using human feedback to continuously improve object recognition accuracy.

[0126] Procedure 800 can feature “receiving expected actions” 822, and this refers to retrieving these actions to determine action difference values ​​instead of the initial detection and verification of a considered object. It is understood that these can be considered concurrent operations, but for clarity, they are treated here as separate operations. Thus, the records can be retrieved from the same library and database 444 as those used to identify the considered actions in the first place.

[0127] Procedure 800 can include “Determining Action Differences” 824. Thus, as mentioned, this process refers either to (1) generating the differences between the expected actions and the actions under consideration, or (2) retrieving these differences if they were already determined during the initial identification of the actions under consideration. This can also include comparing the actions under consideration with undesired actions already contained in the historical undesired action record. If such a match occurs, the precise reason for the mismatch between the action under consideration and the expected action can be identified in the historical data.

[0128] An exemplary approach can be used to train any of the neural networks or other machine learning models mentioned here in order to evaluate the accuracy and efficiency of the networks and models, in order to improve the networks and models over time.

[0129] Procedure 800 can include “Reporting of Actions and Differences to Production Line Operating Systems” 826. In this example, the differences and associated data can be provided to PLCs 428 or another plant operating system. PLC 420 then decides how to respond to the difference between the considered and expected actions and the undesired actions. This may simply involve looking up previous responses if the undesired action was identified by a match with previous undesired actions that have already been experienced (or simulated) and added to a data set. Otherwise, PLC 428 may have programming to determine the appropriate response and may generate instructions or commands to execute the response.For example, the PLC 428 can form instructions to command a machine to stop, initiate an AMR, perform a task, or trigger the next step in production.

[0130] Otherwise, PLC 428 can signal the E-Stop Unit 424 to stop operations on the production line, whether by stopping the entire plant or specific AMRs or machines. PLC 428 can also issue warnings through the Warning Unit 426 and to personnel on the production line, at the plant, the plant systems center, the floor, or any other location that controls or monitors actions on the production line. If the discrepancies indicate a payload state on a payload AMR, then a PLC can send a signal to the Material Unit 422 (if not received directly from the Communication Unit 410) to initiate the retrieval of more payload, such as by signaling the Payload Unit 434 to control an AMR 500 to retrieve more payload.

[0131] Procedure 800 may include “Receiving automatic response instructions” 828, and in the event that a PLC 428 determines that the difference is an urgent matter or user involvement is not required, instructions may be sent to the AMR controller 404 to immediately change the operation of an AMR, or directly to other machines associated with the production line to immediately change the operation or actions with the other machines.

[0132] Otherwise, procedure 800, “Providing Actions and Differences for the AMR System Dashboard,” may include 830, where a user can provide inputs to determine the correct responses. This is typically used for non-urgent matters or after an urgent response has been implemented, and to evaluate the response and determine whether a different response should be implemented if the same undesired action under consideration occurs in the future. This may include PLC 428 outputting a prescribed script for a display to show a user or operator.Based on the data from the system, the PLC 428 can use ladder logic to drive a set of commands to a stack light warning system on machines or AMRs, software AMR dashboard 414, or other connected systems, so that an operator can easily understand whether further action is required based on the observations made by the cameras.

[0133] Method 800 may include “Receiving manual response instructions” 832, wherein a user or operator can provide instructions for a proper response, and including updating the programming code of the AMRs or other devices or systems, if desired.

[0134] Method 800 may include “Transmitting instructions to AMRs” 834 or other devices on or associated with the production line to implement the manual responses or responses with manual contributions.

[0135] The procedure 800 can feature “storing actions, differences and reactions in a historical archive” 836, wherein these elements are stored in a historical archive for tracking, and where relevant, considered actions and a match with an expected action can be added to the records of the database 444, for example, for training and operation of action-differentiating neural networks.

[0136] In this context, relational terms such as first, second, and the like may be used solely to distinguish one entity or action from another, without necessarily requiring or implying any actual relationship or order between such entities or actions. Numerical ordinals such as "first," "second," "third," etc., simply denote different individuals within a plurality and do not imply any order or sequence unless expressly defined by the claim language. The sequence of text in any claim does not imply that process steps must be performed in a temporal or logical order according to such a sequence unless expressly defined by the claim language.The process steps can be exchanged in any order without deviating from the scope of the invention, as long as such an exchange does not contradict the claim language and is not logically nonsensical.

[0137] Furthermore, depending on the context, words like "connected" or "coupled with," used to describe a relationship between different elements or parts of the nozzle, do not imply that a direct physical connection between these elements is necessary unless otherwise stated. For example, two elements can be physically, electronically, logically, or in any other way connected by one or more additional elements.

[0138] Although at least one exemplary embodiment has been presented in the foregoing detailed description, it is understood that a large number of variations exist. It is also understood that the exemplary embodiments are not intended to limit the scope, applicability, or configuration of the description in any way. Rather, the foregoing detailed description will provide the person skilled in the art with a suitable roadmap for implementing the exemplary embodiments. It is understood that various modifications to the function and arrangement of elements can be made without deviating from the scope of the description as set out in the appended claims and their legal equivalents.

Claims

A method comprising: operating, by at least one processor, at least one autonomous vehicle having multiple cameras and arranged to move near a production line arranged to supply a product, wherein the operation comprises: automatically moving the at least one autonomous vehicle near the production line while free from a predetermined fixed route; capturing, by the multiple cameras, at least one sequence of images of an area including at least a part of the production line; automatically detecting, by at least one processor, at least one change associated with the object and capturing at least one sequence of images; and providing instructions, by at least one processor, to perform a response in response to the detected change. The method of claim 1, wherein the object is a production line arranged to provide a product, wherein the change is a movement of an operation associated with the production line, and wherein the response involves reacting to considered actions of the detected operation, and wherein the response is associated with the production line. The method of claim 2, comprising operating the at least one autonomous vehicle to move along the production line in order to keep an object within a field of view of at least one of the cameras. The method of claim 2, comprising operating several autonomous vehicles, each capturing a different field of view of the production line, to track an object moving along the production line and passing through several of the fields of view. The method of claim 2, comprising arranging a first autonomous vehicle to be monitored by one or more cameras of a second autonomous vehicle, and wherein the instructions include modifications of one or more actions to be performed by the first autonomous vehicle depending on the actions under consideration that are detected by the one or more cameras of the second autonomous vehicle. The method of claim 2, comprising: carrying a loaded payload on the at least one autonomous vehicle; automatically detecting when at least part of the loaded payload is removed from the at least one autonomous vehicle or is used in conjunction with the production line or both, using the at least one sequence of images; and instructing the at least one autonomous vehicle to retrieve more payload when the detection indicates that at least part of the loaded payload has been removed. The method of claim 2, comprising generating a 3D map of the production line that changes over time; and determining and tracking the position and movement of the at least one autonomous vehicle on the 3D map using the at least one sequence of images; and using a visual simultaneous localization and mapping (VSLAM) algorithm to generate the 3D map of the production line and to track the movement of the at least one autonomous vehicle. The method of claim 2, wherein several autonomous vehicles provide several sequences of images and wherein the method comprises registering images together from different of the several autonomous vehicles to generate a 3D map of the production line. The method of claim 2, comprising: comparing data from images of predetermined desired expected actions or previously recorded undesired actions or both with the actions under consideration; and determining the instructions depending on the results of the comparison. System comprising: memory, processor circuit forming at least one processor communicatively coupled to the memory and arranged to operate by: operating at least one autonomous vehicle having multiple cameras and arranged to move near a production line arranged to supply a product, wherein the operation comprises: automatically moving the at least one autonomous vehicle near the production line while free from a predetermined fixed route, and capturing, by the multiple cameras, at least one sequence of images of an area including at least a part of the production line; automatically detecting at least one operation associated with the production line in which at least one sequence of images is captured;and providing instructions for carrying out a response in response to considered actions of the recognized processes, and wherein the response is assigned to the production line.

Images