System and method for comprehensive production line monitoring and rapid reaction implementation

Abstract

Systems and methods are provided for comprehensive production line monitoring and rapid reaction implementation. A vehicle, system, medium, and method including operating, by at least one processor, at least one autonomous vehicle having a plurality of cameras and disposed to move in proximity to a production line arranged to provide a product. The operation includes automatically moving the at least one autonomous vehicle in proximity to the production line without a predetermined fixed route and capturing, by the plurality of cameras, at least one sequence of images of an area including the production line. The method includes automatically recognizing, by the at least one processor, at least one operation associated with the production line and captured in the at least one sequence of images and providing, by the at least one processor, instructions to perform a reaction in response to a viewing action of the recognized operation. The reaction is associated with the production line.

Description

Technical Field

[0001] This disclosure relates to production lines for assembling products in a manufacturing plant, and more specifically to the automated monitoring, analysis, reporting, and adjustment of production line operations. Background Technology

[0002] Manufacturing plants monitor production lines or assembly lines to better ensure that product assembly is performed efficiently. This includes using camera array systems at each stage along the production line. Cameras are fixed and positioned, such as overhead, to avoid interfering with other moving objects (such as robots or human operators performing assembly along the production line), stockpiles of materials to be used during assembly, or paths and spaces for delivering payloads to moving vehicles (including autonomous mobile robots (AMPs) and / or automated guided vehicles (AGVs)) along the production line. Better camera positioning and greater integration between camera systems at different stages are needed to capture more informational production perspectives for more accurate efficiency analysis.

[0003] Therefore, it is desirable to provide systems and methods that enable more effective and efficient production line monitoring without significantly interfering with other production operations. Furthermore, other desirable features and characteristics of this disclosure will become apparent from the accompanying drawings and the foregoing introduction, based on the subsequent detailed description and appended claims. Summary of the Invention

[0004] In an example implementation, a method includes operating at least one autonomous vehicle via at least one processor. The at least one autonomous vehicle has multiple cameras and is configured to move near an object to be monitored. This vicinity is located in a three-dimensional space with overhead equipment to be avoided. The operation includes: automatically moving the at least one autonomous vehicle near the object without a predetermined fixed route; and capturing at least one image sequence via the multiple cameras of a region including at least a portion of the object. The method further includes: automatically identifying at least one change associated with the object and captured in the at least one image sequence via at least one processor; and providing instructions via at least one processor to perform a reaction in response to the identified change.

[0005] Similarly, in another example implementation, the objects are arranged as a production line to deliver products, and the changes are the movements of operations associated with the production line. Responses include responding to viewing actions of the identified operations.

[0006] Similarly, according to another example implementation, the method involves operating multiple autonomous vehicles, each capturing a different field of view of the production line to track objects traveling along the production line and passing through multiple fields of view.

[0007] Similarly, according to another example implementation, the method includes positioning a first autonomous vehicle to be monitored by one or more cameras of a second autonomous vehicle. Instructions include modifying one or more actions to be performed by the first autonomous vehicle based on viewing actions captured by the one or more cameras of the second autonomous vehicle.

[0008] According to another example implementation, the method includes: carrying a loaded payload on at least one autonomous vehicle, automatically detecting when at least a portion of the loaded payload is removed from at least one autonomous vehicle or used in connection with a production line, or both, by using at least one image sequence, and instructing at least one autonomous vehicle to retrieve more payload when the detection indicates that at least a portion of the loaded payload has been removed.

[0009] According to another example implementation, the method includes: generating a 3D map of the production line over time, and determining and tracking the position and movement of at least one autonomous vehicle on the 3D map using at least one image sequence.

[0010] Similarly, according to another example implementation, the method involves using a visual simultaneous localization and mapping (VSLAM) algorithm to generate a 3D map of the production line and track the movement of at least one autonomous vehicle.

[0011] Similarly, in another example implementation, multiple autonomous vehicles provide multiple image sequences. The method involves registering images from different autonomous vehicles among the multiple autonomous vehicles together to generate a 3D map of the production line.

[0012] Similarly, according to another example implementation, the method includes: comparing image data of a predetermined expected action or a previously recorded unexpected action, or both, with the viewing action, and determining an instruction based on the result of the comparison.

[0013] In another example implementation, a system includes a memory and processor circuitry forming at least one processor communicatively coupled to the memory and arranged to operate by: operating at least one autonomous vehicle having multiple cameras and configured to move near a production line arranged to deliver products. The operation includes: automatically moving at least one autonomous vehicle near the production line without a predetermined fixed route, and capturing at least one image sequence via the multiple cameras of an area including at least a portion of the production line. The at least one processor is arranged to operate by: automatically identifying at least one operation associated with the production line and captured in the at least one image sequence, and providing instructions to execute a response in response to a viewing action of the identified operation. This response is associated with the production line.

[0014] According to another example implementation, at least one processor is arranged to operate by: determining one or more differences between a viewing action and a previously stored image corresponding to an expected action or an undesired action, or both, using at least one image sequence; notifying a user of the difference on a display device remote from at least one autonomous vehicle; and receiving instructions from the user to modify the planned movement of at least one autonomous vehicle based on the difference.

[0015] In another example implementation, at least one processor is arranged to operate by generating real-time 3D maps and changing the data of the production line and the positions of multiple autonomous vehicles over time, using at least one image sequence.

[0016] According to another example implementation, at least one processor is arranged to operate by comparing images of viewed actions with images of expected or undesired actions or both to determine action differences, and generating content for a response based on those differences, including providing all alternative options for: (1) providing a signal to at least one autonomous vehicle in the autonomous vehicle, (2) providing a signal to at least one non-AMR device near the production line, and (3) providing data to notify a person of the action difference.

[0017] In another example implementation, at least one autonomous vehicle has a body connected to a mobile mechanism, a base, and a payload area arranged above the base. The base includes sidewalls disposed below the payload area and has multiple cameras, each facing outward from the base.

[0018] Similarly, in another example implementation, multiple cameras are positioned on a single autonomous vehicle to capture images horizontally in a 360-degree radius around that vehicle.

[0019] In another example implementation, at least one non-transitory computer-readable medium includes instructions thereon that, when executed by a computing device, cause the computing device to operate in such a way as to operate at least one autonomous vehicle having multiple cameras and configured to move near a production line arranged to deliver products. The operation includes: automatically moving at least one autonomous vehicle near the production line without a predetermined fixed route; and capturing at least one image sequence via the multiple cameras of an area including at least a portion of the production line; automatically identifying at least one operation associated with the production line and captured in the at least one image sequence; and providing instructions to perform a response in response to a viewing action of the identified operation. This response is associated with the production line.

[0020] Similarly, according to another example implementation, the instructions cause the computing device to operate in such a way that one of the autonomous vehicles moves along the production line so that one or more objects moving on the production line remain in the view of multiple cameras on that one autonomous vehicle.

[0021] Similarly, according to another example implementation, the instructions cause the computing device to operate in such a way that multiple autonomous vehicles move along a production line, each of which follows a different object moving along the production line and captures images of the different objects moving along the production line.

[0022] According to another example implementation, at least one autonomous vehicle is arranged to monitor objects associated with the production line, including at least one of the following: human operators, other autonomous vehicles in at least one autonomous vehicle, objects moving in association with the production line, objects added to products, tools used to assemble products, objects moving from or to at least one autonomous vehicle, and the movement of the robot.

[0023] Similarly, according to another example implementation, at least one autonomous vehicle is arranged to track objects associated with at least one of the following production line stages: material preparation, product component manufacturing, product part sub-assembly, product assembly, product quality control testing, product surface treatment, product packaging, product storage, and product transportation. Attached Figure Description

[0024] The present disclosure will be described below in conjunction with the accompanying drawings. The drawings are not drawn to scale, and the reference numerals in the drawings denote similar elements, and in the drawings:

[0025] Figure 1A This is a schematic top view of an example autonomous mobile robot (AMR) system set up at a manufacturing plant according to at least one implementation of this paper;

[0026] Figure 1B This is a schematic top view of another example autonomous mobile robot (AMR) system set up at a manufacturing plant according to at least one implementation of this paper;

[0027] Figure 2 This is a schematic perspective view of an example autonomous mobile robot (AMR) according to at least one implementation of this paper, which monitors object operations on a production line.

[0028] Figure 3 This is a schematic perspective view of an example autonomous mobile robot (AMR) located along a production line according to at least one implementation of this paper.

[0029] Figure 4This is a schematic diagram of an example manufacturing plant system based on at least one implementation of this paper;

[0030] Figure 5 This is a schematic diagram of an example autonomous mobile robot (AMR) according to at least one implementation of this paper;

[0031] Figure 6 Figure 1 to Figure 5 A schematic diagram of an example image processing unit of an autonomous mobile robot (AMR) or factory system according to at least one implementation thereof;

[0032] Figure 7 This is a flowchart illustrating an example method of an autonomous mobile robot (AMR) based on at least one implementation of this paper; and

[0033] Figure 8 This is a flowchart of an example method for an autonomous mobile robot (AMR) system based on at least one implementation of this paper. Detailed Implementation

[0034] The following detailed description presents only exemplary implementations and is not intended to limit this disclosure or its application and use. Furthermore, it is not intended to be bound by the foregoing background technology or any theories presented in the following detailed description.

[0035] The systems and methods disclosed herein provide highly adaptive integrated autonomous vehicles, such as autonomous mobile robots (AMRs), each equipped with sensor and camera arrays. The AMRs monitor any one or more stages of a production line from start to finish to track operations along the line and / or trace any part of, for example, the construction of a single product, while using a single system to analyze image data from multiple AMRs along the production line. Operation of the AMR system is accomplished despite any obstacles along the production line, and the movement of the AMRs is not restricted to maintaining a fixed, predetermined path or route during AMR operation.

[0036] The use of AMRs enables dynamic camera array positioning and, by allowing the AMR's sensors and cameras to "follow" and synchronize with the production line, facilitates vision-based dynamic data analysis and real-time, location-based data analysis. This analysis can then be leveraged through AMR system dashboards or other plant systems, such as programmable logic circuits (PLCs), to improve production line efficiency, send digital inputs to the plant shop system, and trigger responses to captured images in real-time or near real-time. This can include using a single AMR that captures images over time, which can be registered (or stitched together), or using multiple AMRs along the production line that can simultaneously provide images to be stitched together. In either case, a 3D map (or digital or virtual model) of the production line can be generated, and the AMRs can be positioned on this map. Further image analysis can be used to identify production line operations and determine modifications to reduce latency and improve efficiency and performance.

[0037] In some forms, the AMR carries payloads along the production line, including parts to be added to products being assembled and / or tools to be used by operators or robots on the production line. Therefore, the disclosed vision-based AMR system with a mobile AMR can reduce or eliminate reliance on other vision monitoring systems because the payload is carried by the AMR itself, without occupying space that the AMR would otherwise need to avoid. This also allows for the "timely" placement of parts and materials along the production line while increasing the coverage of the camera array.

[0038] Such AMRs can provide real-time location reporting, optional line tracking, and maneuverability, while the vision-based data analytics solution presented in this paper includes real-time location tracking, task time analysis, visual error-proofing, and payload element counting. Combining AMRs with this type of vision-based analytics also allows for real-time repositioning (or location refinement) of the vision system on the AMR to better ensure consistent visual coverage along the entire production line or at desired locations. The AMR system also further integrates the collected data inputs with in-plant data systems to enable real-time manufacturing improvements. Data inputs can be used and analyzed to achieve real-time production line rebalancing, trend analysis, parts ordering, and AMR reallocation.

[0039] As described below, the camera array can be placed on the base or platform of the AMR, where the payload is positioned above the base. With this arrangement, overhead objects (such as cranes, robotic arms, etc.) will not obstruct the camera's field of view.

[0040] Similarly, this approach to operating AMR systems allows manufacturing plant personnel to analyze the integration and interactions of different systems across different sections of a single production line, further enabling plant personnel to modify the production line system to improve overall performance. Furthermore, AMR systems can perform real-time autonomous decisions for production line rebalancing, queue management, and material delivery, thereby improving manufacturing plant efficiency. AMR systems also enable manufacturing plant personnel to directly record human operator actions along the production line, providing an unbiased and unobstructed view of the real interactions between people and machines and the work being performed. AMR systems can also provide plant personnel with data analytics, breaking down value-added and non-value-added workloads at each location, allowing the plant to modify operations at monitored locations to increase value-added workloads and overall performance. AMR systems also enable the simultaneous use of AMR technology and vision-based data analytics solutions within a single package.

[0041] Now refer to Figure 1A Manufacturing plant 100 may include factory system 101 having production line 102 and products 104 and 106, which are assembled along production line 102 as they (or some sub-components of the products) move along the production line as indicated by arrows. Products 104 and 106 are not limited to any particular product, and for example, are automobiles. Human operators 114 and 116 may perform tasks associated with production line 102 and may include performing tasks of assembling products 104 and 106.

[0042] Depending on the products being assembled, production line 102 can perform many different stages of the manufacturing process. This can include any stage of the manufacturing process, including material preparation, raw material handling, material cutting and shaping (including any machining or molding, etc.), component manufacturing, sub-component manufacturing, parts assembly, final product assembly, intermediate and / or final quality control, inspection and testing, surface treatment, finishing or coating (such as painting, powder coating, anodizing, or other surface treatments), drying, curing, final or fine finishing of the product, labeling operations (including adding logos, stamps, or decals), performance testing, packaging, storage, and shipping. Therefore, any single stage or any combination of stages can form production line 102, and there are no particular limitations. For example, one or more parts of production line 102 can be within a building (such as a manufacturing plant or factory), particularly a building with overhead equipment. It is generally difficult (if not impossible) to manipulate overhead monitoring robots to avoid overhead equipment.

[0043] Depending on the product, any suitable equipment (including conveyor belts, cranes, trolleys, handcarts or other vehicles, etc.) can be used to move the assembled products 104 and 106 along production line 102 from one stage to another.

[0044] The factory system 101 may also include an AMR system 120, which has one or more autonomous vehicles, such as AMR 108, including AMRs 110, 112, and 118. The AMR system 120 is particularly well-suited for operation under overhead equipment within a building or manufacturing plant, but can also be used outdoors or in other environments via other alternatives. In one form, the AMR system can be used to monitor many different objects within a building or manufacturing plant, such as infrastructure (building condition), other systems (fluid delivery systems), or indoor manufacturing plant traffic (such as indoor manufacturing plant intersections), to monitor pedestrians, operators, and moving equipment. In one example form, any object within a building, manufacturing plant, factory, or other area (such as a three-dimensional space with overhead equipment to be avoided) can be monitored, provided the autonomous vehicle is free to automatically set its own route to perform monitoring. In another form, the autonomous vehicle travels on the ground via wheels or another means of movement in contact with the ground.

[0045] More specifically, each AMR 108 may have one or more cameras 130, which form a local or AMR camera array 132 on each AMR 108. Here, at least eight cameras 130 are provided on each AMR 108, with two cameras provided on each of the four sides of the AMR 108, but any desired number of cameras and camera arrangement can be used. Collectively, the camera arrays 132 on all AMRs 108 cooperate to form a global or AMR system camera array 140. The cameras 130 are pointed outwards, such that each AMR camera array 132 establishes a converged (or AMR or local) field of view 109, 111, or 113. In one form, each AMR field of view 109, 111, and 113 is 360 degrees horizontally and, depending on the camera and camera position, can be 180 degrees vertically. In one configuration, downward-facing and / or upward-facing cameras (not shown) may also be located on each or each individual AMR 108 to establish a 180-degree or 360-degree vertical range of fields of view 109, 111, and 113 when relevant.

[0046] As illustrated in this example, AMR 108 can be positioned along the production line 102 and can remain relatively stationary to monitor a designated area of ​​the production line, or it can travel along the production line or any other desired path to or away from the production line. In the first case, AMR 108 can be assigned to a specific task or operation performed on the production line, and it does not move along production line 102 as sub-assembly products 104 and 106 move along the production line. In this case, fields of view 109, 111, and 113 overlap each other, allowing airborne camera 130 to maintain a view of continuous production line operation as products or other objects travel through fields of view 109, 111, and 113.

[0047] Then, using camera array 140, AMR 108 or local camera array 132 can be assigned to monitor specific operations or tasks, thereby more precisely monitoring specific observable actions, including object movement. Unless the context is explicit or explains that factory or manufacturing plant personnel are excluded, the term "object" as used herein can include human operators. The monitored operation may be referred to herein as an observable action or actual action, rather than the predetermined expected action described below. Here, AMR 108 can monitor observable or observable actions in any of the aforementioned production stages or any other production stages, and may include operators following a specified work plan, operators installing parts, delivering new parts, identifying safety issues, following operational sequences, use of personal protective equipment (PPE), tool use, task timing, etc. More specifically, such camera monitoring can include at least two main categories: human operator (or personnel) actions and autonomous robot operations. AMR system 120 can monitor personnel tasks, such as loading materials onto conveyors, assembling parts, performing quality checks on products, or adjusting machine settings, to monitor appropriate ergonomic practices, avoid bottlenecks, and comply with safety protocols (such as maintaining a safe distance from moving parts or operating machinery). Additionally, the AMR system 120 can monitor proper processing during assembly and provide real-time prompts for corrective actions. It should be noted that, as used herein, the term "real-time" includes near real-time and refers to time as perceived by humans, and may have processing lag times of less than one second or up to several seconds.

[0048] In addition, the AMR system 120 can monitor autonomous robot movements, such as picking and placing parts, welding, packaging, or palletizing, to name a few. The AMR system 120 can monitor these robot movements to ensure the robot functions correctly, where movements such as arm extension, gripping actions, and material transfer are tracked to ensure proper part placement, avoid collisions, or achieve timeliness. Monitoring robot movements also allows for predictive maintenance, as cameras can detect signs of wear or irregular movement that indicate the need for maintenance before a failure occurs.

[0049] Throughout production line 102, the AMR system 120 can also monitor various stages, such as material handling, the movement of raw materials from the warehouse to the assembly area, checking for inventory issues, or identifying equipment malfunctions. During assembly, the camera array 132 can better capture when parts are correctly aligned and when workers follow assembly instructions to prevent defects. In the final stage, the AMR system 120 can verify the quality of finished products before packaging and shipping, and monitor ongoing quality control and testing operations, ensuring that only products meeting required standards proceed to the next stage, ultimately contributing to improved product quality, faster production times, and improved workplace safety. Many other operational examples monitored by the AMR system 120 can also be used.

[0050] As another example alternative, AMR 108 can move along with the objects or products 104 and 106 at production line 102 (as indicated by the dashed arrows). In this example case, each AMR 108 can be assigned to a specific product 104 to monitor the assembly of that specific product. In this case, each AMR 108 can be assigned to a different product or a different part of a product, with one AMR assigned to capture the front of the product and another AMR assigned to the back of the product. Many variations are envisioned. In this case, a single AMR 108 can monitor a range of different operational and viewing actions that vary as the AMR 108 moves along production line 102, depending on the stage of production line 102.

[0051] Reference Figure 1B In another example arrangement, factory system 103 is similar to factory system 101 and has similar elements numbered the same, which need not be described again. However, in this example, AMR system 121 shows a single AMR 150 tracking product 104 or other objects or operators as product 104 moves along production line 102. As described above, AMR 150 may have a camera (not shown) to monitor the sequence of operations performed on product 104. Furthermore, in this case, AMR 150 can autonomously change its linear path to path 154 to avoid obstacles 152 along the initial path. Once AMR 150 detects obstacle 152, path adjustment can occur relatively quickly. Once the obstacle is overcome, AMR 150 can resume the linear portion of path 154 to continue tracking product 104.

[0052] Reference Figure 2The factory system 200 has a production line 202, which is monitored by an AMR system 220 similar to AMR system 120 to display example camera positions and demonstrate tracking of human operators (or factory personnel). Therefore, in this example, AMR 204 has eight cameras 206, each with a field of view (FOV), shown in dashed lines and overlapped to form a continuous 360-degree total AMR FOV around AMR 204. AMR 204 may have a body 222 with sidewalls 208 on which the cameras are mounted, or windows on the sidewalls 208. Like AMR 108, each different side of the sidewalls 208 (front, back, right, left) has two cameras 206, but many other configurations can also be used alternatively.

[0053] Alternatively, a standalone AMR can have a camera mount, which can be an adjustable mount or a robotic arm, to optimally position the camera for maximum field of view coverage, especially when the AMR 500 is not carrying a payload, and where camera positioning is possible if there are fewer constraints.

[0054] Also as part of this example, human operator 212 can move object 214, whether it is a sub-component of a product being assembled, a tool, or other object. Operator 212 and object 214 are within the field of view 210 of one of the cameras 206 on AMR 204, so that object 214 can be identified and recognized by factory system 200 (using...). Figure 4 (Detailed explanation). The movement of operator 212 and object 214 is recorded over time, allowing for analysis of the operator's execution of object movements.

[0055] Reference Figure 3 In another example configuration, factory system 300 has production line 302, which is monitored by AMR system 320, wherein AMR 308 is also used to transport payload 314 along the production line of production line 302. Factory system 300 has many [unclear text - possibly related to...] Figures 1A to 2The factory system in this example has the same characteristics and does not need to be described again. In this case, production line 302 produces product 304 (as an example, such as a car), and human operator 306 is operating on product 304. The AMR system 320 has an AMR 308, which has eight cameras 310 on its sidewall 312, similar to example cameras 130 and 206, and each camera has a field of view (FOV) as described above. In this example, a fixed object 318 (here, a factory column or other object) can be used as an anchor point for constructing a 3D map of the production line using the cameras 310, as well as for positioning the AMR 308 and detecting and identifying operator 306. In this case, the AMR 308 has an upper surface 316 located below the payload 314 carried by the AMR 308. The upper surface 316 may or may not be in direct contact with the payload 314 and may have many different shapes and mechanisms to receive, hold, and provide elements of the payload 316. Many variations exist, and the AMR systems 120, 121 and 320 described herein are not limited to any particular arrangement for holding or carrying the payload 314.

[0056] Reference Figure 4 The factory system 400 according to at least one implementation thereof may include any production line as described above and may have an AMR system 401, which may include AMR1 to N (as shown in Figures 110, 112 to 402) located in the factory or manufacturing plant workshop and is similar to or the same as the AMR systems 120, 220 and 320 described above. The AMR system 401 may alternatively be considered separate from the factory system 400, since many units of the factory system 400 (such as the control center) may be located remotely from the AMRs in the manufacturing plant or factory workshop.

[0057] The AMR system 401 may have an AMR control 404, which includes at least an image processing unit 406, a motion difference unit 408, a motion unit 430, a camera unit 432, and an optional payload unit 434. The AMR dashboard unit 414 may or may not be considered part of the AMR control 404, and may be considered part of other systems on the plant system 400. The remaining parts of the plant system 400, possibly remote from the AMR, may include at least one of a communication unit 410, a plant data center (or unit) 412, a display device 416, and / or a user interface 418. The plant system 400 may also have a plant operating system 420, which may include at least a materials unit 422, an emergency stop unit 424, an alarm unit 426, other programmable logic circuits (PLCs), or a unit 428 (which operates various systems within the plant and on the plant shop floor and controls machinery and computing equipment, or has mechanisms for initiating reports to personnel managing or performing plant shop operations). It should be understood that the term "factory or manufacturing plant workshop" in this article refers to any location used for operations associated with the production line.

[0058] The factory system 400 may also include one or more processors 440 and memory 442 to operate any unit or system of the factory system 400, and the processors 440 and memory 442 may be considered as part of any unit of the factory system 400 described herein, including any programmable logic controller (PLC) 428. One or more of the processors 440 and memory 442 may be provided for any of these units or systems located away from any other unit or system of the factory system 400.

[0059] More specifically now, processor 440 is provided with computational and control functions for executing any unit and system of factory system 400 and AMR system 401 (including AMR control 404), and may be referred to as control, controller, factory system, computing device, computer, etc. Processor 440 may have circuitry that may be part of or form part of AMR control 404. Processor 440 may include one or more circuits forming any type of processor or multiple processors, including a central processing unit (CPU), digital signal processor (DSP), a single integrated circuit (such as a microprocessor), or any suitable number of integrated circuit devices and / or circuit boards working together to implement the functions of the processing unit. This may include a system-on-a-chip (SoC) and one or more processor cores. Processor 440 may also include image processing technologies, including a graphics processing unit (GPU), image signal processor (ISP), application-specific integrated circuit (AISC), field-programmable gate array (FPGA), neural processing unit (NPU), vision processor (VP), video processing unit (VPU), and deep learning accelerator (DLA). These processors may be shared or dedicated hardware. In addition, dedicated or function-specific processors may be provided for operating neural networks, machine learning, and other structures for image processing, such as those with a graphics processing unit (GPU) or an image signal processor (ISP). During operation, processor 440 executes any unit or system of factory system 400, and the units and systems of factory system 400 may be any combination of hardware, firmware, and / or software. The software portion of a unit or system may be stored on memory 442 and thus controls the general operation of factory system 400 and AMR system 401 when executing the factory system processes described herein, such as those shown in Figures 1 to 12. Figure 3 and Figures 7 to 8 The process and implementation of any of them, as well as the process and implementation further described below in conjunction with them.

[0060] Memory 442 refers to memory in a general sense and can be any storage device, and can include any type of suitable memory. For example, memory 442 can include various types of dynamic random access memory (DRAM), such as SDRAM, various types of static RAM (SRAM), and cache, while memory 442 can also include various types of non-volatile memory (PROM, EPROM, and flash memory). In some examples, memory 442 is located on and / or co-located with processor 440 on the same computer chip. In the depicted implementation, memory 442 stores the aforementioned factory system and AMR system units, as well as one or more databases, to store mapping and image processing data and other stored values ​​156. Additionally, memory 442 can include various different types of direct access memory and / or other memory devices. In one example implementation, memory 442 includes a program product (or unit or system) from which memory 442 can receive a program that executes. Figure 8 This includes one or more implementations of the process and implementation methods further described below in conjunction with the figures. In another example implementation, the program product may be directly stored in memory 442 and / or accessed by auxiliary storage devices (e.g., disks). During operation, the program and accompanying data are stored in memory 442, and the program is executed by processor 440.

[0061] Therefore, by way of example, memory 442 may store data from motion difference unit 408 as well as motion datasets from motion database 444, including both expected motions and corresponding unintended previously captured motions (also referred to as the history or dataset of “accidental” motions) associated with any production line described herein. Processor 440 may be used to operate motion difference unit 408 to determine the difference between the viewed motion and the expected motion (and / or the similarity between the viewed motion and the unintended (or accidental) motion). Such differences and similarities may be compared to image processing thresholds determined experimentally, such as pixel distance or comparisons of the sum of absolute errors (SAD) type.

[0062] Factory system 101 may have hardware that forms or supports processor 440, memory 442, and any other units or systems on factory system 400, and this hardware may include at least one bus to transfer programs (e.g., units or systems), data, status, and other information or signals between various components of factory system 400. The bus may be any suitable physical or logical arrangement for connecting computer systems and components. This includes, but is not limited to, direct hardwired connections, fiber optic, infrared, and wireless bus technologies.

[0063] It should be understood that although this example implementation is described in the context of a full-featured computer system, those skilled in the art will recognize that the apparatus of this disclosure can be distributed as a program product having one or more types of non-transitory computer-readable signal-bearing media for storing the program and its instructions and for performing its distribution, such as a non-transitory computer-readable medium carrying the program and containing computer instructions stored therein for causing a computer processor (such as processor 440) to execute and implement the program. Such a program product can take many forms in memory 442, and this disclosure applies equally regardless of the specific type of computer-readable signal-bearing medium used for performing the distribution. Examples of signal-bearing media forming memory 442 include recordable media such as floppy disks, hard disks, memory cards, and optical disks, and transmission media such as digital and analog communication links. It should be understood that cloud-based storage and / or other technologies may also be utilized in some implementations. As described above, it should also be understood that the processor 440 of the factory system 400 may include various processors located in various locations, which perform various functions for the factory system 400. For example, the processor 440 may be coupled to one or more remote computer systems and / or other control systems or may otherwise utilize these systems. Therefore, this includes each unit and system of the factory system 400 having its own processor as part of the processor 440.

[0064] Example communication unit 410 may include a transceiver and antenna for remote communication with the AMR, other remote systems, servers, devices, modules, or units. It should be noted that any part or component (or unit) of factory system 101 can handle any of the operations described herein related to the AMR production line, and can remotely perform monitoring, analysis of captured images, and implementation of responses when needed. By way of examples, communication unit 410 may receive input (such as signals or packets of image data, and AMR and object location data) via application programming interfaces (APIs), message queue telemetry transport (MQTT) for real-time tasks, representational state transitions (REST), and / or via wireless communications (such as Wi-Fi, Bluetooth, radio frequency (RF), near field communication (NFC), ultra-wideband (UWB), etc.) to a Common Industrial Protocol (CIP) bridge to the factory floor system. Therefore, communication unit 410 can communicate via a network including one or more computer networks (such as the Internet) and communication networks (e.g., satellite-based, cellular, and / or any number of other different types of wireless communication networks). In one form, the communication unit 410 uses a network address with a Uniform Resource Locator (URL) as an API to collect transmitted data (such as images or signals from an AMR) over the Internet.

[0065] One or more displays 416 (such as computer monitors, smartphones, tablets, etc.) can be used to display AMR monitoring data and images from the AMR dashboard unit 414 to a user, to notify the user of significant motion discrepancies and related alarms, and to receive (1) input from the user, i.e., whether to initiate an AMR or other action first, and / or (2) as a response or reaction to motion discrepancies, and to modify the AMR or other object action by communicating with a PLC or other unit. In one form, the display 416 can be any display capable of providing a screen on a computer, smartphone, tablet, or other computing device to view images on the display 416. Such a display 416 can be a digital display, a graphical user interface (GUI), an LED display, a plasma display, an LCD display, an organic light-emitting diode (OLED) display, a thin-film transistor (TFT) display, a head-up display (HUD), a 3D display, a holographic display, a virtual or augmented display, etc. The usage details of the display 416 are explained below.

[0066] As described above, user interface 418 allows communication between the user and other plant systems and AMR system units. Therefore, interface 418 may include a touchscreen or any other display on display device 416, keyboard and / or mouse operation, audio system and / or other suitable interface devices and architecture. User interface 418 may be mobile relative to other units of plant system 400 and may include a smartphone or tablet held by or with personnel on-site (or on the plant floor or production line).

[0067] The factory data center (or unit) 412 can manage or have a memory 442 and / or a motion database 444 to store AMR data and provide inputs received from the AMR and differences from the motion difference unit 408 to the PLC 428 to determine what response to implement based on the differences. Additionally, the factory data center 412 can transmit AMR report data to a display device 416 and / or a user interface 418 for display. The factory data center 412 (whether alone or in conjunction with a PLC or other units) can also determine production line rebalancing, trend analysis, parts ordering, AMR reallocation, etc., based on motion differences. The factory data center 412 can also manage historical archives stored in the memory 442 of the factory operations, including a storage of viewed actions and corresponding expected actions, undesirable actions, motion differences, and the final responses implemented as a result.

[0068] In addition, factory data center 412 can also serve as a technology hub and supervisor, collecting, storing, and analyzing data generated by manufacturing operations. It can integrate data from sensors, AMRs, other machines, and control systems to monitor real-time performance, track production metrics, and optimize processes, which may or may not be related to AMRs. Data center 412 can support predictive maintenance to help prevent equipment failures, automate processes to improve efficiency, manage energy usage, inventory, and supply chain data, and ensure security and regulatory compliance.

[0069] AMR control 404 receives at least image data from image sequences of each camera and AMR position data, but optionally may also receive local AMR maps or 3D maps. In one form, individual AMRs may also have their own AMR controls to generate motion differences within a global scope (including all AMR cameras) or within a local AMR scope of each AMR. In these cases, motion differences may also be provided to a central AMR control 404. Many variations are possible.

[0070] In this example, the AMR does not have its own motion difference capability and is focused on the AMR control 404 located far from the AMR, but as mentioned above, other methods can also be used. Here, input is received via communication unit 410. Then, image processing unit 406 analyzes various images by registering images from different cameras and forming a 3D model or graph of the production line. By way of an example, different local AMR 3D graphs from multiple AMRs are stitched together, or multiple local 3D graphs from a single AMR that is moved to capture different viewpoints. By way of an example, this is performed by using feature matching algorithms such as Visual Simultaneous Localization and Mapping (VSLAM) and / or other algorithms described below. Then, same or other object recognition algorithms can be used to identify operations and movements in the image sequence to generate the identified viewing actions for comparison with predetermined expected actions and / or predetermined undesired actions in the action database 444. Thereafter, motion difference unit 408 determines and reports the difference between the viewing action and the expected action (and the similarity between the viewing action and the undesired action (in use)). The following is through process 800 ( Figure 8 Provide details of the process.

[0071] Motion unit 430 provides motion commands to the AMR and modifies these commands as needed due to differences in motion. Therefore, motion unit 430 can provide a wide range of commands for the AMR to travel to a certain position, while the AMR itself can determine the exact path to that position. When differences in motion indicate that the AMR must follow a certain path, motion unit 430 can control the AMR path generation mechanism. Many variations are envisioned.

[0072] Camera unit 432 can control the operation of all cameras on the AMR by monitoring when the camera is activated or deactivated, thereby ensuring that the camera is guided to the desired direction, focus level, and other camera settings. Camera unit 432 can also provide modifications and updates to image quality, etc.

[0073] When at least one of the AMRs is carrying a payload, an optional payload unit 434 is provided. The payload unit 434 can monitor the payload amount on one or more AMRs and provide instructions for moving the AMR to a payload loading dock or area to retrieve more payload and move it to the desired production line location on the production line.

[0074] For example, the AMR dashboard unit 414 can display motion differences and other data related to AMR performance on the display device 416. The AMR dashboard can also be used to provide lists and views of various available reports, create custom reports requested by the user, display historical data, or provide other reports for the user to view. The AMR dashboard unit 414 can also receive input from the user requesting information or providing instructions to generate a response to motion differences, and this may include updating the programming of various AMRs or plant systems through the AMR dashboard unit 414, but alternatively, other interfaces may be used.

[0075] The factory operating system 420 manages various systems that operate the production line and other areas of the manufacturing plant. For example, the PLC 428 can receive data associated with motion discrepancies and then determine what the appropriate response is, generate instructions for the AMR or other equipment implementing the response, and initiate reports associated with motion discrepancies, including providing data or instructions to other units of the factory operating system 420, as well as user-viewable reports.

[0076] Additionally, in some forms, an example PLC 428 can be provided to handle inputs from sensors, switches, and other devices, control outputs such as motors, actuators, and valves, and manage various operations. This can include sequential control (such as controlling different stages in a production line), process control (such as monitoring and regulating variables such as temperature, pressure, or flow), motion control (coordinating the movement of machines or robotic arms), safety systems (ensuring emergency shutdowns or safety interlocks), material handling (such as picking and placing operations using AMRs, conveyor systems, or robots), and communication (integration with other systems, such as Supervisory Control and Data Acquisition (SCADA) or Manufacturing Execution Systems (MES), for plant-wide data exchange).

[0077] Material unit 422 can receive instructions from PLC 428 to retrieve more material, and then instruct payload unit 434 to control AMR to retrieve more material, and can provide retrieval instructions to another device or person on the production line (or material supply location). Many variations exist.

[0078] Emergency stop unit 424 can receive instructions from PLC 428 to stop all or part of the production line operation, and then implement the instructions by sending signals to the equipment or computing system controlling the production line. Many variations exist.

[0079] Alarm unit 426 can receive instructions from PLC 428 to provide alarms to users and personnel working on or otherwise associated with the production line, alerting them to significant motion discrepancies (or all motion discrepancies). Data from such alarms can be provided to AMR dashboard unit 414 via factory data center 412 to display motion discrepancies (or anomalies) and other relevant data on display device 416 to the user. The following is explained via process 800 (… Figure 8 Provides additional details on the operation of the plant system.

[0080] Reference Figure 5 Example autonomous vehicles (such as autonomous mobile robots (AMRs) 500) are identical or similar to the AMRs in the aforementioned AMR systems 120, 220, and 401. In one example implementation, the AMR 500 may have a controller 501, a camera 504, wheels 526 or other transport mechanisms, a motor 528, a steering mechanism 530, sensors 520, a local user input device 516 (which may have a display device or screen 517 (or additionally or alternatively, speakers and / or microphones of an audio system)), and a data storage unit 518 (which may store programs or code for any unit of the AMR 500). By some alternatives, the data storage unit 518 may also store programs or code for an image processing unit 522 and / or any other unit described herein, which is used to analyze image data from the camera 504 of any AMR 500, and is a complement or alternative to the unit that analyzes image data at the factory system 400. Therefore, such an image processing unit 522 can be located on at least one AMR, some AMRs, or not on any of them, and can analyze only image data from the camera array of its own AMR, or can receive images from other AMRs in a global camera array that includes all AMRs. Additionally, the data storage unit 518 can have any of the hardware structures described above for the memory 442.

[0081] In one example form, the AMR may have wheels 526 for movement, but it may also have any other suitable mechanical transport equipment or system, such as skateboards, rollers, fan propellers, whether for vertical rotation of a fan boat or horizontal rotation of a drone or helicopter, etc. Wheels 526 may be differential-drive, omnidirectional, or all-terrain type wheels, providing mobility without being limited by tracks. Wheels 526 may be operatively connected to axles or other mechanisms connected to one or more motors 528 (including both propulsion and steering motors). Motors 528 may be electric, fuel-powered, or other types of motors. A steering mechanism 530 may also be connected to wheels 526 to steer one or more wheels, these wheels having steering mechanisms, such as tie rods and steering racks, which cause one or more wheels 526 to rotate to control the direction of movement of the AMR 500. Many other configurations and arrangements can be used as long as the AMR can autonomously steer and drive.

[0082] Sensor unit 520 may include the sensors themselves and any processing for collecting sensor data and providing it in a desired format. This may include sensors such as light detection and ranging (LIDAR), other cameras, ultrasonic sensors, and infrared sensors, which help the robot perceive its environment and detect obstacles. Other sensors that may be provided include detection sensors (e.g., radar, sonar, etc.) and / or other sensors (e.g., vehicle position sensors, velocity sensors, accelerometers, gyroscopes, inertial measurement unit (IMU) sensors, braking sensors, steering sensors, etc.). In various implementations, sensor 520 obtains additional information about the production line environment and / or the operation of the AMR 500 itself (e.g., its position, velocity, deceleration, and / or acceleration, etc.) for use, for example, in operating the AMR 500 autonomously based on the operation of the AMR 500 and / or some of its components.

[0083] Through various implementations, cameras 504 used to acquire images of observational or perceptual data of a production line or its surrounding environment may include front, rear, side, top, bottom, and / or panoramic cameras on the AMR 500, including wide-angle, 360-degree, and / or fisheye lens cameras, as well as monocular, stereo, infrared, time-of-flight, thermal, LiDAR cameras, high-resolution cameras (e.g., RGB, depth, thermal cameras), etc. These cameras 504 can capture images, which are then processed as described herein. In various implementations, video camera images are acquired. Additionally or alternatively, still camera images may be acquired.

[0084] The local user input device (or interface) 516 and display device 517 are as described above in the case of user interface 418 and display device 416. Interface 516 can be used to view output data from AMR 500 or input data from AMR 500 to determine the status of AMR 500 or otherwise control any function of AMR 500.

[0085] In one example implementation, the AMR controller 501 has one or more processors 502, a power supply unit 506, a mobility unit 508, a navigation unit 510, an optional local payload unit 512, a communication unit 514, and an AMR local control unit 524.

[0086] Processor 502 can be formed by processor circuitry that operates or forms any unit or system on AMR 500. Therefore, processor 502 is derived from the aforementioned processor 440 ( Figure 4 An alternative architecture is formed. Similarly, communication unit 514 is similar to communication unit 410 of factory system 400 and may include a transceiver and antenna to transmit AMR image data and any AMR-related data generated by local image processing unit 522 or sensor 520, such as position and orientation data, as well as any other AMR data, including status data, performance data, etc. Communication unit 514 also receives any instructions, requests, or other data to control the AMR.

[0087] The power supply unit 506 can control and / or power the AMR 500 to, and can be any suitable power source, including electrical power, whether it is a battery, AC or DC, and more specifically, a lithium-ion battery. Some AMRs can use any one or more of lead-acid batteries, fuel cells, hydrogen fuel cells, and supercapacitors, with or without other batteries. Furthermore, the AMR 500 can be wirelessly charged or inductively charged.

[0088] The mobility unit 508 controls the motor 528, which in turn controls the steering mechanism 530 and wheels 526 (or other transport mechanisms) in use.

[0089] The AMR local control (or unit) 524 controls the actions of the AMR, including receiving and analyzing instructions and commands, whether from its own onboard protocols or from received messages or signals instructing or commanding the AMR to perform certain tasks. In one form, and in various implementations, the AMR local control 524 is coupled to the sensor unit 520, as well as to the mobility unit 508, navigation unit 510, and payload unit 512 (when provided). In various implementations, the AMR local control 524 is also coupled to the data storage unit 518, the display device 517, and the communication unit 514. Instructions may include software protocols for data transmission and coordination with other systems, as well as managing image processing performed by the image processing unit 522 or the navigation unit 510 for self-localization on a production line map, moving the AMR to a target destination according to protocols or received instructions, while avoiding obstacles, etc. Therefore, the AMR local control unit 524 can operate an operating system and algorithms for managing tasks such as path planning, decision-making, and obstacle avoidance, including the operation of the image processing unit 522. The AMR local control unit 524 can also control the operation and settings of the camera array on the AMR, as well as the transmission of image data and position data to the factory system 400 during generation.

[0090] When not dependent on image processing unit 522 or factory system 400 ( Figure 4 When the image processing unit 406 is in operation, the navigation unit 510 may perform local image processing and / or sensor data processing as instructed by the AMR control unit 524 to generate an AMR-level local 3D map. In one configuration, the navigation unit 510 focuses on forming the 3D map and localization, while the image processing unit 522 concentrates on locating and identifying other objects and operations captured by the camera 504 to provide locally defined viewing actions from the AMR 500, rather than having the remote image processing unit 406 determine the viewing actions. Furthermore, it should be understood that the plant system 400 may delegate image processing tasks to the image processing unit 522, the navigation unit 510, and the image processing unit 406 of the plant system 400, and to whichever task assignment is deemed most efficient, timely, or optimal in performance. Additionally, and alternatively, 2D maps may be used instead of 3D maps when necessary.

[0091] When the AMR 500 is carrying a payload, the optional payload unit 512 can control the payload carrying operation. This may include receiving data on the element count of the payload being used and captured in an image by one of the image processing units 522 or 406; determining a target timing or point in time to retrieve more payload; and initiating a retrieval in coordination with the AMR local control unit 524 to move the AMR 500 as needed to retrieve the payload without interrupting monitoring of the production line. Further details on the operation of the AMR 500 are provided below through process 700.

[0092] It should be understood that, in one example embodiment, all units of the AMR controller 501 and data storage unit 518 may be mounted on or within the body of each or each individual AMR 500. Alternatively, any part or component (or unit) of the controller 501 and data storage 518 may be remotely executed when needed, performing any processing related to AMR monitoring as described herein. Specifically, in various implementations, the controller 501 is disposed within the body of the AMR 500. In some implementations, the controller 501 and / or AMR local controls 524 and / or one or more of their components may be disposed within the body of the AMR 500 (such as AMR 204 ( Figure 2 The main body 222) is external, for example, on a remote server, in the cloud, or in another device that performs image processing remotely. It should be understood that controller 501 and / or AMR local control 524 can otherwise interact with... Figure 5 The implementations described differ. For example, controller 501 may be coupled to or otherwise utilize one or more remote computer systems and / or other control systems, such as as part of one or more devices and systems in the aforementioned AMR 500.

[0093] Reference Figure 6 Image processing unit 600 may be the same as or similar to image processing unit 406 or 522. Image processing unit 600 may include a separate registration or stitching unit 602, which is used when registration itself is not performed by mapping unit 604. Image processing unit 600 may also have triangulation unit 606 and motion recognition / verification unit 608. One or more of these units may be considered separate from the image processing unit, or may be considered as AMR 500 ( Figure 5 ) or factory system 400 ( Figure 4 (a part of different units)

[0094] In one example, the image processing unit 522 and AMR local control 524 on the AMR 500 perform initial operations to register images from different cameras to form an AMR or local 3D map, locate the AMR, and perform object recognition to avoid obstacles. Meanwhile, the image processor 406 on the AMR control 404 at the factory system level 400 can perform object recognition and operation recognition to identify viewing actions. In one case, the AMR transmits input with image sequences and AMR location data to the factory system level, although a local 3D map is also sent. Action differences are then determined by the action difference unit 408 located away from the AMR. Alternatively, these tasks can be shared or balanced between the two image processing units 406 and 522 in other ways, such as where feasible, and to improve efficiency and performance, and reduce latency for real-time analysis, thereby enabling rapid response on the production line as needed or desired. Therefore, in the second alternative, the AMR performs more onboard image processing, where one or more individual AMRs enable AMR control 404 to generate a global 3D map, a database of the locations of multiple AMRs, and enable motion difference unit 408 to identify and view motions before sending them to plant system level 400. In the third option, the AMR performs minimal onboard image processing, where remote AMR control 404 can receive image sequences from the AMR camera and perform image processing to generate any 3D map used for locating the AMR. Any desired combination or variation of these can be used.

[0095] Now in more detail, the image registration unit 602, when separated from the mapping unit 604 and located on a separate AMR, receives images from the AMR's own onboard camera array, although it may also receive images from other AMRs to have images from the global camera array of all or multiple AMRs used on or associated with the same production line. The image registration unit 602 then performs registration or stitching, which can be performed using a variety of different feature detection and feature matching algorithms. Such techniques may include Scale Invariant Feature Transform (SIFT), Speeded Robust Feature Transform (SURF), Oriented Fast and Rotationally Brief (ORB), and Kernel Aligned Zero Mean Expansion (AKAZE or Speeded KAZE) to identify unique points in the image that are invariant to scale, rotation, and in some cases, affine transformations. Other feature matching techniques, such as Approximate Nearest Neighbor Fast Library (FLANN) or BRUTEFORCE (Simple Exhaustive Search), can be used to find correspondences between feature descriptors. Additionally, geometric verification methods, such as RANSAC (Random Sample Consensus), can refine the matching by rejecting outliers and ensuring robust matching between image pairs.

[0096] Then, when using a single AMR camera, mapping unit 604 forms a 3D map of at least a portion of the production line. When using images from multiple AMRs, this may cover the entire production line or at least the portion of the production line covered by a global camera array. Image registration and mapping can be performed using Visual Simultaneous Localization and Mapping (VSLAM) or alternatively, SLAM alone, and this may include the use of one or more of the registration techniques described above.

[0097] VSLAM performs simultaneous mapping, localization, and tracking of the AMR over time. Therefore, VSLAM can continuously update both the AMR's position and map in real time as the AMR moves. This can be done using only visual data (captured images), but can also be augmented using sensor data to confirm and refine the AMR's position and other objects as needed. Mapping involves identifying and tracking key features (such as corners, edges, or textures) in the visual data, and then using these features to construct a 3D map of the space. This is continuously updated to track the AMR's movement in real time. Localization also determines the AMR's orientation within the map.

[0098] Therefore, in one form, VSLAM can perform feature detection, feature matching, camera motion estimation (to determine the camera's position and orientation (or pose) relative to the environment), graph construction (which can be 3D graph construction or modeling), and localization (where mapping and localization can be verified, filled in, and refined using other sensors such as LiDAR and / or GPS for more accurate localization and mapping). Furthermore, VSLAM can perform feedback correction, which includes correction for "scale drift" over time, where the estimated map gradually increases or decreases in size due to errors in depth estimation. The AMR processor 502 (and / or 440) may include a graphics SoC with GPU and deep learning capabilities to operate neural networks for processing the image data described above.

[0099] Alternatives to VSLAM, or algorithms that can complement or be part of VLSAM, can be any combination of the aforementioned techniques with feature registration, or any combination of object detection and recognition algorithms, such as algorithms based on any one or more of the following: machine learning, neural networks, convolutional neural networks (CNN), region-based convolutional neural networks (R-CNN), regressive neural networks (RNN), masked R-CNN, look-only once (YOLO), single-trigger multi-box detector (SSD), semantic segmentation such as fully convolutional networks (FSG) and U-networks (e.g., Haar cascade (Viola-Jones (VJ) detector)), histogram of oriented gradients (HOG), MOG (Gaussian mixture) background subtraction, template matching, DPM (deformable part model), GMM (Gaussian mixture model) background subtraction, LDA (linear discriminant analysis), and / or many other algorithms.

[0100] Once localization and 3D mapping are established, the triangulation unit 606 can be used to detect, identify, and recognize other objects associated with or on the production line. Techniques such as Kalman filters, optical flow, and others mentioned herein can be used to track moving objects and maintain their identity over time. Therefore, if VSLAM has not yet performed non-AMR object detection and identification operations during continuous 3D mapping and AMR localization operations, the triangulation unit 606 can also include or establish such non-AMR object detection and identification operations. As described above, in an alternative, non-AMR triangulation and object identification can be performed entirely by the image processing unit 406 while generating a global 3D map by registering images from all or more cameras on all or more AMRs. Alternatively, this can be performed entirely or partially by the image processing unit 522 on one or more AMRs. The object detection and identification techniques described above can be used, and then triangulation can be applied to track the position of objects, where the objects can be any object within the camera's field of view, whether it is a product being assembled, a robot or machine performing assembly or other tasks associated with the production line, or a human operator. This process is executed continuously to track the movement of objects captured in a global 3D map over time, resulting in an extremely accurate and precise model of the objects. In one form, digital or virtual production lines can be generated based on 3D mapping and object tracking.

[0101] Image processing units 406 and / or 522 may then include an action recognition and verification unit 608. At this point, each time step (or stamp) of AMR monitoring using camera images can have a 3D map, forming a series of 3D maps to capture production line operations within a series of map data. The identified object sequence is then analyzed using the various object recognition techniques described above to determine individual viewing actions. This recognition is then verified by comparing the viewing actions with expected actions from database 444 or other databases to identify individual operations, such as the movement of a single robot or a single human operator. Verification is performed by inputting the viewing actions into a trained neural network that compares the viewing actions with expected actions. When this initial comparison is sufficiently close, the identified operation can be labeled or annotated.

[0102] The following sections in processes 700 and 800 describe the follow-up actions performed at plant system level 400 or AMR, including comparison of observed actions and expected actions (as well as undesirable actions), reaction generation, report generation, and history archiving.

[0103] Reference Figure 7 According to at least one implementation described herein, a process 700 for operating an autonomous vehicle (such as an autonomous mobile robot (AMR)) used in the examples herein is provided. Process 700 is described using operations 702-734, which are typically evenly numbered. Where relevant, references can be made to Figures 1 to 734. Figure 6 Systems, processes, vehicles, equipment, vehicle displays, and components.

[0104] Process 700 may include “activating the AMR system” 702. In one form, the AMR system (such as 120, 220, and 401) is automatically activated once power is supplied to the AMR 500 and the unit of the AMR system 401 is started. This may involve initializing the AMR hardware (such as motors, sensors, cameras, etc.) and calibrating the cameras and sensors. The AMR 500 can begin from an initial position, such as at the base or charging location, or otherwise from the payload loading dock location or any other desired location associated with providing the AMR 500 for production line operation.

[0105] Once the AMR 500 and AMR system 401 are activated and running, the AMR 500 can perform continuous monitoring, such as by using a high-speed camera providing 30-120 frames per second (fps), but also 1-30 fps, 15-30 fps, 30 fps, 60 fps, or other desired frame rates. Preprocessing of the raw image data (such as demosaicing, noise reduction, scaling, etc.) can be performed at the camera itself or by the ASR's image processing unit 522.

[0106] At AMR 500, process 700 may include a “detection area map” 704, and as described above, may include the use of VSLAM or other techniques, providing images in real time in one form. The result may be a 2D or 3D map of the production line area, which can be used to form a 3D model of the production line. VSLAM or other algorithms may be executed continuously to generate a map with each frame, or to generate maps at desired intervals (for example, such as once every 10 frames), generating maps at each desired mapping time interval. The 3D map can then be used to track changes in the production line over time. As described above, each AMR 500 may generate its own local AMR-level 3D map, or a global map may be generated at each AMR or one or more AMRs when image data is shared between AMRs.

[0107] Process 700 may include “Locating the AMR on the map” 706, where VSLAM also identifies the AMR and simultaneously tracks its movement over time. Camera-based vision mapping and localization may include using features and objects in the production line environment as fixed anchor locations, such as objects or posts 318 ( Figure 3 ), to anchor the map and determine the AMR position relative to the anchor.

[0108] Process 700 may optionally include "retrieving the payload" 708, and occurs when one or more AMRs 500 are to carry a payload. In this case, the payload is set to be loaded onto the AMR 500. The AMR 500 uses mapping and positioning from VLSAM to generate a path to the payload loading dock or area, or the AMR can determine that it is already at the target payload loading location. Payload loading can be performed automatically by autonomous equipment on or detached from the AMR 500, such as clamps, cranes, conveyors, and / or autonomous forklifts, but may also be performed manually by a human operator using equipment such as trolleys or manual forklifts. Many different examples exist.

[0109] Subsequently, regardless of whether a payload is being carried, process 700 may include "receiving location instructions" 710, where the AMR can perform autonomous path planning to a target destination at or around the production line, monitor the production line, and deliver the payload to the production line upon allocation. For example, the destination instructions can be retrieved from an onboard protocol or received from other units, such as those in the plant system 400. Planning can be performed before, during, or after the AMR 500 is located at the payload dock.

[0110] In one form, operation 710 may include "receiving / determining the initial production line position" 712, and this is provided when the AMR 500 typically maintains at least a single position on the production line along with other AMRs, allowing each AMR to be assigned a different production line segment or area for monitoring, as in AMR system 120 (Figure 1). The AMR 500 can still move back and forth from this position to acquire payloads or perform other tasks, such as to achieve a better view of objects or operations performed, for example, within the assigned production line segment. Such field-of-view adjustments of the AMR can be performed autonomously or manually, and relatively quickly when needed.

[0111] Additionally, operation 710 may include "receiving operation / object assignment" 714, where the AMR 500 is not assigned to a specific section of the production line, but rather to a product being assembled or other object moving along the production line. In this case, the target destination is the AMR's initial position. Once positioned on the production line, the AMR will continuously identify or recognize the assigned object or product and move along the production line, while maintaining the assigned object and product within the local field of view of the local camera array on the AMR. In this case, a single AMR can be used instead of multiple AMRs, or multiple AMRs can be used in parallel, each assigned a moving object to be monitored.

[0112] Other variations can be used, such as assigning objects or products to AMRs only within a certain section of the production line, or moving all AMRs along the production line in a fixed order, regardless of the presence or location of objects and products on the production line. Many other variations exist.

[0113] Process 700 may include “planning a path from the current location to the destination” 715. Then, the latest 3D (or other) maps and positioning data can be used to generate a path from the AMR’s current (or other important and relevant) location to the target destination. Path generation may take into account actual and potential obstacles and traffic on the map, such as other AMRs, objects, and human operators detected on the map, other mapped or known obstacles or paths, such as porches, ramps, elevators, lifts, and other factors such as time consumption, on-time scheduling, speed limits, energy consumption, wear and tear on physical components of the AMR, safety protocols, prohibited AMR travel zones, other designated zones, mission priorities and distances, payload size when overloading the payload expands the clearance required for the AMR, physical impact sensitivity to the payload or other equipment on the AMR (such as bumps during AMR travel), environmental conditions that may affect computer vision during travel (lighting, temperature, etc.), temporary or maintenance obstacles, backup or auxiliary paths (if needed), and multi-AMR path planning sharing and cooperation to avoid paths generated by other AMRs. Many other factors not listed here may also be considered.

[0114] Process 700 may include “traveling to an initial position along the production line” 716, after which the AMR 500 travels along a self-generated path, taking into account the factors described above, without any predetermined fixed route that the AMR cannot deviate from. The AMR can use VSLAM and other technologies described above, along with sensor data, to adjust its position to maintain heading and refine its localization. The AMR travels to the indicated destination, which may in this case be along the production line. The process of autonomously generating a path between the current position and the destination can be repeated as needed, and can use any desired destination provided to the AMR, whether along the production line or other destinations associated with the production line, typically within or above the manufacturing plant or factory site, campus, or building. Alternatively, the AMR may move to other production lines (multiple production lines) or for other reasons as needed.

[0115] In one form, navigation unit 510 can perform path planning, and AMR local control 524 can communicate with AMR control 404 and other plant operating system 420 or users via AMR dashboard 414 to coordinate actions (e.g., sending commands to pause or reroute the AMR). AMR local control 524 can also adjust the AMR's path based on real-time data received from cameras that update any of the above factors.

[0116] When the destination is to deliver a payload to or along a production line, additional operations can be performed autonomously or manually, such as verifying the precise docking or unloading point location and orientation of the AMR to properly unload the payload. This can be done using the AMR's camera, a camera that can see the delivering AMR, and / or other onboard sensors (e.g., proximity sensors, etc.).

[0117] Process 700 may include "monitoring production line operations" 718 and "performing ongoing camera capture" 720, which refers to AMR camera 504 performing continuous image or video capture as described above, while maintaining visibility on specific segments of the production line or on specific products, objects, or operations, also as described above. The captured image sequences (or video sequences) from each camera are then stored in storage 518 for analysis by AMR local controls 524, navigation unit 510, and / or payload unit 512 to determine when a new payload should be retrieved.

[0118] Optionally, process 700 may include "moving along the production line with the assigned object" 722, and path generation as described above is performed by the navigation unit 510 and AMR local controls 524. This may include route monitoring and obstacle avoidance by continuously scanning the environment around the AMR using a camera, lidar, or other sensors described above to detect obstacles in the planned path. The path may be dynamically adjusted, such as when obstacles are detected, and the consistency of the VSLAM 3D map may be checked by verifying that the AMR is correctly positioned (e.g., considering drift correction). The 3D map can then be updated in real time as needed.

[0119] Alternatively, process 700 may optionally include “performing local image processing” 724, where one or more AMRs collect image sequences or AMR-level 3D maps and positioning data from multiple AMRs 500 to generate a global 3D map. When the capacity of the AMR onboard system is available, AMR local controls and other units may also perform operational monitoring of AMR controls 404, including identifying observed operations, comparing them to expected and undesirable operations, and generating discrepancies that can then be provided to the plant system level or analyzed locally to determine responses in response to the discrepancies, such as changing the path or orientation of the AMR. Thus, a single AMR 500 can itself have the capability to generate reports on performance, efficiency, and anomalies in the production process. In one form, local user input devices 516 and display devices 517 can then provide an onboard AMR dashboard to display real-time data, trends, and alarms related to the assembly process, and then receive input from plant floor users or human operators.

[0120] Additionally, process 700 may include "transferring image data to AMR control" 726, where the AMR provides camera input to factory system 400 via onboard communication 514. This camera input takes the form of captured images and the timestamped location of the AMR. In this example case, the input may include the AMR location, and a sequence of captured images or a generated AMR-level 3D map, or both, which is then transferred from AMR 500 to remote factory system 400, specifically AMR control 404. This may include the transfer of raw sensor data as well as analyzed and calculated AMR and / or production line metrics, such as assembly speed, product count, error rate, etc. Such analysis can be performed using image processing unit 522 and AMR local control unit 524.

[0121] Continuing this example, after transmitting camera input and other AMR data to remote factory system 400, process 700 may include a "receive response instruction" 728, wherein an instruction is received at AMR 500 to change the operation of one or more AMRs 500 if a sufficiently significant difference is determined between the observed action and the expected action (or the similarity between the observed action and the undesired action). Difference analysis will be discussed below through process 800 and at the factory system level 400. Here, the received instruction may include changing the planned path of the AMR or changing the location or orientation of the AMR, changing the object, production, or operation assigned to the AMR, or any other action related to the AMR (including emergency stop of the AMR and other moving objects associated with the production line). The instruction may include signals sent to other computing devices, such as robots assembling parts onto products or other machinery related to other stages of assembly or forming and delivering finished products. In another form, instructions may include providing alarms or other data for human operators or personnel to read or hear, whether on a computing device that the operator can view on a monitor, or on machinery or equipment on or associated with a production line, including a monitor or audio speaker on the AMR itself, and this may include printed output on paper or other media.

[0122] In one of the above forms, instructions can be generated because the action of one AMR is observed by another AMR. As an example, one AMR may observe that a different payload AMR is positioned too far from the production line, making it additional time for a human operator to travel to and from the payload AMR to manually unload the payload. Instructions can command the payload AMR to move closer to the production line and the location where the human operator is working, reducing delays and improving efficiency.

[0123] Alternatively, when the payload AMR is carrying a payload, operation 728 may include a "receive payload command" 730 when it is detected that the payload has been fully or sufficiently unloaded and more payload should be retrieved. This may be triggered locally by the AMR's own camera alone, or via commands from the factory system level and via observation from images obtained from another AMR viewing the payload AMR. The command may be a simple code, i.e., to retrieve more payload.

[0124] Process 700 may include "execute action according to instruction" 732, whereby the AMR may change and execute its path, position, orientation, target, task, etc., according to the instruction. This operation 732 may include "obtain the next payload" 734 for the payload AMR, which then determines the path back to the payload loading dock, leaves the production line (or other) current position, and returns to the payload loading dock along the autonomously generated path.

[0125] Now refer to Figure 8 According to at least one implementation described herein, a process 800 for operating the autonomous mobile robot (AMR) system used in the examples herein is provided. Process 800 is described using operations 802-836, which are typically evenly numbered. Where relevant, references can be made to Figures 1 to 836. Figure 7 Systems, processes, vehicles, equipment, displays, and components.

[0126] Process 800 may include “activating the AMR system” 802, and this includes some of the same operations as operation 702 of process 704 of AMR. Here, if it is not already in “always on” mode, the AMR system unit at plant system 400, as well as other systems and units, may be started.

[0127] Process 800 may include "receiving image data from one or more AMRs" 804, and this may include receiving at least some version of the input described together with transmission operation 726, which includes at least an image sequence and timestamped locations of the AMRs, and optionally includes 3D maps generated by the AMRs, sensor data, and calculated AMR and production line performance parameters, such as production line assembly speed, accuracy, etc.

[0128] Process 800 may include “preprocessing image data” 806, and this may include any noise reduction, any other quality enhancement, scaling, etc., so that the received image is in the format expected by image processing unit 406, which is performed to perform viewing action recognition and action difference determination.

[0129] Process 800 may include “generating a 3D map of the production line and locating the AMR position over time” 808, and in one form, this may include “using VSLAM” 810, and may also be performed on the plant system side or at the plant system level. Alternatively or additionally, algorithms other than VSLAM may be used. Operation 808 may include “registering images from multiple cameras and / or multiple AMRs to each other” 812. Thus, when local 3D maps of the AMRs are received from the AMRs at image processing unit 406, these maps can be stitched together to form a global 3D map or model of the entire production line or a portion of the production line covered by all AMRs monitoring the production line. Alternatively, when the AMRs transmit only individual image or video sequences from the respective cameras, image processing unit 406 may stitch the individual video sequences together to form a global 3D map. This can be performed by stitching together all those images with the same or substantially the same timestamp to form a 3D map at each timestamp. In either of these cases, VSLAM and / or other algorithms may be used to generate the global 3D map and locate the AMRs on the 3D global map. This can be done continuously at each timestamp or frame of a video sequence, or, depending on the frame rate, forming the graph at regular intervals (such as once every 10 frames) may suffice. In one form, the 3D graph can be converted into models such as meshes, point clouds, and voxel meshes, to name just a few.

[0130] Process 800 may include "for initialization, providing monitoring assignment to the AMR" 814, and this may include "providing the object to be tracked" 816, such as via AMR system 121 ( Figure 1B As shown in the diagram. In this operation, the data center and PLC 428 can assign products being assembled or other objects to be monitored by one or more AMRs, with or without user involvement (such as through AMR dashboard 414 or other plant system interfaces). This may initially include assignment to a base, pallet, crane, or other moving structure or body to accommodate or receive the products to be monitored. For payload AMRs, operation 816 may also include instructions to cause a substitute AMR to monitor the objects initially assigned to the payload AMR when the payload AMR retrieves more payload. In this case, as part of these instructions, system 400 may assign alternative destinations and subsequent return destinations to the substitute AMR.

[0131] Additionally, operation 814 may include “providing an area to be monitored” 818, as shown by AMR system 120, where the AMR is first assigned a production line segment to be monitored and does not necessarily move out of the assigned segment along the production line unless the AMR has other tasks to perform, such as transporting a payload. In this case, an alternative AMR may be used as described above. Many variations are envisioned. The process 800 then loops back to operation 804 to perform continuous monitoring.

[0132] Process 800 may include “identifying viewing action” 820, and this may involve triangulation by image processing unit 522 to determine the location of other objects, such as human operators, machines, equipment, product sub-components, tools, etc., associated with the production line. Triangulation is performed by determining the location of objects, AMRs, and any fixed anchors (such as anchor 318 above). Figure 3 The distance between the objects is used to determine their location. Object recognition algorithms, such as VSLAM, can also be applied to track all objects (AMR and non-AMR objects and people) on a global 3D map over time.

[0133] Then, an object identification algorithm (if not already executed) can be used in the initial action identification operation to identify the same objects moving over time on the global 3D graph or model, which may be individual view actions. This operation is repeated for each potential individual view action. Each action can be labeled or annotated. This can include semantic object identification and providing labels and / or annotations for the identified view objects.

[0134] Once a sequence of moving objects is detected in an image sequence as an initial or potential viewing object, the viewing operation identification can be verified by comparing the moving object sequence with a dataset representing operations in expected and unexpected states. Database 444 can be a library containing both datasets, as well as the correspondence between expected and unexpected sequences of the same object. A neural network can be used as input to the image data from the image sequence and trained to compare the moving object sequence with the dataset. The neural network can be trained using human feedback to continuously improve the accuracy of object identification.

[0135] Process 800 may include “obtaining the expected actions” 822, and this refers to obtaining these actions to determine action difference values, rather than the initial identification and verification of the viewed object. It should be understood that these operations can be considered simultaneous, but for clarity, they are treated here as separate operations. Therefore, the dataset can be obtained from the same libraries and databases 444 initially used to identify the viewed actions.

[0136] Process 800 may include “determining action differences” 824. Thus, as described above, this operation refers to either: (1) generating differences between the expected action and the viewed action, or (2) retrieving these differences if they were already determined when the viewed action was first identified. This may also include comparing the viewed action with unwanted actions already present in a historical unwanted action dataset. When such a match occurs, the exact cause of the mismatch between the viewed action and the expected action can be identified in the historical data.

[0137] An example method can be used to train any neural network or other machine learning model described in this article to evaluate the accuracy and efficiency of the network and model, thereby improving the network and model over time.

[0138] Process 800 may include “reporting actions and discrepancies to the production line operating system” 826. In this example, the discrepancy and related data may be provided to PLC 428 or another plant operating system. PLC 420 then decides how to react to the discrepancy between the viewed actions and the expected and undesired actions. This may only involve looking up previous reactions when identifying undesired actions by matching them with previous undesired actions that have been experienced (or simulated) and added to the dataset. Alternatively, PLC 428 may be programmed to determine the appropriate reaction and may generate instructions or commands to execute that reaction. For example, PLC 428 may generate instructions to tell the machine to stop, have the AMR perform a task, or trigger the next step in production.

[0139] Additionally, PLC 428 can signal emergency stop unit 424 to halt production line operation, whether by stopping the entire plant or a specific AMR or machine. Furthermore, PLC 428 can issue alarms via alarm unit 426 to personnel at the production line, plant, plant system center, or plant system level, or at other locations controlling or monitoring actions on the production line. When a discrepancy indicates a change in the payload status on the payload AMR, PLC can send a signal to material unit 422 (if not directly received from communication unit 410) to initiate the retrieval of additional payload, such as by signaling payload unit 434 to control AMR 500 to retrieve more payload.

[0140] Process 800 may include "receiving automatic response instruction" 828, and if PLC 428 determines that the difference is an urgent matter or does not require user intervention, it may send an instruction to AMR control 404 to immediately change the operation of AMR, or directly send an instruction to other machinery associated with the production line to immediately change the operation or action of other machinery.

[0141] Additionally, process 800 may include "Providing Actions and Differences to the AMR System Dashboard" 830, where the user can provide input to determine the correct response. This is typically used for non-emergency situations or after an emergency response has been implemented, to evaluate the response and determine whether a different response should be implemented in the event of the same undesirable action in the future. This may include having the PLC 428 issue a pre-written script to cause a display to be shown to the user or operator. Based on the system data, the PLC 428 can use ladder logic to drive a set of commands to the stack light alarm system on the machine or AMR, the software AMR dashboard 414, or other connected systems, allowing the operator to easily understand whether further action is needed based on observations taken by a camera.

[0142] Process 800 may include "receiving manual response instructions" 832, whereby a user or operator may provide instructions for the correct response, and may include updating the programming code of the AMR or other devices or systems when necessary.

[0143] Process 800 may include “transmitting instructions to the AMR” 834, or transmitting instructions to other equipment on or associated with the production line to implement a manual response or a response with manual contribution.

[0144] Process 800 may include "storing actions, differences, and reactions in a history archive" 836, wherein these elements are stored in the history archive for tracking, and where applicable, the viewed actions and their correspondence with expected actions may be added to the dataset of database 444 for training and operation, for example, an action discrimination neural network.

[0145] Here, relational terms (such as first and second, etc.) may be used only to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Numerical ordinal numbers (such as “first,” “second,” and “third,” etc.) simply indicate distinct individuals within a plural and do not imply any order or sequence unless specifically defined by the language of the claims. The textual order in any claim does not imply that process steps must be performed in a chronological or logical order according to such a sequence, unless expressly specified by the language of the claims. Process steps may be interchanged in any order without departing from the scope of the invention, provided that such interchange does not contradict the language of the claims and is not logically meaningless.

[0146] Furthermore, depending on the context, unless otherwise stated, the use of terms such as “connected” or “coupled to” when describing the relationship between different elements or parts of a nozzle does not imply that there must be a direct physical connection between these elements. For example, two elements may be connected to each other physically, electronically, logically, or in any other way by one or more additional elements.

[0147] Although at least one example implementation has been presented in the foregoing detailed description, it should be understood that numerous variations exist. It should also be understood that the example implementations are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing the example implementations. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. A method, the method comprising: At least one autonomous vehicle is operated by at least one processor, the at least one autonomous vehicle having multiple cameras and being configured to move near a production line arranged to deliver products, wherein the operation includes: The at least one autonomous vehicle automatically moves near the production line without a predetermined fixed route; and At least one image sequence is captured by the plurality of cameras, comprising at least a portion of the production line. Automatically identify at least one change associated with an object and captured in the at least one image sequence by at least one processor; and The processor provides instructions to execute the response in response to the identified change.

2. The method of claim 1, wherein the object is a production line arranged to provide products, wherein the change is a movement of an operation associated with the production line, and wherein the response includes responding to a viewing action of the identified operation, and wherein the response is associated with the production line.

3. The method of claim 2, the method comprising operating the at least one autonomous vehicle to move along the production line to maintain an object within the field of view of at least one of the cameras.

4. The method of claim 2, the method comprising operating a plurality of autonomous vehicles, each of the plurality of autonomous vehicles capturing different fields of view of the production line to track objects traveling along the production line and passing through multiple fields of view in the fields of view.

5. The method of claim 2, wherein the method comprises positioning a first autonomous vehicle to be monitored by one or more cameras of a second autonomous vehicle, and The instructions include modifying one or more actions to be performed by the first autonomous vehicle based on the viewing actions captured by the one or more cameras of the second autonomous vehicle.

6. The method according to claim 2, wherein the method comprises: The effective payload is carried on the at least one autonomous vehicle; The system automatically detects when at least a portion of the loaded payload is removed from the at least one autonomous vehicle or used in association with the production line, or both, by using the at least one image sequence; and when the detection indicates that at least a portion of the loaded payload has been removed, the system instructs the at least one autonomous vehicle to retrieve more payload.

7. The method according to claim 2, wherein the method comprises: Generate a 3D graph of the production line as it changes over time; And by using the at least one image sequence to determine and track the position and movement of the at least one autonomous vehicle on the 3D map; The system also uses a Visual Simultaneous Localization and Mapping (VSLAM) algorithm to generate the 3D map of the production line and to track the motion of the at least one autonomous vehicle.

8. The method of claim 2, wherein the plurality of autonomous vehicles provide a plurality of image sequences, and wherein the method includes registering images from different autonomous vehicles among the plurality of autonomous vehicles together to generate a 3D map of the production line.

9. The method according to claim 2, wherein the method comprises: Compare the image data of the predetermined expected action or the previously recorded unexpected action, or both, with the viewed action; And determine the instruction based on the result of the comparison.

10. A system comprising: memory, A processor circuit, the processor circuit forming at least one processor, the at least one processor being communicatively coupled to the memory and arranged to operate in such a way as: Operating at least one autonomous vehicle, the at least one autonomous vehicle having multiple cameras and configured to move near a production line arranged to deliver products, wherein the operation includes: The at least one autonomous vehicle automatically moves near the production line without a predetermined fixed route; and At least one image sequence is captured by the plurality of cameras, comprising at least a portion of the production line. Automatically identify at least one operation associated with the production line and captured in the at least one image sequence; and In response to a viewing action of the identified operation, instructions are provided to execute a reaction, wherein the reaction is associated with the production line.

Images