Vision-based manufacturing monitoring and event analysis

Abstract

The present disclosure provides "vision-based manufacturing monitoring and event analysis." A manufacturing monitoring system includes distributed vision units positioned at manufacturing stations to capture visual data representative of station operations. Each vision unit generates time-referenced event records that include media data describing observed activity and contextual metadata. A central processing system receives and aggregates event records to maintain a representation of manufacturing line operations. An artificial intelligence agent analyzes the aggregated event data relative to a learned reference operational context and provides information associated with events to a user interface. In some implementations, the system associates event records with individual workpieces and can transmit control commands that interrupt operation of manufacturing equipment when observed activity deviates from the reference operational context.

Description

Cross-references to related applications

[0001] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 781,507, filed April 1, 2025, the disclosure of which is hereby incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to a manufacturing operation monitoring system. More specifically, the present invention relates to an artificial intelligence (AI)-based factory operation monitoring system that uses a camera-based AI vision system to provide real-time visibility, event logging, anomaly detection, and performance measurement. Background Technology

[0003] Manufacturers across multiple industries encounter obstacles when attempting to deploy real-time factory monitoring solutions, primarily due to the complexity and rigidity of conventional platforms such as Factory Information Systems (FIS), Supervisory Control and Data Acquisition (SCADA) systems, and Manufacturing Execution Systems (MES). Deploying these platforms typically requires significant investment in programmable logic controllers (PLCs), dedicated sensors, industrial computing hardware, network infrastructure, proprietary software licenses, and specialized system integration services.

[0004] These burdens disproportionately impact small and medium-sized manufacturers (SMMs), for whom comprehensive real-time operational visibility is often impractical or unattainable. Without timely monitoring and analysis, SMMs' ability to detect production bottlenecks, optimize cycle times, or accurately identify the root causes of quality and yield issues is limited.

[0005] Furthermore, traditional monitoring solutions typically rely on dedicated hardware configurations and ongoing maintenance, which are difficult to scale or adapt across heterogeneous manufacturing lines. As modern manufacturing environments evolve towards greater variability and shorter production cycles, manufacturers increasingly require flexible monitoring systems that can provide actionable insights with minimal disruption to existing workflows and infrastructure. Summary of the Invention

[0006] In some implementations, the manufacturing monitoring system includes one or more vision units positioned along the manufacturing line at corresponding manufacturing stations. Each vision unit includes a camera, a processor, a data storage device, and a communication interface. The vision units capture visual data representing operational activities occurring at the manufacturing stations. In some implementations, the vision units perform local processing on the captured data and generate time-referenced event records associated with the observed station activities.

[0007] The generated event logs may include media data and contextual metadata describing the working conditions associated with the observed activity. Event logs may include, for example, timestamps, identifiers of the visual units that captured the event, and information describing the operational context associated with the observed workstation.

[0008] A central processing system (CPU) can be communicatively coupled to the vision units. The CPU may include one or more processors and associated memory storing executable instructions. The CPU can receive and aggregate event records generated by multiple vision units and can maintain an operational representation of the manufacturing line. In some embodiments, the operational representation may include a digital representation reflecting the relationship between a manufacturing station and a sequence of events occurring within the manufacturing process.

[0009] An AI agent, executed by a central processing system, can receive event-related data derived from event logs. The AI ​​agent can analyze the received data relative to a learned reference operating context associated with the operations on the manufacturing line. Based on this analysis, the system can generate operational information associated with observed workstation activities and present this information to one or more user interfaces for viewing or action.

[0010] In some implementations, each vision unit can establish a reference operating context for the corresponding manufacturing station during the calibration phase. During the calibration phase, the vision unit can observe representative operational activities and establish baseline information describing the expected sequence of events or operational characteristics associated with the station.

[0011] In some implementations, the vision unit can generate event logs based on segmented operational activities identified within the captured visual data. The vision unit can then transmit the derived event data to the central processing system without continuously transmitting the original visual data, thereby reducing network bandwidth requirements.

[0012] In some implementations, the vision unit may include additional sensing devices. Such sensing devices may include acoustic sensors, proximity sensors, vibration sensors, temperature sensors, thermal sensors, inertial measurement unit sensors, radar sensors, lidar sensors, or other sensing components configured to capture information associated with operations at the manufacturing station.

[0013] The central processing system can maintain an operational representation of the manufacturing line based on the aggregation of event records received from multiple vision units. This operational representation can reflect the workstation relationships, operational states, and event sequences observed throughout the manufacturing process.

[0014] In some implementations, an AI agent may include one or more trained multimodal analysis models configured to process image data, video data, audio data, sensor data, or a combination thereof.

[0015] In some implementations, as a workpiece moves between manufacturing stations, the system can associate event logs with individual workpieces. The central processing system can maintain a temporal sequence of process paths representing the sequence of stations accessed for each workpiece.

[0016] In some implementations, the central processing system can be implemented within a distributed or cloud-based computing environment.

[0017] In some implementations, the monitoring system may include one or more vision units configured to generate time-referenced event data representing operational activities at corresponding manufacturing stations. A data aggregation system may receive the event data, evaluate the event data relative to a reference operational context associated with the manufacturing line, and generate station-related operational information for presentation via a user interface.

[0018] In another implementation, the manufacturing line control system can receive time-referenced event data from a vision unit observing the manufacturing station and compare the received event data with a stored reference operating scenario associated with the manufacturing line. When the received event data deviates from the reference operating scenario, the system can transmit a stop command to the equipment on the manufacturing line. The stop command can cause one or more machines, conveyors, or robotic devices within the manufacturing line to cease operation.

[0019] In some implementations, the described architecture supports monitoring manufacturing operations without direct integration with programmable logic controllers or other workstation-level control systems. By using distributed vision units that observe operational activities and generate time-referenced event logs representing workstation behavior, the system can be deployed across manufacturing environments with minimal modifications to existing equipment. This architecture facilitates scalable monitoring of heterogeneous manufacturing workstations while supporting event-driven analytics, workpiece traceability across workstations, and centralized operational visibility derived from aggregated event data. Attached Figure Description

[0020] Figure 1 This is a block diagram illustrating an artificial intelligence vision system (AVS).

[0021] Figure 2 A representative manufacturing line is shown, comprising multiple AVS units located at manufacturing stations.

[0022] Figure 3 This is a flowchart illustrating the operations performed during the calibration phase of AVS.

[0023] Figure 4 This is a flowchart illustrating the operations performed during the monitoring phase of AVS.

[0024] Figure 5 A representative event log, including associated event attributes, is shown.

[0025] Figure 6 A representative workflow performed during the monitoring phase is shown.

[0026] Figure 7A and Figure 7B This is a flowchart illustrating the operations performed during the event analysis phase of AVS.

[0027] Figure 8 A representative event report, including associated reporting attributes, is shown.

[0028] Figure 9 An example operational scenario involving a robotic arm at a manufacturing station is shown, which is processed during the event analysis phase.

[0029] Figure 10 This demonstrates a traceability operation supported by AVS for tracking workpieces throughout the manufacturing process.

[0030] Figure 11 This is a high-level block diagram illustrating a representative computing device used to implement one or more functions described herein.

[0031] Figure 12 A representative digital twin system is shown, illustrating the aggregation of workstation-level event data from multiple manufacturing workstations, the maintenance of digital twin representations in a central storage device, and the presentation of role-specific operational information to different users through a user interface. Detailed Implementation

[0032] Detailed embodiments are disclosed herein. However, the disclosed embodiments are provided as illustrative examples and may be implemented in various alternative forms. The drawings are not necessarily drawn to scale, and certain features may be enlarged or reduced to better illustrate particular parts or relationships. Therefore, the structural and functional details described herein are not intended to be limiting, but rather serve as a representative framework to enable those skilled in the art to implement the disclosed subject matter in different configurations.

[0033] The combination of AI-based computer vision, edge computing, and cloud-based analytics offers a compelling alternative to conventional factory floor monitoring methods. However, many existing vision-based systems lack capabilities such as autonomous calibration, contextual interpretation of non-patterned conditions, and integration with digital twin frameworks, limiting their ability to support higher levels of operational intelligence.

[0034] This disclosure addresses these shortcomings by proposing a factory monitoring architecture that provides real-time operational visibility, automated event identification, condition deviation detection, and key performance indicator (KPI) generation through edge-deployed vision units and distributed AI agents. The architecture is designed to be self-configurable and scalable for practical deployment across manufacturing environments of varying sizes and complexities, including SMMs.

[0035] More specifically, this disclosure relates to the use of an artificial intelligence vision system (AVS) to monitor manufacturing lines. AVS operates in conjunction with multimodal AI agents and event-driven analytics to support intelligent operation monitoring and part traceability. Through this coordinated architecture, the manufacturing line is enhanced with context-aware and data-driven insights, enabling continuous visibility and traceability throughout the manufacturing workflow.

[0036] As used herein, certain terms are described for clarity and consistency. The following description illustrates example meanings of selected terms used throughout this disclosure. Unless otherwise expressly stated, these descriptions are not intended to limit the scope of this disclosure or the claims, and the terms described may cover additional structures, configurations, or implementations consistent with the subject matter disclosed.

[0037] A vision unit refers to a sensing device configured to capture visual data associated with operations at a manufacturing station. A vision unit may include one or more cameras, a processor, a memory device, a communication interface, and optional additional sensors. In some implementations, the vision unit performs local processing on the captured data before transmitting it to other system components.

[0038] Event logs or event data refer to data representing observed operational events associated with a manufacturing station or process. Event logs or event data may include timing information, identifiers, media content, metadata, sensor data, or derived analytical information associated with the observed event.

[0039] A reference operating scenario refers to baseline information describing the expected operational behavior of a manufacturing station or line. A reference operating scenario may include expected sequences of events, timing relationships, environmental conditions, operating states, or other situational characteristics associated with normal operation.

[0040] Operation representation refers to a digital representation of a manufacturing process, manufacturing line, or manufacturing station derived from observed operational data. Operation representation can include graphical, structural, or data-based representations of station relationships, operational states, and event sequences.

[0041] An AI agent refers to software or computational logic configured to analyze operational data and generate derived information, interpretations, or actions. An AI agent may include one or more machine learning models, multimodal models, statistical models, rule-based logic, or combinations thereof.

[0042] A stop command refers to a signal, message, instruction, or control indication transmitted to manufacturing equipment that causes a modification, interruption, or cessation of the equipment's operation.

[0043] Figure 1 This is a block diagram illustrating AVS 100. AVS 100 includes one or more AVS units 101. Each AVS unit 101 includes a camera 102, a processor 103, a data storage device 104, and a network connectivity device 105. In some embodiments, the data storage device 104 includes memory storing executable instructions for execution by the processor 103. Each AVS unit 101 may optionally include a microphone 106 or a thermal camera 107.

[0044] AVS unit 101 is arranged along the manufacturing line to observe one or more workstations. Workstations may include machining stations, assembly stations, inspection stations, transfer stations, storage stations, or transport stations. Machining stations perform material or part change operations. Assembly stations assemble components into assemblies. Inspection stations evaluate parts or assemblies against defined standards. Transfer stations move parts between stations. Storage stations temporarily store parts during processing. Transport stations move completed items for downstream disposal.

[0045] AVS 100 also includes a display dashboard 108. The display dashboard 108 presents system status information and operational notifications to the user. AVS unit 101 is communicatively connected to a central processing unit 109. Central processing unit 109 includes a cloud storage system 110 and an AI agent 111. In some embodiments, central processing unit 109 is also communicatively connected to manufacturing equipment via a control interface configured to transmit operating commands, pause commands, or stop commands to one or more workstations, conveyors, robotic devices, or machine tools on the manufacturing line. In some embodiments, central processing unit 109 and cloud storage system 110 operate together as a data aggregation system for receiving, storing, and aggregating event data generated by AVS unit 101.

[0046] In some implementations, each AVS unit 101 performs event identification, time segmentation, and preliminary classification locally at the workstation before transmitting the exported event data to the central processing unit 109. This distributed processing architecture reduces network bandwidth requirements and enables continuous monitoring operations without a continuous cloud connection.

[0047] Figure 2A representative manufacturing line 200 monitored by an AVS 100 is shown. Each AVS unit 101 is physically mounted and positioned to observe one or more workstations 201 along the manufacturing line 200. Each workstation 201 corresponds to a defined operating area within the manufacturing line 200.

[0048] Each AVS unit 101 operates in multiple phases. These phases include a calibration phase, a monitoring phase, and an event assessment phase. During the calibration phase, the AVS unit 101 establishes a reference operating scenario that includes expected event sequences, temporal relationships, and observable workstation characteristics.

[0049] During the monitoring phase, processor 103 performs AI-based image processing and multimodal language modeling. AVS unit 101 observes ongoing activities at workstation 201. Observed activities may include processing operations, assembly actions, inspections, transfers, storage actions, or transport actions. AVS unit 101 processes the captured data locally and transfers the exported data to cloud storage system 110.

[0050] When the operating conditions deviate from the established reference scenario, the AVS 100 initiates an event assessment phase. During the event assessment phase, the AI ​​agent 111 uses a finely tuned multimodal language model to analyze the identified situation to determine its contextual relevance and system response.

[0051] Figure 3 The calibration phase 300 of the AVS 100 is illustrated. During calibration phase 300, each AVS unit 101 determines the type of workstation it observes. Calibration phase 300 begins with an initialization step 301. The AVS 100 then performs an initialization step 302. During initialization step 302, the hardware and software components of the AVS 100 are prepared for operation. The AVS unit 101 establishes network communication and confirms connection to the cloud storage system 110.

[0052] After initialization step 302, AVS 100 captures initial visual data 303 from the observed environment. The initial visual data includes images or videos of the workstation. AVS 100 analyzes the initial visual data to assess the workstation environment 304. Then, AVS 100 determines whether the workstation environment is identified 305. If the workstation environment is identified, AVS 100 proceeds to the workstation type identification step 306. If the workstation environment is not identified, AVS 100 presents a notification and requests manual assistance via display dashboard 108 307.

[0053] During the workstation type identification step 306, the AVS 100 evaluates observable features within its field of view. Observable features include machinery, tools, and repetitive objects. The AVS 100 then determines whether a workstation type has been identified 308. If a workstation type has been identified, the AVS 100 proceeds to the critical event awareness step 309. If a workstation type has not been identified, the AVS 100 requests manual verification or correction 310. The correct workstation classification is then manually entered.

[0054] During the critical event awareness step 309, AVS 100 identifies operational activities associated with the workstation type. AVS 100 builds an event library based on the identified activities 311. AVS 100 monitors the workstation to determine if all critical events have been captured 312. If additional data is needed, AVS 100 continues capturing information or awaits further operator input 313.

[0055] After establishing the event library, the AVS 100 performs calibration verification step 314. The AVS 100 captures a representative operating cycle at the workstation. The captured data defines the baseline timing pattern, motion sequence, and sound profile. The AVS 100 evaluates whether the calibration results are acceptable 315. If the calibration results are unacceptable, the AVS 100 returns to the critical event awareness step 316.

[0056] The calibration phase 300 also includes verifying the functionality of the event logging. Each record entry includes a timestamp, an identifier for the AVS unit 101, and associated media data. The AVS 100 confirms that the recorded information is stored and can be retrieved via the cloud storage system 110.

[0057] When the calibration results are acceptable, the AVS 100 performs the final determination step 317. During the final determination step 317, the calibration data is stored and the AVS 100 is set to active monitoring state 318. The calibration phase 300 ends at the end of the calibration process step 319.

[0058] Figure 4 The monitoring phase 400 of the AVS 100 is shown. Monitoring phase 400 begins after calibration phase 300 is completed. Monitoring phase 400 begins at start monitoring phase step 401. Start monitoring phase step 401 initiates continuous observation of the manufacturing line by the AVS 100.

[0059] Monitoring phase 400 includes a real-time data collection step 402. During real-time data collection step 402, each AVS unit 101 captures visual data from its assigned workstation. Each AVS unit 101 may also capture audio data. The captured data represents the operational activities taking place at the workstation.

[0060] The AVS 100 processes the captured data during event detection and classification step 403. During event detection and classification step 403, the AVS 100 identifies operational events based on a reference pattern established during calibration. Identified events may include machine operation changes, part placement actions, tool interactions, inspection actions, or transfer activities. The AVS 100 assigns each identified event to a predefined event category.

[0061] After event detection and classification, AVS 100 performs event logging step 404. During event logging step 404, AVS 100 generates a log entry for each identified event. Each log entry includes a timestamp and identifier from the AVS unit 101. AVS 100 transmits the event log data to cloud storage system 110 for centralized access and analysis.

[0062] Figure 4 A flowchart representing monitoring phase 400 of the AVS 100 is shown. This phase supports the goal of providing a cost-effective and intelligent plant monitoring solution. The process begins with the initiation of monitoring phase 401, which marks the transition from calibration to continuous observation and data collection across the manufacturing line.

[0063] In the real-time data collection step 402, AVS unit 101 collects visual data and optional audio data in real time from its assigned workstation. Each AVS unit 101 identifies and captures events associated with each workstation type based on an event pattern learned during calibration.

[0064] After collecting the data, the AVS 100 performs event detection and classification 403. The AVS 100 processes the incoming data to identify specific events, such as machine start-up or stop cycles, part placement, tool changes, inspection results, and transfer activities. Each event is categorized according to a predefined template and classified for further analysis.

[0065] Following this step, AVS performs event logging step 404. AVS 100 generates an event log that details the identified events. The log is periodically transmitted to a centralized cloud storage system 110, where it can be accessed for system-level analysis.

[0066] Following event logging step 404, the AVS 100 performs a condition assessment step 405. During condition assessment step 405, the AVS 100 compares the recorded events with the expected operating pattern established during calibration. Deviations may include the absence of expected events, the occurrence of unexpected events, or changes in timing or sequence relative to the expected operating pattern. If the recorded events remain within the expected operating range, the AVS 100 continues with real-time data collection step 402.

[0067] If the condition assessment step 405 identifies a deviation from the expected operating pattern, the AVS 100 initiates a notification step 406. During the notification step 406, the AVS 100 transmits event-related data to the AI ​​agent 111. The AI ​​agent 111 performs a contextual analysis on the identified condition.

[0068] The monitoring phase 400 also includes a continuous KPI tracking step 407. During the continuous KPI tracking step 407, AVS 100 calculates performance metrics based on recorded events. Performance metrics include operation timing, throughput, utilization, and process consistency. Performance metrics are updated as new event data is recorded.

[0069] The AVS 100 performs a feedback and continuous learning step 408. During the feedback and continuous learning step 408, the AVS 100 updates the reference pattern and event model based on operational data and user input. This update supports the refinement of event classifications over time.

[0070] Following feedback and continuous learning step 408, the AVS 100 performs digital twin integration step 409. During digital twin integration step 409, the recorded operational data is associated with a virtual representation of the manufacturing line. The virtual representation reflects the workstation structure, process flow, and current operating conditions.

[0071] Monitoring phase 400 also includes a part or work traceability step 410. During part or work traceability step 410, as a part or work item moves through the manufacturing line, AVS 100 associates recorded events with the individual part or work item.

[0072] Monitoring phase 400 ends at step 411, which marks the end of the monitoring phase. After step 411, the AVS100 remains in monitoring phase 400 unless a recalibration or reconfiguration is initiated.

[0073] Figure 5 A representative event log 500 generated during monitoring phase 400 is shown. Each event log 500 includes a timestamp 501, an identifier 502 of the AVS unit 101 that captured the event, an event type 503, one or more media links 504, and metadata 505. The metadata 505 describes the operational context associated with the event.

[0074] Figure 6 A representative workflow 600 executed during monitoring phase 400 is illustrated. Workflow 600 includes multiple AVS units 601 positioned to observe multiple workstations 602. Each AVS unit 601 captures operational events at the corresponding workstation 602. Operational events include a workstation start event 603a and a workstation stop event 603b.

[0075] AVS unit 601 transmits the captured events to cloud storage system 606. Cloud storage system 606 stores the event data received from AVS unit 601. The stored event data is used to update the digital twin representation and calculate performance metrics.

[0076] Display dashboard 604 presents operational information derived from stored event data. Display dashboard 604 reflects the current operational status and aggregated performance information. The system generates historical summaries based on the stored event data.

[0077] When the operating conditions deviate from the expected pattern, the system provides event-related data to AI Agent 605. AI Agent 605 performs contextual analysis on the event-related data. Contextual analysis supports system-level interpretation of the observed operating conditions.

[0078] In various implementations, the processing associated with AVS unit 601 can be distributed across edge devices and centralized computing resources. For example, certain operations (such as monitoring, event segmentation, and preliminary anomaly detection) can be performed locally by a processor within AVS unit 601. In some implementations, centralized computing resources (such as cloud storage system 606 or other centralized processing systems) can be used to perform calibration processes and higher-level anomaly analysis. Artificial intelligence models for event identification, anomaly detection, or context analysis can be executed locally at AVS unit 601 within a centralized computing resource, or in a hybrid configuration where parts of the analysis are performed at the edge and parts at the center. In some implementations, AVS unit 601 can operate as a standalone monitoring device capable of performing calibration, monitoring, and anomaly detection locally. In other implementations, processing operations can be performed substantially within a centralized computing resource. Thus, the system architecture supports deployments where monitoring, model execution, anomaly detection, and analysis operations can occur at the edge, within a centralized computing resource, or in a combination of both.

[0079] Figure 7A and Figure 7B The event analysis phase 700 performed by AVS 100 is shown. Event analysis phase 700 begins when deviations from the expected operating pattern are identified 701. Deviations are identified by AVS unit 101 during monitoring phase 400.

[0080] After identifying deviation 701, the AVS 100 performs data capture step 702. During data capture step 702, the AVS 100 collects event-related data. Event-related data includes time reference images, video clips, and optional audio data. Event-related data also includes contextual metadata associated with the observed workstation.

[0081] During data transmission step 703, AVS 100 transmits event-related data to AI agent 111. During data reception step 704, AI agent 111 receives the event-related data. During comparison step 705, AI agent 111 compares the received data with a baseline operation pattern.

[0082] Following comparison step 705, AI agent 111 performs event classification step 706. During event classification step 706, AI agent 111 assigns the bias to one of several event categories. Event categories include operational delay category 707a, mechanical condition category 707b, process output condition category 707c, operating environment condition category 707d, or unclassified condition category 707e.

[0083] Following the event classification step 706, the AI ​​agent 111 performs a root cause assessment step 708. During the root cause assessment step 708, the AI ​​agent 111 analyzes image data 709a, analyzes audio data 709b, and cross-references historical operational data 709c. This analysis supports the identification of contributing operational factors.

[0084] Based on the root cause assessment step 708, AI agent 111 identifies key contributing factors during identification step 710. Then, AI agent 111 generates one or more response recommendations during recommendation generation step 711.

[0085] Response recommendations may include maintenance scheduling action 712a, operating parameter adjustment action 712b, or personnel notification action 712c. AI agent 111 assesses whether an immediate response procedure is required during response determination step 713.

[0086] If response determination step 713 indicates that an immediate response procedure is required, AI agent 111 initiates controlled response step 714. Controlled response step 714 may include transmitting control signals via a control interface to pause or stop one or more sections of the manufacturing line. If response determination step 713 indicates that an immediate response procedure is not required, AI agent 111 continues with standard response step 715.

[0087] After executing either the controlled response step 714 or the standard response step 715, AI agent 111 generates an event report during report generation step 716. AI agent 111 then compiles the event report during report compilation step 717.

[0088] After report compilation step 717 is completed, the AVS 100 receives operator input during feedback receiving step 718. Operator input is provided by a human operator or maintenance personnel viewing the incident report. Operator input may include corrections, confirmations, or additional observations.

[0089] AI agent 111 updates one or more analytical models based on received operator input during model update step 719. Model update step 719 may occur according to a predefined schedule or in response to accumulated feedback. Event analysis phase 700 concludes with successive refinement step 720.

[0090] Figure 8 A representative event report 800 generated during event analysis phase 700 is shown. Event report 800 includes a timestamp 801 identifying when the event occurred. Event report 800 also includes an identifier for the AVS unit 802 that captured the event.

[0091] The event report 800 includes an event type 803 associated with the captured event. The event report 800 also includes one or more media links 804. The media link 804 references associated image data, video data, or audio data.

[0092] Event report 800 also includes supporting data 805. Supporting data 805 provides background information related to the operating conditions at the time the event occurred. Event report 800 also includes a root cause explanation 806 generated by AI agent 111.

[0093] The event report 800 also includes an action list 807. The action list 807 identifies one or more actions that are recommended by the system or performed in response to the event.

[0094] Figure 9 A representative operational scenario 900 is shown, processed during the event analysis phase 700. Operational scenario 900 occurs at a workstation including a robotic arm 901. The AVS unit 902 is positioned to observe the operation of the robotic arm 901.

[0095] During normal operation, AVS unit 902 anticipates that part 903 will appear on conveyor belt 904 after the robotic arm 901 completes its movement sequence. The anticipated movement sequence will place part 903 at a defined position 905.

[0096] In the scenario shown, part 903 was not detected at the expected location. AVS 100 identifies deviations from the expected operational pattern. Visual data captured by AVS unit 902 indicates alignment changes of robot arm 901 during movement 906.

[0097] AVS unit 902 transmits the captured data to central processing unit 109. AI agent 111 analyzes the captured data and determines the correlation between deviation and alignment change. AI agent 111 generates event-related information and provides the information to display dashboard 108 for operator viewing.

[0098] Under certain operating conditions, the system can initiate a controlled response 909 based on the determined operating conditions, such as stopping the conveyor belt 904.

[0099] The operational scenario shown demonstrates the coordinated operation of event identification, context analysis, and response determination within the AVS 100. The AVS 100 categorizes observed operating conditions, assesses contributing factors, and supports the generation of response actions via AI agent 111.

[0100] The Event Analysis Phase 700 also supports adaptive model updates based on observed operating conditions and operator input. Figure 3 and Figures 7A to 7B The structured processing logic shown supports the evaluation of cyclic conditions and their integration into subsequent system behavior across the entire manufacturing environment.

[0101] Figure 10 The traceability operation 1000, supported by AVS 100, is illustrated. Traceability operation 1000 tracks individual workpieces as they move through the manufacturing process. Traceability operation 1000 combines monitoring phase 400 and event analysis phase 700 operations.

[0102] Traceability operation 1000 begins with object identification step 1001. During object identification step 1001, AVS unit 101 identifies workpiece 1002 entering station 1003. AVS unit 101 assigns a time reference to workpiece 1002.

[0103] When workpiece 1002 moves forward past station 1003, AVS unit 101 records operation events. Operation events include workpiece entry event 1005a, processing event 1005b, and workpiece departure event 1005c. Each operation event is recorded along with associated metadata.

[0104] Workpiece 1002 moves along manufacturing line 1004 between workstations 1003. Central processing unit 107 aggregates recorded event data associated with workpiece 1002. Central processing unit 107 constructs process path 1006 representing the series of workstations accessed by workpiece 1002.

[0105] When a condition associated with workpiece 1002 is identified at a downstream workstation, the central processing unit 107 retrieves previously recorded data associated with workpiece 1002. The retrieved data is collected from one or more upstream workstations 1008. The retrieved data includes machine settings, operator interactions, environmental conditions, and material batch information.

[0106] AI agent 111 analyzes the retrieved data during causal evaluation step 1009. Causal evaluation step 1009 compares the retrieved data with previously observed operational patterns. Based on the comparison, AI agent 111 generates one or more response recommendations during recommendation formulation step 1010.

[0107] Traceability Operation 1000 supports the association of operating conditions with specific workstations, process parameters, and workpiece history. Traceability Operation 1000 also supports viewing process behavior across multiple workstations.

[0108] By associating time-referenced event data with individual workpieces, AVS 100 maintains continuous visibility of workpiece movement throughout the manufacturing process. Traceability Operation 1000 supports data-driven assessments of process behavior across the entire manufacturing environment.

[0109] Figure 11 A representative computing device 1100 is shown. The computing device 1100 can be used to implement one or more functions described herein. The computing device 1100 can represent a central processing unit 109 or a cloud storage system 110. The computing device 1100 executes computer-readable instructions stored in memory.

[0110] The computing device 1100 includes a processing unit 1101, a system memory 1102, and a bus 1103. The bus 1103 connects the processing unit 1101 to the system memory 1102 and other components of the computing device 1100.

[0111] Bus 1103 represents one or more data communication structures. These data communication structures may include a memory bus, a peripheral bus, or a processor bus. Bus 1103 supports data transfer between components of the computing device 1100.

[0112] Processing unit 1101 executes program instructions stored in system memory 1102. The program instructions perform one or more operations described in this disclosure. Processing unit 1101 may include one or more processors.

[0113] The computing device 1100 includes a computer-readable medium. The computer-readable medium includes volatile and non-volatile media. The computer-readable medium also includes removable and non-removable media.

[0114] System memory 1102 includes random access memory 1104 and cache memory 1105. Computing device 1100 also includes a storage system 1106. Storage system 1106 stores program data and operation data. System memory 1102 stores one or more program products. Each program product includes a program module configured to perform the operations described herein.

[0115] System memory 1102 stores program utility 1107. Program utility 1107 includes one or more program modules 1108. Program module 1108 includes executable instructions and associated data.

[0116] The computing device 1100 communicates with one or more external devices 1109 via one or more input / output interfaces 1110. The external devices 1109 include user interface devices or peripheral devices.

[0117] The computing device 1100 communicates with one or more networks via a network adapter 1111. The network adapter 1111 is connected to a bus 1103. The network adapter 1111 supports data exchange with remote computing systems.

[0118] AVS 100 may include one or more sensors. Sensors include cameras, acoustic sensors, proximity sensors, inertial measurement unit sensors, positioning sensors, radar sensors, and lidar sensors. The selected sensor type depends on the workstation monitored by the AVS unit 101. In some embodiments, non-visual sensors are used to supplement visual observation at workstations with limited line-of-sight or requiring spatial measurements.

[0119] Figure 12 A representative digital twin system 1200 for a manufacturing environment is shown. The digital twin system 1200 includes multiple manufacturing stations 1201, 1202, and 1203.

[0120] The digital twin system 1200 also includes multiple vision and microphone units 1204, 1205, and 1206. Vision and microphone unit 1204 is positioned to sense manufacturing station 1201. Vision and microphone unit 1205 is positioned to sense manufacturing station 1202. Vision and microphone unit 1206 is positioned to sense manufacturing station 1203. Each vision and microphone unit 1204, 1205, and 1206 captures station-related visual and audio data representing operations occurring at the corresponding manufacturing station 1201, 1202, and 1203. Each vision and microphone unit 1204, 1205, and 1206 generates station-related event data based on the captured visual and audio data. The station-related event data may include time-referenced event records and associated media.

[0121] The digital twin system 1200 also includes a cloud / central storage device 1207. The cloud / central storage device 1207 is communicatively connected to vision and microphone units 1204, 1205, and 1206. The cloud / central storage device 1207 receives workstation-related event data generated by the vision and microphone units 1204, 1205, and 1206. The cloud / central storage device 1207 stores the received workstation-related event data for subsequent retrieval. The cloud / central storage device 1207 also supports the aggregation of workstation-related event data across manufacturing workstations 1201, 1202, and 1203.

[0122] The digital twin system 1200 also includes a digital twin 1208. Digital twin 1208 is stored in or accessible via a cloud / central storage device 1207. Digital twin 1208 represents a virtual view of a manufacturing environment based on workstation-related event data received from vision and microphone units 1204, 1205, and 1206. Digital twin 1208 includes multiple virtual workstation representations 1209, 1210, and 1211. Virtual workstation representation 1209 corresponds to manufacturing workstation 1201. Virtual workstation representation 1210 corresponds to manufacturing workstation 1202. Virtual workstation representation 1211 corresponds to manufacturing workstation 1203. Digital twin 1208 is updated based on aggregated workstation-related event data stored in the cloud / central storage device 1207. Digital twin 1208 provides a consolidated view of workstation activities across multiple manufacturing workstations.

[0123] The digital twin system 1200 also includes a first user interface 1212. The first user interface 1212 is configured for a first user role associated with workstation-level monitoring. The first user interface 1212 presents workstation-specific information associated with manufacturing workstation 1201. The workstation-specific information may include cycle time metrics and workstation status metrics for manufacturing workstation 1201. The first user interface 1212 receives data from a cloud / central storage device 1207. The first user interface 1212 presents the workstation-specific information based on workstation-related event data stored in the cloud / central storage device 1207 and reflected in the digital twin 1208.

[0124] The digital twin system 1200 also includes a second user interface 1213. The second user interface 1213 is configured for a second user role associated with multi-workstation viewing. The second user interface 1213 presents aggregated workstation evaluation information associated with manufacturing workstations 1201, 1202, and 1203. The aggregated workstation evaluation information may include the workstation evaluation status of each of manufacturing workstations 1201, 1202, and 1203. The second user interface 1213 receives data from a cloud / central storage device 1207. The second user interface 1213 presents the aggregated workstation evaluation information based on aggregated workstation-related event data stored in the cloud / central storage device 1207 and reflected in the digital twin 1208.

[0125] During operation, vision and microphone units 1204, 1205, and 1206 provide workstation-related event data to cloud / central storage device 1207. Cloud / central storage device 1207 stores and aggregates workstation-related event data across manufacturing workstations 1201, 1202, and 1203. Digital twin 1208 is maintained based on the aggregated workstation-related event data. First user interface 1212 and second user interface 1213 present role-specific information derived from the aggregated workstation-related event data and digital twin 1208.

[0126] The disclosed subject matter can be implemented as a system, method, or computer program product. The computer program product includes a computer-readable storage medium storing program instructions. These program instructions cause a processor to perform the operations described herein.

[0127] A computer-readable storage medium is a tangible means for storing instructions for use by a processing apparatus. The computer-readable storage medium includes electronically, magnetically, optically, electromagnetically, or semiconductor-based storage media, or combinations thereof. The computer-readable storage medium does not include transient signals.

[0128] A non-transitory computer-readable storage medium includes one or more physical storage devices. The physical storage devices store program instructions in electronic, magnetic, optical, or semiconductor form.

[0129] Computer-readable program instructions can be transmitted to a computing device via a network. The network includes wired or wireless communication paths. A network interface receives the program instructions and provides them for execution or storage.

[0130] Flowcharts and block diagrams illustrate representative architectures and operations. Each block represents a functional unit or a portion of executable instructions. Blocks can be executed in different orders. Blocks can be implemented using software, hardware, or a combination thereof.

[0131] Each functional block can be implemented by computer program instructions executed by the processor. The computer program instructions cause the processor to perform operations associated with the functional block.

[0132] The process described herein can be performed continuously or intermittently. The process can be executed by one or more computing devices. Individual process steps can be executed by different computing devices.

[0133] The described embodiments are provided for illustrative purposes. The described embodiments do not limit the scope of this disclosure. Modifications and variations may be made without departing from the scope of the appended claims.

[0134] As used herein, a module or component refers to software, firmware, hardware, or a combination thereof. In a software implementation, a module includes program code that is executed by a processor.

[0135] Program code can be written in assembly language or higher-level programming languages. Program code can be translated into a machine-readable form for execution.

[0136] Computer-executable instructions can be stored in a computer-readable storage medium. Execution of these instructions causes a computing device to perform defined operations.

[0137] Unless otherwise specified, the order of the operations described is not fixed. Operations may be performed in parallel or in a different order. For clarity, the accompanying drawings may omit interactions.

[0138] Any disclosed method can be implemented as computer-executable instructions stored on a computer-readable storage medium. The computer-readable storage medium includes tangible storage media accessible to a computing device.

[0139] Variations, substitutions, and equivalent arrangements may be implemented within the scope of the appended claims. The described embodiments illustrate representative implementations.

[0140] The terminology used herein is descriptive. The terminology does not limit the scope of this disclosure.

[0141] According to the present invention, a manufacturing monitoring system is provided, comprising: one or more vision units, each vision unit including a camera, a local processor, a data storage device, and a network interface, wherein each vision unit can be deployed at a manufacturing station and is configured to capture visual data representing operations at the manufacturing station; a central processing system communicatively coupled to the one or more vision units and including at least one processor and a non-transitory storage medium storing executable instructions; and an artificial intelligence agent executed by the central processing system, wherein the manufacturing monitoring system is configured to cause the one or more vision units to generate time-referenced event records associated with observed station operations, the event records including media data and contextual metadata; the central processing system to aggregate the event records received from the one or more vision units and maintain an operational representation of the manufacturing line; the artificial intelligence agent to receive event-related data from the central processing system and apply learned reference context to the received event-related data; and the manufacturing monitoring system to provide event-related information to a user interface for viewing or action.

[0142] According to an embodiment, the one or more vision units are configured to establish a reference operating scenario for the manufacturing station during a calibration phase performed locally at the one or more vision units.

[0143] According to an embodiment, the one or more vision units are configured to generate the time reference event record based on segmented operational activities observed at the manufacturing station.

[0144] According to an embodiment, the one or more vision units are configured to transmit derived event data to the central processing system independently of the continuous transmission of the original visual data.

[0145] According to an embodiment, each vision unit further includes at least one additional sensor selected from acoustic sensors, proximity sensors, vibration sensors, temperature sensors, thermal sensors, current sensors, radar sensors, or lidar sensors.

[0146] According to an embodiment, each time reference event record includes a timestamp, an identifier of the visual unit, and metadata describing the operational context of the manufacturing station.

[0147] According to an embodiment, the operational representation of the manufacturing line includes a virtual representation reflecting the relationship between manufacturing stations and event sequences.

[0148] According to an embodiment, the virtual representation is updated based on the aggregation of event records received from multiple visual units.

[0149] According to an embodiment, the artificial intelligence agent is implemented using one or more trained multimodal models executed by the central processing system.

[0150] According to an embodiment, the artificial intelligence agent is configured to associate event logs with individual workpieces as the workpieces move between manufacturing stations.

[0151] According to an embodiment, the central processing system maintains the temporal sequence process path for each associated workpiece based on the event records.

[0152] According to an embodiment, the central processing system is implemented using a cloud-based computing environment.

[0153] According to the present invention, a monitoring system for a manufacturing environment is provided, the monitoring system comprising: one or more vision units, the one or more vision units being locatable adjacent to corresponding manufacturing stations on a manufacturing line, each vision unit including a camera, a processor, and a memory, wherein the memory of each vision unit stores executable instructions, which, when executed by the processor of the vision unit, cause the vision unit to generate time reference event data representing operational activities occurring at the corresponding manufacturing station based on visual data captured by the camera; and a data aggregation system communicatively connected to the one or more vision units and including at least one processor and a memory storing executable instructions, wherein, when executed by the at least one processor, the data aggregation system receives the time reference event data from the one or more vision units, evaluates the time reference event data relative to a reference operational context associated with the manufacturing line, and provides station-related operational information to a user interface.

[0154] According to an embodiment, the executable instructions stored in the memory of each vision unit cause the processor of the vision unit to generate the time reference event data based on segmented activities identified within the visual data captured by the camera.

[0155] According to an embodiment, the executable instructions of the data aggregation system cause the at least one processor to apply one or more trained multimodal models to the time reference event data to generate the workstation-related operation information.

[0156] According to an embodiment, the executable instructions of the data aggregation system cause the at least one processor to maintain a digital representation of the manufacturing line based on the time reference event data received from the one or more vision units.

[0157] According to the present invention, a manufacturing line control system is provided, comprising: one or more vision units positioned to observe corresponding workstations on a manufacturing line, each vision unit including a camera, a processor, and a memory storing instructions, the instructions, when executed by the processor, causing the vision unit to generate time reference event data representing activity occurring at the corresponding workstation based on visual data captured by the camera; and a control processor communicatively coupled to the one or more vision units, wherein the control processor is configured to: receive the time reference event data from the one or more vision units; compare the received event data with a stored reference operating context of the manufacturing line; and transmit a stop command to the equipment on the manufacturing line to stop operation at at least one workstation of the manufacturing line when the received event data deviates from the reference operating context.

[0158] According to an embodiment, the stop command causes the conveyor, robot device, or machine tool of the manufacturing line to stop operating.

[0159] According to an embodiment, the reference operating scenario includes a sequence of expected events associated with the operation of the workstation.

[0160] According to an embodiment, when received event data indicates that no expected operational event exists at one of the workstations, the control processor generates the stop command.

Claims

1. A manufacturing monitoring system, comprising: One or more vision units, each vision unit including a camera, a local processor, a data storage device and a network interface, wherein each vision unit may be deployed at a manufacturing station and is configured to capture visual data representing the operation of the manufacturing station; A central processing system, communicatively coupled to the one or more vision units and including at least one processor and a non-transitory storage medium storing executable instructions; as well as An AI agent, executed by the central processing system, wherein the manufacturing monitoring system is configured to cause the one or more vision units to generate time-referenced event records associated with observed workstation operations, the event records including media data and contextual metadata; the central processing system aggregates the event records received from the one or more vision units and maintains an operational representation of the manufacturing line; the AI ​​agent receives event-related data from the central processing system and applies learned reference context to the received event-related data; and the manufacturing monitoring system provides event-related information to a user interface for viewing or action.

2. The manufacturing monitoring system of claim 1, wherein the one or more vision units are configured to establish a reference operating context for the manufacturing station during a calibration phase performed locally at the one or more vision units.

3. The manufacturing monitoring system of claim 1, wherein the one or more vision units are configured to generate the time reference event record based on segmented operational activities observed at the manufacturing station.

4. The manufacturing monitoring system of claim 1, wherein the one or more vision units are configured to transmit derived event data to the central processing system independently of the continuous transmission of raw visual data.

5. The manufacturing monitoring system of claim 1, wherein each vision unit further comprises at least one additional sensor selected from acoustic sensors, proximity sensors, vibration sensors, temperature sensors, thermal sensors, current sensors, radar sensors or lidar sensors.

6. The manufacturing monitoring system of claim 1, wherein each time reference event record includes a timestamp, an identifier of the vision unit, and metadata describing the operational context of the manufacturing station.

7. The manufacturing monitoring system of claim 1, wherein the operational representation of the manufacturing line includes a virtual representation reflecting the relationship between manufacturing stations and event sequences.

8. The manufacturing monitoring system of claim 7, wherein the virtual representation is updated based on the aggregation of event records received from a plurality of vision units.

9. The manufacturing monitoring system of claim 1, wherein the artificial intelligence agent is implemented using one or more trained multimodal models executed by the central processing system.

10. A monitoring system for a manufacturing environment, comprising: One or more vision units, which can be positioned adjacent to corresponding manufacturing stations on a manufacturing line, each vision unit including a camera, a processor, and a memory, wherein the memory of each vision unit stores executable instructions that, when executed by the processor of the vision unit, cause the vision unit to generate time reference event data representing operational activities occurring at the corresponding manufacturing station based on visual data captured by the camera. A data aggregation system communicatively coupled to the one or more vision units and including at least one processor and a memory storing executable instructions, wherein the executable instructions of the data aggregation system, when executed by the at least one processor, cause the data aggregation system to receive time reference event data from the one or more vision units, evaluate the time reference event data relative to a reference operating context associated with the manufacturing line, and provide workstation-related operating information to a user interface.

11. The monitoring system of claim 10, wherein the executable instructions stored in the memory of each vision unit cause the processor of the vision unit to generate the time reference event data based on segmented activities identified within the visual data captured by the camera.

12. The monitoring system of claim 10, wherein the executable instructions of the data aggregation system cause the at least one processor to apply one or more trained multimodal models to the time reference event data to generate the workstation-related operational information.

13. The monitoring system of claim 10, wherein the executable instructions of the data aggregation system cause the at least one processor to maintain a digital representation of the manufacturing line based on the time reference event data received from the one or more vision units.

14. A manufacturing line control system, comprising: One or more vision units are positioned to observe corresponding workstations on a manufacturing line. Each vision unit includes a camera, a processor, and a memory storing instructions that, when executed by the processor, cause the vision unit to generate time-referenced event data representing activity occurring at the corresponding workstation based on visual data captured by the camera. as well as A control processor communicatively coupled to the one or more vision units, wherein the control processor is configured to: receive the time reference event data from the one or more vision units; compare the received event data with a stored reference operating context of the manufacturing line; and transmit a stop command to the equipment of the manufacturing line to stop the operation of at least one station of the manufacturing line when the received event data deviates from the reference operating context.

15. The manufacturing line control system of claim 14, wherein the stop command causes the conveyor, robot device, or machine tool of the manufacturing line to stop operating.

Images