Flexible production line production process state monitoring system based on computer vision technology

Abstract

The invention discloses a flexible production line production process state monitoring system based on a computer vision technology, and relates to the technical field of flexible production line production monitoring, and the system comprises a visual perception module which collects a production video VID of a flexible production line, carries out the target detection and instance segmentation of a target region, and recognizes a workpiece OBJ, equipment DEV and an operator PER in the flexible production line; the flexible production line is monitored through the visual perception module and the trajectory analysis module, when an abnormal workpiece is detected, the production process of the abnormal workpiece is quickly traced, and the root cause of abnormality formation is determined through the root cause diagnosis module; through the analysis of the beat deviation and the process path of each station, the risk assessment can be dynamically carried out to generate the state level STA, the production state can be recognized in time, the production stability is improved, and the downtime caused by abnormal operation or equipment failure is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of flexible production line monitoring technology, specifically a flexible production line production process status monitoring system based on computer vision technology. Background Technology

[0002] As the manufacturing industry continues to develop towards personalized, customized, and small-batch, multi-variety production models, traditional fixed production lines, due to their rigid structure, single production cycle, and high process changeover costs, are gradually becoming unable to meet the demands of modern manufacturing systems for production flexibility and rapid response capabilities. To adapt to the needs of frequent changes in product types and dynamic adjustments to production tasks, flexible production lines are gradually becoming an important component of advanced manufacturing systems. Flexible production lines typically integrate automated equipment, industrial robots, intelligent transmission devices, and information control systems to achieve dynamic allocation and collaborative control of production resources, thereby enabling rapid switching between different product models, improving production efficiency, and reducing changeover time.

[0003] Existing flexible production lines struggle to comprehensively identify workpiece flow processes, personnel operations, and abnormal situations on-site. With frequent product model changes and dynamic adjustments to workstation layouts on flexible production lines, path matching deviations are prone to occur, leading to a decrease in the accuracy of production status identification. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a flexible production line process status monitoring system based on computer vision technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a flexible production line process status monitoring system based on computer vision technology, comprising: The visual perception module acquires production video (VID) from the flexible production line and performs target detection and instance segmentation on the target area to identify workpieces (OBJ), equipment (DEV), and operators (PER) in the flexible production line. The trajectory analysis module extracts motion path data based on the spatial position changes of workpiece OBJ and operator PER in continuous video frames to form trajectory TRK; and extracts the spatiotemporal correlation between each target object based on the trajectory TRK to construct a relationship graph GPH, wherein the relationship graph GPH is used to describe the flow relationship of workpiece OBJ between different equipment DEVs and the interaction relationship between operator PER and equipment DEV. The path cycle time calculation module is used to perform path reasoning based on the relationship graph GPH to generate process path PAT, and calculate cycle time CYC based on the process path PAT; The anomaly tracing and root cause diagnosis module is used to track abnormal workpieces. When a quality anomaly is detected, a product lineage chain GEN is constructed using the identifier UID as an index, and the cause of the anomaly is determined based on rule judgment and historical sample comparison analysis, generating the root cause ranking result RCU. The production status assessment module is used to assess the production process status of the flexible production line based on the cycle time (CYC), identify bottleneck workstations based on cycle time deviation, and generate status level (STA) and alarm information when cycle time deviation or production efficiency decline is detected.

[0006] Preferably, the visual perception module is used for video acquisition of the flexible production line, including: Fixed-view acquisition terminals are deployed at key workstations on the production line to acquire production video VIDs, and lens intrinsic parameter calibration and distortion correction are performed on the production video VIDs. The production video VID is processed with illumination equalization, reflection suppression and noise filtering to obtain the standard image IMG; Based on the standard image IMG, target detection and instance segmentation are performed on the dynamic region to identify target instances and extract their appearance features, spatial location, posture information and timestamps to form observation data OBS; the target instances include workpiece OBJ, equipment DEV and operator PER, and the workpiece OBJ is assigned an identifier UID.

[0007] Preferably, the trajectory analysis module is used to reconstruct the workpiece flow process based on observation data (OBS), including: The observation data OBS is sorted in chronological order, and an association cost matrix is ​​constructed. The association cost matrix is ​​composed of a weighted average of location prediction bias, appearance feature similarity, and workstation topology constraints. Target association is performed based on the association cost matrix to generate workpiece trajectory TRKOBJ and personnel trajectory TRKPER. When the trajectory is interrupted, the trajectory is reconnected by combining target re-identification and accessibility constraints of adjacent workstations. Based on the workstation boundary, the workpiece trajectory TRKOBJ is analyzed for spatial landing point. Combined with speed change and dwell time features, workpiece arrival, start, completion, waiting and abnormal stop events are extracted. Based on the spatial relationship between the personnel trajectory TRKPER and the equipment operation area, the contact duration, and the motion characteristics, personnel-equipment interaction events are extracted. The process event flow (EVT) is generated by sequentially connecting workpiece events and interaction events in chronological order. A relational graph GPH is constructed based on the process event flow EVT. In the relational graph GPH, the workpiece OBJ, equipment DEV, personnel PER, and workstation nodes are graph nodes, and the flow, dwell, operation, and interaction relationships are graph edges. The start and end times, duration, interaction frequency, and path direction are recorded.

[0008] Preferably, the trajectory analysis module employs a trajectory reconstruction algorithm based on multi-feature fusion in the execution of target association and trajectory construction, including: A state vector is constructed based on the target's spatial location, and the predicted location is calculated using a Kalman prediction model. The target's appearance feature vector is extracted, and the feature similarity between targets in adjacent frames is calculated. An accessibility constraint matrix is ​​constructed based on the flexible production line workstation topology to limit the spatial range where the target can appear in adjacent time intervals. A target association cost function C is then constructed. ; Where Dpos represents the location prediction distance; Dapp represents the appearance feature distance; Dtop represents the topological constraint cost; and α, β, and γ represent the corresponding weight coefficients. The cross-frame target matching relationship is determined by minimizing the target association cost function C, and the Hungarian matching algorithm is used to complete the global optimal association of multiple targets. When the trajectory is temporarily occluded or lost, the feature similarity between the historical feature vector and the current candidate target is calculated by the target re-identification network, and the trajectory is completed by combining the nearest neighbor station constraint to obtain the workpiece trajectory TRKOBJ and the personnel trajectory TRKPER.

[0009] Preferably, the path cycle time calculation module is used to derive the process path based on the relationship graph GPH and the process event flow EVT, including: Using the identifier UID as an index, the sequence of workstation nodes corresponding to the workpiece OBJ and their order relationship are extracted from the relationship graph GPH, and the single-piece process path PAT is reconstructed from the process event flow EVT. Based on the process path PAT, the flow time of workpiece OBJ between adjacent stations and the dwell time in a single station are calculated to generate the single-piece cycle time CYC1. Within the statistical window W, perform statistical analysis on the single-piece cycle time CYC1 of multiple workpieces OBJ, calculate the average cycle time and dispersion index corresponding to each station to generate the station cycle time CYC2.

[0010] Preferably, the anomaly tracing and root cause diagnosis module is used to track abnormal workpieces, including: When the quality inspection station outputs a non-conforming mark, the process path PAT, process event flow EVT, station cycle time CYC2, and trajectory TRK of the workpiece OBJ are retrieved using the identifier UID as the index to construct the production process file of the workpiece OBJ; based on the file, the dwell time of the workpiece OBJ at each station, the operator PER, the equipment DEV, and the batch information are associated to form the product genealogy chain GEN. Diagnostic analysis of the product lineage chain GEN is performed based on preset anomaly detection rules, including: Perform node sequence matching on the process path PAT, calculate the path edit distance DIST, and determine the missing process anomaly when DIST exceeds the threshold THR1. Extract the arrival time, start time and completion time from the process event flow EVT, calculate the deviation DEV between the station dwell time TIM and the baseline cycle time BAS, and if DEV is greater than the threshold THR2, it is judged as an abnormal timeout. Rework behavior is identified by the number of repetitions and time intervals of workstation nodes in the process path PAT. If the number of repetitions exceeds the threshold THR3, it is judged as abnormal rework. Based on the spatial relationship between the personnel trajectory TRKPER and the equipment operation area and the duration of the interaction, personnel-equipment interaction events are extracted. If the interaction is missing or the duration is less than the threshold THR4, it is judged as an operation missing anomaly. The rate of change of beat (VAR) is calculated based on the single-piece beat (CYC1) of the workpiece OBJ. If the VAR exceeds the threshold THR5, it is determined to be an abnormal beat change.

[0011] Preferably, the anomaly tracing and root cause diagnosis module is used to track abnormal workpieces and further includes: When the quality inspection station outputs a non-conforming mark, the workpiece OBJ marked as non-conforming is used as the center. Based on the matching conditions of the same batch, related samples are extracted from the historical production database to form a candidate sample set. Based on the final quality result, the candidate samples are divided into a normal sample set NOR and an abnormal sample set ABN. Statistical feature values ​​of the normal sample set NOR and the abnormal sample set ABN are extracted in terms of feature dimensions such as processing time, waiting time, workstation dwell time, personnel interaction frequency, continuous equipment running time, equipment switching times and rework times. The deviation between abnormal samples and normal samples is calculated and expressed by the difference in standard deviation. The deviation of each feature dimension is mapped to the contribution rate CON of each candidate factor to the abnormal results, and the root cause ranking results RCU are generated in descending order of contribution rate. Extract the process path PAT of multiple abnormal workpieces OBJ within the preset time window WIN, analyze the overlapping segments of the paths, and determine whether there are common equipment DEV, operator PER, or workstation dwell intervals. If there are duplicate nodes in multiple abnormal paths, mark the node as a systemic risk source RSK.

[0012] Preferably, the production status assessment module assesses the production status of the flexible production line, including: Read the current workstation cycle time CYC2 and the baseline cycle time BAS, and calculate the cycle time deviation: ; The Dcyc is fused with the top N root cause contribution rates CON and systemic risk sources RSK in the root cause ranking result RCU to calculate the workstation risk score Rj, using the following formula: ; Among them, w1, w2, and w3 are weighting coefficients; Based on the workstation risk score Rj and the duration of the abnormality, the production line status is divided into three levels: normal, warning, and abnormal, and the status level STA is output. Based on the primary root cause type in the RCU, corresponding action instructions (OPTs) are generated, including: when the primary root cause is equipment-related, outputting equipment maintenance and load transfer alarms; when the primary root cause is personnel operation-related, outputting personnel verification and workstation reassignment alarms; and when the primary root cause is process path-related, outputting path correction alarms.

[0013] This invention provides a flexible production line process status monitoring system based on computer vision technology, which has the following advantages compared with the prior art: This invention monitors the flexible production line through a visual perception module and a trajectory analysis module. When an abnormal workpiece is detected, the production process of the abnormal workpiece is quickly traced and the root cause diagnosis module is used to determine the root cause of the abnormality. By analyzing the cycle deviation of each workstation and the process path, risk assessment can be dynamically performed to generate a status level (STA), which can identify the production status in a timely manner, improve production stability, and effectively reduce downtime caused by abnormal operation or equipment failure. This invention, through anomaly tracing and root cause analysis, combined with historical sample comparison and real-time data analysis, can accurately pinpoint the causes of abnormal workpieces and generate corresponding handling suggestions for different root cause types such as equipment failure, improper operation by personnel, or process problems. This mechanism can take timely measures to prevent the recurrence of anomalies, thereby ensuring the efficient and stable operation of the production line. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the principle framework of the present invention; Figure 2 This is a schematic diagram of the system structure framework of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1This application provides a flexible production line process status monitoring system based on computer vision technology, comprising: The visual perception module acquires production video (VID) from the flexible production line and performs target detection and instance segmentation on the target area to identify workpieces (OBJ), equipment (DEV), and operators (PER) in the flexible production line. The trajectory analysis module extracts motion path data based on the spatial position changes of workpiece OBJ and operator PER in continuous video frames to form trajectory TRK; and extracts the spatiotemporal correlation between each target object based on the trajectory TRK to construct a relationship graph GPH, wherein the relationship graph GPH is used to describe the flow relationship of workpiece OBJ between different equipment DEVs and the interaction relationship between operator PER and equipment DEV. The path cycle time calculation module is used to perform path reasoning based on the relationship graph GPH to generate process path PAT, and calculate cycle time CYC based on the process path PAT; The anomaly tracing and root cause diagnosis module is used to track abnormal workpieces. When a quality anomaly is detected, a product lineage chain GEN is constructed using the identifier UID as an index, and the cause of the anomaly is determined based on rule judgment and historical sample comparison analysis, generating the root cause ranking result RCU. The production status assessment module is used to assess the production process status of the flexible production line based on the cycle time (CYC), identify bottleneck workstations based on cycle time deviation, and generate status level (STA) and alarm information when cycle time deviation or production efficiency decline is detected.

[0017] In a preferred embodiment of the present invention, the visual perception module is used to acquire video of the flexible production line, including: Fixed-view acquisition terminals are deployed at key workstations on the production line to acquire production video VIDs, and lens intrinsic parameter calibration and distortion correction are performed on the production video VIDs. The production video VID is processed with illumination equalization, reflection suppression and noise filtering to obtain the standard image IMG; Based on the standard image IMG, target detection and instance segmentation are performed on the dynamic region to identify target instances and extract their appearance features, spatial location, posture information and timestamps to form observation data OBS; the target instances include workpiece OBJ, equipment DEV and operator PER, and the workpiece OBJ is assigned an identifier UID.

[0018] Specifically, illumination equalization processing is performed on video frames to reduce the impact of different shifts and lighting environments on the image brightness distribution; at the same time, a highlight region suppression algorithm is used to weaken reflected light from reflective areas on the equipment surface to reduce interference from metal equipment or bright surfaces; and a spatial domain filtering algorithm is used to perform noise filtering on video frames to remove random noise generated by image sensors or environmental interference, thereby obtaining a standard image IMG with stable brightness distribution and a high signal-to-noise ratio.

[0019] After obtaining the standard image (IMG), dynamic regions are extracted using background modeling methods, and target detection and instance segmentation are performed on these dynamic regions. Specifically, a deep learning target detection network is used to identify target instances in the image, and an instance segmentation algorithm is used to obtain the precise contour regions of each target, thereby identifying workpieces (OBJ), equipment (DEV), and operators (PER) in the flexible production line. For each target instance, its appearance feature vector, spatial coordinates, posture information, and corresponding timestamp are extracted. The appearance feature vector is used for subsequent cross-frame target association, the spatial coordinates are used for trajectory calculation, and the posture information is used to identify personnel operation behavior.

[0020] For each identified workpiece OBJ, a unique identifier UID is assigned. The UID is maintained through a cross-frame target matching mechanism. When the workpiece OBJ moves or is briefly occluded in consecutive video frames, the target identity is re-associated through appearance feature matching and spatial location prediction, thereby ensuring the unique identification of the same workpiece in different time frames.

[0021] The trajectory analysis module is used to reconstruct the workpiece flow process based on observation data (OBS), including: The observation data OBS is sorted in chronological order, and an association cost matrix is ​​constructed. The association cost matrix is ​​composed of a weighted average of location prediction bias, appearance feature similarity, and workstation topology constraints. Target association is performed based on the association cost matrix to generate workpiece trajectory TRKOBJ and personnel trajectory TRKPER. When the trajectory is interrupted, the trajectory is reconnected by combining target re-identification and accessibility constraints of adjacent workstations. Based on the workstation boundary, the workpiece trajectory TRKOBJ is analyzed for spatial landing point. Combined with speed change and dwell time features, workpiece arrival, start, completion, waiting and abnormal stop events are extracted. Based on the spatial relationship between the personnel trajectory TRKPER and the equipment operation area, the contact duration, and the motion characteristics, personnel-equipment interaction events are extracted. The process event flow (EVT) is generated by sequentially connecting workpiece events and interaction events in chronological order. A relational graph GPH is constructed based on the process event flow EVT. In the relational graph GPH, the workpiece OBJ, equipment DEV, personnel PER, and workstation nodes are graph nodes, and the flow, dwell, operation, and interaction relationships are graph edges. The start and end times, duration, interaction frequency, and path direction are recorded.

[0022] Specifically, the observation data OBS is sorted according to timestamps, and the target observation results in consecutive time frames are uniformly numbered. For each target instance in a time frame, its spatial coordinates, appearance feature vector, target category, and time information are extracted, and a cross-frame target association problem is constructed based on this. For the target set in adjacent time frames, an association cost matrix is ​​established, where each element in the matrix represents the matching cost between the target in the previous frame and the target in the current frame. The matching cost is calculated by weighting multiple factors, including position prediction bias, appearance feature similarity, and workstation topology constraints. Specifically, the position prediction bias is obtained by calculating the predicted position of the target in the next time frame using a Kalman prediction model based on motion state vectors, and calculating the Euclidean distance with the actual observation position; the appearance feature similarity is obtained by calculating the cosine similarity or feature distance between the target appearance feature vectors; and the workstation topology constraints are based on the workstation connection relationship of the flexible production line, restricting the reachable area of ​​the target in adjacent time frames, thereby avoiding erroneous matching across unreachable workstations.

[0023] After constructing the association cost matrix, a multi-target matching algorithm is used to associate targets across frames, determining the correspondence between targets in consecutive video frames, thereby generating the workpiece motion trajectory TRKOBJ and the operator motion trajectory TRKPER. When the trajectory is interrupted between consecutive frames due to occlusion, detection failure, or movement outside the field of view, a target re-identification algorithm is used to perform feature matching on candidate targets, calculate the similarity between the historical trajectory feature vector and the current candidate target, and combine the workstation adjacency relationship to reconnect the trajectories, thereby restoring the complete trajectory.

[0024] After obtaining the workpiece trajectory TRKOBJ, spatial landing point analysis is performed on the trajectory based on a preset station boundary model. Specifically, the behavior state of the workpiece in each station is identified by detecting whether the center point of the workpiece trajectory enters a certain station area, and by combining the target speed change characteristics and dwell time information. When the workpiece trajectory first enters a station boundary area, it is identified as an arrival event; when the workpiece dwells in the area for a period of time that reaches the preset processing preparation time and the speed drops below the threshold, it is identified as a start-up event; when the workpiece leaves the station boundary area, it is identified as a completion event; when the workpiece dwells in the station area for a period of time that significantly exceeds the baseline processing time, it is identified as a waiting event; when the dwell time exceeds the preset abnormal threshold, it is identified as an abnormal stop event.

[0025] Meanwhile, for the operator trajectory TRKPER, the spatial proximity between the operator's trajectory and the equipment operating area is analyzed, and the interaction behavior between the operator and the equipment is identified by combining the operator's dwell time and movement changes within the equipment control area. When the operator's trajectory enters the equipment operating area and remains there for more than a set time, it is identified as a operator-equipment interaction event.

[0026] Subsequently, workpiece-related events and personnel interaction events are organized and linked in chronological order to form a Process Event Flow (EVT). The EVT is used to record the process execution sequence of the workpiece and personnel operation behaviors during the production process.

[0027] Further construct the production relationship graph (GPH). Specifically, use workpiece (OBJ), equipment (DEV), operator (PER), and each workstation node as graph nodes, and use workpiece flow relationships, workstation dwell relationships, operator operation relationships, and operator-equipment interaction relationships as graph edges. For each relationship edge, record its start time, end time, duration, interaction frequency, and path direction.

[0028] The trajectory analysis module employs a trajectory reconstruction algorithm based on multi-feature fusion in the execution of target association and trajectory construction, including: A state vector is constructed based on the target's spatial location, and the predicted location is calculated using a Kalman prediction model. The target's appearance feature vector is extracted, and the feature similarity between targets in adjacent frames is calculated. An accessibility constraint matrix is ​​constructed based on the flexible production line workstation topology to limit the spatial range where the target can appear in adjacent time intervals. A target association cost function C is then constructed. ; Where Dpos represents the location prediction distance; Dapp represents the appearance feature distance; Dtop represents the topological constraint cost; and α, β, and γ represent the corresponding weight coefficients. The cross-frame target matching relationship is determined by minimizing the target association cost function C, and the Hungarian matching algorithm is used to complete the global optimal association of multiple targets. When the trajectory is temporarily occluded or lost, the feature similarity between the historical feature vector and the current candidate target is calculated by the target re-identification network, and the trajectory is completed by combining the nearest neighbor station constraint to obtain the workpiece trajectory TRKOBJ and the personnel trajectory TRKPER.

[0029] Specifically, after obtaining the observation data OBS output by the visual perception module, a motion state vector is constructed for each target instance. The state vector includes the target's spatial position coordinates, velocity components, and motion direction information in the current time frame. Based on the state vector, a Kalman prediction model is used to calculate the target's predicted position in the next time frame, thus obtaining the target's predicted position vector. By calculating the Euclidean distance between the predicted position and the observed position in the current frame, the predicted position distance Dpos is obtained, which reflects the continuity of the target's motion trajectory.

[0030] Furthermore, an appearance feature vector is extracted for each target instance. This appearance feature vector is obtained through a deep feature extraction network, and the similarity of target features in adjacent video frames is calculated to obtain the appearance feature distance (Dapp). This appearance feature distance measures the similarity between two targets in their appearance features, helping to determine whether targets across frames belong to the same instance.

[0031] Furthermore, a workstation topology model is constructed based on the workstation layout structure of the flexible production line. This topology model describes the spatial adjacency relationships between workstations and the flow path of workpieces within the production line. Based on this topology model, an accessibility constraint matrix is ​​constructed. By determining whether the spatial position of the target between adjacent time frames conforms to the reachable path between workstations, the topology constraint cost Dtop is calculated to avoid erroneous matching of the target across unreachable workstations during trajectory association.

[0032] After obtaining the location prediction distance Dpos, appearance feature distance Dapp, and topology constraint cost Dtop, a target association cost function C is constructed. A comprehensive matching cost is formed by weighted fusion of these three types of costs. Here, α, β, and γ represent the weight coefficients of location constraints, appearance feature constraints, and topology constraints, respectively. A target association cost matrix is ​​constructed by calculating the cost function values ​​between different targets.

[0033] Subsequently, by minimizing the target association cost function C, the optimal target matching relationship is solved globally using the Hungarian matching algorithm, thereby achieving cross-frame association of multiple targets and generating continuous target motion trajectories. For the identified workpiece target, a workpiece trajectory TRKOBJ is generated; for the operator, a corresponding personnel trajectory TRKPER is generated.

[0034] As a preferred embodiment of the present invention, the path cycle time calculation module is used to derive the process path based on the relationship graph GPH and the process event flow EVT, including: Using the identifier UID as an index, the sequence of workstation nodes corresponding to the workpiece OBJ and their order relationship are extracted from the relationship graph GPH, and the single-piece process path PAT is reconstructed from the process event flow EVT. Based on the process path PAT, the flow time of workpiece OBJ between adjacent stations and the dwell time in a single station are calculated to generate the single-piece cycle time CYC1. Within the statistical window W, perform statistical analysis on the single-piece cycle time CYC1 of multiple workpieces OBJ, calculate the average cycle time and dispersion index corresponding to each station to generate the station cycle time CYC2.

[0035] Further calculations are made of the transit time of workpiece OBJ between each workstation and its dwell time within each workstation. Specifically, the transit time between workstations is obtained by calculating the difference between the departure time and arrival time of the workpiece between two adjacent workstation nodes. Simultaneously, the dwell time of the workpiece within that workstation is obtained by calculating the time difference between the completion time and arrival time within the same workstation. By summarizing the above time data, a single-piece production cycle CYC1 can be generated, representing the complete production cycle of a single workpiece from entering to leaving a workstation.

[0036] To reflect the overall operating efficiency of the production line, statistical analysis is performed on the single-piece cycle time (CYC1) of multiple workpieces within a preset statistical window W. This statistical window can be set according to a fixed number of workpiece samples or a fixed time interval. First, the cycle time data of different workpieces are grouped according to workstation nodes. Then, the mean, standard deviation, and dispersion index are calculated for the cycle time samples of the same workstation to reflect the average processing efficiency and production fluctuations of that workstation. Through statistical analysis of all workstations, the corresponding workstation cycle time (CYC2) is finally generated.

[0037] In a preferred embodiment of the present invention, the anomaly tracing and root cause diagnosis module is used to track abnormal workpieces, including: When the quality inspection station outputs a non-conforming mark, the process path PAT, process event flow EVT, station cycle time CYC2, and trajectory TRK of the workpiece OBJ are retrieved using the identifier UID as the index to construct the production process file of the workpiece OBJ; based on the file, the dwell time of the workpiece OBJ at each station, the operator PER, the equipment DEV, and the batch information are associated to form the product genealogy chain GEN. Diagnostic analysis of the product lineage chain GEN is performed based on preset anomaly detection rules, including: Perform node sequence matching on the process path PAT, calculate the path edit distance DIST, and determine the missing process anomaly when DIST exceeds the threshold THR1. Extract the arrival time, start time and completion time from the process event flow EVT, calculate the deviation DEV between the station dwell time TIM and the baseline cycle time BAS, and if DEV is greater than the threshold THR2, it is judged as an abnormal timeout. Rework behavior is identified by the number of repetitions and time intervals of workstation nodes in the process path PAT. If the number of repetitions exceeds the threshold THR3, it is judged as abnormal rework. Based on the spatial relationship between the personnel trajectory TRKPER and the equipment operation area and the duration of the interaction, personnel-equipment interaction events are extracted. If the interaction is missing or the duration is less than the threshold THR4, it is judged as an operation missing anomaly. The rate of change of beat (VAR) is calculated based on the single-piece beat (CYC1) of the workpiece OBJ. If the VAR exceeds the threshold THR5, it is determined to be an abnormal beat change.

[0038] Specifically, after the quality inspection station outputs a non-conforming mark, the process path PAT corresponding to the workpiece is first extracted from the product lineage chain GEN using the workpiece's unique identifier (UID) as an index. This path is then matched against a preset standard process path (STD). During the matching process, a path edit distance algorithm is used to calculate the node differences between the two paths. The edit distance DIST is used to characterize the structural deviation between the two process paths. When the edit distance DIST is greater than a preset threshold THR1, it is determined that the workpiece has a missing process abnormality during production.

[0039] Furthermore, the arrival time, start time, and completion time of the workpiece at each workstation are extracted from the process event flow (EVT), and the actual dwell time (TIM) of the workpiece at each workstation is calculated accordingly. The dwell time (TIM) is compared with the baseline cycle time (BAS) of the corresponding workstation to calculate the dwell time deviation (DEV). When the deviation (DEV) exceeds a preset threshold (THR2), it is determined that the workpiece has experienced a timeout at that workstation.

[0040] Furthermore, by traversing the sequence of workstation nodes in the process path PAT, the number of times the same workstation node appears in the path is counted, and rework behavior is identified by combining the time interval between adjacent nodes. When the number of times the same workstation node appears repeatedly exceeds the preset threshold THR3, it is determined that the workpiece has undergone abnormal rework during the production process.

[0041] Furthermore, personnel operation behavior is identified based on the spatial proximity between the personnel trajectory TRKPER and the equipment operation area. Specifically, by detecting whether the personnel trajectory enters the equipment operation area and calculating the continuous dwell time of the personnel in that area, personnel-equipment interaction events are extracted. When it is detected that no personnel-equipment interaction event occurs at the corresponding equipment station during the processing stage, or the interaction duration is lower than the minimum operation duration threshold THR4, it is determined that there is an operation missing anomaly at that station.

[0042] Furthermore, the production rhythm of the workpiece is analyzed based on the single-piece cycle time CYC1. Specifically, the rate of change (VAR) of the single-piece cycle time CYC1 between consecutive workpieces OBJ is calculated to reflect the fluctuation of the production cycle time. When the rate of change (VAR) exceeds the preset threshold THR5, it is determined that an abnormal change in cycle time has occurred in the production process.

[0043] In a preferred embodiment of the present invention, the anomaly tracing and root cause diagnosis module is used to track abnormal workpieces, and further includes: When the quality inspection station outputs a non-conforming mark, the workpiece OBJ marked as non-conforming is used as the center. Based on the matching conditions of the same batch, related samples are extracted from the historical production database to form a candidate sample set. Based on the final quality result, the candidate samples are divided into a normal sample set NOR and an abnormal sample set ABN. Statistical feature values ​​of the normal sample set NOR and the abnormal sample set ABN are extracted in terms of feature dimensions such as processing time, waiting time, workstation dwell time, personnel interaction frequency, continuous equipment running time, equipment switching times and rework times. The deviation between abnormal samples and normal samples is calculated and expressed by the difference in standard deviation. The deviation of each feature dimension is mapped to the contribution rate CON of each candidate factor to the abnormal results, and the root cause ranking results RCU are generated in descending order of contribution rate. Extract the process path PAT of multiple abnormal workpieces OBJ within the preset time window WIN, analyze the overlapping segments of the paths, and determine whether there are common equipment DEV, operator PER, or workstation dwell intervals. If there are duplicate nodes in multiple abnormal paths, mark the node as a source of risk RSK.

[0044] Specifically, multiple production characteristic parameters are extracted from the normal sample set (NOR) and the abnormal sample set (ABN), including processing time, waiting time, workstation dwell time, personnel interaction frequency, continuous equipment operation time, equipment changeover times, and rework times. For each characteristic dimension, its statistical characteristic value in the normal sample set (NOR) and its statistical characteristic value in the abnormal sample set (ABN) are calculated, and the degree of deviation between the two types of samples is further calculated. In this embodiment, the degree of deviation is quantified by the difference in the standard deviation of the corresponding features of the two types of samples, thereby obtaining the degree of change of each characteristic dimension under abnormal conditions relative to normal conditions.

[0045] Furthermore, after obtaining the deviation degree of each feature dimension, the deviation degree is mapped to the contribution rate CON of the candidate factors to the abnormal results. Specifically, by normalizing the deviation degree of each feature dimension and combining it with the frequency of occurrence of the feature in the abnormal samples, a weighted calculation is performed on each candidate factor to obtain the contribution rate of each candidate factor to the abnormal results. The candidate factors are ranked from high to low according to the contribution rate CON to generate the root cause ranking result RCU, thereby identifying the key factors with the greatest impact on the formation of anomalies.

[0046] Furthermore, to identify potential risk factors, the process paths (PATs) corresponding to multiple abnormal workpieces (OBJ) are extracted within a preset time window (WIN), and path overlap segment analysis is performed on each abnormal path. Specifically, by comparing the station node sequences in different abnormal workpiece paths, the equipment DEV, operator PER, or station dwell interval that repeatedly appear in multiple abnormal paths are identified. When a node is jointly contained by multiple abnormal paths within the time window, that node is marked as a systemic risk source (RSK).

[0047] As a preferred embodiment of the present invention, the production status assessment module assesses the production status of the flexible production line, including: Read the current workstation cycle time CYC2 and the baseline cycle time BAS, and calculate the cycle time deviation using the following formula: ; The Dcyc is fused with the top N root cause contribution rates CON and risk sources RSK in the root cause ranking result RCU to calculate the workstation risk score Rj, using the following formula: ; Among them, w1, w2, and w3 are weighting coefficients; Based on the workstation risk score Rj and the duration of the abnormality, the production line status is divided into three levels: normal, warning, and abnormal, and the status level STA is output. Based on the primary root cause type in the RCU, corresponding action instructions (OPTs) are generated, including: when the primary root cause is equipment-related, outputting equipment maintenance and load transfer alarms; when the primary root cause is personnel operation-related, outputting personnel verification and workstation reassignment alarms; and when the primary root cause is process path-related, outputting path correction alarms.

[0048] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0049] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0050] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A flexible production line process status monitoring system based on computer vision technology, characterized in that, include: The visual perception module acquires production video (VID) from the flexible production line and performs target detection and instance segmentation on the target area to identify workpieces (OBJ), equipment (DEV), and operators (PER) in the flexible production line. The trajectory analysis module extracts motion path data based on the spatial position changes of workpiece OBJ and operator PER in continuous video frames to form trajectory TRK; and extracts the spatiotemporal correlation between each target object based on the trajectory TRK to construct a relationship graph GPH, wherein the relationship graph GPH is used to describe the flow relationship of workpiece OBJ between different equipment DEVs and the interaction relationship between operator PER and equipment DEV. The path cycle time calculation module is used to perform path reasoning based on the relationship graph GPH to generate process path PAT, and calculate cycle time CYC based on the process path PAT; The anomaly tracing and root cause diagnosis module is used to track abnormal workpieces. When a quality anomaly is detected, a product lineage chain GEN is constructed using the identifier UID as an index, and the cause of the anomaly is determined based on rule judgment and historical sample comparison analysis, generating the root cause ranking result RCU. The production status assessment module is used to assess the production process status of the flexible production line based on the cycle time (CYC), identify bottleneck workstations based on cycle time deviation, and generate status level (STA) and alarm information when cycle time deviation or production efficiency decline is detected.

2. The flexible production line process status monitoring system based on computer vision technology according to claim 1, characterized in that, The visual perception module is used to acquire video from the flexible production line, including: Fixed-view acquisition terminals are deployed at key workstations on the production line to acquire production video VIDs, and lens intrinsic parameter calibration and distortion correction are performed on the production video VIDs. The production video VID is processed with illumination equalization, reflection suppression and noise filtering to obtain the standard image IMG; Based on the standard image IMG, target detection and instance segmentation are performed on the dynamic region to identify target instances and extract their appearance features, spatial location, posture information and timestamps to form observation data OBS; the target instances include workpiece OBJ, equipment DEV and operator PER, and the workpiece OBJ is assigned an identifier UID.

3. The flexible production line process status monitoring system based on computer vision technology according to claim 1, characterized in that, The trajectory analysis module is used to reconstruct the workpiece flow process based on observation data (OBS), including: The observation data OBS is sorted in chronological order, and an association cost matrix is ​​constructed. The association cost matrix is ​​composed of a weighted average of location prediction bias, appearance feature similarity, and workstation topology constraints. Target association is performed based on the association cost matrix to generate workpiece trajectory TRKOBJ and personnel trajectory TRKPER. When the trajectory is interrupted, the trajectory is reconnected by combining target re-identification and accessibility constraints of adjacent workstations. Based on the workstation boundary, the workpiece trajectory TRKOBJ is analyzed for spatial landing point. Combined with speed change and dwell time features, workpiece arrival, start, completion, waiting and abnormal stop events are extracted. Based on the spatial relationship between the personnel trajectory TRKPER and the equipment operation area, the contact duration, and the motion characteristics, personnel-equipment interaction events are extracted. The process event flow (EVT) is generated by sequentially connecting workpiece events and interaction events in chronological order. A relational graph GPH is constructed based on the process event flow EVT. In the relational graph GPH, the workpiece OBJ, equipment DEV, personnel PER, and workstation nodes are graph nodes, and the flow, dwell, operation, and interaction relationships are graph edges. The start and end times, duration, interaction frequency, and path direction are recorded.

4. The flexible production line process status monitoring system based on computer vision technology according to claim 1, characterized in that, The trajectory analysis module employs a trajectory reconstruction algorithm based on multi-feature fusion in the execution of target association and trajectory construction, including: A state vector is constructed based on the target's spatial location, and the predicted location is calculated using a Kalman prediction model. The target's appearance feature vector is extracted, and the feature similarity between targets in adjacent frames is calculated. An accessibility constraint matrix is ​​constructed based on the flexible production line workstation topology to limit the spatial range where the target can appear in adjacent time intervals. A target association cost function C is then constructed. ； Where Dpos represents the location prediction distance; Dapp represents the appearance feature distance; Dtop represents the topological constraint cost; and α, β, and γ represent the corresponding weight coefficients. The cross-frame target matching relationship is determined by minimizing the target association cost function C, and the Hungarian matching algorithm is used to complete the global optimal association of multiple targets. When the trajectory is temporarily occluded or lost, the feature similarity between the historical feature vector and the current candidate target is calculated by the target re-identification network, and the trajectory is completed by combining the nearest neighbor station constraint to obtain the workpiece trajectory TRKOBJ and the personnel trajectory TRKPER.

5. The flexible production line process status monitoring system based on computer vision technology according to claim 1, characterized in that, The path cycle time calculation module is used to derive the process path based on the relationship graph GPH and the process event flow EVT, including: Using the identifier UID as an index, the sequence of workstation nodes corresponding to the workpiece OBJ and their order relationship are extracted from the relationship graph GPH, and the single-piece process path PAT is reconstructed from the process event flow EVT. Based on the process path PAT, the flow time of workpiece OBJ between adjacent stations and the dwell time in a single station are calculated to generate the single-piece cycle time CYC1. Within the statistical window W, perform statistical analysis on the single-piece cycle time CYC1 of multiple workpieces OBJ, calculate the average cycle time and dispersion index corresponding to each station to generate the station cycle time CYC2.

6. The flexible production line process status monitoring system based on computer vision technology according to claim 1, characterized in that, The anomaly tracing and root cause diagnosis module is used to track abnormal workpieces, including: When the quality inspection station outputs a non-conforming mark, the process path PAT, process event flow EVT, station cycle time CYC2, and trajectory TRK of the workpiece OBJ are retrieved using the identifier UID as the index to construct the production process file of the workpiece OBJ; based on the file, the dwell time of the workpiece OBJ at each station, the operator PER, the equipment DEV, and the batch information are associated to form the product genealogy chain GEN. Diagnostic analysis of the product lineage chain GEN is performed based on preset anomaly detection rules, including: Perform node sequence matching on the process path PAT, calculate the path edit distance DIST, and determine the missing process anomaly when DIST exceeds the threshold THR1. Extract the arrival time, start time and completion time from the process event flow EVT, calculate the deviation DEV between the station dwell time TIM and the baseline cycle time BAS, and if DEV is greater than the threshold THR2, it is judged as an abnormal timeout. Rework behavior is identified by the number of repetitions and time intervals of workstation nodes in the process path PAT. If the number of repetitions exceeds the threshold THR3, it is judged as abnormal rework. Based on the spatial relationship between the personnel trajectory TRKPER and the equipment operation area and the duration of the interaction, personnel-equipment interaction events are extracted. If the interaction is missing or the duration is less than the threshold THR4, it is judged as an operation missing anomaly. The rate of change of beat (VAR) is calculated based on the single-piece beat (CYC1) of the workpiece OBJ. If the VAR exceeds the threshold THR5, it is determined to be an abnormal beat change.

7. The flexible production line process status monitoring system based on computer vision technology according to claim 1, characterized in that, The anomaly tracing and root cause diagnosis module is used to track abnormal workpieces and also includes: When the quality inspection station outputs a non-conforming mark, the workpiece OBJ marked as non-conforming is used as the center. Based on the matching conditions of the same batch, related samples are extracted from the historical production database to form a candidate sample set. Based on the final quality result, the candidate samples are divided into a normal sample set NOR and an abnormal sample set ABN. Statistical feature values ​​of the normal sample set NOR and the abnormal sample set ABN are extracted in terms of feature dimensions such as processing time, waiting time, workstation dwell time, personnel interaction frequency, continuous equipment running time, equipment switching times and rework times. The deviation between abnormal samples and normal samples is calculated and expressed by the difference in standard deviation. The deviation of each feature dimension is mapped to the contribution rate CON of each candidate factor to the abnormal results, and the root cause ranking results RCU are generated in descending order of contribution rate. Extract the process path PAT of multiple abnormal workpieces OBJ within the preset time window WIN, analyze the overlapping segments of the paths, and determine whether there are common equipment DEV, operator PER, or workstation dwell intervals. If there are duplicate nodes in multiple abnormal paths, mark the node as a systemic risk source RSK.

8. The flexible production line process status monitoring system based on computer vision technology according to claim 1, characterized in that, The production status assessment module assesses the production status of the flexible production line, including: Read the current workstation cycle time CYC2 and the baseline cycle time BAS, and calculate the cycle time deviation: ； The Dcyc is fused with the top N root cause contribution rates CON and systemic risk sources RSK in the root cause ranking result RCU to calculate the workstation risk score Rj, using the following formula: ； Among them, w1, w2, and w3 are weighting coefficients; Based on the workstation risk score Rj and the duration of the abnormality, the production line status is divided into three levels: normal, warning, and abnormal, and the status level STA is output. Based on the primary root cause type in the RCU, corresponding action instructions (OPTs) are generated, including: when the primary root cause is equipment-related, outputting equipment maintenance and load transfer alarms; when the primary root cause is personnel operation-related, outputting personnel verification and workstation reassignment alarms; and when the primary root cause is process path-related, outputting path correction alarms.

Images