A production line production state identification method, device, equipment and medium

Abstract

The application discloses a production line production state recognition method, device, equipment and medium. The application belongs to the technical field of state supervision. The method comprises the following steps: acquiring a production line monitoring video; recognizing a work station area in the monitoring image frame; performing producer recognition and production material detection on the recognized work station area to obtain a producer in the work station area and production material associated with the producer; performing key point detection and action fitting on the producer in the work station area to determine an action sequence of the producer and determine an effective action duration of the producer; recognizing a kinematic state of the production material associated with the producer in the production line monitoring video; and performing production work station idle state recognition according to the effective action duration and the kinematic state, and triggering a production line idle state alarm. Through the above production line production state recognition method, the recognition cost can be reduced, the efficiency of production line production state recognition and the accuracy of the recognition result can be improved.

Description

Technical Field

[0001] This application belongs to the field of status monitoring technology, specifically relating to a method, device, equipment, and medium for identifying the production status of a production line. Background Technology

[0002] Production line status identification refers to the process of identifying the operational load of a production line at present or within a specific time period. By monitoring and analyzing the production line's operational load, the current busy / idle status of the production line can be determined, thereby assessing the utilization rate of the production line's capacity. This facilitates optimized production management, improved efficiency, and the allocation of production resources, enhancing the rationality of production line operation and maintenance. Consequently, production line status identification has gradually become a hot topic in the industry.

[0003] In related technologies, the identification of production line status usually utilizes a human-machine collaborative on-site observation method. This involves periodically acquiring operating parameters such as current, voltage, and temperature of equipment on the production line to analyze the operating status of each piece of equipment, manually recording the downtime of each piece of equipment, and determining the busy / idle status of the production line based on the downtime. Alternatively, the downtime of materials and producers can be determined through manual on-site observation to determine the busy / idle status of the production line.

[0004] However, the relevant technologies rely on manual analysis, which is inefficient in analyzing the production status of the production line. Furthermore, the acquisition of equipment operating parameters requires a large number of monitoring devices, and the analysis results of equipment operating status cannot accurately reflect the operating status of the production line. This results in high costs and inaccurate identification results for identifying the production status of the production line. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, equipment, and medium for identifying the production status of a production line. This method addresses the problems of low identification efficiency, high identification cost, and inaccurate identification results when using existing technologies to identify the production status of a production line. By detecting the producer and associated production materials in the workstation area identified in the production line monitoring image frame, the effective action duration of the producer is determined. Based on the effective action duration and the kinematic state of the production materials, the idle status of the production workstation is identified. This achieves the goal of automatically identifying the production line status without the need for equipment parameter detection devices or manual intervention, thereby reducing identification costs and improving the efficiency and accuracy of production line status identification.

[0006] In a first aspect, embodiments of this application provide a method for identifying the production status of a production line, the method comprising:

[0007] Obtain production line monitoring videos;

[0008] Identify the workstation area in the monitoring image frame, which is an image frame from the production line monitoring video;

[0009] Producer identification and production data detection are performed on the identified workstation areas to obtain the producers in the workstation areas and the production data associated with the producers.

[0010] Key point detection and motion fitting are performed on the producers in the workstation area to determine the motion sequence of the producers, and the effective motion duration of the producers is determined based on the motion sequence.

[0011] Identify the kinematic state of production materials associated with producers in production line monitoring videos;

[0012] Based on the effective action duration and kinematic state, the idle status of the production station is identified, and an idle status alarm for the production line is triggered.

[0013] This solution determines the effective action duration of producers by detecting producers and associated production materials in the workstation areas identified in the production line monitoring image frames. Based on the effective action duration and the kinematic state of the production materials, it identifies the idle status of the production workstations. This achieves the goal of automatically identifying the production line status without the need for equipment parameter detection devices or manual intervention, thereby reducing identification costs and improving the efficiency and accuracy of production line status identification.

[0014] Furthermore, producer identification and production data detection are performed on the identified workstation areas to obtain the producers in the workstation areas and the production data associated with the producers, including:

[0015] The association detection model is used to identify producers and detect production materials in the identified workstation areas, thereby obtaining the producers in the workstation areas and the production materials associated with the producers.

[0016] Among them, the association detection model is trained based on training samples, and production materials that are not associated with producers in the training samples have negative sample labels.

[0017] The beneficial effect of this solution is that by pre-training the association detection model with training samples bearing negative sample labels, the association detection model can be used to identify producers and production materials associated with producers. This can achieve the purpose of filtering out production materials in the workstation area that are not associated with producers, avoiding the influence of production materials from other workstations on the identification results of the current workstation's operating status, and improving the accuracy of the identification results of the operating status of each production workstation.

[0018] Furthermore, by using an association detection model to detect production materials, the production materials associated with producers are obtained, including:

[0019] The global attention mechanism in the association detection model is used to calculate the association weights between producers and production materials in the workstation area;

[0020] The production materials associated with the producer are determined based on the association weight.

[0021] The beneficial effect of this scheme is that by using the global attention mechanism in the association detection model to calculate the association weight between producers and production materials in the workstation area, the production materials associated with the producers can be determined, which can further improve the comprehensiveness of the identification of associated production materials and the accuracy of image recognition results.

[0022] Furthermore, keypoint detection and motion fitting are performed on the producers in the workstation area to determine the producer's motion sequence, including:

[0023] Read the coordinates of the torso joints and the hand key points from the key point detection results of the producers in the workstation area;

[0024] The coordinates of key points of the torso joints and key points of the hands are fused with the pre-built 3D model of the producer to obtain the instantaneous posture of the producer in the monitoring image frame.

[0025] The producer's action sequence is determined based on instantaneous posture.

[0026] The beneficial effect of this scheme is that by reading the coordinates of the trunk joint key points and the hand key points in the key point detection results, the key point coordinates are fused with the pre-built 3D model of the producer to obtain the instantaneous posture of the producer in the monitoring image frame, and the action sequence of the producer is determined. This can achieve the purpose of continuous modeling of the producer's action sequence, and improve the timeliness and accuracy of the recognition of the producer's action sequence.

[0027] Furthermore, after determining the producer's action sequence based on instantaneous pose, the method also includes:

[0028] Identify the production stage corresponding to the production line monitoring video;

[0029] Obtain the standard sequence of actions for the production phase;

[0030] The producer's action sequence is validated based on the standard action sequence.

[0031] If the producer's action sequence is found to be in violation based on the results of the normative verification, the effective action duration determination process is stopped and a production line violation alarm is issued.

[0032] The beneficial effects of this solution are that by identifying the production stage corresponding to the production line monitoring video to obtain the standard action sequence, the action sequence of the producer is verified for standardization. If the producer's action sequence violates the rules, the effective action duration determination process is stopped and an alarm for production line violation is issued. This can avoid production safety issues and subsequent misjudgments of effective action duration caused by producer action sequence violations, thereby improving the intelligence of production line production status monitoring and the accuracy of monitoring results.

[0033] Furthermore, the effective action duration of the producer is determined based on the action sequence, including:

[0034] Identify monitoring image frames in an action sequence that show changes in action;

[0035] The effective action duration of the producer is determined based on the timing information of the monitored image frames.

[0036] The beneficial effect of this scheme is that by identifying monitoring image frames with action changes in the action sequence, and determining the effective action duration of the producer based on the temporal information of the monitoring image frames, the steps for identifying effective actions can be simplified, and the efficiency of calculating the effective action duration can be improved.

[0037] Furthermore, identifying the kinematic state of production materials associated with the producer in the production line monitoring video includes:

[0038] Determine whether the production materials have changed position based on the position information of the production materials associated with the producer in multiple consecutive monitoring image frames of the production line monitoring video.

[0039] When the means of production do not change position, their kinematic state is determined to be a static state.

[0040] The beneficial effect of this scheme is that by determining whether the production materials have changed position based on their position information in multiple consecutive monitoring image frames, and if the production materials have not changed position, the kinematic state of the production materials is determined to be stationary, which can improve the efficiency and accuracy of identifying the kinematic state of the production materials.

[0041] Secondly, embodiments of this application provide a device for identifying the production status of a production line, the device comprising:

[0042] The video acquisition module is used to acquire monitoring videos of the production line;

[0043] The workstation recognition module is used to monitor the workstation area in the image frame, which is an image frame in the production line monitoring video.

[0044] The image recognition module is used to identify producers and detect production materials in the identified workstation area, thereby obtaining the producers in the workstation area and the production materials associated with the producers.

[0045] The motion fitting module is used to perform key point detection and motion fitting on the producers in the workstation area, determine the motion sequence of the producers, and determine the effective motion duration of the producers based on the motion sequence.

[0046] The kinematic state recognition module is used to identify the kinematic state of production materials associated with the producer in the production line monitoring video;

[0047] The status alarm module is used to identify the idle status of the production station based on the effective action duration and kinematic state, and trigger the production line idle status alarm.

[0048] This solution determines the effective action duration of producers by detecting producers and associated production materials in the workstation areas identified in the production line monitoring image frames. Based on the effective action duration and the kinematic state of the production materials, it identifies the idle status of the production workstations. This achieves the goal of automatically identifying the production line status without the need for equipment parameter detection devices or manual intervention, thereby reducing identification costs and improving the efficiency and accuracy of production line status identification.

[0049] Furthermore, the image recognition module is specifically used for:

[0050] The association detection model is used to identify producers and detect production materials in the identified workstation areas, thereby obtaining the producers in the workstation areas and the production materials associated with the producers.

[0051] Among them, the association detection model is trained based on training samples, and production materials that are not associated with producers in the training samples have negative sample labels.

[0052] The beneficial effect of this solution is that by pre-training the association detection model with training samples bearing negative sample labels, the association detection model can be used to identify producers and production materials associated with producers. This can achieve the purpose of filtering out production materials in the workstation area that are not associated with producers, avoiding the influence of production materials from other workstations on the identification results of the current workstation's operating status, and improving the accuracy of the identification results of the operating status of each production workstation.

[0053] Furthermore, the image recognition module is specifically used for:

[0054] The global attention mechanism in the association detection model is used to calculate the association weights between producers and production materials in the workstation area;

[0055] The production materials associated with the producer are determined based on the association weight.

[0056] The beneficial effect of this scheme is that by using the global attention mechanism in the association detection model to calculate the association weight between producers and production materials in the workstation area, the production materials associated with the producers can be determined, which can further improve the comprehensiveness of the identification of associated production materials and the accuracy of image recognition results.

[0057] Furthermore, the motion fitting module is specifically used for:

[0058] Read the coordinates of the torso joints and the hand key points from the key point detection results of the producers in the workstation area;

[0059] The coordinates of key points of the torso joints and key points of the hands are fused with the pre-built 3D model of the producer to obtain the instantaneous posture of the producer in the monitoring image frame.

[0060] The producer's action sequence is determined based on instantaneous posture.

[0061] The beneficial effect of this scheme is that by reading the coordinates of the trunk joint key points and the hand key points in the key point detection results, the key point coordinates are fused with the pre-built 3D model of the producer to obtain the instantaneous posture of the producer in the monitoring image frame, and the action sequence of the producer is determined. This can achieve the purpose of continuous modeling of the producer's action sequence, and improve the timeliness and accuracy of the recognition of the producer's action sequence.

[0062] Furthermore, the device also includes:

[0063] The production stage identification module is used to identify the production stage corresponding to the production line monitoring video.

[0064] The standard action sequence acquisition module is used to acquire the standard action sequence during the production phase.

[0065] The normative verification module is used to perform normative verification on the producer's action sequence based on the standard action sequence;

[0066] The violation alarm module is used to stop the effective action duration determination process and issue a production line violation alarm when the producer's action sequence is determined to be in violation based on the results of normative verification.

[0067] The beneficial effects of this solution are that by identifying the production stage corresponding to the production line monitoring video to obtain the standard action sequence, the action sequence of the producer is verified for standardization. If the producer's action sequence violates the rules, the effective action duration determination process is stopped and an alarm for production line violation is issued. This can avoid production safety issues and subsequent misjudgments of effective action duration caused by producer action sequence violations, thereby improving the intelligence of production line production status monitoring and the accuracy of monitoring results.

[0068] Furthermore, the motion fitting module is specifically used for:

[0069] Identify monitoring image frames in an action sequence that show changes in action;

[0070] The effective action duration of the producer is determined based on the timing information of the monitored image frames.

[0071] The beneficial effect of this scheme is that by identifying monitoring image frames with action changes in the action sequence, and determining the effective action duration of the producer based on the temporal information of the monitoring image frames, the steps for identifying effective actions can be simplified, and the efficiency of calculating the effective action duration can be improved.

[0072] Furthermore, the kinematic state recognition module is specifically used for:

[0073] Determine whether the production materials have changed position based on the position information of the production materials associated with the producer in multiple consecutive monitoring image frames of the production line monitoring video.

[0074] When the means of production do not change position, their kinematic state is determined to be a static state.

[0075] The beneficial effect of this scheme is that by determining whether the production materials have changed position based on their position information in multiple consecutive monitoring image frames, and if the production materials have not changed position, the kinematic state of the production materials is determined to be stationary, which can improve the efficiency and accuracy of identifying the kinematic state of the production materials.

[0076] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0077] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0078] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0079] In this embodiment, production line monitoring video is acquired; workstation areas in monitoring image frames are identified, where the monitoring image frames are image frames in the production line monitoring video; producer identification and production material detection are performed on the identified workstation areas to obtain the producers in the workstation areas and the production materials associated with the producers; key point detection and motion fitting are performed on the producers in the workstation areas to determine the action sequence of the producers, and the effective action duration of the producers is determined based on the action sequence; the kinematic state of the production materials associated with the producers in the production line monitoring video is identified; based on the effective action duration and kinematic state, the idle state of the production workstation is identified, and an idle state alarm for the production line is triggered. The above-described method for identifying the production line status solves the problems of low efficiency, high cost, and inaccurate results when using existing technologies to identify the production line status. By detecting the producer and associated production materials in the workstation area identified in the production line monitoring image frame, the effective action duration of the producer is determined. Based on the effective action duration and the kinematic state of the production materials, the idle status of the production workstation is identified. This method achieves the goal of automatically identifying the production line status without the need for equipment parameter detection devices or manual intervention, thereby reducing identification costs and improving the efficiency and accuracy of production line status identification. Attached Figure Description

[0080] Figure 1 This is a flowchart illustrating the method for identifying the production status of a production line provided in an embodiment of this application;

[0081] Figure 2 This is a schematic diagram of the identification of the producer's workstation area provided in this application;

[0082] Figure 3 This is a flowchart illustrating the method for identifying the production status of a production line provided in an embodiment of this application;

[0083] Figure 4 This is a schematic diagram provided in this application for identifying producers and related means of production;

[0084] Figure 5 This is a schematic diagram illustrating the determination of an action sequence provided in an embodiment of this application;

[0085] Figure 6 This is a schematic diagram illustrating the calculation of the instantaneous attitude of the producer provided in this application;

[0086] Figure 7This is a schematic diagram illustrating the determination of the producer's action sequence provided in this application;

[0087] Figure 8 This is a schematic diagram of the structure of the production line production status identification device provided in the embodiments of this application;

[0088] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0089] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0090] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0091] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0092] The following description, in conjunction with the accompanying drawings, details the method, apparatus, equipment, and medium for identifying the production status of a production line provided in this application, through specific embodiments and application scenarios.

[0093] In existing technologies, the identification of production line status typically utilizes a human-machine collaborative on-site observation method. This involves periodically acquiring operating parameters such as current, voltage, and temperature of equipment on the production line to analyze the operating status of each device, and manually recording the downtime of each device. The busy / idle status of the production line is determined based on the downtime, or the downtime of materials and producers is determined through manual on-site observation. However, these technologies rely on manual analysis, resulting in low efficiency in analyzing production line status. Furthermore, acquiring equipment operating parameters requires numerous monitoring devices, and the analysis results cannot accurately reflect the production line's operating status. This leads to problems such as high cost and inaccurate identification results for production line status identification.

[0094] This solution determines the effective action duration of producers by detecting producers and associated production materials in the workstation areas identified in the production line monitoring image frames. Based on the effective action duration and the kinematic state of the production materials, it identifies the idle status of the production workstations. This achieves the goal of automatically identifying the production line status without the need for equipment parameter detection devices or manual intervention, thereby reducing identification costs and improving the efficiency and accuracy of production line status identification.

[0095] Figure 1 This is a flowchart illustrating the production status identification method for a production line provided in an embodiment of this application. Figure 1 As shown, the specific steps include the following:

[0096] S101, acquire production line monitoring video.

[0097] Firstly, this solution can be used in scenarios involving monitoring the production status of a production line, particularly monitoring the action sequences of producers and the duration of idle time for production materials to automatically identify the production line's status. By detecting producers and associated production materials in the workstation areas identified in the production line monitoring image frames, the effective action duration of producers is determined. Based on the effective action duration and the kinematic state of the production materials associated with the producers, the idle status of the production workstations is identified. This achieves the goal of automatically identifying the production line's status without the need for equipment parameter detection devices or manual intervention, reducing identification costs and improving the efficiency and accuracy of production line status identification.

[0098] Based on the above usage scenarios, it is understood that the executing entity of this application can be an electronic device with image recognition and computing capabilities, such as a mobile phone, computer and other smart terminals, without further limitations.

[0099] Among them, the production line monitoring video can be video collected by cameras, including videos of producers and production materials on the production line.

[0100] In one embodiment, production line monitoring video can be obtained through a monitoring video transmission interface.

[0101] S102, identify the workstation area in the monitoring image frame, where the monitoring image frame is an image frame in the production line monitoring video.

[0102] The monitoring image frame can be an image frame from the production line monitoring video. A workstation area can be the area where a producer repeatedly operates while performing production tasks. The workstation area includes the producer and production materials, and one workstation area corresponds to one producer.

[0103] In one embodiment, production line monitoring video can be acquired through a monitoring video transmission interface and input into a pre-trained workstation area recognition model. The workstation area recognition model is then used to identify the workstation area in each frame of the production line monitoring video.

[0104] Figure 2 This is a schematic diagram of the identification of the producer's workstation area provided in this application.

[0105] like Figure 2 As shown, video stream data of all workstation areas on the production line can be acquired in real time from the monitoring equipment network. The convolutional neural network in the workstation area recognition model extracts image features from each frame of the monitoring image data. The extracted results are then input into a bounding box regression structure to predict the range of the workstation area in the monitoring image frame. During the model training phase, data augmentation techniques such as image rotation, blurring, and scaling are used on the training samples of the monitoring image frames to improve the generalization ability of the workstation area recognition model and address the issue of handling small sample data.

[0106] S103, perform producer identification and production data detection on the identified workstation area to obtain the producers in the workstation area and the production data associated with the producers.

[0107] Production materials can be raw materials and / or tools on the production line. Production materials associated with a producer can be those processed or used by the producer in the workstation area. A monitoring image in a workstation area may simultaneously include production materials from the producer in that workstation area and production materials from producers in other workstation areas. To avoid identifying production materials from other workstations and causing subsequent errors in production status identification, this solution only identifies production materials associated with the current producer in the workstation area.

[0108] In one embodiment, image recognition technology can be used to identify producers and production materials in each workstation area based on the contour features of the human body and the production materials, thus obtaining the producers and production materials in the workstation area image. Due to the shooting angle of the production line monitoring video camera, the workstation area image may contain image data of production materials unrelated to the producers. In this case, the distance between the production materials and the producers can be calculated based on the position coordinates of each production material in the workstation area and the position coordinates of the producers in the workstation area. Production materials with a distance greater than a preset association distance threshold are then filtered to obtain the production materials associated with the producers. The preset association distance threshold can be determined based on the activity range of the producers or the coverage area of ​​the workstation area on the production line. Production materials unrelated to the producers can be production materials in other workstation areas.

[0109] In one embodiment, a pre-trained model can also be used to perform image segmentation and feature extraction on the workstation area image, extract global contextual features of the workstation area image, and identify the producers in the workstation area and the production materials associated with the producers.

[0110] This solution combines video surveillance and location awareness technology. By structuring the video space, it associates the camera's field of view with the location awareness grid. By leveraging strong prior knowledge of the production line and work area, it achieves precise division of workstation areas and personnel positioning, breaking through the limitations of traditional video surveillance that relies solely on image analysis, and it does not depend on manually labeled data.

[0111] S104, perform key point detection and motion fitting on the producers in the workstation area, determine the motion sequence of the producers, and determine the effective motion duration of the producers based on the motion sequence.

[0112] Key points can be points located on the producer's body that reflect the producer's movement characteristics. Examples include the head, shoulders, elbows, hands, and knees. Motion fitting involves determining the producer's real-time movements based on the positions of these key points. For example, a first line can be drawn connecting the shoulder and waist key points, and a second line connecting the knee and waist key points. The angle between these two lines can be determined based on the coordinates of each key point, and this angle can be used to simulate the movements of the producer's upper and lower body to determine if the producer is bending over. The producer's motion sequence can be a sequential combination of changes in the producer's posture over time. This sequence includes the producer's posture in each monitoring image frame. Since each monitoring image frame has a corresponding monitoring time point in the production line monitoring video, the producer's motion sequence can include the monitoring time point corresponding to each posture. Effective actions are those performed by the producer while carrying out production tasks. Examples include picking up tools, picking up parts, and tightening screws. The duration of an effective action is the duration for which the producer performs an effective action.

[0113] In one embodiment, a key point monitoring algorithm can be used to detect key points of the producers in the workstation area, and motion fitting can be performed on the producers based on the key point detection results. The motion fitting results are combined according to the order of the monitoring image frames to determine the motion sequence of the producers. Based on the motion sequence, the corresponding actions of the producers can be determined, and it can be identified whether the actions are valid. Then, the effective action duration of the producers can be determined based on the monitoring time point corresponding to each posture in the action sequence of valid actions.

[0114] This solution integrates 3D pose estimation with dynamic material tracking for collaborative analysis. It improves the accuracy of pose estimation through key point fusion algorithms, enabling 3D pose calculation of hands and body joints. Simultaneously, it combines a material tracking module to monitor the movement trajectory and state changes of raw materials through sensors, forming a dynamic "human-material" correlation model to calculate the producer's action sequence in real time and reduce the time spent on video review.

[0115] In one embodiment, determining the effective action duration of a producer based on an action sequence includes: identifying monitoring image frames in the action sequence that show action changes; and determining the effective action duration of the producer based on the timing information of the monitoring image frames.

[0116] Among these, the monitoring image frames exhibiting changes in motion can be adjacent or consecutive monitoring image frames with different producer postures. The timing information of the monitoring image frames can include their order within the production line monitoring video and the corresponding monitoring time point.

[0117] In one embodiment, monitoring image frames with motion changes in the action sequence can be identified based on whether the postures of adjacent monitoring image frames in the action sequence change. The duration of the motion change is calculated based on the timing information of the monitoring image frames with motion changes, and this duration is determined as the effective action duration of the producer. This scheme determines whether the producer is in a stationary state by identifying whether there are motion changes in the producer, identifies actions with motion changes as effective actions, and determines the duration of the motion change as the effective action duration.

[0118] This solution simplifies the process of identifying effective actions and improves the efficiency of calculating effective action duration by identifying monitoring image frames with action changes in the action sequence and determining the effective action duration based on the timing information of the monitoring image frames.

[0119] S105, Identify the kinematic state of production materials associated with the producer in the production line monitoring video.

[0120] The kinematic state of production materials can be information describing whether the production materials move on the production line. The kinematic state can include both moving and stationary states.

[0121] In one embodiment, the corresponding positions of production materials at different timestamps in the production line monitoring video can be identified. Based on whether the corresponding positions of production materials at different timestamps are the same, it can be determined whether the production materials have moved, thus obtaining the kinematic state of the production materials associated with the producer.

[0122] S106, based on the effective action duration and kinematic state, identifies the idle status of the production station and triggers an idle status alarm for the production line.

[0123] The idle status of a production workstation can be defined as the workstation area being in a non-working state. The idle status of a production workstation can be determined based on whether the corresponding producer and / or production materials are in a static or waiting state. A production line idle status alarm can be used to inform production line monitoring personnel or terminals that the production line is currently idle. Production line monitoring personnel can analyze production line faults and resource allocation issues based on the real-time production status, improving the rationality and timeliness of production line scheduling and management.

[0124] In one embodiment, the ratio of effective action duration to the total monitoring duration of the production line monitoring video can be calculated. Whether the producer at the production station is idle is determined based on whether the ratio is greater than a preset effective duration percentage threshold. Whether the production materials at the production station are idle is determined based on whether their kinematic state is stationary. If the producer and / or the production materials associated with the producer are idle, a production line idle state alarm is triggered.

[0125] This solution continuously models the production line status based on the time-series information of the production line monitoring video, constructing a production status time-series analysis model. It dynamically tracks the continuity of producer actions and the flow rhythm of production materials. Based on the total monitoring time and effective action time of the production line monitoring video, it determines the proportion of effective action time and idle time for producers. It statistically analyzes producer actions during working hours, distinguishing between effective operation time and inactive body time. Furthermore, it statistically analyzes whether the idle time of production materials in the workstation area exceeds a preset threshold based on the kinematic state of the production materials. The busy / idle level of the production line is quantified by the proportion of effective action time and idle time, or the proportion of effective action time and total monitoring time, along with the idle time of production materials, providing effective reference data for production line monitoring and analysis. The comparison thresholds for the proportion of producer action time and the comparison thresholds for the idle time of production materials in this solution can be adaptively adjusted according to the production line type; no further restrictions are imposed here.

[0126] This solution utilizes widely deployed monitoring equipment in the production space to identify all work areas in real time, detecting producers and production materials. Through multi-source sensing fusion, high-precision attitude calculation, and continuous state modeling, it achieves a leap from "passive monitoring" to "proactive intelligent analysis." Its improvement lies in transforming discrete monitoring data into structured production knowledge, providing real-time, multi-dimensional decision-making support for lean management. Compared to existing methods, this solution uses only existing monitoring cameras on the production line. By performing continuous frame analysis (<100ms) on the video streams from these cameras, it boasts advantages such as low deployment cost, strong environmental adaptability, and real-time response capabilities, achieving a data processing latency of ≤200ms, meeting the dynamic adjustment needs of high-speed production lines. The model in this solution is adaptive, supports cross-industry migration, and can formulate production status judgment standards according to industry needs, enabling fully automated determination and threshold optimization of production line busy / idle status.

[0127] In one embodiment, identifying the kinematic state of production materials associated with the producer in a production line monitoring video includes: determining whether the production materials have undergone a positional change based on the positional information of the production materials associated with the producer in multiple consecutive monitoring image frames of the production line monitoring video; and determining that the kinematic state of the production materials is a stationary state if the production materials have not undergone a positional change.

[0128] In one embodiment, the position coordinates of the production materials associated with the producer in each monitoring image frame of the production line monitoring video can be obtained. By comparing whether the position coordinates in multiple consecutive monitoring image frames have changed, it can be determined whether the position of the production materials has changed during the production line monitoring process. If the position of the production materials has not changed, the kinematic state of the production materials is determined to be a stationary state.

[0129] This solution determines whether the production materials have changed position based on their position information in multiple consecutive monitoring image frames. If the production materials have not changed position, the kinematic state of the production materials is determined to be stationary, which can improve the efficiency and accuracy of identifying the kinematic state of the production materials.

[0130] The technical solution provided in this application embodiment acquires production line monitoring video; identifies workstation areas in monitoring image frames, where the monitoring image frames are image frames in the production line monitoring video; performs producer identification and production material detection on the identified workstation areas to obtain the producers in the workstation areas and the production materials associated with the producers; performs key point detection and motion fitting on the producers in the workstation areas to determine the action sequence of the producers, and determines the effective action duration of the producers based on the action sequence; identifies the kinematic state of the production materials associated with the producers in the production line monitoring video; and identifies the idle state of the production workstations based on the effective action duration and kinematic state, and triggers an idle state alarm for the production line. The above-described method for identifying the production line status solves the problems of low efficiency, high cost, and inaccurate results when using existing technologies to identify the production line status. By detecting the producer and associated production materials in the workstation area identified in the production line monitoring image frame, the effective action duration of the producer is determined. Based on the effective action duration and the kinematic state of the production materials, the idle status of the production workstation is identified. This method achieves the goal of automatically identifying the production line status without the need for equipment parameter detection devices or manual intervention, thereby reducing identification costs and improving the efficiency and accuracy of production line status identification.

[0131] Figure 3 This is a flowchart illustrating the production status identification method for a production line provided in an embodiment of this application. Figure 3 As shown, the specific steps include the following:

[0132] S301, acquire production line monitoring video.

[0133] S302, Identify the workstation area in the monitoring image frame, wherein the monitoring image frame is an image frame in the monitoring video of the production line.

[0134] S303, using an association detection model to identify producers and detect production materials in the identified workstation area, to obtain the producers in the workstation area and the production materials associated with the producers; wherein, the association detection model is trained based on training samples, and the production materials in the training samples that are not associated with the producers have negative sample labels.

[0135] The association detection model can be a reinforcement learning model capable of identifying producers and associated production materials within a workstation area. Training samples can be multi-frame monitoring images labeled with both producer and production material tags, used for image recognition training of the association detection model. Production materials unrelated to producers in the training samples can be production materials located in the current workstation area but belonging to other workstation areas. Negative sample labels can be labels used to provide negative feedback to the association detection model regarding the identification results of unrelated production materials. Negative sample labels can be used to label images of production materials unrelated to producers in the workstation area, reducing the association weights calculated by the association detection model during model training, allowing the model to directly output the detection results of production materials associated with producers.

[0136] In one embodiment, an image of the workstation area can be input into a pre-trained association detection model. This model automatically identifies producers and production materials within the workstation area, thus obtaining the producers and associated production materials. The association detection model in this solution is trained based on training samples, and production materials unrelated to producers in the training samples are labeled with negative samples.

[0137] In one embodiment, an association detection model is used to identify producers and production materials in the identified workstation area to obtain the producers in the workstation area and the production materials associated with the producers. This includes: using the encoder in the association detection model to perform global detection of producers and production materials in the identified workstation area to obtain the producers and production materials in the workstation area; using the global attention mechanism in the association detection model to calculate the association weight between producers and production materials in the workstation area; and determining the production materials associated with the producers based on the association weight.

[0138] Global detection can be an operation that performs a full-coverage analysis of the entire workstation area image. Global attention mechanism can be a computational method where the model dynamically calculates the dependencies between all elements during the processing of sequence data, thereby capturing global contextual information. In this scheme, the global attention mechanism can be a computational method used by the association detection model to determine the production materials associated with the producer by calculating the association relationship between the producer and production material blocks when identifying producers and production materials. Association weight can be data used to describe the degree of association between producers and production materials.

[0139] In one embodiment, the encoder in the association detection model can be used to perform image segmentation on the identified workstation area and position encoding on the segmented image blocks to extract global features of producers and production materials in the workstation area, thereby obtaining the producers and production materials in the workstation area. The global attention mechanism in the association detection model is used to calculate the association weight between producers and production materials in the workstation area, and the production materials with a high degree of association with the producers are determined based on the association weight.

[0140] Figure 4 This is a schematic diagram provided in this application for identifying producers and related means of production.

[0141] like Figure 4 As shown, this scheme can use ViT (Vision Transformer, a visual self-attention model) to identify producers and production materials. By employing a Transformer encoder structure, the image of the workstation area is divided into fixed-size image blocks, and the position of each image block in the workstation area image is encoded to provide global features for detecting producers and production materials. The encoder captures the global context of the workstation area image through a self-attention mechanism, improving the detection capability of small targets and dense scenes. Through the global attention mechanism, the association between producers and production materials is preserved, and the influence of other workstations or objects other than production materials in the production environment on the image recognition results is shielded, increasing the robustness of the algorithm.

[0142] This solution utilizes the global attention mechanism in the association detection model to calculate the association weights between producers and production materials in the workstation area, thereby determining the production materials associated with producers. This can further improve the comprehensiveness of the identification of associated production materials and the accuracy of image recognition results.

[0143] S304 performs key point detection and motion fitting on the producers in the workstation area to determine the motion sequence of the producers, and determines the effective motion duration of the producers based on the motion sequence.

[0144] S305 identifies the idle status of the production station based on the effective action duration and kinematic state, and triggers an idle status alarm for the production line.

[0145] The technical solution provided in this application pre-trains the association detection model using training samples with negative sample labels, and uses the association detection model to identify producers and production materials associated with producers. This achieves the purpose of filtering out production materials in the workstation area that are not associated with producers, avoiding the influence of production materials from other workstations on the identification results of the current workstation's operating status, and improving the accuracy of the identification results of the operating status of each production workstation.

[0146] Figure 5 This is a schematic diagram illustrating the determination of an action sequence provided in an embodiment of this application. For example... Figure 5 As shown, the specific steps include the following:

[0147] S501, read the coordinates of the torso joint key points and the hand key points from the key point detection results of the producer in the workstation area.

[0148] The trunk joint key points can be points on the producer's torso and limbs that can affect the producer's posture or movement recognition results. The hand key points can be points at the producer's hand joints that can affect the producer's hand movement recognition results. In this scheme, the preferred number of trunk joint key points is 20, and the preferred number of key points for each hand is 21. After key point detection of the producer in the workstation area, the detection results obtained in this scheme can include trunk joint key points and their corresponding coordinates, as well as hand key points and their corresponding coordinates.

[0149] In one embodiment, after identifying the producer in the workstation area, the image region corresponding to the producer can be separated, and a convolutional neural network can be used to detect key points of the torso joints and hands in the producer, and the coordinates of the torso joints and hands in the key point detection results can be read.

[0150] S502 integrates the coordinates of key points of the torso joints and key points of the hands with the pre-built 3D model of the producer to obtain the instantaneous posture of the producer in the monitoring image frame.

[0151] The fusion computation can be a calculation that adjusts a pre-built 3D model of the producer according to key point coordinates to reconstruct the producer's action posture in each frame of the monitoring image. The instantaneous posture can be the corresponding full-body posture of the producer in each frame of the monitoring image.

[0152] In one embodiment, the coordinates of key points of the torso joints and key points of the hands can be fused with a pre-built stereo model of the producer to regress the producer's body posture. The full-body posture of the producer in the monitoring image frame can be predicted by calculating the bending angles of each joint of the producer.

[0153] Figure 6 This is a schematic diagram of the calculation of the instantaneous attitude of the producer provided in this application.

[0154] like Figure 6As shown in the image, the two sets of white dots on the left are the trunk joint keypoints obtained from keypoint detection of the producer. These keypoints are connected according to human structural features, and the 2D pixel coordinates of each keypoint in the connection result are fused with a pre-constructed 3D model to regress the producer's body posture. By calculating the bending angles of each joint, the overall posture is predicted, resulting in a 3D model identical to the producer's posture. After keypoint detection of the producer's hands, the 2D pixel coordinates of the hand keypoints are fused with the aforementioned 3D model of the producer's body posture to obtain a 3D model with identical hand details and body posture.

[0155] S503 determines the producer's action sequence based on instantaneous posture.

[0156] In one embodiment, after obtaining the instantaneous posture of the producer in each monitoring image frame, the sequence of the producer's actions can be determined based on the timing information of the monitoring image frames in the production line monitoring video.

[0157] Figure 7 This is a schematic diagram of the sequence of producer actions provided in this application.

[0158] like Figure 7 As shown, after fusing the 2D pixel coordinates of key points in the producer with the pre-built 3D model to obtain the producer's instantaneous 3D pose, the instantaneous poses are combined and connected according to the order in which they appear to obtain a coherent sequence of actions of the producer.

[0159] In one embodiment, after determining the producer's action sequence based on instantaneous posture, the method further includes: identifying the production stage corresponding to the production line monitoring video; obtaining the standard action sequence of the production stage; performing a normative verification on the producer's action sequence based on the standard action sequence; and stopping the effective action duration determination process and issuing a production line violation alarm if the producer's action sequence is determined to be in violation based on the result of the normative verification.

[0160] The production stage refers to the process of dividing a product into multiple continuous, logically independent links or steps according to the technological flow. The standard action sequence can be a pre-defined set of precise actions required to complete the aforementioned production stage. The standard action sequence includes the instantaneous postures corresponding to instantaneous actions and the execution order between multiple instantaneous postures. Standardization verification is an operation that judges whether the producer's actual action sequence conforms to the requirements of the standard action sequence. The result of standardization verification includes whether the producer's actual action sequence conforms to the requirements of the standard action sequence or not. Production line violation alarms can be used to inform production line monitoring personnel or end-product producers of violations in their action sequences.

[0161] In one embodiment, the production stage corresponding to the production line monitoring video can be identified based on the processing status of the production workshop or production materials corresponding to the production line monitoring video, and a pre-stored standard action sequence corresponding to the production stage can be obtained. The standard action sequence may include the producer's standard joint movement trajectory, standard posture data, standard operation rhythm, standard tool operability, and standard consumption rate of production materials, etc. Each actual posture in the producer's action sequence is compared with the standard posture in the standard action sequence in real time to verify the producer's action standardization. If the producer's action sequence is found to be inconsistent with the standard action sequence in terms of order or posture, the result of the standardization verification is determined to be that the producer's actual action sequence does not meet the requirements of the standard action sequence. At this time, it can be determined that the producer's action sequence is in violation. To ensure the safety of the production line process and avoid subsequent misjudgments of the effective action duration, the process of determining the effective action duration needs to be stopped, and a production line violation alarm needs to be issued.

[0162] This solution obtains standard action sequences by identifying the production stage corresponding to the production line monitoring video, performs standardization verification on the producer's action sequences, and stops the effective action duration determination process and issues a production line violation alarm when the producer's action sequence violates regulations. This can avoid production safety issues and subsequent misjudgments of effective action duration caused by producer action sequence violations, and improves the intelligence of production line status monitoring and the accuracy of monitoring results.

[0163] This solution uses time series analysis and behavioral pattern learning to determine the busy / idle status of producers during production line operation. It continuously models the joint movement trajectories and operational rhythms of producers, and uses 3D posture estimation algorithms to identify subtle movements (such as tool operation standardization). Combined with dynamic material information, it achieves quantitative analysis of the production process, comprehensively judging work status and avoiding misjudgments based on instantaneous movements. Furthermore, this solution is scalable in terms of application scenarios and functions, adaptable to various industrial scenarios, and can provide extended functions such as safety and quality monitoring based on continuous high-precision movements. For example, posture estimation can detect violations (such as climbing not according to specifications); track the standardization of assembly movements, link them to material usage records, and locate quality defects; and generate targeted training programs based on posture data comparing the operational differences between new and experienced employees.

[0164] The technical solution provided in this application embodiment reads the coordinates of key points of the torso joints and key points of the hands in the key point detection results, and then fuses the key point coordinates with the pre-built 3D model of the producer to obtain the instantaneous posture of the producer in the monitoring image frame and determine the action sequence of the producer. This can achieve the purpose of continuously modeling the action sequence of the producer and improve the timeliness and accuracy of the recognition of the action sequence of the producer.

[0165] Figure 8 This is a schematic diagram of the production line production status identification device provided in an embodiment of this application. For example... Figure 8 As shown, it specifically includes the following:

[0166] Video acquisition module 801 is used to acquire monitoring videos of the production line;

[0167] The workstation identification module 802 is used to monitor the workstation area in the image frame, and the monitored image frame is the image frame in the production line monitoring video.

[0168] The image recognition module 803 is used to identify producers and detect production materials in the identified workstation area, so as to obtain the producers in the workstation area and the production materials associated with the producers.

[0169] The motion fitting module 804 is used to perform key point detection and motion fitting on the producers in the workstation area, determine the motion sequence of the producers, and determine the effective motion duration of the producers based on the motion sequence.

[0170] The kinematic state recognition module 805 is used to identify the kinematic state of production materials associated with the producer in the production line monitoring video.

[0171] The status alarm module 806 is used to identify the idle status of the production station based on the effective action duration and kinematic state, and trigger the production line idle status alarm.

[0172] Furthermore, the image recognition module 803 is specifically used for:

[0173] The association detection model is used to identify producers and detect production materials in the identified workstation areas, thereby obtaining the producers in the workstation areas and the production materials associated with the producers.

[0174] Among them, the association detection model is trained based on training samples, and production materials that are not associated with producers in the training samples have negative sample labels.

[0175] Furthermore, the image recognition module 803 is specifically used for:

[0176] The global attention mechanism in the association detection model is used to calculate the association weights between producers and production materials in the workstation area;

[0177] The production materials associated with the producer are determined based on the association weight.

[0178] Furthermore, the motion fitting module 804 is specifically used for:

[0179] Read the coordinates of the torso joints and the hand key points from the key point detection results of the producers in the workstation area;

[0180] The coordinates of key points of the torso joints and key points of the hands are fused with the pre-built 3D model of the producer to obtain the instantaneous posture of the producer in the monitoring image frame.

[0181] The producer's action sequence is determined based on instantaneous posture.

[0182] Furthermore, the device also includes:

[0183] The production stage identification module is used to identify the production stage corresponding to the production line monitoring video.

[0184] The standard action sequence acquisition module is used to acquire the standard action sequence during the production phase.

[0185] The normative verification module is used to perform normative verification on the producer's action sequence based on the standard action sequence;

[0186] The violation alarm module is used to stop the effective action duration determination process and issue a production line violation alarm when the producer's action sequence is determined to be in violation based on the results of normative verification.

[0187] Furthermore, the motion fitting module 804 is specifically used for:

[0188] Identify monitoring image frames in an action sequence that show changes in action;

[0189] The effective action duration of the producer is determined based on the timing information of the monitored image frames.

[0190] Furthermore, the kinematic state recognition module 805 is specifically used for:

[0191] Determine whether the production materials have changed position based on the position information of the production materials associated with the producer in multiple consecutive monitoring image frames of the production line monitoring video.

[0192] When the means of production do not change position, their kinematic state is determined to be a static state.

[0193] The technical solution provided in this application includes a video acquisition module for acquiring production line monitoring video; a workstation identification module for monitoring workstation areas in image frames, where the monitoring image frames are image frames from the production line monitoring video; an image recognition module for identifying producers and detecting production materials in the identified workstation areas to obtain the producers in the workstation areas and the production materials associated with them; a motion fitting module for detecting key points and fitting motions of producers in the workstation areas to determine the motion sequences of producers and the effective motion duration of producers based on the motion sequences; a kinematic state recognition module for identifying the kinematic state of the production materials associated with producers in the production line monitoring video; and a status alarm module for identifying the idle status of production workstations based on the effective motion duration and kinematic state, and triggering an idle status alarm for the production line. The aforementioned production line production status identification device can solve the problems of low identification efficiency, high identification cost, and inaccurate identification results when using existing technologies to identify the production line production status. By detecting the producer and associated production materials in the workstation area identified in the production line monitoring image frame, the effective action duration of the producer is determined. Based on the effective action duration and the kinematic state of the production materials, the idle status of the production workstation is identified. This achieves the goal of automatically identifying the production line production status without the need for setting up equipment parameter detection devices or manual intervention, thereby reducing identification costs and improving the efficiency and accuracy of production line production status identification.

[0194] The production line status identification device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0195] The production line status identification device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0196] The production line production status identification device provided in this application embodiment can realize the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.

[0197] Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 9 As shown, this application embodiment also provides an electronic device 900, including a processor 901, a memory 902, and a program or instructions stored in the memory 902 and executable on the processor 901. When the program or instructions are executed by the processor 901, they implement the various processes of the above-described production line production status identification method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0198] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0199] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described method for identifying the production status of a production line and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0200] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0201] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described production line production status identification method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0202] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0203] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0204] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0205] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0206] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. A method for identifying the production status of a production line, characterized in that, The method includes: Obtain production line monitoring videos; Identify the workstation area in the monitoring image frame, wherein the monitoring image frame is an image frame in the production line monitoring video; Producer identification and production data detection are performed on the identified workstation areas to obtain the producers in the workstation areas and the production data associated with the producers. Key point detection and motion fitting are performed on the producers in the workstation area to determine the motion sequence of the producers, and the effective motion duration of the producers is determined based on the motion sequence. Identify the kinematic state of the production materials associated with the producer in the production line monitoring video; Based on the effective action duration and the kinematic state, the idle status of the production station is identified, and an idle status alarm for the production line is triggered.

2. The method according to claim 1, characterized in that, The process of identifying producers and detecting production data in the identified workstation areas to obtain the producers in the workstation areas and the production data associated with the producers includes: The association detection model is used to identify producers and detect production materials in the identified workstation areas to obtain the producers in the workstation areas and the production materials associated with the producers. The association detection model is trained based on training samples, in which production materials unrelated to the producer are labeled with negative samples.

3. The method according to claim 2, characterized in that, The process of using an association detection model to identify producers and detect production materials in the identified workstation areas yields the producers in the workstation areas and the production materials associated with those producers, including: The encoder in the association detection model is used to perform global detection of producers and production materials in the identified workstation area to obtain the producers and production materials in the workstation area. The global attention mechanism in the association detection model is used to calculate the association weights between producers and production materials in the workstation area; The production materials associated with the producer are determined based on the association weight.

4. The method according to claim 1, characterized in that, Key point detection and motion fitting are performed on the producers in the workstation area to determine the motion sequence of the producers, including: Read the coordinates of the torso joints and the coordinates of the hands from the results of key point detection of the producers in the workstation area. The coordinates of the key points of the torso joints and the key points of the hands are fused with the pre-constructed 3D model of the producer to obtain the instantaneous posture of the producer in the monitoring image frame. The producer's action sequence is determined based on the instantaneous posture.

5. The method according to claim 4, characterized in that, After determining the producer's action sequence based on the instantaneous posture, the method further includes: Identify the production stage corresponding to the production line monitoring video; Obtain the standard action sequence for the production stage; The producer's action sequence is validated based on the standard action sequence. If the producer's action sequence is found to be in violation based on the results of the normative verification, the effective action duration determination process is stopped and a production line violation alarm is issued.

6. The method according to claim 1, characterized in that, Determining the effective action duration of the producer based on the action sequence includes: Identify monitoring image frames in the action sequence that exhibit action changes; The effective action duration of the producer is determined based on the timing information of the monitored image frames.

7. The method according to claim 1, characterized in that, The identification of the kinematic state of the production materials associated with the producer in the production line monitoring video includes: Based on the position information of the production materials associated with the producer in multiple consecutive monitoring image frames of the production line monitoring video, it is determined whether the position of the production materials has changed; When the production materials do not change position, the kinematic state of the production materials is determined to be a stationary state.

8. A device for identifying the production status of a production line, characterized in that, The device includes: The video acquisition module is used to acquire monitoring videos of the production line. The workstation identification module is used to monitor the workstation area in the image frame, wherein the monitoring image frame is an image frame in the production line monitoring video; The image recognition module is used to identify producers and detect production materials in the identified workstation area to obtain the producers in the workstation area and the production materials associated with the producers. The motion fitting module is used to perform key point detection and motion fitting on the producers in the workstation area, determine the motion sequence of the producers, and determine the effective motion duration of the producers based on the motion sequence. A kinematic state recognition module is used to identify the kinematic state of the production materials associated with the producer in the production line monitoring video; The status alarm module is used to identify the idle status of the production station based on the effective action duration and the kinematic state, and trigger an idle status alarm for the production line.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method for identifying the production status of a production line as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method for identifying the production status of a production line as described in any one of claims 1-7.

Images