Method, system and device for collecting and recognizing production plant behavior

Abstract

The application discloses a kind of production workshop behavior acquisition and identification method, system and equipment, by modeling processing to the physical entity of workshop, create workshop digital twin, utilize workshop digital twin real simulation workshop production flow and operation;Through multi-angle camera, the multi-view video of each station in workshop is obtained, the operator on station is identified, the personnel digital twin of operator is created, the actual operation action of operator in workshop is simulated through personnel digital twin, the actual position of each operator is accurately positioned, and the behavior identification result of personnel digital twin is identified, finally realize the comparison judgment between the behavior identification result of each operator in workshop and the behavior requirement of station preset, determine whether the behavior identification result of operator on each station meets requirement.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition technology, and is particularly applicable to methods, systems and equipment for collecting and recognizing behavior in production workshops. Background Technology

[0002] The rapid development of the manufacturing industry has placed higher demands on the intelligence and automation levels of production workshops. To better monitor and manage the production process, various sensors (such as cameras, infrared sensors, and sound sensors) are deployed in the production workshop to collect real-time information on the behavior of personnel, equipment, and materials. By processing and analyzing the collected image data, it is possible to identify and assess aspects such as personnel behavior, equipment status, and material flow within the workshop.

[0003] Currently, single-point cameras are commonly used to collect video data of workers during workshop behavior recognition. This video data is then analyzed to identify worker behavior. However, due to the insufficient coverage and limited field of view of single-point cameras, the behavior recognition results from the collected video data are incomplete. For example, actions may be obscured, or blind spots may not be captured, leading to low accuracy in behavior recognition. Furthermore, the operational requirements of personnel at different workstations in different workshops vary, and data collected by single-point cameras is insufficient to accurately identify the behavior of personnel at different workstations. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and equipment for collecting and recognizing behaviors in production workshops, in order to solve the problem that the accuracy of behavior recognition in workshops is low and cannot meet the needs of different workstations in different workshops.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The method for collecting and identifying behaviors in a production workshop according to the present invention includes the following steps:

[0007] S1. Create a 3D model of the workshop based on the basic attribute data of the workshop entity, and map and associate dynamic production data with the 3D model of the workshop to obtain a digital twin of the workshop.

[0008] S2, acquire multi-view videos of each workstation in the workshop, extract personnel targets at each workstation, and obtain personnel information for each workstation;

[0009] S3, create digital twins of personnel at each workstation based on personnel information at each workstation; load the digital twins of personnel at each workstation into the workshop digital twin to obtain the workshop production digital twin;

[0010] S4, In the digital twin of the workshop production, perform behavior recognition on the digital twins of personnel at each workstation to obtain the behavior recognition results of personnel at each workstation;

[0011] S5, based on the preset behavioral requirements at each workstation, determines whether the behavior recognition results of personnel at each workstation are abnormal; if so, it generates an abnormal alarm and marks and records the abnormal behavior recognition results.

[0012] Furthermore, in step S3, the step of creating digital twins of personnel at each workstation based on personnel information at each workstation includes:

[0013] S3.1, based on the convolutional neural network algorithm, extract skeleton features from multi-angle videos of the workstation to obtain several skeleton nodes of the personnel;

[0014] S3.2, Create a digital twin of a person based on several skeleton nodes of the person, and create a tag for the digital twin of the person based on the person information; the person information includes at least: identity information, job position and job responsibilities.

[0015] Furthermore, step S4 specifically includes,

[0016] S4.1, based on the digital twin of workshop production, construct the global coordinate system of the workshop and the sub-coordinate system of each workstation in the workshop;

[0017] S4.2, Determine the global position of each person's digital twin in the workshop global coordinate system;

[0018] S4.3, Determine the sub-coordinate system where each person's digital twin is located based on the global position of each person's digital twin;

[0019] S4.4, Extract the skeleton data of the digital twin of the person in the sub-coordinate system where the digital twin of the person is located; the skeleton data includes: skeleton nodes and the coordinates of the skeleton nodes in the sub-coordinate system;

[0020] S4.5, acquire motion data of the digital twin of the person, and construct joint point dataset, bone length dataset and core joint point dataset based on the skeleton data and motion data of the digital twin of the person;

[0021] S4.6 Input the joint point dataset, bone length dataset and core joint point dataset into the pre-trained behavior recognition network for feature recognition to obtain the first recognition result corresponding to the joint point dataset, the second recognition result corresponding to the bone length dataset and the third recognition result corresponding to the core joint point dataset.

[0022] S4.7, Based on the first identification result, the second identification result and the third identification result, determine the behavior identification result of the digital twin of the person.

[0023] Furthermore, the motion data includes multiple motion frames of the digital twin of the person; the keypoint dataset includes the skeleton nodes of the digital twin of the person in multiple motion frames, the coordinates of each skeleton node, and the coordinate changes of skeleton nodes between adjacent frames; the bone length dataset includes the bone length between skeleton nodes of the digital twin of the person calculated from the coordinates of each skeleton node, and the coordinate changes of skeleton nodes between adjacent frames; the core keypoint dataset construction method includes:

[0024] S4.5.1 Extract several core joints from several skeleton nodes of the personnel digital twin;

[0025] S4.5.2, Determine the coordinates of the core joints and the changes in the coordinates of the core joints in adjacent frames;

[0026] S4.5.3, the core joint dataset is constructed from the coordinates of the core joints, the coordinate changes of the core joints, and the bone lengths between the core joints.

[0027] Furthermore, the behavior recognition network includes a first recognition unit and a second recognition unit connected in series; the first recognition unit includes multiple graph convolutional modules connected in series; the second recognition unit includes multiple temporal convolutional modules and a fusion module connected in parallel, and the outputs of the multiple parallel temporal convolutional modules are simultaneously used as the inputs of the fusion module.

[0028] Furthermore, step S4.7 specifically includes,

[0029] S4.7.1, based on the entropy weight method, weights are assigned to the first identification result, the second identification result, and the third identification result to obtain the first weight corresponding to the first identification result, the second weight corresponding to the second identification result, and the third weight corresponding to the third identification result;

[0030] S4.7.2, weighted summation, to obtain the behavior recognition results of the digital twin of the person.

[0031] Furthermore, the core joints include hand joints, knee joints, and spinal joints.

[0032] Furthermore, the workshop entity mentioned in step S1 includes equipment, tools, instruments, and safety facilities; the basic attribute data of the workshop entity includes identification information, technical specifications, status information, and location information.

[0033] The present invention discloses a system for collecting and recognizing behaviors in a production workshop, and a method for implementing the aforementioned method for collecting and recognizing behaviors in a production workshop, comprising:

[0034] The first construction module is used to acquire dynamic production data of the physical entity of the workshop and construct a digital twin of the workshop based on the dynamic production data of the physical entity of the workshop.

[0035] The target extraction module is used to acquire multi-view videos of each workstation in the workshop, extract targets from the multi-view videos, and obtain personnel information at each workstation.

[0036] The second construction module is used to create digital twins of personnel at each workstation based on the personnel information at each workstation, and load the digital twins of personnel at each workstation into the workshop digital twin to obtain the workshop production digital twin.

[0037] The behavior recognition module is used to identify the behavior of the digital twins of personnel at each workstation in the workshop production digital twin, and obtain the behavior recognition results of the personnel at each workstation.

[0038] The behavior analysis module is used to determine whether the behavior recognition results of personnel at each workstation are abnormal based on the preset behavior requirements at each workstation; if so, it generates an abnormal alarm and marks and records the abnormal behavior recognition results.

[0039] The present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for collecting and recognizing behaviors in the production workshop.

[0040] The advantages of this invention lie in its ability to create a digital twin of the workshop by modeling its physical entity, realistically simulating the production process and operations. By acquiring multi-view videos of each workstation using multi-angle cameras, the invention identifies the operators at each workstation, creating digital twins of each operator. These digital twins realistically simulate the operators' actual actions in the workshop, accurately locating each operator's position. The invention also identifies the behavioral recognition results of these digital twins, ultimately comparing them with the preset behavioral requirements of each workstation to determine if the results meet the requirements. This invention utilizes multi-angle cameras to acquire multi-view videos and combines this with a big data human skeleton feature extraction model to solve the problem of low accuracy in workshop personnel behavior recognition. Furthermore, based on digital twin technology, by pre-setting behaviors for each workstation within the digital twin system, it can meet the needs of recognizing the operational behaviors of personnel at different workstations in different workshops. Attached Figure Description

[0041] Figure 1 This is a flowchart of the method for collecting and recognizing behaviors in the production workshop as described in this invention.

[0042] Figure 2 This is a block diagram of the production workshop behavior collection and recognition system described in this invention. Detailed Implementation

[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, the method for collecting and identifying behaviors in a production workshop according to the present invention includes the following steps:

[0045] S1. Obtain the basic attribute data of the physical entity of the workshop, and create a 3D model of the workshop based on the basic attribute data of the workshop entity. At the same time, map and associate the dynamic production data of the physical entity of the workshop with the 3D model of the workshop to obtain a digital twin of the workshop.

[0046] The physical entities in a workshop mainly include equipment and machinery (e.g., injection molding machines, stamping machines, lathes, milling machines, grinding machines, welding machines, printing machines, etc., as well as auxiliary equipment such as handling robots, automated assembly lines, conveyor belts, elevators, and storage racks), tools and instruments (e.g., electronic scales, optical measuring instruments, chromatographs, etc., measuring tools and testing instruments), and safety facilities (e.g., protective covers, safety doors, emergency stop buttons, guardrails, etc., fire extinguishers, fire hydrants, etc., prohibition signs, warning signs, instruction signs, etc.). The basic attribute data of these physical entities mainly includes identification information, technical specifications, status information, and location information.

[0047] The modeling tools for 3D models can be various computer-aided design software such as CAD that can be used to build 3D models.

[0048] Dynamic production data can be various real-time or near real-time data related to the physical entities in the workshop. This data reflects the current state, operating status, and various parameters of the physical entities during the production process; for example:

[0049] Equipment status data includes, but is not limited to, information such as whether the equipment is running, whether it is in a faulty state, and whether it needs maintenance;

[0050] Production process data: including but not limited to parameters such as temperature, pressure, and cooling time of the injection molding machine, and the number of stampings and stamping force of the stamping machine;

[0051] Material flow data: including but not limited to the type, quantity, and speed of materials on the conveyor belt;

[0052] Environmental monitoring data includes, but is not limited to, workshop temperature, humidity, and air quality.

[0053] Robot motion data: including but not limited to the current position, speed, and target position of the transport robot.

[0054] S2: Acquire multi-view videos of each workstation in the workshop, extract personnel targets at each workstation, and obtain personnel information for each workstation. Specifically, cameras can be installed in different locations in the workshop to capture videos from different angles, or the same camera can be rotated to capture videos from different angles; angles can include front, side, back, etc., to ensure that the movements and postures of personnel can be fully captured.

[0055] S3: Create digital twins of personnel at each workstation based on their information; load these digital twins into the workshop digital twin to obtain the workshop production digital twin.

[0056] Creating digital twins of personnel at each workstation based on their information includes the following steps:

[0057] S3.1, based on a convolutional neural network algorithm, extracts skeleton features from personnel in multi-angle videos of various workstations, obtaining several skeleton nodes for each worker. Specifically, the acquired multi-angle videos are first preprocessed, including denoising, correction, and standardization, to improve the accuracy and efficiency of subsequent processing. A convolutional neural network architecture, such as ResNet or VGG, is then selected to process the video frames and generate feature maps. These feature maps contain key information from the video, and the skeleton nodes of the personnel, including joints and limb endpoints, are located on these feature maps to describe their posture and movements.

[0058] S3.2, create a digital twin of the person based on several skeleton nodes of the person, and create tags for the digital twin of the person based on the person's information. The person's information includes at least: identity information, job position, and job responsibilities.

[0059] S4. In the digital twin of the workshop production, behavior recognition is performed on the digital twins of personnel at each workstation to obtain the behavior recognition results for each personnel at each workstation. Specifically, this includes the following steps:

[0060] S4.1, based on the digital twin of the workshop production, constructs a global coordinate system for the workshop and sub-coordinate systems for each workstation. Specifically, it defines the overall spatial scope of the workshop and establishes a three-dimensional global coordinate system. Based on the actual location and layout of each workstation, it divides the global coordinate system into sub-coordinate system regions for each workstation, determines the reference points and directions of the sub-coordinate systems, and ensures accurate relative relationships with the global coordinate system. This provides a unified spatial reference framework, facilitating unified management of the entire workshop. The establishment of sub-coordinate systems helps to more accurately locate and analyze activities at each workstation, and is beneficial for subsequent data integration and cross-workstation analysis.

[0061] S4.2, Determine the global position of each person's digital twin in the workshop global coordinate system. Real-time position information of personnel is captured using sensors, cameras, and other devices, and this captured position information is then converted to the workshop global coordinate system.

[0062] S4.3, Based on the global position of each person's digital twin, determine the sub-coordinate system in which each person's digital twin is located, providing basic data for subsequent sub-coordinate system positioning and behavior analysis.

[0063] S4.4 Extract the skeleton data of the personnel digital twin within the sub-coordinate system where the personnel digital twin resides. The skeleton data includes skeleton nodes and their coordinates within the sub-coordinate system. Transforming the skeleton node data to the sub-coordinate system for subsequent processing reduces data processing complexity, improves computational efficiency, and makes data analysis in the sub-coordinate system more targeted and precise, facilitating in-depth analysis of behavior at specific workstations.

[0064] S4.5 Acquire motion data from the digital twin of the personnel. This motion data includes multiple motion frames of the digital twin, i.e., multi-view videos of various workstations in the workshop. Based on the skeleton data and motion data of the digital twin, construct a joint point dataset, a bone length dataset, and a core joint point dataset. Extract joint point coordinates from the skeleton data in different motion states to construct a joint point dataset; calculate the distance between adjacent joint points to construct a bone length dataset; identify and extract core joint points crucial for behavior recognition to construct a core joint point dataset.

[0065] The keypoint dataset includes the skeleton nodes of the digital twins of people in multiple moving images, the coordinates of each skeleton node, and the coordinate changes of skeleton nodes between adjacent images.

[0066] The bone length dataset includes the bone length between skeleton nodes of the human digital twin calculated from the coordinates of each skeleton node, as well as the coordinate change of skeleton nodes between adjacent frames.

[0067] The core joints include hand joints, knee joints, and spinal nodes. The core joint data set is derived by extracting several core joints from several skeletal nodes of a digital twin of a person; then, the coordinates of these core joints and the coordinate changes of core joints within adjacent frames are determined; and the bone length information between core joints is determined based on their coordinates. The coordinates of the core joints, the coordinate changes of the core joints, and the bone lengths between the core joints are then combined to form the core joint dataset.

[0068] By constructing multi-dimensional datasets (joint point dataset, bone length dataset, and core joint point dataset), rich basic information is provided for behavior recognition. The bone length dataset reflects the inherent characteristics of human body structure, which helps to improve recognition accuracy. The core joint point dataset focuses on key movement parts, simplifying the data processing process.

[0069] S4.6 Input the joint point dataset, bone length dataset, and core joint point dataset into the pre-trained behavior recognition network for feature recognition, and obtain the first recognition result corresponding to the joint point dataset, the second recognition result corresponding to the bone length dataset, and the third recognition result corresponding to the core joint point dataset.

[0070] This invention comprises three behavior recognition networks. The first behavior recognition network takes a joint point dataset as input and outputs a first recognition result. The second behavior recognition network takes a bone length dataset as input and outputs a second recognition result. The third behavior recognition network takes a core joint point dataset as input and outputs a third recognition result.

[0071] Each behavior recognition subnetwork includes a first recognition unit and a second recognition unit. The output of the first recognition unit serves as the input of the second recognition unit. The first recognition unit includes multiple cascaded graph convolutional modules. The second recognition unit includes multiple parallel temporal convolutional modules and a fusion module. The outputs of the multiple parallel temporal convolutional modules simultaneously serve as the input of the fusion module.

[0072] S4.7, Based on the first identification result, the second identification result and the third identification result, determine the behavior identification result of the digital twin of the person.

[0073] This invention employs the entropy weighting method to assign weights to the first, second, and third identification results, obtaining a first weight corresponding to the first identification result, a second weight corresponding to the second identification result, and a third weight corresponding to the third identification result. The first identification result is then weighted and summed with its first weight, the second identification result with its second weight, and the third identification result with its third weight to obtain the behavioral identification result of the person's digital twin.

[0074] S5, based on the preset behavioral requirements at each workstation, determines whether the behavior recognition results of personnel at each workstation are abnormal; if so, it generates an abnormal alarm and marks and records the abnormal behavior recognition results.

[0075] like Figure 2 As shown, the system for collecting and recognizing behaviors in a production workshop according to the present invention includes a method for collecting and recognizing behaviors in a production workshop, comprising:

[0076] The first construction module is used to acquire dynamic production data of the physical entity in the workshop, and to construct a digital twin of the workshop based on the dynamic production data of the physical entity in the workshop.

[0077] The target extraction module is used to acquire multi-view videos of each workstation in the workshop, extract targets from the multi-view videos, and obtain personnel information at each workstation.

[0078] The second construction module is used to create digital twins of personnel at each workstation based on the personnel information at each workstation, and load the digital twins of personnel at each workstation into the workshop digital twin to obtain the workshop production digital twin.

[0079] The behavior recognition module is used to identify the behavior of the digital twins of personnel at each workstation in the workshop production digital twin, and obtain the behavior recognition results of the personnel at each workstation.

[0080] The behavior analysis module is used to determine whether the behavior recognition results of personnel at each workstation are abnormal based on the preset behavior requirements at each workstation; if so, it generates an abnormal alarm and marks and records the abnormal behavior recognition results.

[0081] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for collecting and recognizing production workshop behavior.

Claims

1. A method for collecting and identifying behaviors in a production workshop, characterized in that, Includes the following steps: S1. Create a 3D model of the workshop based on the basic attribute data of the workshop entity, and map and associate the dynamic production data of the workshop entity with the 3D model of the workshop to obtain a digital twin of the workshop. S2, acquire multi-view videos of each workstation in the workshop, extract the personnel at each workstation from the multi-view videos, and obtain personnel information at each workstation; S3, create digital twins of personnel at each workstation based on personnel information at each workstation; load the digital twins of personnel at each workstation into the workshop digital twin to obtain the workshop production digital twin; S4, In the digital twin of the workshop production, perform behavior recognition on the digital twins of personnel at each workstation to obtain the behavior recognition results of personnel at each workstation; Step S4 specifically includes, S4.1, based on the digital twin of workshop production, construct the global coordinate system of the workshop and the sub-coordinate system of each workstation in the workshop; S4.2, Determine the global position of each person's digital twin in the workshop global coordinate system; S4.3, Determine the sub-coordinate system where each person's digital twin is located based on the global position of each person's digital twin; S4.4, Extract the skeleton data of the digital twin of the person in the sub-coordinate system where the digital twin of the person is located; the skeleton data includes: skeleton nodes and the coordinates of the skeleton nodes in the sub-coordinate system; S4.5, acquire motion data of the digital twin of the person, and construct joint point dataset, bone length dataset and core joint point dataset based on the skeleton data and motion data of the digital twin of the person; S4.6 Input the joint point dataset, bone length dataset and core joint point dataset into the pre-trained behavior recognition network for feature recognition to obtain the first recognition result corresponding to the joint point dataset, the second recognition result corresponding to the bone length dataset and the third recognition result corresponding to the core joint point dataset. The behavior recognition network includes a first recognition unit and a second recognition unit connected in series; the first recognition unit includes multiple graph convolutional modules connected in series; the second recognition unit includes multiple temporal convolutional modules and a fusion module connected in parallel, and the outputs of the multiple parallel temporal convolutional modules are simultaneously used as the inputs of the fusion module. S4.7, Based on the first identification result, the second identification result, and the third identification result, determine the behavior identification result of the digital twin of the person; S5, based on the preset behavioral requirements at each workstation, determines whether the behavior recognition results of personnel at each workstation are abnormal; if so, it generates an abnormal alarm and marks and records the abnormal behavior recognition results.

2. The method for collecting and identifying behaviors in a production workshop according to claim 1, characterized in that: In step S3, creating digital twins of personnel at each workstation based on their information includes: S3.1, based on the convolutional neural network algorithm, extract skeleton features from multi-angle videos of workers at their workstations to obtain several skeleton nodes of the workers; S3.2, Create a digital twin of a person based on several skeleton nodes of the person, and create a tag for the digital twin of the person based on the person information; the person information includes at least: identity information, job position and job responsibilities.

3. The method for collecting and identifying behaviors in a production workshop according to claim 1, characterized in that: The motion data includes multiple motion frames of the digital twin of the person; the key point dataset includes the skeleton nodes of the digital twin of the person in multiple motion frames, the coordinates of each skeleton node, and the coordinate changes of the skeleton nodes between adjacent frames; the bone length dataset includes the bone length between the skeleton nodes of the digital twin of the person calculated from the coordinates of each skeleton node, and the coordinate changes of the skeleton nodes between adjacent frames. The method for constructing the core key point dataset includes: S4.5.1 Extract several core joints from several skeleton nodes of the personnel digital twin; S4.5.2, Determine the coordinates of the core joints and the changes in the coordinates of the core joints in adjacent frames; S4.5.3, the core joint dataset is constructed from the coordinates of the core joints, the coordinate changes of the core joints, and the bone lengths between the core joints.

4. The method for collecting and identifying behaviors in a production workshop according to claim 1, characterized in that: Step S4.7 specifically includes, S4.7.1, based on the entropy weight method, weights are assigned to the first identification result, the second identification result, and the third identification result to obtain the first weight corresponding to the first identification result, the second weight corresponding to the second identification result, and the third weight corresponding to the third identification result; S4.7.2, weighted summation, to obtain the behavior recognition results of the digital twin of the person.

5. The method for collecting and identifying behaviors in a production workshop according to claim 1, characterized in that: The core joints include hand joints, knee joints, and spinal joints.

6. The method for collecting and identifying behaviors in a production workshop according to claim 1, characterized in that: The workshop entity mentioned in step S1 includes equipment, tools, instruments, and safety facilities; the basic attribute data of the workshop entity includes identification information, technical specifications, status information, and location information.

7. A system for collecting and recognizing behaviors in a production workshop, the system being used to implement the method for collecting and recognizing behaviors in a production workshop as described in any one of claims 1-6, characterized in that, The system includes: The first construction module is used to acquire dynamic production data of the physical entity of the workshop and construct a digital twin of the workshop based on the dynamic production data of the physical entity of the workshop. The target extraction module is used to acquire multi-view videos of each workstation in the workshop, extract targets from the multi-view videos, and obtain personnel information at each workstation. The second construction module is used to create digital twins of personnel at each workstation based on the personnel information at each workstation, and load the digital twins of personnel at each workstation into the workshop digital twin to obtain the workshop production digital twin. The behavior recognition module is used to identify the behavior of the digital twins of personnel at each workstation in the workshop production digital twin, and obtain the behavior recognition results of the personnel at each workstation. The behavior analysis module is used to determine whether the behavior recognition results of personnel at each workstation are abnormal based on the preset behavior requirements at each workstation; if so, it generates an abnormal alarm and marks and records the abnormal behavior recognition results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for collecting and recognizing production workshop behavior as described in any one of claims 1-6.

Images