Workshop personnel violation action identification method and device fusing RGB and depth image information

Abstract

The invention provides a workshop personnel violation action recognition method and device fusing RGB and depth image information, and belongs to the technical field of personnel violation action recognition. The method comprises the steps of obtaining video streams, internal reference streams and external reference streams of an RGB camera and a depth camera, aligning RGB images in the RGB video streams to depth images in the depth video streams frame by frame, extracting portrait frames and skeleton sequence data of portraits in the aligned RGB images frame by frame, obtaining three-dimensional coordinates of double-shoulder center points in the skeleton sequence data in combination with the depth images, and obtaining the three-dimensional coordinates of the double-shoulder center points in the three-dimensional coordinates. And according to the portrait frame, the skeleton sequence data and the shoulder center point three-dimensional coordinates, extracting a plurality of statistical magnitudes, establishing an action rule model, converting the portrait frame, the skeleton sequence data, the shoulder center point three-dimensional coordinates and the statistical magnitudes into action types and confidence coefficients, and performing violation action judgment. The method does not need to collect a large number of images for training, and can recognize motion details.

Description

Technical Field

[0001] The present invention relates to the technical field of personnel illegal action recognition, and in particular to a method and device for recognizing illegal actions of workshop personnel by integrating RGB and depth image information. Background Art

[0002] The standardization of workshop workers' movements is the basis for ensuring safe production. At present, the standardization of movements mainly relies on the team leader to check the surveillance video or regularly visit the production site. However, manual duty will lead to a waste of labor resources, and it cannot guarantee that all illegal movements can be warned in time. Therefore, it is necessary to provide a solution that uses algorithms to automatically analyze the movements of workshop personnel from the surveillance video stream.

[0003] In recent years, human action recognition algorithms based on RGB video streams have made great progress and have been applied in human-computer interaction, motion reconstruction, autonomous driving and other fields. Among them, the recognition algorithm based on skeleton sequence is the most widely used because of its strong robustness, high maturity and fast processing speed. Skeleton sequence refers to the 2D or 3D coordinates of key human nodes (such as hips, knees, elbows and other joints) extracted from RGB video streams. Existing recognition algorithms based on skeleton sequence generally include two parts: skeleton sequence extraction and action recognition. First, skeleton sequences are extracted from video frames using large posture recognition models such as YOLOv8 model and Openpose model. Then, the skeleton sequence is used as the input of deep classification neural network to obtain relevant human actions. Large posture recognition models such as YOLOv8 can already be plug-and-play, and can achieve high accuracy without videos in specific scenarios as training data. The research on deep classification neural networks needs to be improved, and researchers have developed various deep learning models according to the needs of specific scenarios. For example, patent CN117612264A proposed an action recognition method based on graph convolutional neural network; patent CN117953586A proposed an action recognition method based on spatiotemporal trajectory graph (JTG, Joints Trajectory Graph); patents CN117011936A and CN116612533A respectively proposed an action recognition method based on transformer network. However, the training of these neural networks requires a large number of bone sequences as input, and operations such as collecting RGB video streams, converting them into bone sequences, and annotating action types will cost a lot of manpower and time, which is not conducive to application and promotion in workshops.

[0004] The types of illegal actions in the workshop are relatively simple and the scenarios are relatively fixed. Using rule models instead of deep learning models for calculations can greatly save data collection and training time. In addition, after new illegal actions are added, the rule model only needs to be simply modified to meet the monitoring needs again, without the need for retraining. Compared with deep learning models, it is more scalable.

[0005] On the other hand, the skeleton sequence extracted from the RGB video stream only focuses on the relative position relationship between the key nodes of the human body, ignoring the real spatial coordinates between these key nodes, which makes it impossible to be used to recognize some fine movements. Take sitting on a chair and sitting on a table as examples. The former is a normal action and the latter is an illegal action, but their skeleton sequence data are similar and cannot be distinguished based on the RGB video stream alone. Summary of the invention

[0006] In order to solve the technical problems existing in the prior art, the present invention provides a method and device for identifying illegal actions of workshop personnel by integrating RGB and depth image information, obtaining the video stream, internal reference stream and external reference stream of RGB camera and depth camera, aligning the RGB image in the RGB video stream to the depth image in the depth video stream frame by frame, extracting the portrait frame and skeleton sequence data of the portrait in the aligned RGB image frame by frame, combining the depth image, obtaining the three-dimensional coordinates of the center points of the shoulders in the skeleton sequence data, extracting several statistics according to the portrait frame, skeleton sequence data and the three-dimensional coordinates of the center points of the shoulders, establishing an action rule model, converting the portrait frame, skeleton sequence data, the three-dimensional coordinates of the center points of the shoulders and the statistics into action types and confidences, and making illegal action judgments. The present invention does not need to collect a large number of images for training, and can recognize action details at the same time.

[0007] The present invention provides a method for identifying illegal actions of workshop personnel by integrating RGB and depth image information, comprising the following steps:

[0008] Step 1), use the RGB camera to obtain RGB video stream, RGB internal reference stream and RGB external reference stream, use the depth camera to obtain depth video stream, depth internal reference stream and depth external reference stream; RGB internal reference stream and RGB external reference stream refer to the sequence formed by the internal reference matrix and external reference matrix of the RGB camera collected continuously within a time interval; the depth internal reference stream and the depth external reference stream refer to the sequence formed by the internal reference matrix and external reference matrix of the depth camera collected continuously within a time interval;

[0009] Step 2), according to the RGB internal reference stream, the RGB external reference stream, the depth internal reference stream and the depth external reference stream, align the RGB image in the RGB video stream to the depth image in the depth video stream frame by frame;

[0010] Step 3), using the YOLOv8-pose model to extract the portrait frame and skeleton sequence data of the portrait in the aligned RGB image frame by frame;

[0011] Step 4), combining the depth image, obtaining the three-dimensional coordinates of the center points of both shoulders in the bone sequence data;

[0012] Step 5), extracting several statistics based on the portrait frame, the skeleton sequence data and the three-dimensional coordinates of the center points of the shoulders, the statistics including: a Boolean statistic of whether there are all zero values ​​in the coordinate points of the parts involved in the human body movements, point coordinates, angles and distances;

[0013] Step 6), establish an action rule model, convert the portrait frame, skeleton sequence data, three-dimensional coordinates of the shoulder center points and the extracted statistics into a preset action type and its confidence level, and compare the action with the preset illegal action to determine the illegal action based on the converted action type and its confidence level.

[0014] Preferably, in step 1), the RGB camera and the depth camera start collecting data at the same time, and the collection frame rates of the RGB video stream, RGB internal reference stream, RGB external reference stream, depth video stream, depth internal reference stream and depth external reference stream are the same.

[0015] Preferably, in step 2), the RGB image is aligned to the depth image by pixel indexing.

[0016] Preferably, in step 2), the RGB image is aligned to the depth image by pixel index, specifically including:

[0017] Assume that p depth is the depth image Fr depth A pixel point on the depth (x),p depth (y)), the depth value is Z depth ;

[0018] Assume that p rgb is the RGB image Fr rgb The same pixel point on the rgb (x),p rgb (y)), p depth With p rgb The relationship between is expressed by the following equation:

[0019]

[0020] Where R = R rgb R cepth -1 , T = T rgb -R rgb R depth-1 T depth , The x, y, and z coordinates of the aligned RGB image are temporarily stored; traverse Fr depth For each pixel on Fr rgb The RGB image is aligned to the depth image, and the aligned RGB image is recorded as Fr rgb ′,Fr rgb ′ and Fr deptg Each pixel of has a one-to-one correspondence;

[0021] R rgb The external parameters of the RGB camera include a rotation matrix of size 3×3. The external parameters of the RGB camera also include a transfer vector T of length 3 rgb ;

[0022] H depth is the intrinsic parameter matrix of the depth camera;

[0023] R depth The external parameters of the depth camera include a rotation matrix of size 3×3; T depth The extrinsic parameters for the depth camera include a transfer vector of length 3.

[0024] Preferably, in step 3), the extracted portrait frame contains the aspect ratio HW_ratio, and the extracted skeleton sequence data includes 17 key nodes of the human body, namely, nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; these key nodes are in the aligned RGB image Fr rgb The coordinates on ′ are (x1, y1), (x2, y2), (x3, y3), (x0, y4), (x5, y5), (x6, y6), (x7, y7), (x8, y8), (x9, y9), (x 10 ,y 10 )、(x 11 ,y 11 )、(x 12 ,y 12 )、(x 13 ,y 13 )、(x 14 ,y 14 )、(x 15 ,y 15 )、(x 16 ,y 1b )、(x 17 ,y 17 ).

[0025] Preferably, in step 4), the center point of both shoulders is at Frrgb The coordinates (x shc ,y shc ) is expressed as:

[0026]

[0027] Use this coordinate to get the depth value in the depth image, recorded as Z depth (shc) ;

[0028] Taking the optical center of the depth camera as the reference origin, the real-world 3D coordinates of the center points of the shoulders are true (shc) The calculation formula is:

[0029]

[0030] p true (shc) (z) = Z depth (shc)

[0031] in,

[0032] Z depth (shc) =Fr depth ((x6+x7) / 2,(y6+y7) / 2);

[0033] p true (shc) (x), p true (shc) (y) and p true (shc) (z) are the three-dimensional coordinates p true (shc) The x value, y value and z value of

[0034] Among them, fx depth and fy depth are the focal lengths of the depth camera in the x and y directions, ppx depth and ppy depth They are the x-coordinate and y-coordinate of the optical center of the depth camera on the imaging plane, and the intrinsic parameter matrix of the depth camera is recorded as H depth , expressed as:

[0035]

[0036] Preferably, in step 5), a Boolean statistic Bool c Determine whether there are all zero values ​​in the left shoulder, right shoulder, left hip, right hip, left knee, and right knee coordinate points. If so, set it to false; if not, set it to true; use Boolean statistics Boola Determine whether there are all zero values ​​in the left ankle and right ankle coordinate points. If so, set it to false; if not, set it to true.

[0037] Preferably, in step 5), the point coordinate statistics are divided by the center point of the shoulders (x shc ,y shc ), the coordinates of the center points of the wrists, hips, knees, and ankles are calculated and recorded as (x wrc ,y wrc )、(x hic ,y hic )、(x knc ,y knc ) and (x anc ,y anc ), their calculation formulas are:

[0038]

[0039] Preferably, in step 5), the angle statistics include the angle between the shoulder-hip ray and the horizontal ray, the angle between the hip-knee line and the horizontal ray, the angle between the shoulder-hip-knee ray and the horizontal ray, and the angle between the hip-knee-ankle ray and the horizontal ray, which are respectively denoted as upper_angle, lower_angle, shk_angle and hka_angle; their calculation formulas are:

[0040]

[0041] Among them, x shc_hic =x shc -x hic , x knc_hic =x knc -x hic ,y shc_hic =y shc -y hic ,y knc_hic =y knc -y hic , x hic_knc =x hic -x knc , x anc_knc =x anc -x knc ,y hic_knc =y hic -y knc ,y anc_knc =y anc -yk nc .

[0042] Preferably, in step 5), the distance statistics include the distance between the shoulder and the hip in the y-axis direction and the distance between the wrist and the ankle in the y-axis direction, denoted as D shc_hic and D wrc_anc , and their calculation formulas are respectively:

[0043]

[0044] where x wrc_anc = x wrc - x anc , y wrc_anc = y wrc - y anc .

[0045] Preferably, the preset action types include: incomplete, standing, bending and standing, sitting on the ground, sitting on a chair, sitting on a table, crawling through equipment, and others; the violation actions include: sitting on the ground, sitting on a table, and crawling through equipment.

[0046] Preferably, in step 6), the determination and confidence calculation method for incomplete in the action type are as follows: When Bool c is false, the action is determined to be incomplete, and the confidence V incom of incomplete is set to 1.

[0047] Preferably, in step 6), the determination and confidence calculation method for standing are as follows:

[0048] First, when HW_ratio > 1.5, 70 < upper_angle ≤ 110, 70 < lower_angle ≤ 110, the action is determined to be standing;

[0049] Then, different confidence calculation methods are executed in different cases:

[0050] When Bool a = True, 150 < hka_angle ≤ 180, the calculation formula for the confidence V stand of standing is as follows:

[0051] V stand = 0.7 - |90 - lower_angle| * 0.02 + |180 - hka_angle| * 0.01

[0052] Otherwise, the calculation formula for the confidence V stand of standing is as follows:

[0053] V stand = 0.7 - |90 - lower_angle| * 0.02.

[0054] Preferably, in step 6), the determination and confidence calculation method for bending and standing are as follows:

[0055] First, when HW_ratio > 1.5, upper_angle ≤ 70 or upper_angle > 110, and 70 < lower_angle ≤ 110, the action is determined as bending and standing;

[0056] Then, different confidence calculation methods are executed in different cases:

[0057] When Bool a = True and 150 < hka_angle ≤ 180, the confidence V pro_stand of bending and standing is calculated as follows:

[0058] V pro_stand = 0.7 - |90 - lower_angle| * 0.02 + |180 - hka_angle| * 0.01

[0059] Otherwise, the confidence V pro _ stand of bending and standing is calculated as follows:

[0060] V pro_standx 0.7 - |90 - lower_angle| * 0.02,

[0061] Preferably, in step 6), sitting on the ground, sitting on a chair, and sitting at a table all belong to the large category of sitting postures, and the determination and confidence calculation methods are as follows:

[0062] First, when 70 < upper_angle ≤ 110, lower_angle ≤ 70 or lower_angle > 110, and 35 < shk_angle ≤ 170, the action is determined as the large category of sitting postures;

[0063] Then, each sitting posture has different detailed determination statements and confidence calculation methods:

[0064] When the additional condition is p true (shc) (y) > -300, it is determined as sitting on the ground, and the confidence V cross_sit of sitting on the ground is calculated as follows:

[0065]

[0066] When the additional condition is p true (shc) (y) ≤ -300, p true (shc)When (y) > -700, it is determined that the person is sitting on the chair, and different confidence calculation methods are executed according to different situations:

[0067] When the further additional condition Bool a = True, and 60 < hka_angle ≤ 150, the confidence level V of sitting on the chair normal_sit is calculated as follows:

[0068] V normal_sit = 0.7 - |90 - shk_angle| * 0.005 + (60 - |90 - shk_angle|) * 0.05

[0069] Otherwise, the confidence level V of sitting on the chair normal_sit is calculated as follows:

[0070] V normal_sit = 0.7 - |90 - shk_angle| * 0.005

[0071] When the additional condition is p true (shc) When (y) ≤ -700, it is determined that the person is sitting on the table, and different confidence calculation methods are executed according to different situations:

[0072] When the further additional condition Bool a = True, and 60 < hka_angle ≤ 150, the confidence level V of sitting on the table upper_sit is calculated as follows:

[0073] V upper_sit = 0.7 - |90 - shk_angle| * 0.005 + (60 - |90 - shk_angle|) * 0.05

[0074] Otherwise, the confidence level V of sitting on the table upper_sit is calculated as follows:

[0075] V upper_sit = 0.7 - |90 - shk_angle| * 0.005.

[0076] Preferably, in step 6), the determination and confidence calculation method for drilling through the device are as follows:

[0077] When HW ratio ≤ 1.5, upper_angle ≤ 70 or upper_angle > 110, p true (shc) (y) > -250, y wrc > y hic at this time, the action is determined to be drilling through the device, and the confidence level V of drilling through the devicekneel Set to 1.

[0078] Preferably, in step 6), actions that do not meet the above conditions are uniformly classified as others, and the confidence V of other actions is other Set to 1.

[0079] The present invention also provides a device for identifying illegal actions of workshop personnel by fusing RGB and depth image information, comprising a processor, which is capable of executing the steps of the above-mentioned method for identifying illegal actions of workshop personnel by fusing RGB and depth image information.

[0080] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0081] 1. The present invention uses a rule model instead of a deep learning model to process bone sequence data. It does not require large-scale collection of labeled data, which saves a lot of labor and time costs and is conducive to promotion in workshops. At the same time, when new illegal actions need to be expanded, only simple rule modifications are required to meet monitoring needs again without retraining. Compared with deep learning models, it is more scalable.

[0082] 2. The present invention integrates the information in RGB images and depth images, and integrates the skeleton sequence information and three-dimensional coordinate information, making it possible to recognize certain fine movements. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A flowchart of identifying illegal actions of workshop personnel by integrating RGB and depth image information according to an embodiment of the present invention;

[0084] Figure 2 (a) is an aligned RGB image according to an embodiment of the present invention, and (b) is a depth image according to an embodiment of the present invention;

[0085] Figure 3 An RGB image including a portrait frame of a workshop personnel according to an embodiment of the present invention;

[0086] Figure 4(a) is an incomplete image of an embodiment of the present invention, (b) is an image of a standing image of an embodiment of the present invention, (c) is an image of a standing bent image of an embodiment of the present invention, (d) is an image of a sitting on the ground of an embodiment of the present invention, (e) is an image of a sitting on a chair of an embodiment of the present invention, (f) is an image of a sitting on a table of an embodiment of the present invention, (g) is an image of a drilling device of an embodiment of the present invention, and (h) is an image of other actions of an embodiment of the present invention; each figure shows the recognized action type and confidence level, which can be seen at a glance;

[0087] Figure 5 This is a flowchart of identifying illegal actions of workshop personnel by fusing RGB and depth image information according to another embodiment of the present invention. DETAILED DESCRIPTION

[0088] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0089] like Figure 5 The present invention provides a method for identifying illegal actions of workshop personnel by integrating RGB and depth image information, comprising the following steps:

[0090] Step 1), use the RGB camera to obtain RGB video stream, RGB internal reference stream and RGB external reference stream, use the depth camera to obtain depth video stream, depth internal reference stream and depth external reference stream; RGB internal reference stream and RGB external reference stream refer to the sequence formed by the internal reference matrix and external reference matrix of the RGB camera collected continuously within a time interval; the depth internal reference stream and the depth external reference stream refer to the sequence formed by the internal reference matrix and external reference matrix of the depth camera collected continuously within a time interval;

[0091] Step 2), according to the RGB internal reference stream, the RGB external reference stream, the depth internal reference stream and the depth external reference stream, align the RGB image in the RGB video stream to the depth image in the depth video stream frame by frame;

[0092] Step 3), using the YOLOv8-pose model to extract the portrait frame and skeleton sequence data of the portrait in the aligned RGB image frame by frame;

[0093] Step 4), combining the depth image, obtaining the three-dimensional coordinates of the center points of both shoulders in the bone sequence data;

[0094] Step 5), extracting several statistics based on the portrait frame, the skeleton sequence data and the three-dimensional coordinates of the center points of the shoulders, the statistics including: a Boolean statistic of whether there are all zero values ​​in the coordinate points of the parts involved in the human body movements, point coordinates, angles and distances;

[0095] Step 6), establish an action rule model, convert the portrait frame, skeleton sequence data, three-dimensional coordinates of the shoulder center points and the extracted statistics into a preset action type and its confidence level, and compare the action with the preset illegal action to determine the illegal action based on the converted action type and its confidence level.

[0096] Confidence is a variable that describes the degree of certainty of action determination, and its value range is between 0 and 1. The closer to 0, the less certain the current action type is, and the closer to 1, the more certain the current action type is. The confidence is not less than 0 and not greater than 1. The present invention determines the action type independently for each frame, first determines the action type, and then assigns confidence to the action type.

[0097] According to a specific embodiment of the present invention, in step 1), the RGB camera and the depth camera start to collect data at the same time, and the RGB video stream, RGB internal reference stream, RGB external reference stream, depth video stream, depth internal reference stream and depth external reference stream have the same collection frame rate. The RGB video stream, RGB internal reference stream and RGB external reference stream are provided in real time by the RGB camera. The depth video stream, depth internal reference stream and depth external reference stream are provided in real time by the depth camera.

[0098] According to a specific embodiment of the present invention, in step 2), the RGB image is aligned to the depth image through pixel indexing.

[0099] According to a specific embodiment of the present invention, in step 2), the RGB image is aligned to the depth image by pixel index, specifically comprising:

[0100] Assume that p depth is the depth image Fr depth A pixel point on the depth (x),p deptn (y)), the depth value is Z d ep t h ;

[0101] Assume that p rgb is the RGB image Fr rgb The same pixel point on the rgb (x),p rgb (y)), p depth With p rgb The relationship between is expressed by the following equation:

[0102]

[0103] Where R = R rgb R depth -1 , T = T rgb -Rrgb R depth -1 T depth , The x, y, and z coordinates of the aligned RGB image are temporarily stored; traverse Fr depth For each pixel on Fr rgb The RGB image is aligned to the depth image, and the aligned RGB image is recorded as Fr rgb ′,Fr rgb ′ and Fr deptg Each pixel of has a one-to-one correspondence;

[0104] R rgb The external parameters of the RGB camera include a rotation matrix of size 3×3. The external parameters of the RGB camera also include a transfer vector T of length 3 rgb ;

[0105] H depth is the intrinsic parameter matrix of the depth camera;

[0106] R depth The external parameters of the depth camera include a rotation matrix of size 3×3; T depth The extrinsic parameters for the depth camera include a transfer vector of length 3.

[0107] According to a specific embodiment of the present invention, in step 3), the extracted portrait frame contains the aspect ratio HW_ratio, and the extracted skeleton sequence data includes 17 key nodes of the human body, namely, nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; these key nodes are in the aligned RGB image Fr rgb The coordinates on ′ are (x1, y1), (x2, y2), (x3, y3), (x4, y4), (x5, y5), (x6, y6), (x7, y7), (x8, y8), (x9, y9), (x 10 ,y 10 )、(x 11 ,y 11 )、(x 12 ,y 12 )、(x 13 ,y 13 )、(x 14 ,y 14 )、(x 15 ,y 15 )、(x 16 ,y 16 )、(x 17 ,y 17 ).

[0108] According to a specific embodiment of the present invention, in step 4), the center point of the shoulders is at Fr rgb The coordinates (x shc ,y shc ) is expressed as:

[0109]

[0110] Use this coordinate to get the depth value in the depth image, recorded as Z depth (shc) ;

[0111] Taking the optical center of the depth camera as the reference origin, the real-world 3D coordinates of the center points of the shoulders are true (shc) The calculation formula is:

[0112]

[0113] p true (shc) (z) = Z depth (shc)

[0114] in,

[0115] Z depth (shc) =F depth ((x6+x7) / 2,(y6+y7) / 2);

[0116] p true (shc) (x), p true (shc) (y) and p true (shc) (z) are the three-dimensional coordinates p true (shc) The x value, y value and z value of

[0117] Among them, fx depth and fy depth are the focal lengths of the depth camera in the x and y directions, ppx depth and ppy depth They are the x-coordinate and y-coordinate of the optical center of the depth camera on the imaging plane, and the intrinsic parameter matrix of the depth camera is recorded as H depth , expressed as:

[0118]

[0119] According to a specific embodiment of the present invention, in step 5), a Boolean statistic Bool cDetermine whether there are all zero values ​​in the left shoulder, right shoulder, left hip, right hip, left knee, and right knee coordinate points. If so, set it to false; if not, set it to true; use Boolean statistics Bool a Determine whether there are all zero values ​​in the left ankle and right ankle coordinate points. If so, set it to false; if not, set it to true.

[0120] According to a specific embodiment of the present invention, in step 5), the point coordinate statistics are divided by the center point of the shoulders (x shc ,y shc ), the coordinates of the center points of the wrists, hips, knees, and ankles are calculated and recorded as (x wrc ,y wrc )、(x hic ,y hic )、(x knc ,y knc ) and (x anc ,y anc ), their calculation formulas are:

[0121]

[0122] According to a specific embodiment of the present invention, in step 5), the angle statistics include the angle between the shoulder-hip ray and the horizontal ray, the angle between the hip-knee line and the horizontal ray, the angle between the shoulder-hip-knee ray and the horizontal ray, and the angle between the hip-knee-ankle ray and the horizontal ray, which are respectively denoted as upper_angle, lower_angle, shk_angle and hka_angle; their calculation formulas are:

[0123]

[0124] Among them, x shc_hic =x shc -x hic , x knc_hic =x knc -x hic ,y shc_hic =y shc -y hic ,y knc_hic =y knc -y hic , x hic_knc =x hic -x knc , x anc_knc =x anc -x knc ,y hic_knc =y hic -y knc ,y anc_knc =yanc -yk nc 。

[0125] According to a specific embodiment of the present invention, in step 5), the distance statistics include the distance between the shoulder and the hip in the y-axis direction and the distance between the wrist and the ankle in the y-axis direction, which are denoted as D shc_hic and D wrc_anc , and their calculation formulas are respectively:

[0126]

[0127] where x wrc_anc = x wrc - x anc , y wrc_anc = y wrc - y anc 。

[0128] According to a specific embodiment of the present invention, the preset action types include: incomplete, standing, bending and standing, sitting on the ground, sitting on a chair, sitting on a table, crawling through equipment, and others; the violation actions include: sitting on the ground, sitting on a table, and crawling through equipment.

[0129] According to a specific embodiment of the present invention, in step 6), the determination and confidence calculation method for incomplete in the action type are as follows: when Bool c is false, the action is determined to be incomplete, and the confidence of incompleteness V incom is set to 1.

[0130] According to a specific embodiment of the present invention, in step 6), the determination and confidence calculation method for standing are as follows:

[0131] First, when HW_ratio > 1.5, 70 < upper_angle ≤ 110, 70 < lower_angle ≤ 110, the action is determined to be standing;

[0132] Then, different confidence calculation methods are executed in different cases:

[0133] When Bool a = True, 150 < hka_angle ≤ 180, the calculation formula for the confidence of standing V stand is as follows:

[0134] V stand = 0.7 - |90 - lower_angle| * 0.02 + |180 - hka_angle| * 0.01

[0135] Otherwise, the calculation formula for the confidence of standing V stand is as follows:

[0136] V stand = 0.7 - |90 - lower_angle| * 0.02.

[0137] According to a specific embodiment of the present invention, in step 6), the determination and confidence calculation method for bending and standing are as follows:

[0138] First, when HW_ratio > 1.5, upper_angle ≤ 70 or upper_angle > 110, 70 < lower_angle ≤ 110, the action is determined as bending and standing;

[0139] Then, different confidence calculation methods are executed in different cases:

[0140] When Bool a = True, 150 < hka_angle ≤ 180, the confidence V pro_stand of bending and standing is calculated as follows:

[0141] V pro_stand = 0.7 - |90 - lower_angle| * 0.02 + |180 - hka_angle| * 0.01

[0142] Otherwise, the confidence V pro _ stand of bending and standing is calculated as follows:

[0143] V pro_stand = 0.7 - |90 - lower_angle| * 0.02.

[0144] According to a specific embodiment of the present invention, in step 6), sitting on the ground, sitting on a chair, and sitting at a table all belong to the sitting posture category, and the determination and confidence calculation methods are as follows:

[0145] First, when 70 < upper_angle ≤ 110, lower_angle ≤ 70 or lower_angle > 110, 35 < shk_angle ≤ 170, the action is determined as the sitting posture category;

[0146] Then, each sitting posture has different detailed determination statements and confidence calculation methods:

[0147] When the additional condition is ptrue (shc) (y) > -300, it is determined as sitting on the ground, and the confidence V cross_sit of sitting on the ground is calculated as follows:

[0148]

[0149] When the additional condition is p true (shc) (y) ≤ -300, p true (shc) (y) > -700, it is determined that the person is sitting on the chair, and different confidence calculation methods are executed according to different situations:

[0150] When the further additional condition Bool a = True, 60 < hka_angle ≤ 150, the confidence V of sitting on the chair normal_sit The calculation formula is as follows:

[0151] V normal_sit = 0.7 - |90 - shk_angle| * 0.005 + (60 - |90 - shk_angle|) * 0.05

[0152] Otherwise, the confidence V of sitting on the chair normal_sit The calculation formula is as follows:

[0153] V normal_sit = 0.7 - |90 - shk_angle| * 0.005

[0154] When the additional condition is p true (shc) (y) ≤ -700, it is determined that the person is sitting on the table, and different confidence calculation methods are executed according to different situations:

[0155] When the further additional condition Bool a = True, 60 < hka_angle ≤ 150, the confidence V of sitting on the table upper_sit The calculation formula is as follows:

[0156] V upper_sit = 0.7 - |90 - shk_angle| * 0.005 + (60 - |90 - shk_angle|) * 0.05

[0157] Otherwise, the confidence V of sitting on the table upper_sit The calculation formula is as follows:

[0158] V upper_si t = 0.7 - |90 - shk_angle| * 0.005.

[0159] According to a specific implementation of the present invention, in step 6), the determination and confidence calculation method of the drilling equipment are as follows:

[0160] When HW ratio ≤ 1.5, upper_angle ≤ 70 or upper_angle > 110, ptrue (shc) (y)>-250,y wrc >y hic When , the action is judged as drilling through the device, and the confidence level of drilling through the device is V kneel Set to 1.

[0161] According to a specific embodiment of the present invention, in step 6), the actions that do not meet the above conditions are uniformly classified as others, and the confidence V of other actions is other Set to 1.

[0162] The present invention also provides a device for identifying illegal actions of workshop personnel by fusing RGB and depth image information, comprising a processor, which is capable of executing the steps of the above-mentioned method for identifying illegal actions of workshop personnel by fusing RGB and depth image information.

[0163] Example 1

[0164] The present invention provides a method for identifying illegal actions of workshop personnel by integrating RGB and depth image information, comprising the following steps:

[0165] Step 1), use the RGB camera to obtain the RGB video stream, RGB internal reference stream and RGB external reference stream, and use the depth camera to obtain the depth video stream, depth internal reference stream and depth external reference stream;

[0166] Step 2), according to the RGB internal reference stream, the RGB external reference stream, the depth internal reference stream and the depth external reference stream, align the RGB image in the RGB video stream to the depth image in the depth video stream frame by frame;

[0167] Step 3), using the YOLOv8-pose model to extract the portrait frame and skeleton sequence data of the portrait in the aligned RGB image frame by frame;

[0168] Step 4), combining the depth image, obtaining the three-dimensional coordinates of the center points of both shoulders in the bone sequence data;

[0169] Step 5), extracting several statistics based on the portrait frame, the skeleton sequence data and the three-dimensional coordinates of the center points of the shoulders, the statistics including: a Boolean statistic of whether there are all zero values ​​in the coordinate points of the parts involved in the human body movements, point coordinates, angles and distances;

[0170] Step 6), establish an action rule model, convert the portrait frame, skeleton sequence data, three-dimensional coordinates of the shoulder center points and the extracted statistics into a preset action type and its confidence level, and compare the action with the preset illegal action to determine the illegal action based on the converted action type and its confidence level.

[0171] Example 2

[0172] The present invention provides a method for identifying illegal actions of workshop personnel by integrating RGB and depth image information, comprising the following steps:

[0173] Step 1), use the RGB camera to obtain the RGB video stream, RGB internal reference stream and RGB external reference stream, and use the depth camera to obtain the depth video stream, depth internal reference stream and depth external reference stream;

[0174] Step 2), according to the RGB internal reference stream, the RGB external reference stream, the depth internal reference stream and the depth external reference stream, align the RGB image in the RGB video stream to the depth image in the depth video stream frame by frame;

[0175] Step 3), use the YOLOv8-pose model (the YOLOv8-pose model is an open source large model provided by the Ultralytics machine learning platform) to extract the portrait frame and skeleton sequence data of the portrait in the aligned RGB image frame by frame;

[0176] Step 4), combining the depth image, obtaining the three-dimensional coordinates of the center points of both shoulders in the bone sequence data;

[0177] Step 5), extracting several statistics based on the portrait frame, the skeleton sequence data and the three-dimensional coordinates of the center points of the shoulders, the statistics including: a Boolean statistic of whether there are all zero values ​​in the coordinate points of the parts involved in the human body movements, point coordinates, angles and distances;

[0178] Step 6) establishes an action rule model, converts the portrait frame, skeleton sequence data, three-dimensional coordinates of the center points of the shoulders, and the extracted statistics into preset action types and their confidences, and compares the action with the preset illegal actions to determine the illegal actions based on the converted action types and their confidences. The preset action types include: incomplete, standing, standing with bent waist, sitting on the ground, sitting on a chair, sitting on a table, drilling through equipment, and others, and the illegal actions include: sitting on the ground, sitting on a table, and drilling through equipment.

[0179] Among them, in step 1), the RGB camera and the depth camera start to collect data at the same time, and the RGB video stream, RGB internal parameter stream, RGB external parameter stream, depth video stream, depth internal parameter stream and depth external parameter stream have the same collection frame rate. The frame rate should be controlled between 10 frames / s and 30 frames / s, which can not only ensure the continuity of the picture, but also meet the requirements of real-time processing of subsequent algorithms.

[0180] Among them, in step 2), the RGB image is aligned to the depth image through pixel index, specifically including:

[0181] Assume that p depth is the depth image Fr depth A pixel point on the depth (x),pdepth (y)), the depth value is Z depth ;

[0182] Assume that p rgb is the RGB image Fr rgb The same pixel point on the rgb (x),p rgb (y)), p depth With p rgb The relationship between is expressed by the following equation:

[0183]

[0184] Where R = R rgb R depth -1 , T = T rgb -R rgb R depth -1 T depth , The x, y, and z coordinates of the aligned RGB image are temporarily stored; traverse Fr depth For each pixel on Fr rgb The RGB image is aligned to the depth image, and the aligned RGB image is recorded as Fr rgb ′,Fr rgb ′ and Fr depth Each pixel of has a one-to-one correspondence;

[0185] R rgb The external parameters of the RGB camera include a rotation matrix of size 3×3. The external parameters of the RGB camera also include a transfer vector T of length 3 rgb ; The intrinsic parameter matrix of the RGB camera is recorded as H rgb , expressed as:

[0186]

[0187] where fx rgb and fy rgb are the focal lengths of the RGB camera in the x and y directions, ppx rgb and ppy rgb They are the x-coordinate and y-coordinate of the optical center of the RGB camera (i.e. the center point of the camera lens) on the imaging plane.

[0188] H depth is the intrinsic parameter matrix of the depth camera;

[0189] R depth The external parameters of the depth camera include a rotation matrix of size 3×3; T depthThe extrinsic parameters for the depth camera include a transfer vector of length 3.

[0190] Among them, in step 3), the extracted portrait frame contains the aspect ratio HW_ratio, and the extracted bone sequence data includes 17 key nodes of the human body, namely nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; these key nodes are in the aligned RGB image Fr rgb The coordinates on ′ are (x1, y1), (x2, y2), (x3, y3), (x4, y4), (x5, y5), (x6, y6), (x7, y7), (x8, y8), (x9, y4), (x 10 ,y 11 )、(x 11 ,y 11 )、(x 12 ,y 12 )、(x 13 ,y 13 )、(x 14 ,y 14 )、(x 15 ,y 15 )、(x 16 ,y 16 )、(x 17 ,y 17 ).

[0191] Among them, in step 4), the center point of both shoulders is at Fr rgb The coordinates (x shc ,y shc ) is expressed as:

[0192]

[0193] Use this coordinate to get the depth value in the depth image, recorded as Z depth (shc) ;

[0194] Taking the optical center of the depth camera as the reference origin, the real-world 3D coordinates of the center points of the shoulders are true (shc) The calculation formula is:

[0195]

[0196] p true (shc) (z) = Z depth (shc)

[0197] in,

[0198] Zdepth (shc) =Fr depth ((x6+x7) / 2,(y6+y7) / 2);

[0199] p true (shc) (x), p true (shc) (y) and p true (shc) (z) are the three-dimensional coordinates p true (shc) The x value, y value and z value of

[0200] Among them, fx depth and fy depth are the focal lengths of the depth camera in the x and y directions, ppx depth and ppy depth They are the x-coordinate and y-coordinate of the optical center of the depth camera (i.e., the center point of the camera lens) on the imaging plane, and the intrinsic parameter matrix of the depth camera is recorded as H depth , expressed as:

[0201]

[0202] Among them, in step 5), the Boolean statistic Bool c Determine whether there are all zero values ​​in the left shoulder, right shoulder, left hip, right hip, left knee, and right knee coordinate points. If so, set it to false; if not, set it to true; use Boolean statistics Bool a Determine whether there are all zero values ​​in the left ankle and right ankle coordinate points. If so, set it to false; if not, set it to true.

[0203] Among them, in step 5), the point coordinate statistics are divided by the center point of the shoulders (x shc ,y shc ), the coordinates of the center points of the wrists, hips, knees, and ankles are calculated and recorded as (x wrc ,y wrc )、(x hic ,y hic )、(x knc ,y knc ) and (x anc ,y anc ), their calculation formulas are:

[0204]

[0205] Wherein, in step 5), the angle statistics include the angle between the shoulder-hip ray and the horizontal ray, the angle between the hip-knee line and the horizontal ray, the angle between the shoulder-hip-knee ray and the horizontal ray, and the angle between the hip-knee-ankle ray and the horizontal ray, which are respectively denoted as upper_angle, lower_angle, shk_angle and hka_angle; their calculation formulas are:

[0206]

[0207] Among them, x shc_hic =x shc -x hic , x knc_hic =x knc -x hic ,y shc_hic =y shc -y hic ,y knc_hic =y knc -y hic , x hic_knc =x hic -x knc , x anc_knc =x anc -x knc ,y hic_knc =y hic -y knc ,y anc_knc =y anc -yk nc .

[0208] In step 5), the distance statistics include the distance between the shoulder and the hip in the y-axis direction and the distance between the wrist and the ankle in the y-axis direction, which are respectively denoted as D shc_hic and D wrc_anc , and their calculation formulas are:

[0209]

[0210] Among them, x wrc_anc =x wrc -x anc ,y wrc_anc =y wrc -y anc .

[0211] Among them, in step 6), the incomplete judgment and confidence calculation method of the action type is as follows: when Bool c When it is false, the action is judged as incomplete, and the confidence level of incompleteness is V incom Set to 1.

[0212] Among them, in step 6), the determination and confidence calculation method of standing are as follows:

[0213] First, when HW_ratio > 1.5, 70 < upper_angle ≤ 110, and 70 < lower_angle ≤ 110, the action is determined as standing;

[0214] Then, different confidence calculation methods are executed in different cases:

[0215] When Bool a = True and 150 < hka_angle ≤ 180, the confidence V stand of standing is calculated as follows:

[0216] V stand = 0.7 - |90 - lower_angle| * 0.02 + |180 - hka_angle| * 0.01

[0217] Otherwise, the confidence V stand of standing is calculated as follows:

[0218] V stand = 0.7 - |90 - lower_angle| * 0.02

[0219] Among them, in step 6), the determination and confidence calculation method of bending and standing are as follows:

[0220] First, when HW_ratio > 1.5, upper_angle ≤ 70 or upper_angle > 110, and 70 < lower_angle ≤ 110, the action is determined as bending and standing;

[0221] Then, different confidence calculation methods are executed in different cases:

[0222] When Bool a = True and 150 < hka_angle ≤ 180, the confidence V pro_stand of bending and standing is calculated as follows:

[0223] V pro_stand = 0.7 - |90 - lower_angle| * 0.02 + |180 - hka_angle| * 0.01

[0224] Otherwise, the confidence V pro_stand of bending and standing is calculated as follows:

[0225] V pro_stand = 0.7 - |90 - lower_angle| * 0.02.

[0226] Among them, in step 6), sitting on the ground, sitting on a chair, and sitting at a table all belong to the large category of sitting postures. The determination and confidence calculation methods are as follows:

[0227] First, when 70 < upper_angle ≤ 110, lower_angle ≤ 70 or lower_angle > 110, 35 < shk_angle ≤ 170, the action is determined as the large category of sitting posture;

[0228] Then, each sitting posture has different detailed determination statements and confidence calculation methods:

[0229] When the additional condition is p true (shc) (y) > -300, it is determined as sitting on the ground. The confidence V cross _ sit of sitting on the ground is calculated as follows:

[0230]

[0231] When the additional condition is p true (shc) (y) ≤ -300, p true (shc) (y) > -700, it is determined as sitting on a chair, and different confidence calculation methods are executed in different cases:

[0232] When the further additional condition Bool a = True, 60 < hka_angle ≤ 150, the confidence V normal_sit of sitting on a chair is calculated as follows:

[0233] V normal_sit = 0.7 - |90 - shk_angle| * 0.005 + (60 - |90 - shk_angle|) * 0.05

[0234] Otherwise, the confidence V normal_sit of sitting on a chair is calculated as follows:

[0235] V normal_sit = 0.7 - |90 - shk_angle| * 0.005

[0236] When the additional condition is p true (shc) (y) ≤ -700, it is determined as sitting at a table, and different confidence calculation methods are executed in different cases:

[0237] When the further additional condition Bool a= True, when 60 < hka_angle ≤ 150, the confidence level V of sitting on the table upper_sit The calculation formula is as follows:

[0238] V upper_sit = 0.7 - |90 - shk_angle| * 0.005 + (60 - |90 - shk_angle|) * 0.05

[0239] Otherwise, the confidence level V of sitting on the table upper_sit The calculation formula is as follows:

[0240] V upper_sit = 0.7 - |90 - shk_angle * 0.005.

[0241] Among them, in step 6), the determination and confidence level calculation method for drilling through the equipment are as follows:

[0242] When HW ratio ≤ 1.5, upper_angle ≤ 70 or upper_angle > 110, p true (shc) (y) > -250, y wrc > y hic When, the action is determined to be drilling through the equipment, and the confidence level V of drilling through the equipment kneel Is set to 1.

[0243] Among them, in step 6), actions that do not meet the above conditions are uniformly classified as others, and the confidence level V of other actions other Is set to 1.

[0244] Embodiment 3

[0245] As Figure 1-4 , the present invention provides a method for identifying illegal actions of workshop personnel by fusing RGB and depth image information, which can identify several preset action types such as incomplete, standing, standing bent, sitting on the ground, sitting on a chair, sitting on a table, drilling through the equipment and others, and identify sitting on the ground, sitting on the table and drilling through the equipment as illegal actions, including the following steps:

[0246] Step 1), data acquisition step: Use an RGB camera and a depth camera to simultaneously shoot 8 videos in the workshop. These videos contain 7 different action types such as incomplete, standing, standing bent, sitting on the ground, sitting on a chair, sitting on a table, drilling through the equipment and others. The start and end times of the videos collected by the RGB camera and the depth camera are the same, and the frame rate is controlled at 30 frames.

[0247] Taking the first frame of the first video as an example, the internal parameter matrix H of the RGB camera rgb Is where fx rgb and fy rgb They are 609.012 and 608.862 respectively, and the focal lengths in the x and y directions are very close. rgb and ppy rgb They are 316.630 and 244.241 respectively.

[0248] The rotation matrix R of the RGB camera rgb for Transfer vector T rgb is [-0.015 0-0.001]. The intrinsic parameter matrix H of the depth image depth for where fx depth and fy depth Both are 388.463, and the focal lengths in the x and y directions are exactly the same; ppx depth and ppy depth They are 322.331 and 237.565 respectively. The rotation matrix R of the depth camera depth for Transfer vector T depth is [0 0 0].

[0249] Step 2), aligning the RGB image to the depth image step: aligning the RGB image in the RGB video stream to the depth image in the depth video stream frame by frame according to the RGB internal reference stream, the RGB external reference stream, the depth internal reference stream and the depth external reference stream;

[0250] Taking the first frame of the first video as an example, the aligned RGB image Fr rgb ′ and depth image Fr depth like Figure 2 As shown in the figure, the image resolution is 640 × 480. After alignment, each pixel of the two images corresponds one to one.

[0251] Step 3), extracting portrait frames and skeleton sequence data: using the YOLOv8-pose model (the YOLOv8-pose model is an open source large model provided by the Ultralytics machine learning platform) to extract the portrait frames and skeleton sequence data of the portraits in the aligned RGB images frame by frame;

[0252] Taking the first frame of the first video as an example, a worker portrait frame is extracted from the RGB image using the YOLOv8-pose model ( Figure 3 ), and its aspect ratio parameter HW_ratio is 5.194. The bone sequence, that is, the coordinates (x1, y1) to (x 17 ,y 17), (71,68), (76,63), (65,61), (0,0), (51,64), (62,103), (48,102), (0,0), (54,157), (0,0), (71,187), (54,192), (45,193), (50,251), (46,255), (30,304), (34,314). It can be seen that due to occlusion, the left ear, left elbow and left wrist are not detected, and their coordinates are all (0,0);

[0253] Step 4), obtaining the three-dimensional coordinates of the center points of both shoulders: combining the depth image, obtaining the three-dimensional coordinates of the center points of both shoulders in the bone sequence data;

[0254] Taking the first frame of the first video as an example, the coordinates of the center point of the shoulders (x shc ,y shc ) is (55,103). The depth value obtained in the depth image using this coordinate is 3813. The real-world 3D coordinates of the center points of the shoulders are p true (shc) is (-1638.056,-890.786,3813.000);

[0255] Step 5), extract statistics: extract several statistics based on the portrait frame, skeleton sequence data and the three-dimensional coordinates of the center points of the shoulders, including: Boolean statistics of whether there are all zero values ​​in the coordinate points of the parts involved in the human body movements, point coordinates, angles and distances.

[0256] Taking the first frame of the first video as an example, since there are no all-zero values ​​in the left shoulder, right shoulder, left hip, right hip, left knee, and right knee coordinate points, the Boolean statistic Bool c Is true. Because there are no all-zero values ​​in the left ankle and right ankle coordinate points, the Boolean statistic Bool a True. Except for the coordinates of the center points of the shoulders (x shc ,y shc ), the point coordinate statistics also need to calculate the center point of both wrists (x wrc ,y wrc ), double hip center point (x hic ,y hic ), center point of both knees (x knc ,y knc ) and the center point of both ankles (x anc ,y anc) are (36,94), (50,193), (48,253) and (32,309) respectively. The angle statistics shoulder-hip angle upper_angle, hip-knee angle lower_angle, shoulder-hip-knee angle shk_angle and hip-knee-ankle angle hka_angle are 86.503, 88.580, 177.923 and 165.475 respectively. Distance statistics shoulder-hip distance D shc_hic and wrist-ankle distance D wrc_anc 90 and 215.5 respectively.

[0257] Step 6), establish a rule model to determine illegal actions: establish a rule model to convert the portrait frame, skeleton sequence data, three-dimensional coordinates of the center points of both shoulders and the extracted statistics into 8 specific action types (incomplete, standing, bending over, sitting on the ground, sitting on a chair, sitting on a table, drilling through equipment and others) and their confidence levels.

[0258] In order to better explain the rule model, eight typical frames of action types are selected as examples. Figure 4 In (a), because the coordinates of the left and right knees are zero, Bool c If it is false, the action type is judged as incomplete, and the confidence V incom Set to 1. Figure 4 In (b), HW_ratio is 3.861, upper_angle is 86.077, lower_angle is 90.428, Bool a True, hka_angle is 162.783, so the action type is determined to be standing, and the confidence V stand The calculated result is 0.864. Figure 4 In (c), HW_ratio is 2.587, upper_angle is 67.051, lower_angle is 107.319, Bool a True, hka_angle is 156.672, so the action type is determined to be bending and standing, and the confidence level is V pro_stand The calculated result is 0.587. Figure 4 In (d), upper_angle is 96.230, lower_angle is 142.977, and shk_angle is 59.207, so the action is first determined to be a sitting posture. true (shc) (y) is -140.807, so the action is further determined to be sitting on the ground, and the confidence V cross_sit The calculated result is 0.619. Figure 4In (e), upper_angle is 97.658, lower_angle is 164.219, and shk_angle is 113.438, so the action is first determined to be a sitting posture. Bool a is true, hka_angle is 111.607, p true (shc) (y) is -558.479, so the action is further determined to be sitting on the chair, with a confidence level of V normal_sit The calculated result is 0.766. Figure 4 In (f), upper_angle is 84.957, lower_angle is 16.837, and shk_angle is 111.880, so the action is first determined to be a sitting posture. a is true, hka_angle is 115.955, p true (shc) (y) is -721.531, so the action is further determined to be sitting on the table, with a confidence level of V uoper_sit is 0.781. Figure 4 In (g), HW_ratio is 0.7015, upper_angle is 17.981, and p true (shc) (y) is 49.4, y wrc >y hic , so the action is determined to be drilling through the equipment, and the confidence level is V kneel Set to 1. Figure 4 In (h), the statistics calculated in the image do not meet any of the above conditions, and the action is judged as other, with a confidence level of V other Set to 1.

[0259] Finally, the illegal action is determined based on whether the action type belongs to the preset illegal action.

[0260] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for identifying illegal actions of workshop personnel by integrating RGB and depth image information, characterized in that: It includes the following steps: Step 1), use an RGB camera to obtain an RGB video stream, an RGB intrinsic stream, and an RGB extrinsic stream, and use a depth camera to obtain a depth video stream, a depth intrinsic stream, and a depth extrinsic stream; Step 2), according to the RGB intrinsic stream, the RGB extrinsic stream, the depth intrinsic stream, and the depth extrinsic stream, align the RGB images in the RGB video stream to the depth images in the depth video stream frame by frame; Step 3), use the YOLOv8-pose model to extract the portrait frame and the skeleton sequence data of the person in the aligned RGB image frame by frame; Step 4), combine with the depth image to obtain the three-dimensional coordinates of the center points of the two shoulders in the skeleton sequence data; Step 5), according to the portrait frame, the skeleton sequence data, and the three-dimensional coordinates of the center points of the two shoulders, extract several statistics, including: a boolean statistic on whether there are all-zero values among the coordinate points of the parts involved in the human action, point coordinates, angles, and distances; Step 6), establish an action rule model, convert the portrait frame, the skeleton sequence data, the three-dimensional coordinates of the center points of the two shoulders, and the extracted statistics into preset action types and their confidence levels, and compare the actions with the preset violation actions to determine violation actions.

2. The method for identifying illegal actions of workshop personnel by integrating RGB and depth image information according to claim 1, characterized in that: In step 1), the RGB camera and the depth camera start collecting data simultaneously, and the acquisition frame rates of the RGB video stream, the RGB intrinsic stream, the RGB extrinsic stream, the depth video stream, the depth intrinsic stream, and the depth extrinsic stream are the same.

3. The method for identifying illegal actions of workshop personnel by integrating RGB and depth image information according to claim 1, characterized in that: In step 2), the RGB image is aligned to the depth image through pixel point indexing, specifically including: Assume that p depth is the depth image Fr depth A pixel point on the depth (x),p depth (y)), the depth value is Z depth ; Assume that p rgb is the RGB image Fr rgb The same pixel point on the rgb (x),p rgb (y)), p aepth With p rgb The relationship between is expressed by the following equation: Where R = R rgb R depth -1 , T=T rgb -R rgb R depth -1 T depth , The x, y, and z coordinates of the aligned RGB image are temporarily stored; traverse Fr depth For each pixel on Fr rgb The RGB image is aligned to the depth image, and the aligned RGB image is recorded as Fr rbb ′,Fr rgb ′ and Fr depth Each pixel of has a one-to-one correspondence; R rgb The external parameters of the RGB camera include a rotation matrix of size 3×3. The external parameters of the RGB camera also include a transfer vector T of length 3 rgb ; H depth is the intrinsic parameter matrix of the depth camera; R depth The external parameters of the depth camera include a rotation matrix of size 3×3; T depth The extrinsic parameters for the depth camera include a transfer vector of length 3.

4. The method for identifying illegal actions of workshop personnel by integrating RGB and depth image information according to claim 3 is characterized in that: In step 3), the extracted portrait frame contains the aspect ratio HW_ratio, and the extracted skeleton sequence data includes 17 key nodes of the human body, namely nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle; these key nodes are in the aligned RGB image Fr rgb The coordinates on ′ are (x1, y1), (x2, y2), (x3, y3), (x4, y4), (x5, y5), (x6, y6), (x7, y7), (x8, y8), (x9, y9), (x 10 ,y 10 )、(x 11 ,y 11 )、(x 12 ,y 12 )、(x 13 ,y 13 )、(x 14 ,y 14 )、(x 15 ,y 15 )、(x 16 ,y 16 )、(x 17 ,y 17 ); In step 5), use the Boolean statistic Bool c Determine whether there are all zero values ​​in the left shoulder, right shoulder, left hip, right hip, left knee, and right knee coordinate points. If so, set it to false; if not, set it to true; use Boolean statistics Bool a Determine whether there are all zero values ​​in the left ankle and right ankle coordinate points. If so, set it to false; if not, set it to true.

5. The method for identifying illegal actions of workshop personnel by integrating RGB and depth image information according to claim 4 is characterized in that: In step 4), the center point of both shoulders is at Fr rgb The coordinates (x shc ,y shc ) is expressed as: Use this coordinate to get the depth value in the depth image, recorded as Z depth (shc) ; Taking the optical center of the depth camera as the reference origin, the real-world 3D coordinates of the center points of the shoulders are true (shc) The calculation formula is: p true (shc) (from)=from depth (shc) Among them, Z depth (shc) =Fr depth ((x6+x7) / 2,(y6+y7) / 2); p true (shc) (x), p true (shc) (y) and p true (shc) (z) are the three-dimensional coordinates p true (shc) The x, y and z values ​​of Among them, fx depth and fy depth are the focal lengths of the depth camera in the x and y directions, ppx depth and ppy depth They are the x-coordinate and y-coordinate of the optical center of the depth camera on the imaging plane, and the intrinsic parameter matrix of the depth camera is recorded as H depth , expressed as:

6. The method for identifying illegal actions of workshop personnel by integrating RGB and depth image information according to claim 5, characterized in that: In step 5), the point coordinate statistics are divided by the center point of the shoulders (x shc ,y shc ), the coordinates of the center points of the wrists, hips, knees, and ankles are calculated and recorded as (x wrc ,y wrc )、(x hic ,y hic )、(x knc ,y knc ) and (x anc ,y anc ), their calculation formulas are: In step 5), the angle statistics include the angle between the shoulder-hip ray and the horizontal ray, the angle between the hip-knee line and the horizontal ray, the angle between the shoulder-hip-knee ray and the horizontal ray, and the angle between the hip-knee-ankle ray and the horizontal ray, which are denoted as upper_angle, lower_angle, shk_angle, and hka_angle respectively; their calculation formulas are respectively: where x shc_hic = x shc - x hic and x knc_hic = x knc - x hic and y shc_hic = y shc - y hic and y knc_hic = y knc - y hic and x hic_knc = x hic - x knc and x anc_knc = x anc - x knc and y hic_knc = y hic - y knc and y anc_knc = y anc - y knc ; In step 5), the distance statistics include the distance between shoulder and hip in the y-axis direction and the distance between wrist and ankle in the y-axis direction, which are denoted as D shc_hic and D wrc_anc , and their calculation formulas are: where x wrc_anc = x wrc - x anc , y wrc_anc = y wrc - y anc .

7. The method for identifying illegal actions of workshop personnel by integrating RGB and depth image information according to claim 6, characterized in that: The preset action types include: incomplete, standing, standing bent, sitting on the ground, sitting on a chair, sitting on a table, crawling through equipment, and others; the violation actions include: sitting on the ground, sitting on a table, and crawling through equipment.

8. The method for identifying illegal actions of workshop personnel by integrating RGB and depth image information according to claim 7, characterized in that: In step 6), the incomplete judgment and confidence calculation of the action type are as follows: When Bool c When it is false, the action is judged as incomplete, and the confidence level of incompleteness is V incom Set to 1; in step 6), actions that do not meet the above conditions are uniformly classified as others, and the confidence of other actions is V other Set to 1.

9. The method for identifying illegal actions of workshop personnel by integrating RGB and depth image information according to claim 7, characterized in that: In step 6), the determination and confidence calculation method for standing are as follows: First, when HW_ratio > 1.5, 70 < upper_angle ≤ 110, 70 < lower_angle ≤ 110, the action is determined to be standing; Then, different confidence calculation methods are executed in different cases: When Bool a = True and 150 < hka_angle ≤ 180, the confidence level V stand is calculated as follows: In stand =0.7-|90-lower_angle|*0.02+|180-hka_angle|*0.01 Otherwise, the standing confidence V stand The calculation formula is as follows: V stand =0.7-|90-lower_angle|*0.02 In step 6), the determination and confidence calculation method for standing bent are as follows: First, when HW_ratio > 1.5, upper_angle ≤ 70 or upper_angle > 110, 70 < lower_angle ≤ 110, the action is determined to be standing bent; Then, different confidence calculation methods are executed in different cases: When Bool a = True and 150 < hka_angle ≤ 180, the confidence level V pro_stand of bending and standing is calculated as follows: In pro_stand =0.7-|90-lower_angle|*0.02+|180-hka_angle|*0.01 Otherwise, the confidence level V of the bent-over standing position is pro_stand The calculation method is as follows: V pro_stand =0.7-|90-lower_angle|*0.02 In step 6), sitting on the ground, sitting on a chair, and sitting on a table all belong to the sitting posture category, and the determination and confidence calculation methods are as follows: First, when 70 < upper_angle ≤ 110, lower_angle ≤ 70 or lower_angle > 110, 35 < shk_angle ≤ 170, the action is determined as the sitting posture category; Then, each sitting posture has different detailed determination statements and confidence calculation methods: When the additional condition is p true (shc) When (y)>-300, it is judged as sitting on the ground, and the confidence level of sitting on the ground is V cross_sit The calculation formula is as follows: When the additional condition is p true (shc) (y)≤-300, p true (shc) When (y)>-700, it is judged as sitting on a chair, and different confidence calculation methods are performed according to different situations: When further adding the condition Bool a = True, when 60 < hka_angle ≤ 150, the confidence level V of sitting on the chair normal_sit is calculated as follows: V normal_sit =0.7-|90-shk_angle|*0.005+(60-|90-shk_abgle|)*0.05 Otherwise, the confidence V of sitting on the chair normal_sit The calculation formula is as follows: V normal_sit =0.7-|90-shk_angle|*0.005 When the additional condition is p true (shc) When (y)≤-700, it is determined as sitting on the table, and different confidence calculation methods are performed according to different situations: When the further additional condition Bool a = True, when 60 < hka_angle ≤ 150, the confidence level V upper_sit is calculated as follows: V upper_sit =0.7-|90-shk_angle|*0.005+(60-|90-shk_angle|)*0.05 Otherwise, the confidence V sitting on the table upper_sit The calculation formula is as follows: V upper_sit =0.7-|90-shk_angle|*0.005 In step 6), the determination and confidence calculation method for passing through the device are as follows: When HW ratio ≤1.5, upper_angle≤70 or upper_angle>110, p true (shc) (y)>-250,y wrc >y hic When , the action is judged as drilling through the device, and the confidence level of drilling through the device is V kneel Set to 1.

10. A device for identifying illegal actions of workshop personnel by integrating RGB and depth image information, characterized in that: Including a processor, the processor is capable of executing the steps of the method for identifying illegal actions of workshop personnel by fusing RGB and depth image information according to any one of claims 1-9.

Images