Escalator passenger abnormal behavior data construction method based on digital twinning

Abstract

The application discloses an escalator passenger abnormal behavior data construction method based on digital twinning, which comprises the following steps: firstly, constructing a digital twinning environment of escalator passenger abnormal behavior; secondly, modeling human body behavior in the digital twinning environment to generate a large number of abnormal behavior data sets; thirdly, training a human body posture recognition network by using the generated data sets, and identifying two-dimensional human body posture data conversion to three-dimensional space; then, completing posture binding of virtual passengers through three-dimensional space posture information, realizing synchronization of real passenger behavior and virtual passenger action; finally, realizing passenger behavior prewarning in reality by human body posture recognition results. The application solves the problem of scarce abnormal dangerous behavior data in reality, re-trains a human body posture recognition network to realize actual passenger behavior recognition, synchronously maps actual passenger behavior to a digital virtual space by using virtual reality technology, realizes monitoring and prewarning of passenger dangerous behavior, and protects the life and property safety of the people.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of escalator monitoring, and relates to an escalator passenger abnormal behavior data construction method based on digital twinning. BACKGROUND

[0002] With the continuous deepening of research in the field of computer vision, the lack of dangerous abnormal behavior target data in reality has become an obstacle to the monitoring and early warning of human safety behavior in the field of machine vision. The concept of digital twinning is to simulate and express the information of the real world in the digital information world, to build a digital twin to realize high-fidelity modeling of the real world, and the digital twin can run according to the physical laws of the real world, can realize self-perception, monitoring, early warning and decision-making processing, and realizes the two-way coordination, dynamic interaction and evolution prediction between the real world and the virtual digital world. With the continuous development of modern sensing technology, communication technology, artificial intelligence and virtual reality technology, digital twinning has the conditions to be realized. Using digital twinning to solve the problem of data missing in practice will become an important research direction in the future.

[0003] By the end of 2021, the number of escalators in China has reached 8.7998 million. With the increase in the number of escalators, the relationship between escalator safety and public safety has gradually attracted widespread attention. At present, there are mainly two kinds of escalator safety monitoring methods: one is to assign special security personnel at the entrance and exit of the escalator to maintain the on-site order of the escalator operation. The manual maintenance method will consume a lot of human resources, and even if a danger occurs, the device cannot be stopped immediately. The other is to detect the human body through infrared, ultrasonic and other devices and play the prompt voice of "stand steady and hold well, pay attention to safety". The above methods cannot realize passenger danger early warning and response, and lack effective measures to deal with escalator safety accidents. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the application provides an escalator passenger abnormal behavior data construction method based on digital twinning, which constructs a virtual body in a digital twinning environment for the abnormal behavior of passengers, solves the problem of lack of abnormal dangerous behavior data in reality, re-trains a human posture recognition network to realize recognition of actual passenger behavior, uses virtual reality technology to synchronize the actual passenger behavior to a digital virtual space, realizes monitoring and early warning of passenger dangerous behavior, and protects the life and property safety of the people.

[0005] The technical scheme adopted by the application to solve the technical problems is:

[0006] An escalator passenger abnormal behavior data construction method based on digital twinning, the method comprising the following steps:

[0007] First step: escalator passenger abnormal behavior digital twin environment construction, the process is as follows:

[0008] Step (1.1) geometric model construction: the establishment of geometric model is the basic link of digital twin construction, which determines the final implementation effect of digital twin, determines the degree of realism of digital twin, respectively constructs escalator geometric model and virtual person geometric model;

[0009] Step (1.2) scene construction: virtual assembly is carried out on different parts of the geometric model to construct the same main body as the reality;

[0010] Step (1.3) data transmission: twin data is the data used to drive the twin model in digital space with real world real data, wherein the twin data is the video data of passenger behavior and the running data of escalator;

[0011] Second step: modeling human behavior in digital twin environment, generating a large number of abnormal behavior data sets, the process is as follows:

[0012] Step (2.1) human kinematics description, in virtual space, in order to make human motion conform to the physical laws of the real world, the human body is simplified as a skeletal model, and the complete motion posture is composed of the limited rotation displacement of each joint;

[0013] Step (2.2) forward motion (FK), human posture forward dynamics usually refers to using the relative displacement and relative rotation of each joint as input to calculate the whole body posture after motion, which can be represented by the following formula: Q=FK(R,T)

[0014]

[0015] Wherein, M is the number of body joints, represents the three-dimensional pose of the i th joint after motion, represents the i th joint position information in the relative displacement matrix, pa(i) represents the parent joint index of the i th joint, R pa(i),i represents the rotation of the i th joint relative to the parent joint, FK represents the process of recursively passing displacement matrix and rotation matrix from root node to all child nodes, which can be represented by the following formula:

[0016]

[0017] In the formula, R i is the global rotation of the i th joint relative to the static pose space;

[0018] Step (2.3) inverse motion (IK), that is, inverse kinematics is the reverse process of forward kinematics. By inputting the expected human joint posture​ and displacement T, the relative rotation R of the joint can be calculated, which is expressed as:

[0019] R = IK(P, T)

[0020] where p i represents the desired pose of the input i-th joint, the resulting rotation should satisfy the following condition:

[0021]

[0022] Step (2.4) human skeleton model, which is a simplified model in digital twin for representing actual passenger behavior, is a tree-like hierarchical structure composed of joint nodes;

[0023] Step (2.5) passenger abnormal behavior data generation;

[0024] Step (2.6) a large amount of abnormal behavior data is generated according to the type of passenger abnormal behavior;

[0025] Third step: use the large amount of abnormal passenger behavior generated in the second step to train the human pose recognition network I-OpenPose in the application, which uses VGG19 feature extraction network, and introduces feature pyramid into VGG19 to fuse the extracted image features Feature Map. Feature Map is input into two branches PAFs and PCM. PAFs is used to express the direction of the pixel points in the pose, and the recognized two-dimensional pose information is mapped to three-dimensional space. In the fourth step, the three-dimensional pose information is used to bind the virtual passenger skeleton model, realizing the behavior synchronization of real passengers and virtual passengers;

[0026] where PAFs is the joint affinity field, and PCM is the joint confidence map;

[0027] Fourth step: the 18 joint coordinates in the three-dimensional space pose information are bound to the virtual character's skeleton joint, realizing the synchronization of real passenger behavior and virtual passenger action;

[0028] Fifth step: the human pose recognition network trained in the third step is input with the actual escalator passenger monitoring video to realize the passenger behavior early warning.

[0029] Further, the process of step (1.1) is as follows:

[0030] Step (1.1.1) escalator geometric model drawing, according to the eight systems of escalator, namely truss, step system, handrail system, guide rail system, handrail device, safety protection device, electrical control system and lubrication system, the three-dimensional modeling software is used to construct the eight system modules of escalator;

[0031] Step (1.1.1.1) escalator geometry model driven, the escalator step, drive wheel, handrail belt drive wheel, drive motor, step, roller chain and roller in the model are driven by real-time data;

[0032] Step (1.1.1.2) motion component attribute definition, respectively, for the escalator step, drive wheel, handrail belt drive wheel, drive motor, step, roller chain and roller definition, and define the motion component data interface;

[0033] Wherein, the motion component attribute: the physical engine realizes the operability of the motion component displacement rotation;

[0034] Wherein, define the motion component data interface: the physical engine realizes the start, stop and speed control of the movable component through real data;

[0035] Step (1.1.2) human geometry model construction;

[0036] Step (1.1.2.1) draw virtual human geometry model, according to the behavior of the person, simplify the human model into a virtual model with 18 joint nodes limited rotation displacement. The joint nodes are 0 neck, 1 nose, 2 left eye, 3 right eye, 4 left ear, 5 right ear, 6 left shoulder, 7 right shoulder, 8 left elbow, 9 right elbow, 10 left wrist, 11 right wrist, 12 left hip, 13 right hip, 14 left knee, 15 right knee, 16 left ankle, 17 right ankle;

[0037] Step (1.1.2.2) virtual person driving, connecting the above virtual person joint nodes to form a human skeleton model, using the three-dimensional space posture information of the fourth step to bind the posture of the virtual human model in the physical engine, and controlling the motion of the virtual human in the physical engine;

[0038] Step (1.1.2.3) reorient the virtual human space position in the above physical engine, set the initial position of the virtual human appearance;

[0039] Wherein, reorientation is to reset the virtual human joint node coordinate space relationship to the joint node space distribution of the initial behavior state.

[0040] In the step (1.2), virtual assembly is to assemble each system module of the escalator in the physical engine, and finally form a virtual model corresponding to the actual escalator;

[0041] Step (1.2.1) mirror expansion two elevators in the physical engine according to the single down elevator, redefine each component of the expanded elevator, and form a bidirectional carrying elevator;

[0042] Step (1.2.2) add a walking staircase between the two elevators;

[0043] Step (1.2.3) redefines the material and color of each module of the elevator;

[0044] The process of the step (1.3) is as follows:

[0045] Step (1.3.1) The passenger behavior video data is subjected to human body posture recognition in the third step, and the data is transmitted to the digital twin client. A bidirectional communication is established through Socket (which is used for bidirectional communication between application processes on different hosts in the network), and TCP protocol (a connection-oriented, byte stream-based transport layer communication protocol) is used to ensure stable and real-time data transmission;

[0046] Step (1.3.2) The escalator operation data is obtained by connecting the control mainboard of the escalator through the communication bus, and the real-time data of the escalator operation is transmitted to the client computer through the upper computer controller;

[0047] Step (1.3.3) Define and drive each part of the escalator, and map the virtual character motion in real time through the results of passenger behavior recognition;

[0048] Step (1.4) Visual interface design: the middle is the digital twin virtual model, and the passenger behavior information and the escalator operation information are displayed on both sides;

[0049] The process of the step (2.4) is as follows:

[0050] Step (2.4.1) Skeleton joint extraction, the performance form of human body posture motion is the spatial transformation of human body joints, and the skeleton joint extraction is to extract the spatial transformation relationship of the tree-like hierarchical structure composed of all joints;

[0051] Step (2.4.2) Joint mapping, the skeleton system in the physics engine is a tree structure, in order to drive the limb action of the virtual character, the joint labels obtained before need to be associated with the corresponding joints in the physics engine skeleton tree;

[0052] Step (2.4.3) Coordinate transformation, since each original skeleton joint has an independent coordinate system, coordinate transformation makes all skeleton joints in the same coordinate system;

[0053] Wherein, the coordinate transformation is O γ = TO ε , O γ is the space coordinate system after coordinate transformation, O ε is the initial space coordinate system, and T is the space transfer matrix;

[0054] Step (2.4.4) Data import, import the joint data to realize the virtual character posture reproduction;

[0055] Step (2.4.5) The key frame refinement action between the initial input action and the end output action needs to be solved by using the inverse kinematics of step (2.3);

[0056] Wherein, the key frame is the frame where the key action in the human pose change is located;

[0057] Step (2.4.6) The key frame data is imported frame by frame, which can be solved by using the forward kinematics of step (2.2), that is, the virtual character can be used to realize the action reproduction of the real person.

[0058] The process of the step (2.5) is as follows:

[0059] Step (2.5.1) Import the virtual character into the physical engine and perform bone reorientation to realize the mapping of different bone joints;

[0060] Wherein, the bone reorientation is to re-match the relationship between the bone joints by matching setting;

[0061] Step (2.5.2) Move the character bone joints to make character actions, and use the inverse kinematics of step (2.3) to obtain the key frame refinement action between the initial input action and the output action made;

[0062] Step (2.5.3) Generate corresponding animation sequences for multiple groups of key frame actions by using the forward kinematics of step (2.2);

[0063] Step (2.5.4) State integration, integrate multiple animation sequences into the same virtual character by using state machine in the physical engine;

[0064] Wherein, the state machine in the physical engine is a tool for transitioning from one action behavior of the virtual character to another action behavior;

[0065] Step (2.5.5) Event triggering, place different character roles in the virtual scene, and assign trigger events to the corresponding actions;

[0066] Step (2.5.6) Data enhancement, realize data enhancement by changing the scene, virtual camera angle and field of view.

[0067] The process of the third step is as follows:

[0068] Step (3.1) Extract the original image features by VGG19, and fuse them with the image features extracted by the feature pyramid of step (3.2);

[0069] Wherein, VGG19 is a kind of convolutional neural network for object recognition, each layer of neural network will use the output of the last layer to further extract more complex features until the complexity can be used to identify objects, so each layer can be regarded as a lot of local feature extractor;

[0070] Step (3.2) feature pyramid extraction image features and step (3.1) VGG19 extracted original image features are fused into input image features Feature Map;

[0071] Step (3.3) the feature map Feature Map obtained in step (3.2) is output to the next layer, there are two branches PAFs (affinity field of joint) and PCM (joint thermogram), each stage outputs a Loss (loss function);

[0072] Step (3.4) maps the coordinates of the two-dimensional pose joint to three-dimensional space;

[0073] Preferably, the process of step (3.2) is as follows:

[0074] Step (3.2.1) feature pyramid (FPN) selects all layers from Conv3_4 (the first VGG) as the network of FPN, wherein Conv6 is an added fully connected layer FC-4096;

[0075] Step (3.2.2) Conv3_4, Conv4_4, Conv5_4, Conv6 are VGG layers, and M3, M4, M5, M6 are connection layers obtained by 1*1 convolution;

[0076] Wherein, Conv3_4 is the fourth convolution kernel of the third convolution layer, and Conv4_4 and Conv5_4 are the same;

[0077] Step (3.2.3) through the up-sampling layer Upsample, the same channel number as the next level is obtained, and the addition operation is performed to obtain the addition layer L3, L4, L5;

[0078] Step (3.2.4) after 3*3 convolution operation, 7*7, 14*14, 28*28, 56*56 are output, and finally the prediction layer P3, P4, P5, P6 is obtained;

[0079] Step (3.2.5) the prediction layer P3, P4, P5, P6 obtained is fused with the image features extracted by VGG19 to obtain the feature map Feature Map;

[0080] The process of step (3.3) is as follows:

[0081] The iterative formula for the branch PAFs in step (3.3.1) is as follows:

[0082] S t =ρ t (F,L t-1 ,S t-1 ),t≥2

[0083] Where, ρ t Let F represent the iteration relationship of stage t, where F is the feature map, L represents the partial affinity field, S represents the two-dimensional confidence level, and t represents the total number of confidence map stages.

[0084] Step (3.3.2) The iterative formula for branch PCM is:

[0085] L t =φ t (F,L t-1 ,S t-1 ),t≥2

[0086] Where, φ t This represents the iterative relationship at stage t;

[0087] The loss function for the PAFs in step (3.3.3) is:

[0088]

[0089] in, The loss function for the branches PAFs, W is a binary code for the actual joint location information map. At the image pixel position p, there is no label, W(p) is 0, otherwise it is 1; The square of the difference between the predicted value and the actual value;

[0090] The loss function for the PCM branch in step (3.3.4) is:

[0091]

[0092] in, The loss function for the branches PAFs, For the actual joint affinity field, The square of the difference between the predicted value and the actual value;

[0093] Step (3.3.5) generates a confidence map for calculating S from the image of the two-dimensional joints, using... Let X represent all confidence graphs generated by k people. j,k Let j represent the j-th joint point of the k-th person in the image. The maximum value of the above formula represents the confidence map of the final joint position of the multi-person; wherein the predicted value of the p point is:

[0094]

[0095] Wherein, exp is the natural constant e, δ is the diffusion of the control peak, is the square of the vector module value from the p point to the kth person j part;

[0096] Step (3.3.6) X j1,k and X j2,k Indicates two joints, and there is a pixel point p on the limb, then The value of the kth person is a unit vector from j2 to j1, and the vector field of the p point is:

[0097]

[0098] Wherein, ||X j2,k -X j1,k ||2 is the length of the limb from the j1 part to the j2 part of the kth person;

[0099] Step (3.3.7) the average affinity field of all persons obtained finally is:

[0100]

[0101] Wherein, n c (p) represents the number of non-zero vectors at p in all persons;

[0102] Step (3.3.8) in the multi-person scene, the score of the limb is calculated by using the following formula, and the case with the maximum correlation confidence is found;

[0103]

[0104] Wherein, E is the correlation confidence, ||d j2 -d j1 ||2 is the distance between the body parts d j2 ,d j1 , and p(u) interpolates the position of the body parts d j2 ,d j1 :

[0105] p(u) = (1-u)d j1 +d j2

[0106] Wherein, the integral value is approximated by sampling and equidistant summation of u;

[0107] Step (3.3.9) changes the multi-person detection problem into a bipartite graph matching problem, obtains the optimal solution of the connected points, and finally obtains all possible limb prediction results to connect all the joint points of the human body;

[0108] Wherein, the bipartite graph matching is a subset of edges selected in a way that no two edges share a node, and the goal is to find a maximum weight matching for the selected edges;

[0109] The process of the step (3.4) is as follows:

[0110] Step (3.4.1) the joint points of the human body posture recognition have 18 point data, and the human body skeleton data is formed from the 18 point data, and the input is two-dimensional coordinates (x Mi ,y Mi )∈R * 2D , and the output is three-dimensional space (x Ni ,y Ni ,z Ni )∈R * 3D ;

[0111] Wherein, R * 2D is a two-dimensional space, and R * 3D is a three-dimensional space;

[0112] Step (3.4.2) the input human body posture data is a two-dimensional joint point skeleton model, and the output skeleton model mapped to the three-dimensional space is a three-dimensional posture;

[0113] The process of the fourth step is as follows:

[0114] Step (4.1) the joint point coordinates of the three-dimensional posture of the skeleton model are bound to the joint points of the virtual character model of the physical engine through space coordinate transformation;

[0115] Step (4.2) the behavior of the human body posture is synchronized into the virtual character through the forward motion calculation of step (2.2).

[0116] The technical concept of the application is: first, a digital twin environment of escalator passenger abnormal behavior is constructed; second, human behavior modeling is performed in the digital twin environment to generate a large number of abnormal behavior data sets; then, the human body posture recognition network is trained using the generated data set to convert two-dimensional human body posture data to three-dimensional space; then, the posture binding of the virtual passenger is completed through the three-dimensional space posture information, realizing the synchronization of the real passenger behavior and the virtual passenger action; finally, the human body posture recognition result realizes the warning of the passenger behavior in reality.

[0117] The beneficial effects of the present application mainly manifest in: a virtual body is constructed in a digital twin environment for abnormal behaviors of passengers, solving the problem of data scarcity of abnormal dangerous behaviors in reality, retraining a human posture recognition network to realize recognition of actual passenger behaviors, using virtual reality technology to synchronously map the actual passenger behaviors to a digital virtual space, realizing monitoring and early warning of dangerous behaviors of passengers, and protecting the life and property safety of the people. BRIEF DESCRIPTION OF DRAWINGS

[0118] Figure 1 A escalator passenger abnormal behavior data construction result in digital twin is shown. DETAILED DESCRIPTION

[0119] The present application will be further described below in conjunction with the drawings.

[0120] Reference Figure 1 A escalator passenger abnormal behavior data construction method based on digital twin, comprising the following steps:

[0121] Step 1: Escalator passenger abnormal behavior digital twin environment construction;

[0122] Step (1.1) Geometric model construction: The establishment of the geometric model is the basic link of the construction of the digital twin, which determines the final implementation effect of the digital twin and determines the degree of realism of the digital twin. The geometric model of the escalator and the virtual passenger geometric model are constructed respectively;

[0123] Step (1.1.1) Escalator geometric model drawing, according to the eight systems of the escalator, namely truss, step system, handrail belt system, guide rail system, handrail device, safety protection device, electrical control system and lubrication system, the eight system modules of the escalator are constructed by using three-dimensional modeling software;

[0124] Step (1.1.1.1) Escalator geometric model driving, the step pedal, drive wheel, handrail belt drive wheel, drive motor, step pedal, roller chain and roller in the model are driven by real-time data;

[0125] Step (1.1.1.2) Define the properties of the motion components, define the step pedal, drive wheel, handrail belt drive wheel, drive motor, step pedal, roller chain and roller, and define the motion component data interface;

[0126] Among them, the motion component attribute: the operability of the motion component displacement rotation in UE;

[0127] Among them, define the motion component data interface: realize the start, stop and speed control of the movable component through real data in UE;

[0128] Wherein, UE (Unreal Engine) is a real-time interactive rendering engine for game development, architecture, VR, etc.

[0129] Step (1.1.2) person geometry model construction;

[0130] Step (1.1.2.1) draw a virtual person geometry model, according to the behavior of the person, simplify the human model into a virtual person model with 18 joint nodes limited rotation displacement. The joint nodes are 0 neck, 1 nose, 2 left eye, 3 right eye, 4 left ear, 5 right ear, 6 left shoulder, 7 right shoulder, 8 left elbow, 9 right elbow, 10 left wrist, 11 right wrist, 12 left hip bone, 13 right hip bone, 14 left knee, 15 right knee, 16 left ankle, 17 right ankle;

[0131] Step (1.1.2.2) virtual person driving, connecting the above-mentioned virtual person joint nodes to form a human skeleton model, using the three-dimensional space posture information of the fourth step to bind the posture of the virtual person model in UE, controlling the movement of the virtual person in UE;

[0132] Step (1.1.2.3) reorient the spatial position of the virtual person in the above-mentioned UE, set the initial position of the virtual person appearing;

[0133] Wherein, reorientation is to reset the spatial relationship of the virtual person's joint node coordinates to the spatial distribution of the joint nodes in the initial behavior state;

[0134] Step (1.2) scene construction: virtual assembly of different parts of the geometry model, construction of the same main body as reality;

[0135] Wherein, virtual assembly is to assemble each system module of the escalator in UE, and finally form a virtual model corresponding to the actual escalator;

[0136] Step (1.2.1) mirror expansion of a single down escalator into two escalators in UE, redefine each component of the expanded escalator to form a bidirectional carrying escalator;

[0137] Step (1.2.2) add a walking staircase between the two escalators;

[0138] Step (1.2.3) redefine the material and color of each module of the escalator;

[0139] Step (1.3) data transmission: twin data is the data used to drive the twin model in digital space with real data in the real world, wherein the twin data is the video data of passenger behavior and the running data of the escalator;

[0140] Step (1.3.1) Passenger behavior video data is recognized by human body posture in the third step, and the data is transmitted to the digital twin client. A two-way communication is established through Socket (two-way communication between application processes on different hosts in the network) and TCP protocol (a connection-oriented, byte stream-based transport layer communication protocol) to ensure stable and real-time data transmission.

[0141] Step (1.3.2) The running data of escalator is obtained by connecting the control mainboard of the escalator to the communication bus. The real-time data of the escalator running is transmitted to the client computer through the host controller.

[0142] Step (1.3.3) Define and drive each part of the escalator, and map the virtual character motion in real time through the results of passenger behavior recognition.

[0143] Step (1.4) Visual interface design: the middle is the escalator and the passengers, and the left and right sides display passenger behavior information and escalator running information.

[0144] Step 2: Model human behavior in the digital twin environment and generate a large number of abnormal behavior data sets.

[0145] Step (2.1) Human kinematics description: In order to make the simulated human behavior in the digital twin virtual space conform to human kinematics, it is necessary to refer to actual human behavior. In reality, human motion is the result of muscle and bone interaction. In virtual space, in order to make human motion conform to the physical laws of the real world, the human body is simplified as a skeletal model, and the complete motion posture is composed of the finite rotation displacement of each joint.

[0146] Step (2.2) Forward kinematics (FK): Human posture forward dynamics usually refers to the process of calculating the whole body posture after motion using the relative displacement and relative rotation of each joint as input, which can be represented by the following formula:

[0147] Q=FK(R,T)

[0148] Where M is the number of body joints, represents the three-dimensional pose of the i-th joint after motion, represents the i-th joint position information in the relative displacement matrix, pa(i) represents the parent joint index of the i-th joint, R pa(i),i represents the rotation of the i-th joint relative to the parent joint, and FK represents the process of recursively passing the displacement matrix and rotation matrix from the root node to all child nodes, which can be represented by the following formula:

[0149] ​

[0150] where R i is the global rotation of the ith joint with respect to the regular rest pose space;

[0151] Step (2.3) inverse kinematics (IK), i.e. inverse kinematics is the reverse process of forward kinematics. By inputting the desired human joint pose and displacement T, the relative rotation R of the joint can be calculated. This process is expressed as:

[0152] R = IK(P, T)

[0153] where p i represents the desired pose of the ith joint input, and the resulting rotation should satisfy the following conditions:

[0154]

[0155] Step (2.3.1) inverse kinematics Jacobian matrix algorithm;

[0156]

[0157] where the angles of the M joint nodes θ = (θ1, θ2, …, θ M ) and the spatial coordinates of the end joint node e = (e x , e y , e z ) T The relationship between θ and e is:

[0158] θ = f -1 (e)

[0159] dθ = J + de

[0160] where f -1 is the functional relationship between θ and e, and J + is the pseudo-inverse of the Jacobian matrix;

[0161] Step (2.3.2) pseudo-inverse of the Jacobian matrix:

[0162] J + = (JJ T ) -1 J T

[0163] where the Jacobian matrix needs to be iteratively calculated to obtain the approximate solution of inverse kinematics;

[0164] Step (2.4) human skeletal model, which is a simplified model in digital twin for the purpose of representing the actual passenger behavior, consisting of a tree-like hierarchical structure of joint nodes;

[0165] Step (2.4.1) Skeleton joint extraction, the performance form of human posture motion is the space transformation of human joint, and the skeleton joint extraction is to extract the space transformation relationship of the tree structure composed of all joints;

[0166] Step (2.4.2) Joint mapping, the skeleton system in the UE is a tree structure, in order to drive the limb action of the virtual character, the joint label obtained before needs to be associated with the corresponding joint in the UE skeleton tree;

[0167] Step (2.4.3) Coordinate transformation, since each original skeleton joint has an independent coordinate system, the coordinate transformation makes all the skeleton joints in the same coordinate system;

[0168] Wherein, the coordinate transformation is O γ = TO ε , O γ is the space coordinate system after the coordinate transformation, O ε is the initial space coordinate system, and T is a space transfer matrix;

[0169] Step (2.4.4) Data import, the joint data is imported to realize the posture reproduction of the virtual character;

[0170] Step (2.4.5) The joints of the human skeleton have internal correlation, and the key frame refinement action between the initial input action and the end output action needs to be solved by using the inverse motion of step (2.3);

[0171] Wherein, the key frame is the frame in which the key action of the human posture change is located;

[0172] Step (2.4.6) Frame-by-frame import of joint data is solved by step (2.2) forward motion, that is, the action reproduction of the virtual character can be realized by using the virtual character;

[0173] Step (2.5) Passenger abnormal behavior data generation;

[0174] Step (2.5.1) Import the virtual character in the UE and perform skeleton reorientation to realize the mapping of different skeleton joints;

[0175] Wherein, the skeleton reorientation is to re-match the relationship between the skeleton joints by matching setting;

[0176] Step (2.5.2) Moving the character skeleton joint to make the character action, and using the inverse motion of step (2.3) to obtain the key frame refinement action between the initial input action and the output action made;

[0177] Step (2.5.3) Generate corresponding animation sequences by using the forward motion of step (2.2) to multiple key frame actions;

[0178] Step (2.5.4) state integration, by using state machine in UE, multiple sets of animation sequences are integrated into the same virtual character;

[0179] Wherein, the state machine is a tool in UE for a virtual character to transition from one action behavior to another;

[0180] Step (2.5.5) event triggering, different character roles are placed in the virtual scene, and corresponding actions are assigned with triggering events;

[0181] Step (2.5.6) data enhancement, data enhancement is achieved by changing the scene, virtual camera angle, field of view, etc;

[0182] Step (2.6) according to the type of passenger abnormal behavior, a large amount of abnormal behavior data is generated, such as Figure 1 as shown;

[0183] Step 3: use the large amount of passenger abnormal behavior data generated in step 2 to train the human pose recognition network I-OpenPose in the present application, the principle of which uses VGG19 feature extraction network, and introduces feature pyramid into VGG19 to fuse the extracted image features (Feature Map), Feature Map as input, into two branches PAFs (Part Affinity Fields) and PCM (Part Confidence Maps), use PAFs to express the direction of pixel points on the pose, map the recognized two-dimensional pose information to three-dimensional space, and use three-dimensional pose information to bind the virtual passenger skeletal model in step 4, realize the behavior synchronization of real passengers and virtual passengers;

[0184] Wherein, PAFs is a joint affinity field, and PCM is a joint confidence map;

[0185] Step (3.1) extract the original image features by VGG19, and fuse them with the image features extracted by the feature pyramid in step (3.2);

[0186] Wherein, VGG19 is a convolutional neural network for object recognition, each layer of neural network uses the output of the previous layer to further extract more complex features until the complexity is sufficient to recognize objects, so each layer can be regarded as a local feature extractor;

[0187] Step (3.2) feature pyramid extracts image features, which are fused with the original image features extracted by VGG19 in step (3.1) as input image features (Feature Map);

[0188] Step (3.2.1) Feature Pyramid Network (FPN) selects all layers from Conv3_4 (the first VGG) as the network of FPN, wherein Conv6 is an added fully connected layer FC-4096;

[0189] Step (3.2.2) Conv3_4, Conv4_4, Conv5_4, Conv6 are VGG layers, and M3, M4, M5, M6 are connection layers obtained by 1*1 convolution;

[0190] Wherein, Conv3_4 is the fourth convolution kernel of the third convolution layer, and Conv4_4 and Conv5_4 are the same;

[0191] Step (3.2.3) After the up-sampling layer Upsample, the same channel number as the next level is obtained, and an addition operation is performed to obtain the addition layer L3, L4, L5;

[0192] Step (3.2.4) After 3*3 convolution operation, 7*7, 14*14, 28*28, 56*56 are output, and finally the prediction layer P3, P4, P5, P6 is obtained;

[0193] Step (3.2.5) The prediction layer P3, P4, P5, P6 obtained is fused with the image features extracted by VGG19 to obtain the feature map Feature Map;

[0194] Step (3.3) The feature map Feature Map obtained in step (3.2) is output to the next layer, and there are two branches PAFs (affinity field of the joint) and PCM (heat map of the joint), and one Loss (loss function) is output at each stage;

[0195] Step (3.3.1) The iteration formula of branch PAFs is:

[0196] S t =ρ t (F,L t-1 ,S t-1 ),t≥2

[0197] Wherein, ρ t represents the iteration relationship of stage t, F is the feature map Feature Map, L represents the partial affinity, S represents the two-dimensional confidence, and t represents the total number of confidence map stages;

[0198] Step (3.3.2) wherein, the iteration formula of branch PCM is:

[0199] L t =φ t (F,L t-1 ,S t-1 ),t≥2

[0200] where φ t denotes the iterative relationship at stage t;

[0201] The loss function of branch PAFs in step (3.3.3) is:

[0202]

[0203] where, is the loss function of branch PAFs, is the real joint part position map, and W is a binary code, which is 0 at image pixel position p and 1 otherwise; is the square of the difference between the predicted value and the real value;

[0204] The loss function of branch PCM in step (3.3.4) is:

[0205]

[0206] where, is the loss function of branch PAFs, is the real joint part affinity field, is the square of the difference between the predicted value and the real value;

[0207] Step (3.3.5) generates a confidence map of S from the image of two-dimensional joints, which is represented by , where X j,k represents the jth joint of the kth person in the image. The maximum value of is used to represent the final confidence map obtained by multiple joint parts; where the predicted value at point p is:

[0208]

[0209] where exp is the base of e exponential, δ is the control of peak diffusion, is the square of the vector modulus value from point p to the jth part of the kth person;

[0210] Step (3.3.6) X j1,k and X j2,k represent two joints, and there is a pixel point p on the limb, then is the value of the unit vector from j1 to j2 of the kth person, and the vector field at point p is:

[0211]

[0212] where ||X j2,k -X j1,k||2 is the length of the limb of the kth person from j2 part to j1 part;

[0213] The average affinity field of all people obtained in step (3.3.7) is:

[0214]

[0215] Wherein, n c (p) represents the number of non-zero vectors at p in all people;

[0216] Step (3.3.8) in the multi-person scene, the score of the limb is calculated by using the following formula to find the case with the maximum association confidence;

[0217]

[0218] Wherein, E is the association confidence, ||d j2 -d j1 ||2 is the distance of two parts of the body d j2 ,d j1 , and p(u) interpolates the position of the body part d j2 ,d j1 :

[0219] p(u) = (1-u)d j1 +d j2

[0220] Wherein, the integral value is approximated by sampling and equidistant summation of u;

[0221] Step (3.3.9) changes the multi-person detection problem into a bipartite graph matching problem to obtain the optimal solution of the connected points, and finally obtains all possible limb prediction results to connect all the joints of the human body;

[0222] Wherein, the bipartite graph matching is a subset of edges selected in a way that no two edges share a node, and the goal is to find a maximum weight matching for the selected edges;

[0223] Step (3.4) maps the joint coordinates of the two-dimensional pose to the three-dimensional space;

[0224] Step (3.4.1) the joint data of human pose recognition has 18 points, and the human skeleton data is formed by the data of the 18 points, the input is two-dimensional joint coordinates (x Mi ,y Mi )∈R * 2D , and the output is three-dimensional space coordinates (x Ni ,y Ni ,z Ni )∈R *3D ;

[0225] wherein R * 2D is a two-dimensional space, R * 3D is a three-dimensional space;

[0226] Step (3.4.2) inputs the human body posture data as a two-dimensional joint node skeleton model, and maps the output skeleton model in three-dimensional space to a three-dimensional posture;

[0227] Fourth step: through the binding of the 18 joint node coordinates of the three-dimensional space posture information to the virtual character's skeleton joint nodes, the real passenger behavior is synchronized with the virtual passenger action;

[0228] Step (4.1) binds the joint node coordinates of the three-dimensional posture of the skeleton model through spatial coordinate transformation to the joint nodes of the virtual character model in the UE;

[0229] Step (4.2) synchronizes the human body posture behavior to the virtual character through the forward motion calculation of step (2.2);

[0230] Fifth step: input the actual escalator monitoring video into the human body posture recognition network trained in the third step to realize the early warning of passenger behavior;

[0231] The scheme of the embodiment constructs a virtual body in the digital twin environment for the abnormal behavior of passengers, solves the problem of data scarcity of abnormal dangerous behavior in reality, re-trains the human body posture recognition network to realize the recognition of actual passenger behavior, uses virtual reality technology to synchronize and map the actual passenger behavior to the digital virtual space, realizes the monitoring and early warning of dangerous passenger behavior, and protects the life and property safety of the people.

[0232] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and is only for the purpose of description. The protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to the equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A method for constructing escalator passenger abnormal behavior data based on digital twinning, characterized in that, The method comprises the following steps: First step: escalator passenger abnormal behavior digital twin environment construction, the process is as follows: Step (1.1) geometric model construction: the establishment of geometric model is the basic link of digital twin construction, which determines the final implementation effect of digital twin, determines the degree of realism of digital twin, respectively, escalator geometric model and virtual person geometric model are constructed; Step (1.2) scene construction: different parts of the geometric model are virtually assembled to construct the same main body as the reality; Step (1.3) data transmission: twin data is the data used to drive the twin model in digital space with real world real data, wherein the twin data is the video data of passenger behavior and the running data of escalator; Second step: modeling human behavior in digital twin environment, generating a large number of abnormal behavior data sets, the process is as follows: Step (2.1) human kinematics description, in virtual space, in order to make human motion comply with the physical laws of the real world, the human body is simplified as a skeletal model, which is composed of the finite rotation displacement of each joint to form a complete motion posture; Step (2.2) Forward Kinematics (FK), human pose forward dynamics refers to the use of the relative displacement of each joint and relative rotation as input, the process of calculating the whole body pose after motion can be represented by the following formula: Q = FK (R, T) where M is the number of body joints, denotes the three-dimensional pose after the i-th joint motion, denotes the i-th joint position information in the relative displacement matrix, pa(i) denotes the parent joint index of the i-th joint, R pa(i),i denotes the rotation of the i-th joint relative to the parent joint, FK represents a process of recursively passing the displacement matrix and the rotation matrix from the root node to all child nodes, and is represented by the following equation: wherein R i is the global rotation of the ith joint with respect to the rest pose space; Step (2.3) inverse kinematics (IK), i.e. inverse kinematics is the reverse process of forward kinematics, by inputting the desired human joint poses and displacement T, the relative rotation R of the joint can be calculated, which is expressed as: R = IK (P, T) where p i represents the desired pose of the input i-th joint, the resulting rotation should satisfy the following condition: Step (2.4) human skeleton model, which is a simplified model in digital twin to represent the actual passenger behavior, composed of a tree-like hierarchical structure of joint nodes; Step (2.5) passenger abnormal behavior data generation; Step (2.6) a large number of abnormal behavior data are generated according to the types of passenger abnormal behavior; Third step: using a large number of abnormal passenger behaviors generated in the second step to train human posture recognition network I-OpenPose, which adopts VGG19 feature extraction network, and introduces feature pyramid into VGG19 to fuse image feature Feature Map. Feature Map is input into two branches PAFs and PCM, PAFs is used to express the direction of pixel points in posture, and the recognized two-dimensional posture information is mapped to three-dimensional space. In the fourth step, three-dimensional posture information is used to bind virtual passenger skeletal model to realize the behavior synchronization of real passenger and virtual passenger; Wherein, PAFs is joint affinity field, and PCM is joint confidence map; Fourth step: 18 joint coordinates in three-dimensional space are bound to the virtual human skeletal joint to realize the behavior synchronization of real passenger and virtual passenger; Fifth step: the human posture recognition network trained in the third step is input with passenger monitoring video of actual escalator to realize passenger behavior warning.

2. The escalator passenger abnormal behavior data construction method based on digital twinning of claim 1, wherein, The process of step (1.1) is as follows: Step (1.1.1) escalator geometric model drawing, according to the eight systems of escalator, including truss, step system, handrail belt system, guide rail system, handrail device, safety protection device, electrical control system and lubrication system, three-dimensional modeling software is used to construct eight system modules of escalator; Step (1.1.1.1) escalator geometric model driving, the model of step pedal, driving wheel, handrail belt driving wheel, driving motor, step pedal, roller chain and roller are driven by real-time data; Step (1.1.1.2) defines the motion component attributes, respectively defines the step tread, drive wheel, handrail belt drive wheel, drive motor, step tread, roller chain and roller, and defines the motion component data interface; Wherein, the motion component attribute: the operability of the physical engine to realize the displacement and rotation of the motion component; Wherein, the definition of the motion component data interface: the physical engine realizes the start, stop and speed control of the movable component through real data; Step (1.1.2) constructs a character geometric model; Step (1.1.2.1) draws a virtual character geometric model, simplifies the human body model into a virtual model with 18 joint nodes with limited rotational displacement according to the behavior of the character, and the joint nodes are 0 neck, 1 nose, 2 left eye, 3 right eye, 4 left ear, 5 right ear, 6 left shoulder, 7 right shoulder, 8 left elbow, 9 right elbow, 10 left wrist, 11 right wrist, 12 left hip bone, 13 right hip bone, 14 left knee, 15 right knee, 16 left ankle, 17 right ankle; Step (1.1.2.2) virtual character driving, connecting the above-mentioned virtual character joint nodes to form a human skeleton model, using the three-dimensional space posture information of the fourth step to bind the posture of the virtual character model in the physical engine, and controlling the movement of the virtual character in the physical engine; Step (1.1.2.3) reorients the virtual character space position in the above-mentioned physical engine, and sets the initial position of the virtual character; Wherein, the reorientation is to reset the virtual character joint node coordinate space relationship to the joint node space distribution of the initial behavior state.

3. The escalator passenger abnormal behavior data construction method based on digital twinning of claim 2, wherein, In the step (1.2), the virtual assembly is to assemble each system module of the escalator in the physical engine, and finally form a virtual model corresponding to the actual escalator; Step (1.2.1) expands two elevators according to mirror image in the physical engine for a single down escalator, redefines each component of the expanded elevator, and forms a bidirectional carrying elevator; Step (1.2.2) adds a walking staircase between the two elevators; Step (1.2.3) redefines the material and color of each module of the elevator.

4. The escalator passenger abnormal behavior data construction method based on digital twinning of claim 3, wherein, The process of step (1.3) is as follows: Step (1.3.1) The passenger behavior video data is transmitted to the digital twin client after the human body posture recognition in the third step, and a bidirectional communication is established through Socket, using TCP protocol to ensure stable and real-time data transmission; Step (1.3.2) The running data of the escalator is obtained by connecting to the control mainboard of the escalator through the communication bus, and the real-time data of the escalator running is transmitted to the client computer through the host controller; Step (1.3.3) define and drive each part of the escalator, and map the virtual character movement in real time through the results of passenger behavior recognition; Step (1.4) visual interface design: the middle is the digital twin virtual model, and the left and right sides display passenger behavior information and escalator running information.

5. The escalator passenger abnormal behavior data construction method based on digital twinning according to any one of claims 1 to 4, characterized in that, The process of step (2.4) is as follows: Step (2.4.1) Skeleton joint extraction, the performance form of human posture motion is the space transformation of human joint, and the skeleton joint extraction is to extract the space transformation relationship of the tree structure composed of all joints; Step (2.4.2) Joint mapping, the skeleton system in the physical engine is a tree structure, in order to drive the limb action of the virtual character, the joint label obtained before needs to be associated with the corresponding joint in the physical engine skeleton tree; Step (2.4.3) Coordinate transformation, since each original skeleton joint has an independent coordinate system, the coordinate transformation makes all the skeleton joints in the same coordinate system; Wherein, the coordinate transformation is O γ = TO ε , O γ is the space coordinate system after coordinate transformation, O ε is the initial space coordinate system, and T is a space transfer matrix; Step (2.4.4) Data import, the joint data is imported to realize the posture reproduction of the virtual character; Step (2.4.5) There is an inherent correlation between the human skeleton joints, and the key frame refinement action between the initial input action and the end output action needs to be solved by using the inverse motion of step (2.3); Wherein, the key frame is the frame where the key action of the human posture change is located; Step (2.4.6) Frame-by-frame import of joint data can realize the action reproduction of the virtual character by using the virtual character to solve the forward motion of step (2.2).

6. The escalator passenger abnormal behavior data construction method based on digital twinning of claim 5, wherein, The process of step (2.5) is as follows: Step (2.5.1) Import the virtual character in the physical engine and perform skeleton reorientation to realize the mapping of different skeleton joints; Wherein, the skeleton reorientation is to match the relationship between the skeleton joints by matching the settings; Step (2.5.2) Move the character skeleton joint to make the character action, and use the inverse motion of step (2.3) to obtain the key frame refinement action between the initial input action and the output action made; Step (2.5.3) Generate corresponding animation sequences by forward motion of step (2.2) for multiple key frame actions; Step (2.5.4) State integration, integrate multiple animation sequences into the same virtual character by using state machine in the physical engine; Wherein, the state machine in the physical engine is a tool for the transition from one action behavior of the virtual character to another action behavior; Step (2.5.5) Event triggering, place different character roles in the virtual scene, and assign trigger events to the corresponding actions; Step (2.5.6) Data enhancement, realize data enhancement by changing the scene, virtual camera angle and field of view.

7. The escalator passenger abnormal behavior data construction method based on digital twinning according to any one of claims 1 to 4, characterized in that, The process of the third step is as follows: Step (3.1) Extract the original image features by VGG19, and fuse them with the image features extracted by step (3.2) feature pyramid; Wherein, VGG19 is a kind of convolutional neural network used for object recognition, and each layer of neural network uses the output of the previous layer to further extract more complex features until the complexity can be used for object recognition, so each layer can be regarded as a local feature extractor; Step (3.2) Feature pyramid extracts image features, and the original image features extracted by step (3.1) VGG19 are fused into input image features Feature Map; Step (3.3) outputs the feature map obtained in step (3.2) to the next layer, and there are two branches, the affinity field of PAFs node and the heat of PCM node, and each stage outputs a loss; Step (3.4) maps the two-dimensional pose joint coordinates to a three-dimensional space.

8. The escalator passenger abnormal behavior data construction method based on digital twinning of claim 7, wherein, The process of step (3.2) is as follows: Step (3.2.1) selects all layers from Conv3_4 as the network of FPN, wherein Conv6 is an added full connection layer FC-4096; Step (3.2.2) Conv3_4, Conv4_4, Conv5_4, and Conv6 are VGG layers, and M3, M4, M5, and M6 are connection layers obtained by 1*1 convolution; wherein, Conv3_4 is the fourth convolution kernel of the third convolution layer, and Conv4_4 and Conv5_4 are the same; Step (3.2.3) is performed through the up-sampling layer Upsample to make the number of channels the same as the next level, and an addition operation is performed to obtain the addition layer L3, L4, and L5; Step (3.2.4) is performed through a 3*3 convolution operation, and 7*7, 14*14, 28*28, and 56*56 are output, and finally the prediction layer P3, P4, P5, and P6 are obtained; Step (3.2.5) fuses the obtained prediction layer P3, P4, P5, and P6 with the image features extracted by VGG19 to obtain the feature map Feature Map.

9. The escalator passenger abnormal behavior data construction method based on digital twinning of claim 8, wherein, The process of step (3.3) is as follows: Step (3.3.1) the iteration formula of branch PAFs is: S t = p t (F, L t-1 , S t-1 ), t≥2 wherein, ρ t represents the iterative relationship of stage t, F is a feature map, L represents a partial affinity field, S represents a two-dimensional confidence, and t represents the total number of confidence map stages. Step (3.3.2) wherein, the iteration formula of branch PCM is: L t = φ t (F, L t-1 , S t-1 ), t≥2 wherein φ t denotes the iteration relation at stage t; Step (3.3.3) the loss function of branch PAFs is: wherein, is the loss function for branch PAFs, is the ground truth joint part position map, W is a binary encoding, unannotated at image pixel position p, W(p) is 0, otherwise 1; is the squared difference between predicted and ground truth values; Step (3.3.4) the loss function of branch PCM is: wherein, is a loss function for branch PAFs, is a real joint part affinity field, is a squared difference between predicted and real values; Step (3.3.5) generates a confidence map of S from the image of the two-dimensional joint node, using to represent all the confidence maps generated by k persons, using X j,k to represent the jth joint node of the kth person in the image, and using to represent the final confidence map obtained by the multi-person joint node part; wherein the prediction value at the p point is: where exp is the natural constant e, and δ is the spread of the control peak, is the squared magnitude of the vector from point p to the jth location of the kth person. Step (3.3.6) X j1,k and X j2,k represents two joints, and there is a pixel point p on the limb, then The value of the kth person is a unit vector from j2 to j1, and the vector field of point p is: wherein ||X j2,k - X j1,k ||2is the length of the limb of the kth person from j1to j2 Step (3.3.7) the average affinity field of all people is finally obtained: where n c (p) denotes the number of non-zero vectors in all persons at p; Step (3.3.8) in a multi-person scene, the following formula is used to calculate the score of the limb to find the case with the maximum association confidence; where E is the correlation confidence, ||d j2 - d j1 is the distance of the body part d j2 , p(u) interpolates the position of the body part d j1 . j2 , p(u) interpolates the position of the body part d j1 . p(u) = (1 - u)d j1 + d j2 wherein, the integral value is approximated by sampling and equally spaced summation of u; Step (3.3.9) changes the multi-person detection problem into a bipartite graph matching problem to obtain the optimal solution of the connected points, and finally obtains all possible limb prediction results to connect all the joints of the human body; wherein, bipartite graph matching is a subset of edges selected in a way that no two edges share a node, and the goal is to find a maximum weight matching for the selected edges.

10. The escalator passenger abnormal behavior data construction method based on digital twinning of claim 7, wherein, The process of step (3.4) is as follows: The joint data of the human body posture recognition in step (3.4.1) has 18 points, and the human body skeleton data is formed from the 18 points, the input is two-dimensional coordinates (x Mi ,y Mi )∈R * 2D , and the output is three-dimensional space (x Ni ,y Ni ,z Ni )∈R * 3D ; wherein R * 2D is two-dimensional space, R * 3D is three-dimensional space; Step (3.4.2) inputs the human pose data as a two-dimensional joint skeleton model, and the output skeleton model in the three-dimensional space is a three-dimensional pose.

Images