Escalator risk state monitoring method based on digital twinning

Abstract

The application discloses an escalator risk state monitoring method based on digital twinning, comprising the following steps: step 1): obtaining feature data of an actual escalator subject and constructing a digital twinning escalator operation model for carrying passengers; step 2): constructing a virtual person simplified model and generating a passenger risk state data set in a real boarding environment; step 3): obtaining image data of an actual boarding escalator passenger and obtaining human body joint nodes in the actual boarding passenger image data through a human body posture recognition model, and converting the human body joint nodes into angle data after association; step 4): calculating the similarity of actions in the actual boarding passenger image data and the actions in the passenger risk state data set and quantifying into a score, and when the score is greater than a set threshold, the escalator executes a preset response measure. The monitoring method provided by the application can realize real-time monitoring of passenger states when boarding an escalator, so as to realize timely response of passenger risk states.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of escalator safety monitoring, and particularly relates to an escalator risk state monitoring method based on digital twinning. BACKGROUND

[0002] Digital twinning is a simulation process integrating multi-discipline, multi-physical quantity, multi-scale and multi-probability, which fully utilizes physical models, sensor updates, operation history and other data, and completes mapping in a virtual space, thereby reflecting the whole life cycle process of the corresponding entity equipment. Digital twinning is to digitize all elements such as people, objects and events in the physical world, and to reconstruct a corresponding "virtual world" in the network space, thereby forming a pattern of coexistence and integration of the physical world in the physical dimension and the virtual world in the information dimension. This is to create a virtual model for a physical object in a digital way, thereby simulating its behavior in a real environment.

[0003] At present, the passenger risk state monitoring of the escalator is mainly performed by manual operation or infrared, ultrasonic and other sensors, and the timely response of the passenger risk state cannot be realized.

[0004] Therefore, the purpose of the present application is to provide an escalator passenger risk state monitoring method based on digital twinning, which can realize real-time monitoring of the passenger state when riding the escalator, so as to realize the timely response of the passenger risk state. SUMMARY

[0005] To solve the above problems, the present application provides an escalator risk state monitoring method based on digital twinning, which can realize real-time monitoring and identification of the passenger state of the escalator, and further ensure the timely response of the passenger risk state.

[0006] In order to achieve the above technical purposes, the technical scheme adopted by the present application is as follows:

[0007] An escalator risk state monitoring method based on digital twinning, comprising:

[0008] Step 1): obtaining feature data of an actual escalator body and constructing a digital twinning escalator operation model carrying passengers;

[0009] Step 2): constructing a virtual person simplified model and generating a passenger risk state data set in a real riding environment;

[0010] Step 3): obtaining image data of an actual riding escalator passenger, and obtaining a human body joint node in the actual riding passenger image data through a human body posture recognition model, and converting the human body joint node into angle data after association;

[0011] Step 4): Calculate the similarity of the action in the actual passenger image data and the action in the passenger risk state data set, and quantify it into a score. When the score is greater than a set threshold, the escalator executes a preset response measure.

[0012] As a possible implementation, further, the human posture recognition model selects the pose-body_25caffe model of OpenPose human posture recognition.

[0013] As a possible implementation, further, the digital twin escalator running space model carrying passengers in step 1) specifically includes:

[0014] According to the obtained feature data of the actual body of the escalator, a geometric model of the escalator is constructed, and different parts of the geometric model are virtually assembled to construct a body identical to the actual escalator;

[0015] Obtain real-time running data of the actual escalator, and drive the digital twin model according to the obtained real-time running data of the escalator.

[0016] As a possible implementation, further, the virtual character model is constructed in step 2), and a passenger risk state data set corresponding to the actual scene is generated, specifically including:

[0017] Obtain actual human behavior data, decompose the posture, decompose into a plurality of human joint nodes, and associate the human joint nodes to generate a virtual character simplified model;

[0018] By adjusting the position of the joint node in the virtual character simplified model, a passenger risk state set in the real boarding environment is generated;

[0019] Convert the joint node coordinates of each risk action in the passenger risk state set into joint angles to generate a passenger risk state data set containing a joint angle data set of each risk action.

[0020] As a possible implementation, further, in step 3), image data of actual passengers boarding the escalator is obtained, and human joint nodes in the actual boarding passenger image data are obtained through a human posture recognition model, and the human joint nodes are associated and converted into angle data, specifically including:

[0021] Obtain image data of passengers boarding the escalator through a camera module, and obtain human joint nodes in the actual boarding passenger image data through a human posture recognition model frame by frame, and convert the human joint nodes into joint angle data after association.

[0022] As a possible implementation, further, the step 4) calculates the similarity of the action in the actual passenger image data and the action in the passenger risk state data set, and quantifies it into a score, specifically including the following steps:

[0023] The joint angle data obtained from the first video frame is taken as a reference value A r The difference value A θ between the reference value A r and the joint angle A s of each risk action in the passenger risk state data set is calculated θ The expression is as follows:

[0024] A θ = |Ar-As|

[0025] Take the minimum value in A θ as the set threshold θ;

[0026] Obtain the human joint angle data A n in the current video frame, calculate the difference value δθ between the current human joint angle data A n of the corresponding joint part and the reference value A r ;

[0027] δθ = |An-Ar|

[0028] When δθ>θ, the similarity score of the passenger action in the current video frame and the action in the passenger risk state data set is calculated.

[0029] As a possible implementation, further, the similarity score of the passenger action in the current video frame and the action in the passenger risk state data set is calculated, and the calculation formula is specifically as follows:

[0030]

[0031] Wherein, S represents the similarity score of the passenger action in the current video frame and the action in the passenger risk state data set, j represents the total number of joint angles; m i represents the weight of the i-th joint;

[0032] U i represents the score of the i-th joint, and the expression is as follows:

[0033]

[0034] Wherein, A ni represents the angle of the i-th joint of the current human body, and A si represents the angle of the corresponding i-th joint in the passenger risk state data set.

[0035] As a possible implementation, further, the preset response measure in step 4) includes a voice risk prompt, escalator deceleration, and escalator stop.

[0036] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the escalator risk state monitoring method based on digital twinning when executing the program.

[0037] The application further provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the escalator risk state monitoring method based on digital twinning.

[0038] By adopting the technical scheme, the application has the beneficial effects compared with the prior art:

[0039] The escalator passenger risk state monitoring method based on digital twinning can realize real-time monitoring of passenger state when riding an escalator, so as to realize timely response of the passenger risk state. The risk state is screened by comparing the difference δθ with the joint angle threshold θ, and the video frame with a potential risk state is extracted. This method can greatly reduce the calculation amount and improve the identification efficiency of dangerous actions in the video, so as to ensure the timely response of the passenger risk state. In addition, the application quantifies the action similarity into a score, effectively evaluates the action similarity, and accurately judges the passenger risk state. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0041] Figure 1 is a brief flowchart of the application;

[0042] Figure 2 is a joint point diagram extracted by the human posture recognition model;

[0043] Figure 3 is a schematic diagram of an electronic device provided by the embodiment of the application;

[0044] Figure 4 is a schematic diagram of a computer-readable storage medium provided by the embodiment of the application. DETAILED DESCRIPTION

[0045] The application will be described in further detail below with reference to the drawings and embodiments. It is particularly pointed out that the following embodiments are only for illustrating the application, but not for limiting the scope of the application. Similarly, the following embodiments are only part of the embodiments of the application, not all embodiments, and all other embodiments obtained by those skilled in the art without creative labor are within the scope of the application.

[0046] Referring to the drawings and embodiments Figure 1 The embodiment provides an escalator risk state monitoring method based on digital twinning, which comprises the following steps:

[0047] Step 1): Obtain the feature data of the actual body of the escalator, and construct a digital twin escalator running model carrying passengers;

[0048] The escalator is composed of a ladder path (a modified plate conveyor) and handrails (a modified belt conveyor) on both sides. Its main components include steps, traction chains and chain wheels, guide rail systems, main drive systems (including motors, speed reducers, brakes, and intermediate transmission links, etc.) driving main shafts, ladder path tensioning devices, handrail systems, comb plates, escalator skeletons, and electrical systems, etc. The main body feature data to be obtained in the scheme includes size specification parameters of each elevator component, as well as lifting height, inclination angle, step width, step pitch, and traction chain pitch, etc.

[0049] Among them, the digital twin escalator running model carrying passengers is constructed, specifically comprising the following steps:

[0050] According to the obtained feature data of the actual body of the escalator, a geometric model of the escalator is constructed, and different parts of the geometric model are virtually assembled to construct a body identical to the actual escalator;

[0051] Obtain real-time running data of the actual escalator (wherein the real-time running data includes conveying speed and electrical system related data, etc.), and drive the digital twin model according to the obtained real-time running data of the escalator.

[0052] Step 2): Construct a virtual person simplified model, and generate a passenger risk state data set in a real boarding environment, specifically comprising the following steps:

[0053] Obtain actual human behavior data, decompose the posture into a plurality of human body joint nodes, and correspondingly associate the human body joint nodes according to the human body structure (as shown in the accompanying drawings), to generate a virtual person simplified model; Figure 2

[0054] ​The selection of the human body joint nodes can be adjusted according to actual conditions. In this embodiment, 17 joint nodes are selected and associated to generate a virtual human simplified model. The 17 joint nodes specifically include:

[0055] 1-left eye; 2-right eye; 3-left ear; 4-right ear; 5-mouth; 6-left shoulder; 7-right shoulder; 8-left elbow; 9-right elbow; 10-left wrist; 11-right wrist; 12-left hip; 13-right hip; 14-left knee; 15-right knee; 16-left ankle; and 17-right ankle. In this embodiment, the head point is located through the left eye 1, the right eye 2, the left ear 3, the right ear 4, and the mouth 5, and the corresponding joint angles are calculated by associating the left and right shoulders.

[0056] By adjusting the positions of the joint nodes in the virtual human simplified model, a passenger risk state set in a real ride environment is generated. The passenger risk state set includes various different risk actions, such as leaning forward, leaning backward, squatting, and the like.

[0057] The joint node coordinates of each risk action in the passenger risk state set are converted into joint angles to generate a passenger risk state data set including a joint angle data set of each risk action. Specifically, when generating the joint angle data set of each risk action, the angle values of each joint under the risk action are obtained and sorted into a risk action joint angle data set under the risk action.

[0058] For ease of understanding, the risk state of leaning backward of the body when the passenger rides an elevator is taken as an example:

[0059] First, a body leaning backward risk action human model is formed by adjusting the positions of the joint nodes in the virtual human simplified model. Then, the angle values of each joint under the leaning backward state are obtained by associating the joint nodes, and the angle values of all joints under the leaning backward state are extracted to generate a joint angle data set under the “body leaning backward” risk action. Then, the joint angle data sets under different risk actions are integrated to generate a passenger risk state data set including a joint angle data set of each risk action.

[0060] Step 3): Obtain image data of an actual ride escalator passenger, and obtain human body joint nodes in the actual ride passenger image data through a human body posture recognition model, and convert the human body joint nodes into angle data after association, specifically:

[0061] Image data of a ride escalator passenger is obtained through a camera module. In this embodiment, an IP camera is used as an image source to obtain real-time image data of a ride escalator passenger. Then, human body joint nodes in the actual ride passenger image data are obtained frame by frame through a human body posture recognition model. In this embodiment, the pose-body_25caffe model of the OpenPose human body posture recognition is used to convert the human body joint nodes into joint angle data after association.

[0062] Step 4): Calculate the similarity of the action in the actual passenger image data and the action in the passenger risk state data set, and quantify it into a score; specifically including the following steps:

[0063] Take the joint angle data in the obtained passenger first video frame as the reference value A r , calculate the difference A r between the joint angle reference value A s and the joint angle A θ of each risk action in the passenger risk state data set, A θ The expression is as follows:

[0064] A θ = |Ar-As|

[0065] Take the minimum value of A θ as the set threshold θ (where θ is the joint angle threshold value);

[0066] Obtain the human body joint angle data A n in the current video frame (where it is required that the current video frame ≠ the first video frame), calculate the difference δθ between the current human body joint angle data A n and the reference value A r ;

[0067] δθ = |An-Ar|

[0068] When δθ> θ, calculate the similarity score of the passenger action in the current video frame and the action in the passenger risk state data set. In this scheme, the first video frame is taken as the reference, and the difference between its joint angle A s and the joint angle threshold value θ is taken as the joint angle threshold value θ, and then the difference δθ between the current human body joint angle and the reference value is calculated. Only when δθ> θ, the corresponding action is calculated for the similarity score of the action in the passenger risk state data set. By comparing δθ and the angle threshold value θ, the risk state is screened, and the video frame with potential risk state is extracted. This method can greatly reduce the calculation amount and improve the recognition efficiency of dangerous actions in the video. Then, the similarity of the action in the passenger video frame with potential risk state and the risk action is quantitatively scored, which effectively realizes the effective evaluation of the passenger risk state.

[0069] Wherein, the similarity score of the passenger action in the current video frame and the action in the passenger risk state data set, the calculation formula is as follows:

[0070]

[0071] Wherein, S represents the similarity score of the passenger action in the current video frame and the action in the passenger risk state data set, j represents the total number of joint angles; m i represents the weight of the ith joint; wherein, m i is an empirical value, which is set according to the criticality of the ith joint angle under the corresponding risk action.

[0072] U i represents the score of the ith joint, and its expression is as follows:

[0073]

[0074] Wherein, A ni represents the angle of the ith joint of the current human body, A si represents the angle of the ith joint in the passenger risk state data set.

[0075] When the score S is greater than the set score threshold, the escalator executes the preset response measures. The preset response measures include voice risk prompt, escalator speed reduction, escalator stop, etc. The specific operation principle is as follows:

[0076] When the passenger gets on the elevator, the real-time image data of the passenger is obtained through the IP camera, and the joint angle data in the first frame data of the passenger is taken as the reference value A r . The difference A θ between the reference value A r and the joint angle A s of each risk action in the passenger risk state data set is calculated, and the minimum value in A θ is taken as the joint angle threshold θ. Then the human body joint angle data A n in the current video frame is obtained, the difference δθ between the current human body joint angle data A n and the reference value A r is calculated, and it is judged whether δθ is greater than θ; when δθ is greater than θ, the video frame has potential risk, then the similarity score of the action in the passenger video frame with potential risk and the action in the passenger risk state data set is calculated, and the risk state of the passenger is further evaluated through the quantified action similarity score; when the action similarity score exceeds the set value, it is determined that the passenger is in a risk state (risk state includes: body leaning forward, leaning back, squatting, falling sideways, etc.); when it is determined that the passenger is in a risk state, the preset response measures including voice risk prompt, escalator speed reduction, escalator stop, etc. are executed to ensure the personal safety of the user.

[0077] Refer to the attached Figure 3As shown, the embodiment of the present application also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to realize the above-mentioned escalator risk state monitoring method based on digital twinning.

[0078] Referring to the drawings Figure 4 As shown, the embodiment of the present application also provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to realize the above-mentioned escalator risk state monitoring method based on digital twinning.

[0079] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0080] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the embodiments of the methods of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0081] The above only describes some embodiments of the present application, and does not limit the protection scope of the present application, and any equivalent device or equivalent process transformation made by using the content of the specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An escalator risk state monitoring method based on digital twinning, characterized by, include: Step 1): Obtain the feature data of the actual escalator and construct a digital twin escalator operation model carrying passengers; Step 2): Construct a simplified model of the virtual character and generate a dataset of passenger risk states in a real-world travel environment; Step 3): Obtain image data of actual passengers riding the escalator, and obtain human body joints in the image data of actual passengers through a human posture recognition model. Then, associate the human body joints and convert them into angle data. Step 4): Calculate the similarity between actions in the actual passenger image data and actions in the passenger risk status dataset, and quantify this similarity into a score. This includes the following steps: The joint angle data obtained from the first video frame is used as the reference value A. r Calculate the reference value A for the corresponding joint. r Joint angle A of each risk action in the passenger risk status dataset s The difference A θ A θ The expression is as follows: Take the minimum value in A θ as the set threshold θ; Get human joint angle data A in the current video frame n Calculate the current joint angle data A of the corresponding joint location. n Compared with reference value A r The difference δθ; When δθ > θ, calculate the similarity score between the passenger's actions in the current video frame and the actions in the passenger risk state dataset. The specific calculation formula is as follows: where S represents the similarity score of the passenger motion in the previous video frame to the motion in the passenger risk state dataset, j represents the total number of joint angles; m i represents the weight of the ith joint. U i Scorei represents the score of the i-th joint, expressed as follows: wherein A ni represents the angle of the i th joint of the current human body, A si represents the angle of the i th joint in the passenger risk state data set; When the score exceeds the set threshold, the escalator will execute a preset response.

2. The digital-twin-based escalator risk state monitoring method according to claim 1, characterized by, The human pose recognition model used is the pose-body_25 caffe model of OpenPose human pose recognition.

3. The escalator risk state monitoring method based on digital twinning according to claim 1, characterized in that, Step 1) involves constructing a digital twin escalator operating space model to carry passengers, specifically including: Based on the feature data of the actual escalator body obtained, a geometric model of the escalator is constructed, and different parts of the geometric model are virtually assembled to construct a body identical to the actual escalator. Acquire real-time operating data of actual escalators and drive digital twin models based on the acquired real-time operating data of escalators.

4. The escalator risk state monitoring method based on digital twinning of claim 1, wherein, Step 2) involves constructing a virtual character model and generating a passenger risk status dataset corresponding to the actual scenario, specifically including: Acquire real-world human behavior data, decompose the posture into several human joints, and associate the human joints to generate a simplified virtual human model. By adjusting the positions of key points in a simplified model of a virtual character, a set of passenger risk states in a real-world riding environment is generated. The coordinates of the joints of each risk action in the passenger risk status set are converted into joint angles to generate a passenger risk status dataset containing the joint angle datasets of each risk action.

5. The escalator risk state monitoring method based on digital twinning according to claim 1, characterized in that, Step 3) involves acquiring image data of actual passengers riding the escalator, and using a human posture recognition model to obtain human joints from the actual passenger image data. These human joints are then correlated and converted into angle data. Specifically, this includes: The camera module acquires image data of passengers riding the escalator, and the human body joints in the actual passenger image data are obtained frame by frame through the human posture recognition model, and then the data are correlated and converted into joint angle data.

6. The escalator risk state monitoring method based on digital twinning according to claim 1, characterized in that, The preset response measures in step 4) include voice risk warnings, escalator deceleration, and escalator stopping.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the escalator risk status monitoring method based on digital twin as described in any one of claims 1 to 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When executed by the processor, the program implements the digital twin-based escalator risk status monitoring method as described in any one of claims 1 to 6.

Images