A method for monitoring abnormal behavior of escalator passengers based on numerical twin
Through numerical twin technology, the escalator space model and virtual character model are constructed, and passenger behavior is monitored and early warning is solved, which is the problem that escalators in the existing technology cannot respond to passenger abnormal behavior in a timely manner, and improve passenger safety guarantees.
Patent Information
- Application Number
- CN202311244781.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-25
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-09-25
AI Technical Summary
The monitoring methods of passenger abnormal behavior of existing escalators mainly rely on artificial or infrared and ultrasound sensors, which cannot achieve timely response and cannot effectively ensure passenger safety.
Using the abnormal behavior monitoring method of escalator passengers based on numerical twins, the passenger behavior video data is obtained by constructing a numerical twin escalator space model and virtual character model, and the human posture recognition model is used to extract the coordinate change data of the joint nodes to predict potential dangerous behaviors. Calculate the pose similarity for automatic response.
Timely monitoring and early warning of passenger abnormal behaviors has been achieved, passenger safety guarantees have been improved, and abnormal behaviors have been accurately judged through action similarity assessment.
Smart Images

Figure CN117454288B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of escalator monitoring, and in particular to a method for monitoring abnormal behavior of escalator passengers based on numerical twins. Background Art
[0002] Digital twins make full use of data such as physical models, sensor updates, and operation history, integrate multi-disciplinary, multi-physical quantity, multi-scale, and multi-probability simulation processes, and complete mapping in virtual space, thereby reflecting the entire life cycle of the corresponding physical equipment. Digital twins digitize all elements of the physical world, such as people, objects, and events, and recreate a corresponding "virtual world" in cyberspace, thereby forming a pattern in which the physical world in the physical dimension and the virtual world in the information dimension coexist and blend the virtual and the real. This is to create a virtual model for a physical object in a digital way to simulate its behavior in a real environment.
[0003] At present, the monitoring method of abnormal passenger behavior in escalators is mainly through manual monitoring or infrared, ultrasonic and other sensor methods, which cannot achieve timely response to abnormal passenger behavior.
[0004] Based on this, the present invention provides an abnormal behavior monitoring method for escalator passengers based on numerical twins, which can effectively monitor the abnormal behavior of passengers when riding escalators, and respond before predicting potential dangerous behavior, providing effective protection for passenger safety. Summary of the invention
[0005] To solve the above problems, the present invention provides an escalator passenger abnormal behavior monitoring method based on numerical twins to achieve timely response and early warning of passenger abnormal behavior.
[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:
[0007] A method for monitoring abnormal behavior of escalator passengers based on numerical twins, comprising:
[0008] Step 1): construct a numerical twin escalator space model based on the physical escalator, and drive the escalator space model with the escalator twin data to obtain a digital twin escalator operation model;
[0009] Step 2): Build a virtual character model and pre-generate a passenger abnormal status dataset;
[0010] Step 3): Obtain the video data of passengers riding the escalator, obtain the coordinates of the passenger's human joints through the human posture recognition model, and extract the multi-frame human joint coordinate change data based on the passenger's posture change in combination with the running state of the digital twin escalator running model;
[0011] Step 4): predicting the relative positions of the joints after a preset time according to the changing trend of the relative positions of the joints of the human body between multiple frames;
[0012] Step 5): Based on the predicted relative positions of the joints, the similarity between the passenger posture after the preset time and the posture behavior in the passenger abnormal state dataset is calculated and quantified into a score. When the score is greater than the set threshold, the escalator executes the preset response measures.
[0013] As a possible implementation method, further, in step 1), a numerical twin escalator space model is constructed based on the physical escalator, specifically: characteristic data of the actual main object of the escalator is obtained, and the characteristic data is mapped through a preset relationship to construct a digital twin escalator space model.
[0014] As a possible implementation manner, further, the escalator twin data in step 1) is real-time operation data of the escalator.
[0015] As a possible implementation, further, in step 2), a virtual character model is constructed, and a passenger abnormal state data set is pre-generated, specifically including:
[0016] Acquire the abnormal behavior data of passengers in advance, decompose their posture into several human joints, and associate the joints through the human skeleton connection relationship to obtain a virtual character model;
[0017] Taking the pre-selected joint points on the torso of the virtual character model as the coordinate origin, the coordinates of the remaining joint points are located to generate a passenger abnormal state data set containing different abnormal behavior joint point coordinate sets.
[0018] As a preferred implementation, preferably, the human posture recognition model in step 3) uses the BlazePose module in MediaPipe.
[0019] As a possible implementation, further, in step 3), the behavior video data of passengers riding the escalator is obtained, the coordinates of the passenger's human joints are obtained through the human posture recognition model, and the multi-frame human joint coordinate change data based on the passenger's posture change is extracted in combination with the running state of the digital twin escalator running model, specifically including:
[0020] The image acquisition module is used to obtain the video data of passengers' behavior on the escalator, and the human body joint coordinate data of the passengers is obtained through the human body posture recognition model;
[0021] According to the running direction and speed of the digital twin escalator operation model, the passenger's human joint coordinate change data obtained by the human posture recognition model is reversely superimposed to obtain multi-frame human joint coordinate change data based on the passenger posture changes.
[0022] As a possible implementation, further, in step 4), according to the change trend of the relative positions of the human body joints between multiple frames, predicting the relative positions of the joints after the user presets the time length specifically includes:
[0023] The vector function r(t) about the change of the coordinates of the human joints within the set time length is calculated based on the obtained data of the change of the coordinates of the human joints between multiple frames of the preset time length based on the change of the passenger's posture; wherein, the expression of r(t) is as follows:
[0024] r(t)=(f(t),g(t),h(t))=f(t)i+g(t)j+h(t)k
[0025] f(t) represents the function of the projection of the human joint coordinates on the x-axis over time t; g(t) represents the function of the projection of the human joint coordinates on the y-axis over time t; h(t) represents the function of the projection of the human joint coordinates on the z-axis over time t; i, j, k represent the unit vectors on the x, y, and z axes, respectively;
[0026] Derivate the vector function r(t) and calculate the change r'(t) of the coordinate vector function of the human joint point at the current moment n );
[0027] r'(t)=f'(t)i+g'(t)j+h'(t)k
[0028] r'(t n )=f'(t n )i+g'(t n )j+h'(t n )
[0029] Among them, t n Indicates the current moment, f'(t), g'(t), and h'(t) are the derivatives of f(t), g(t), and h(t), respectively;
[0030] The change r'(t n ), obtain the relative position coordinates (x f ,y f , z f ).
[0031] As a possible implementation, further, by using the change r'(tn ), obtain the relative position coordinates (x f ,y f , z f ), specifically:
[0032] Calculate t n +Δt time, the human body joint coordinate point vector r(t n +Δt), the specific expression is as follows:
[0033] r(t n +Δt)=[f(t n )+f'(t n )·Δt]i+[g(t n )+g'(t n )·Δt]j+[h(t n )+h'(t n )·Δt]k
[0034] Among them, (x f ,y f , z f )=(f(t n )+f'(t n )·Δt,g(t n )+g'(t n )·Δt,h(t n )+h'(t n )·Δt).
[0035] As a possible implementation, further, in step 5), according to the predicted relative positions of the joint points, the similarity between the passenger posture after a preset time and the posture behavior in the passenger abnormal state data set is calculated and quantified into a component, specifically including:
[0036] Taking the pre-selected joint point on the virtual character model in the passenger abnormal state data set as the origin, obtain the distance from the virtual character model to the origin of each joint point except the pre-selected joint point under the same posture, and calculate the deformation coefficient Ai of each joint point of the passenger body according to the obtained distance data;
[0037] According to the obtained deformation coefficient Ai of each joint point of the passenger's body, the relative position coordinates (x fi ',y fi ', z fi '); where (x fi ',y fi ', z fi ') is as follows:
[0038] (x fi ',y fi', z fi ')
[0039] =(Ai·x fi , Ai·y fi , Ai·z fi )
[0040] =(Ai·[f i (t n )+f i '(t n )·Δt],Ai·[g i (t n )+g i '(t n )·Δt],Ai·[h i (t n )+h i '(t n )·Δt])
[0041] Among them, x fi ,y fi 、z fi Respectively represent the x, y, and z coordinates of the i-th joint point after the preset time length; f i (t n ), g i (t n ),h i (t n ) represent the current x, y, and z coordinates of the passenger’s i-th joint point; f i '(t n ), g i '(t n ),h i '(t n ) represent the changes of the passenger’s i-th joint point in the x, y, and z axes at the current moment respectively;
[0042] The joint points that are identical to the pre-selected joint points of the virtual character model in the passenger abnormal state data set are located in the joint points after the preset deformation time, and then the coordinates (x fi ',y fi ', z fi ') to make the located joint points coincide with the coordinate origin, and obtain the coordinates of each joint point after translation (x ni ,y ni , z ni );(x ni ,y ni , z ni ) is expressed as follows:
[0043] (x ni ,yni , z ni )
[0044] =(Ai·[f i (t n )+f i '(t n )·Δt]-a, Ai·[g i (t n )+g i '(t n )·Δt]-b, Ai·[h i (t n )+h i '(t n )·Δt]-c)
[0045] Among them, x ni represents the x-coordinate of the ith joint point after translation, y ni represents the y coordinate of the i-th joint point after translation, z ni represents the z coordinate of the ith joint point after translation; a represents the translation amount of each joint point along the x-axis; b represents the translation amount of each joint point along the y-axis; c represents the translation amount of each joint point along the z-axis;
[0046] According to the coordinates of each joint point after translation, the similarity score S between the passenger posture after the preset time and the posture behavior in the passenger abnormal state data set is calculated. The calculation method is as follows:
[0047]
[0048] when When i =1; when hour,
[0049] Among them, m i Indicates the weight of the i-th joint point of different abnormal behaviors; L si is a set threshold, which represents the distance from the i-th joint point in the passenger abnormal state data set; L i Represents the coordinates of the i-th joint point after translation (x ni ,y ni , z ni ) to the distance of the corresponding joint point coordinates in the passenger abnormal state dataset; n represents the total number of joint points.
[0050] Based on the above, the present invention also provides a computer-readable storage medium, in which at least one instruction, at least one program, code set or instruction set is stored, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement an escalator passenger abnormal behavior monitoring method based on numerical twins as mentioned above.
[0051] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:
[0052] The method for monitoring abnormal behavior of escalator passengers based on numerical twins provided by the present invention can effectively monitor the abnormal behavior of passengers when they are riding on the escalator, and respond before potential dangerous behavior occurs, thereby providing effective protection for passenger safety. In addition, the present invention quantifies the similarity between the passenger posture after a preset time and the posture behavior in the passenger abnormal state data set into a fraction, thereby achieving an effective evaluation of the action similarity, so as to facilitate the accurate judgment of the abnormal behavior of passengers. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0054] Figure 1 It is a schematic diagram of a brief implementation process of the present invention;
[0055] Figure 2 This is a schematic diagram of the joint points extracted by BlazePose recognition;
[0056] Figure 3 It is a schematic diagram of a computer-readable storage medium provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0057] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.
[0058] See attached Figure 1 As shown, this embodiment provides a method for monitoring abnormal behavior of escalator passengers based on numerical twins, including:
[0059] Step 1): construct a numerical twin escalator space model based on the physical escalator, and drive the escalator space model with the escalator twin data to obtain a digital twin escalator operation model; specifically: obtain the feature data of the actual main object of the escalator, construct a digital twin escalator space model through a preset mapping relationship, and drive the escalator space model with the escalator twin data to obtain a digital twin escalator operation model. Among them, the escalator twin data is the real-time operation data of the escalator.
[0060] The escalator is composed of the ladder path (modified plate conveyor) and the handrails on both sides (modified belt conveyor). Its main components include steps, traction chains and sprockets, guide rail system, main transmission system (including motor, reduction device, brake and intermediate transmission link, etc.), driving main shaft, ladder path tensioning device, handrail system, comb plate, escalator frame and electrical system, etc. The main characteristic data to be obtained in this scheme include: the size specification parameters of each escalator component, as well as the lifting height, inclination angle, step width, step pitch and traction chain pitch, etc.
[0061] Step 2): Build a virtual character model and pre-generate a passenger abnormal status data set; specifically including:
[0062] Acquire the abnormal behavior data of passengers in advance, decompose their posture into several human joints, and associate the joints through the human skeleton connection relationship to obtain a virtual character model;
[0063] The selection of human joint points can be adjusted according to actual conditions. Figure 2 As shown, in this embodiment, 17 joint points are selected and associated to generate a virtual character model. The 17 joint points specifically include:
[0064] 1-left eye; 2-right eye; 3-left ear; 4-right ear; 5-mouth; 6-left shoulder; 7-right shoulder; 8-left elbow; 9-right elbow; 10-left wrist; 11-right wrist; 12-left hip, 13-right hip, 14-left knee, 15-right knee, 16-left ankle, 17-right ankle. In this embodiment, the head point is located by left eye 1, right eye 2, left ear 3, right ear 4, and mouth 5.
[0065] The preselected joint point on the torso of the virtual character model is taken as the coordinate origin. In this embodiment, the preselected joint point is 6-left shoulder; that is, the coordinates of the remaining joint points are located with the 6-left shoulder joint point as the coordinate origin, and a passenger abnormal state data set containing joint point coordinate sets of different abnormal behaviors (such as human torso leaning forward, leaning back, etc.) is generated.
[0066] Step 3): Obtain the video data of passengers riding the escalator, obtain the coordinates of the passenger's human joints through the human posture recognition model, and extract the multi-frame human joint coordinate change data based on the passenger's posture change in combination with the running state of the digital twin escalator running model, specifically including:
[0067] The image acquisition module is used to obtain the video data of the passenger's behavior on the escalator, and the human body joint coordinate data of the passenger is obtained through the human body posture recognition model. In this embodiment, the human body posture recognition model uses the BlazePose module in MediaPipe to extract the position and posture information of key joints, which can represent the movement of each joint of the human body.
[0068] According to the running direction and speed of the digital twin escalator operation model, the passenger's human joint coordinate change data obtained by the human posture recognition model is reversely superimposed to eliminate the human joint coordinate change caused by the escalator operation, and obtain the multi-frame human joint coordinate change data based on the passenger posture change.
[0069] Under normal circumstances, the human body posture recognition model obtains the coordinate change data of the passenger's human body joints caused by the passenger's posture change and the escalator operation. Therefore, it is necessary to eliminate the coordinate change of the human body joints caused by the escalator operation in order to obtain the coordinate change of the human body joints caused by the "passenger posture change".
[0070] Taking the second video frame as an example, the coordinates of the passenger's body joint points obtained by the human posture recognition model are (x2, y2, z2). According to the operating data of the escalator operation model, the movement of the escalator from the first frame to the second frame is (Δx, Δy, Δz). In the second video frame, the coordinates of the body joint points based on the change of the passenger posture should be (x2-Δx, y2-Δy, z2-Δz).
[0071] Step 4): According to the changing trend of the relative positions of the human joints between multiple frames, predict the relative positions of the joints after the user presets the time, specifically including:
[0072] The obtained human joint point coordinate change data of the preset time period between multiple frames based on the passenger posture change is used to calculate the vector function r(t) about the human joint point coordinate change within the set time period (wherein the set time period is the time period from the current moment to the set time period); wherein, the expression of r(t) is as follows:
[0073] r(t)=(f(t),g(t),h(t))=f(t)i+g(t)j+h(t)k
[0074] f(t) represents the function of the projection of the human joint coordinates on the x-axis over time t; g(t) represents the function of the projection of the human joint coordinates on the y-axis over time t; h(t) represents the function of the projection of the human joint coordinates on the z-axis over time t; i, j, k represent the unit vectors on the x, y, and z axes, respectively;
[0075] Derivate the vector function r(t) and calculate the change r'(t) of the coordinate vector function of the human joint point at the current moment n );
[0076] r'(t)=f'(t)i+g'(t)j+h'(t)k
[0077] r'(t n )=f'(t n )i+g'(t n )j+h'(t n )
[0078] Among them, t n Indicates the current moment, f'(t), g'(t), and h'(t) are the derivatives of f(t), g(t), and h(t), respectively;
[0079] The change r'(t n ), get the relative position coordinates (x f ,y f , z f ), wherein the preset time length is represented by Δt; this step is specifically as follows:
[0080] Calculate t n +Δt time, the human body joint coordinate point vector r(t n +Δt), the specific expression is as follows:
[0081] r(t n +Δt)=[f(t n )+f'(t n )·Δt]i+[g(t n )+g'(t n )·Δt]j+[h(t n )+h'(t n )·Δt]k
[0082] Among them, (x f ,y f , z f )=(f(t n )+f'(t n )·Δt,g(t n )+g'(t n)·Δt,h(t n )+h'(t n )·Δt). By predicting the passenger’s behavior after a preset period of time, a response warning can be issued before potential dangerous behavior occurs, providing effective protection for passenger safety.
[0083] Step 5): Based on the predicted relative positions of the joints, calculate the similarity between the passenger posture after the preset time and the posture behavior in the passenger abnormal state data set, and quantify it into a component, specifically including:
[0084] Taking the pre-selected joint point on the virtual character model in the passenger abnormal state data set as the origin, obtain the distance from the virtual character model and the passenger's body to the origin from each remaining joint point except the pre-selected joint point under the same posture (the same posture means that the virtual character model and the passenger's body have the same joint angles), where the passenger joint point data can be selected from the joint point coordinate data in any video frame, preferably the first video frame.
[0085] And calculate the deformation coefficient Ai of each joint point of the passenger's body based on the obtained distance data;
[0086] According to the obtained deformation coefficient Ai of each joint point of the passenger's body, the relative position coordinates (x fi ',y fi ', z fi '); where (x fi ',y fi ', z fi ') is as follows:
[0087] (x fi ',y fi ', z fi ')
[0088] =(Ai·x fi , Ai·y fi , Ai·z fi )
[0089] =(Ai·[f i (t n )+f i '(t n )·Δt],Ai·[g i (t n )+g i '(t n )·Δt],Ai·[h i (t n )+h i '(t n )·Δt])
[0090] Among them, xfi ,y fi 、z fi Respectively represent the x, y, and z coordinates of the i-th joint point after the preset time length; f i (t) represents the function of the projection of the coordinates of the i-th joint point on the x-axis over time t; g i (t) represents the function of the projection of the coordinates of the i-th joint point on the y-axis over time t; h i (t) represents the function of the projection of the coordinates of the i-th joint point on the z-axis over time t, f i '(t), g i '(t),h i '(t) are f i (t), g i (t), h i The derivative of (t); t n Indicates the current moment; f i (t n ), g i (t n ),h i (t n ) represent the current x, y, and z coordinates of the passenger’s i-th joint point; f i '(t n ), g i '(t n ),h i '(t n ) represent the changes of the passenger’s i-th joint point in the x, y, and z axes at the current moment. The introduction of the deformation coefficient Ai can eliminate the influence of different passenger body shapes on the quantitative score of the similarity between passenger posture and posture behavior in the passenger abnormal state dataset, thereby improving the accuracy of action similarity evaluation.
[0091] The joint points that are identical to the pre-selected joint points of the virtual character model in the passenger abnormal state data set are located in the joint points after the preset deformation time, and then the coordinates (x fi ',y fi ', z fi ') to make the located joint point coincide with the origin of the coordinates. In this embodiment, when the pre-selected joint point of the virtual character model in the passenger abnormal state data set is 6-left shoulder, therefore, in this step, the coordinates of each joint point after the preset deformation time are synchronously moved so that the 6-left shoulder joint point after the preset deformation time coincides with the origin of the coordinates, and the coordinates of each joint point after translation (x ni ,y ni , z ni ); where (x ni ,y ni , z ni ) is expressed as follows:
[0092] (x ni ,y ni , z ni )
[0093] =(Ai·[f i (t n )+f i '(t n )·Δt]-a, Ai·[g i (t n )+g i '(t n )·Δt]-b, Ai·[h i (t n )+h i '(t n )·Δt]-c)
[0094] Among them, x ni represents the x-coordinate of the ith joint point after translation, y ni represents the y coordinate of the i-th joint point after translation, z ni represents the z coordinate of the ith joint point after translation; a represents the translation amount of each joint point along the x-axis; b represents the translation amount of each joint point along the y-axis; c represents the translation amount of each joint point along the z-axis;
[0095] According to the coordinates of each joint point after translation, the similarity score S between the passenger posture after the preset time and the posture behavior in the passenger abnormal state data set is calculated. When calculating the similarity score S, the pre-selected joint point as the origin of the coordinate (i.e., the 6-left shoulder joint point) does not participate in the calculation of the similarity score S; therefore, the similarity score S is calculated as follows:
[0096]
[0097] when When i =1; when hour,
[0098] Among them, m i Represents the weight of the i-th joint point of different abnormal behaviors (where m i is an empirical value, which is set according to the criticality of the i-th joint angle in the corresponding abnormal behavior); L si is a set threshold, which represents the distance from the i-th joint point in the passenger abnormal state data set; L i Represents the coordinates of the i-th joint point after translation (x ni ,y ni , z ni) to the coordinates of the corresponding joint point in the passenger abnormal state data set (i.e., the i-th joint point in the passenger abnormal state data); n represents the total number of joint points.
[0099] When the score is greater than the set threshold, the escalator executes the preset response measures, which are:
[0100] The behaviors in the real-time passenger video are scored for similarity with each abnormal behavior in the passenger abnormal status data set, and a judgment is made one by one whether they exceed the set score threshold of the corresponding abnormal behavior. If exceeded, the escalator executes the preset response measures. The preset response measures include voice risk warning, escalator deceleration, escalator stop, etc.
[0101] See attached Figure 3 As shown, an embodiment of the present invention also provides a computer-readable storage medium, in which at least one instruction, at least one program, code set or instruction set is stored, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the above-mentioned escalator passenger abnormal behavior monitoring method based on numerical twins.
[0102] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0104] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for monitoring abnormal behavior of escalator passengers based on numerical twins, characterized in that: include: Step 1): construct a numerical twin escalator space model based on the physical escalator, and drive the escalator space model with the escalator twin data to obtain a digital twin escalator operation model; Step 2): Build a virtual character model and pre-generate a passenger abnormal status dataset; Step 3): Obtain the video data of passengers riding the escalator, obtain the coordinates of the passenger's human joints through the human posture recognition model, and extract the multi-frame human joint coordinate change data based on the passenger's posture change in combination with the running state of the digital twin escalator running model; Step 4): predicting the relative positions of the joints after a preset time according to the changing trend of the relative positions of the joints of the human body between multiple frames; Step 5): Based on the predicted relative positions of the joints, calculate the similarity between the passenger posture after the preset time and the posture behavior in the passenger abnormal state data set, and quantify it into a component, specifically including: Taking the pre-selected joint point on the virtual character model in the passenger abnormal state data set as the origin, obtain the distance from the virtual character model to the origin of each joint point except the pre-selected joint point under the same posture, and calculate the deformation coefficient Ai of each joint point of the passenger body according to the obtained distance data; According to the obtained deformation coefficient Ai of each joint point of the passenger's body, the relative position coordinates (x fi ',y fi ', z fi '); where (x fi ',y fi ', z fi ') is as follows: (x fi ’,y fi ’,z fi ’) =(Ai·x fi ,Ai·y fi ,Ai·z fi ) =(Ai·[f i (t n )+f i ’(t n )·Δt],Ai·[g i (t n )+g i ’(t n )·Δt],Ai·[h i (t n )+h i ’(t n )·Δt]) Among them, x fi ,y fi 、z fi Respectively represent the x, y, and z coordinates of the i-th joint point after the preset time; t n Indicates the current moment; f i (t n ), g i (t n ),h i (t n ) represent the current x, y, and z coordinates of the passenger’s i-th joint point; f i '(t n ), g i '(t n ),h i '(t n ) represent the changes of the passenger’s i-th joint point in the x, y, and z axes at the current moment respectively; The joint points that are the same as the pre-selected joint points of the virtual character model in the passenger abnormal state data set are located in the joint points after the preset deformation time, and then the coordinates (x fi ',y fi ', z fi ') to make the located joint points coincide with the coordinate origin, and obtain the coordinates of each joint point after translation (x ni ,y ni , z ni );(x ni ,y ni , z ni ) is expressed as follows: (x ni ,y ni ,z ni ) =(Ai·[f i (t n )+f i ’(t n )·Δt]-a,Ai·[g i (t n )+g i ’(t n )·Δt]-b,Ai·[h i (t n )+h i ’(t n )·Δt]-c) Among them, x ni represents the x-coordinate of the ith joint point after translation, y ni represents the y coordinate of the i-th joint point after translation, z ni represents the z coordinate of the ith joint point after translation; a represents the translation amount of each joint point along the x-axis; b represents the translation amount of each joint point along the y-axis; c represents the translation amount of each joint point along the z-axis; According to the coordinates of each joint point after translation, the similarity score S between the passenger posture after the preset time and the posture behavior in the passenger abnormal state data set is calculated. The calculation method is as follows: when When i =1; when hour, Among them, m i Indicates the weight of the i-th joint point of different abnormal behaviors; L si is a set threshold, which represents the distance from the i-th joint point in the passenger abnormal state data set; L i Represents the coordinates of the i-th joint point after translation (x ni ,y ni , z ni ) to the distance of the corresponding joint point coordinates in the passenger abnormal state data set; n represents the total number of joint points; When the score is greater than the set threshold, the escalator executes the preset response measures.
2. According to claim 1, a method for monitoring abnormal behavior of escalator passengers based on numerical twinning is characterized in that: In step 1), a numerical twin escalator space model is constructed based on the physical escalator, specifically: characteristic data of the actual main object of the escalator is obtained, and the characteristic data is mapped through a preset relationship to construct a digital twin escalator space model.
3. The method for monitoring abnormal behavior of escalator passengers based on numerical twinning according to claim 1 is characterized in that: The escalator twin data described in step 1) is the real-time operation data of the escalator.
4. The method for monitoring abnormal behavior of escalator passengers based on numerical twinning according to claim 1 is characterized in that: In step 2), a virtual character model is constructed and a passenger abnormal status data set is pre-generated, specifically including: Acquire the abnormal behavior data of passengers in advance, decompose their posture into several human joints, and associate the joints through the human skeleton connection relationship to obtain a virtual character model; Taking the pre-selected joint points on the torso of the virtual character model as the coordinate origin, the coordinates of the remaining joint points are located to generate a passenger abnormal state data set containing different abnormal behavior joint point coordinate sets.
5. The method for monitoring abnormal behavior of escalator passengers based on numerical twinning according to claim 1 is characterized in that: In step 3), the human posture recognition model uses the BlazePose module in MediaPipe.
6. The method for monitoring abnormal behavior of escalator passengers based on numerical twinning according to claim 1 is characterized in that: In step 3), the behavior video data of passengers riding the escalator is obtained, the coordinates of the passenger's human joints are obtained through the human posture recognition model, and the multi-frame human joint coordinate change data based on the passenger's posture change is extracted in combination with the running state of the digital twin escalator running model, specifically including: The image acquisition module is used to obtain the video data of the passenger's behavior on the escalator, and the human body joint coordinate data of the passenger is obtained through the human body posture recognition model; According to the running direction and speed of the digital twin escalator operation model, the passenger's human joint coordinate change data obtained by the human posture recognition model is reversely superimposed to obtain multi-frame human joint coordinate change data based on the passenger posture changes.
7. The method for monitoring abnormal behavior of escalator passengers based on numerical twinning according to claim 4 is characterized in that: Step 4) predicts the relative positions of the joints after a preset time according to the changing trend of the relative positions of the joints of the human body between multiple frames, specifically including: The vector function r(t) about the change of the coordinates of the human joints within the set time length is calculated based on the obtained data of the change of the coordinates of the human joints between multiple frames of the preset time length based on the change of the passenger's posture; wherein, the expression of r(t) is as follows: r(t)=(f(t),g(t),h(t))=f(t)i+g(t)j+h(t)k f(t) represents the function of the projection of the human joint coordinates on the x-axis over time t; g(t) represents the function of the projection of the human joint coordinates on the y-axis over time t; h(t) represents the function of the projection of the human joint coordinates on the z-axis over time t; i, j, k represent the unit vectors on the x, y, and z axes, respectively; Derivate the vector function r(t) and calculate the change r'(t) of the coordinate vector function of the human joint point at the current moment n ); r'(t)=f'(t)i+g'(t)j+h'(t)k r’(t n )=f’(t n )i+g’(t n )j+h’(t n )k Among them, t n Indicates the current moment, f'(t), g'(t), and h'(t) are the derivatives of f(t), g(t), and h(t), respectively; The change r'(t n ), obtain the relative position coordinates (x f ,y f , z f ).
8. The method for monitoring abnormal behavior of escalator passengers based on numerical twin according to claim 7 is characterized in that: The change r'(t n ), obtain the relative position coordinates (x f ,y f , z f ), specifically: Calculate t n +Δt time, the human joint coordinate point vector r(t n +Δt), the specific expression is as follows: r(t n +Δt)=[f(t n )+f’(t n )·Δt]i+[g(t n )+g’(t n )·Δt]j+[h(t n )+h’(t n )·Δt]k where (x f , y f , z f ) = (f(t n ) + f'(t n )·Δt, g(t n ) + g'(t n )·Δt, h(t n ) + h'(t n )·Δt).
9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement an escalator passenger abnormal behavior monitoring method based on numerical twins as described in one of claims 1 to 8.
