Monitoring system of passenger transportation device, passenger transportation device and monitoring method thereof

Abstract

Provided herein is a monitoring system of a passenger transportation device, the passenger transportation device and a monitoring method thereof. The monitoring system comprises: an imaging sensor and / or a depth sensing sensor configured to sense a monitoring area of the passenger transportation device to obtain a data frame; and a processing device configured to perform data processing on the data frame to monitor whether the monitoring area is abnormal, and comprises: a background acquisition module configured to acquire a background model based on a data frame sensed when the monitoring area is in a normal state; a foreground detection module configured to compare a real-time sensed data frame with the background model to obtain a foreground object; a foreground feature extraction module configured to extract a corresponding foreground object state feature from the foreground object; and a state judgment module configured to judge whether the foreground object is in an abnormal state based on at least the foreground object state feature, and determine that the foreground object is in an abnormal state if the judgment is "yes".

Description

[0001] This application is a divisional application of Chinese patent application No. 201610610026.7, filed on July 29, 2016, entitled “Monitoring System for Passenger Transport Device, Passenger Transport Device and Monitoring Method Thereof”. Technical Field

[0002] This invention relates to the field of passenger transport devices, and more specifically, to a monitoring system and method for passenger transport devices. Background Technology

[0003] Passenger transport devices are commonplace in daily life, serving as tools to improve passenger mobility between floors or shorten walking distances. Examples include escalators, typically used in shopping malls, and moving walkways, often found in large airports.

[0004] Due to their extremely wide applicability and user base, such tools inevitably present various safety hazards during use. One scenario is that certain groups may be unable to use passenger transport due to personal reasons, such as unaccompanied elderly people, disabled people, stray pets, or people carrying oversized items. Another scenario is that some individuals who can use passenger transport may be in situations that could lead to safety hazards, such as suddenly falling on the transport, performing dangerous actions, or overcrowding due to high passenger volume. Effective detection of these behaviors to eliminate safety hazards can further improve the application of passenger transport.

[0005] Furthermore, determining whether a specific event posing a safety hazard has occurred or is about to occur on passenger transport equipment requires processing a large amount of redundant data. If some of this data could be filtered out before analysis and judgment without affecting the accuracy of the judgment, the application of passenger transport equipment would be further improved.

[0006] Furthermore, since people nowadays usually carry one or more wearable devices, if these devices can be used reasonably to monitor passenger transport facilities, the application of passenger transport facilities can be further improved. Summary of the Invention

[0007] The purpose of this invention is to provide monitoring systems for passenger transport devices that can avoid or promptly prevent various forms of safety problems from occurring.

[0008] The present invention also aims to provide monitoring methods for passenger transport devices that can avoid or promptly prevent various forms of safety problems from occurring.

[0009] The present invention also aims to provide a passenger transport device that can avoid or promptly prevent various forms of safety problems from occurring.

[0010] According to one aspect of the present invention, a monitoring system for a passenger transport device is provided, comprising: an imaging sensor and / or a depth sensing sensor for sensing a monitoring area of ​​the passenger transport device to acquire data frames; and a processing device for processing the data frames to monitor whether the monitoring area is abnormal, the processing device being configured to include: a background acquisition module for acquiring a background model based on data frames sensed when the monitoring area is in a normal state; a foreground detection module for comparing real-time sensed data frames with the background model to obtain a foreground object; a foreground feature extraction module for extracting corresponding foreground object identification features from the foreground object; and a state judgment module for determining whether the foreground object belongs to an abnormal group based at least on the foreground object identification features, and determining that the foreground object belongs to an abnormal group if the determination is "yes".

[0011] According to another aspect of the present invention, a monitoring method for a passenger transport device is also provided, comprising: S100, a data frame acquisition step: sensing a monitoring area of ​​the passenger transport device to acquire data frames; S200, a background acquisition step: acquiring a background model based on the data frames sensed in the monitoring area under normal conditions; S300, a foreground detection step: comparing the real-time sensed data frames with the background model to obtain a foreground object; S400, a foreground feature extraction step: extracting corresponding foreground object identification features from the foreground object; and S500, a state judgment step: judging whether the foreground object belongs to an abnormal group based at least on the foreground object identification features, and determining that the foreground object belongs to an abnormal group if the judgment is "yes".

[0012] According to another aspect of the present invention, a monitoring system for a passenger transport device is also provided, comprising: an imaging sensor and / or a depth sensing sensor for sensing a monitoring area of ​​the passenger transport device to acquire data frames; and a processing device for processing the data frames to monitor whether there is an anomaly in the monitoring area, the processing device being configured to include: a background acquisition module for acquiring a background model based on data frames sensed when the monitoring area is in a normal state; a foreground detection module for comparing real-time sensed data frames with the background model to obtain a foreground object; a foreground feature extraction module for extracting corresponding foreground object state features from the foreground object; and a state judgment module for judging whether the foreground object is in an abnormal state based at least on the foreground object state features, and determining that the foreground object is in an abnormal state if the judgment is "yes".

[0013] According to another aspect of the present invention, a monitoring method for a passenger transport device is also provided, comprising: S100, a data frame acquisition step: sensing a monitoring area of ​​the passenger transport device to acquire data frames; S200, a background acquisition step: acquiring a background model based on the data frames sensed in the monitoring area under normal conditions; S300, a foreground detection step: comparing the real-time sensed data frames with the background model to obtain a foreground object; S400, a foreground feature extraction step: extracting corresponding foreground object state features from the foreground object; and S500, a state judgment step: judging whether the foreground object is in an abnormal state based at least on the foreground object state features, and determining that the foreground object is in an abnormal state if the judgment is "yes".

[0014] According to another aspect of the present invention, a monitoring system for a passenger transport device is also provided, comprising: an imaging sensor and / or a depth sensing sensor for sensing a monitoring area of ​​the passenger transport device to obtain a first data frame; a receiving device for receiving a second data frame from a wearable device; the second data frame being obtained by a data sensor of the wearable device sensing the wearable device and its wearer; and a processing device for analyzing and processing the first data frame and the second data frame to monitor whether the operating passenger transport device and / or the wearable device wearer are in a normal state.

[0015] According to another aspect of the present invention, a method for monitoring a passenger transport device is also provided, comprising: S100, a first data frame acquisition step: sensing a monitoring area of ​​the passenger transport device to acquire a first data frame; S200, a second data frame acquisition step: receiving a second data frame from a wearable device; the second data frame is acquired by the data sensor of the wearable device sensing the wearable device and its wearer; and S300, a processing step: analyzing and processing the first data frame and the second data frame to monitor whether the operating passenger transport device and / or the wearable device wearer are in a normal state.

[0016] According to another aspect of the present invention, a monitoring system for a passenger transport device is also provided, comprising: a depth sensing sensor for sensing a monitoring area of ​​the passenger transport device to obtain a first data frame; an auxiliary sensor group for sensing the monitoring area of ​​the passenger transport device to obtain a second data frame group; a preprocessing device for performing probability distribution processing on the data frames in the second data frame group to obtain a preprocessed second data frame group; and a processing device for analyzing and processing the first data frame and the preprocessed second data frame group to monitor whether a specific event occurs in the operating passenger transport device.

[0017] According to another aspect of the present invention, a method for monitoring a passenger transport device is also provided, comprising: S100, a first data frame acquisition step, sensing a monitoring area of ​​the passenger transport device to acquire a first data frame; S200, a second data frame group acquisition step, sensing a monitoring area of ​​the passenger transport device to acquire a second data frame group; S300, a preprocessing step, performing probability distribution processing on the data frames in the second data frame group to acquire a preprocessed second data frame group; and S400, a state determination step, used to analyze and process the first data frame and the preprocessed second data frame group to monitor whether a specific event has occurred in the operating passenger transport device.

[0018] According to another aspect of the invention, a passenger transport device is also provided, which includes the monitoring system as described above. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of an embodiment of the passenger transport device and its monitoring system of the present invention.

[0020] Figure 2 This is a schematic diagram of another embodiment of the passenger transport device and its monitoring system of the present invention.

[0021] Figure 3This is a schematic diagram of yet another embodiment of the passenger transport device and its monitoring system of the present invention.

[0022] Figure 4 This is a schematic diagram of another embodiment of the passenger transport device and monitoring system of the present invention.

[0023] Figure 5 This is a schematic diagram of the control steps of an embodiment of the monitoring method for the moving handrail of the passenger transport device of the present invention.

[0024] Figure 6 This is a schematic diagram of the control steps of another embodiment of the monitoring method for the moving handrail of the passenger transport device of the present invention.

[0025] Figure 7 This is a schematic diagram of the control steps of another embodiment of the monitoring method for the movable handrail of the passenger transport device of the present invention.

[0026] Figure 8 This is a schematic diagram of control steps for another embodiment of the monitoring method for the moving handrail of the passenger transport device of the present invention. Detailed Implementation

[0027] The invention will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments of the invention are illustrated. However, the invention may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to make this disclosure thorough and complete, and to fully convey the inventive concept to those skilled in the art. In the drawings, the same reference numerals refer to the same elements or components, and therefore, descriptions of them will be omitted.

[0028] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different processing devices and / or microcontroller devices.

[0029] In this invention, the passenger transport device includes escalators and moving walkways. The following illustrated embodiments use escalators as an example to describe in detail the monitoring system and method of the passenger transport device according to embodiments of the present invention. However, it should be understood that the monitoring system and method for escalators in the following embodiments can also be applied analogously to moving walkways, and any necessary improvements, such as adaptability modifications, can be learned by those skilled in the art through the teachings of these embodiments.

[0030] In this invention, the acquisition of various data frames relies more or less on imaging sensors and / or depth sensing sensors, and the two types of sensors used will be further explained below.

[0031] As an example, the imaging sensor can be any type of 2D image sensor. It should be understood that any image sensor capable of capturing image frames including pixel brightness information can be used here. Of course, an image sensor capable of capturing image frames including pixel brightness information and color information (e.g., RGB information) can also be used here.

[0032] Depth sensing sensors can be any 1D, 2D, 3D depth sensor or combination thereof. Such sensors can operate in optical, electromagnetic, or acoustic modes capable of generating depth maps (also known as point clouds or occupancy grids) of corresponding dimensions. Various depth sensing sensor technologies and devices include, but are not limited to, structured light measurement, phase-shift measurement, time-of-flight measurement, stereo triangulation, optical triangulation plates, light field cameras, coded aperture cameras, computational imaging techniques, simultaneous localization and mapping (SLAM), imaging radar, imaging sonar, scanning LiDAR, flash LiDAR, passive infrared (PIR) sensors, and small focal plane arrays (FPAs), or combinations thereof. Different technologies may include active (transmitting and receiving signals) or passive (receiving signals only) modes and can operate in electromagnetic or acoustic modes (such as visual, infrared, etc.). Using depth sensing can have specific advantages beyond conventional 2D imaging, and using infrared sensing can have specific benefits beyond visible spectrum imaging. Alternatively or additionally, the sensor can be an infrared sensor with one or more pixel spatial resolutions, such as a passive infrared (PIR) sensor or a small IR focal plane array (FPA).

[0033] It should be noted that there are qualitative and quantitative differences between 2D imaging sensors (such as conventional security cameras) and 1D, 2D, or 3D depth sensing sensors in the extent to which depth sensing offers many advantages. In 2D imaging, the reflected colors (a mixture of wavelengths) from the first object in each radial direction from the imager are captured. The 2D image can then include a combined spectrum of the source illumination and the spectral reflectance coefficients of objects in the scene. A 2D image can only be roughly interpreted as a picture by a person. In 1D, 2D, or 3D depth sensing sensors, color (spectral) information is absent; more precisely, the distance (depth, range) to the first reflecting object in either the radial (1D) or directional (2D, 3D) direction from the sensor is captured. 1D, 2D, and 3D techniques may have inherent maximum detectable range limits and may have spatial resolution relatively lower than typical 2D imagers. Compared to conventional 2D imaging, using 1D, 2D, or 3D depth sensing can advantageously provide improved operation, better separation of occluded objects, and better privacy protection, in terms of relative immunity to ambient lighting problems. Using infrared sensing can have specific benefits beyond visible spectrum imaging. For example, 2D images cannot be converted into depth maps, and depth maps cannot be converted into 2D images (e.g., artificially assigning continuous colors or grayscale to continuous depths would roughly resemble how a person sees a 2D image to roughly interpret a depth map, which is not an image in the conventional sense).

[0034] See appendix Figure 1 The illustration shows a monitoring system 1200 for a passenger transport device 1100. The monitoring system 1200 includes an imaging sensor and / or a depth sensor 1210 for sensing the monitoring area of ​​the passenger transport device 1100 to acquire data frames. Its operating principle has been described in detail above and will not be repeated here. Furthermore, the monitoring system 1200 also includes a processing unit 1220 for processing the data frames to monitor whether the monitoring area is abnormal. This enables real-time monitoring of the monitoring area of ​​the passenger transport device 1100 to provide necessary warnings or maintenance, thereby eliminating safety hazards caused by the presence of various abnormal groups.

[0035] Specifically, the processing device 1220 is configured to include: a background acquisition module 1221, a foreground detection module 1222, a foreground feature extraction module 1223, and a state determination module 1224. The background acquisition module 1221 acquires a background model based on data frames sensed while the monitoring area is in a normal state. The foreground detection module 1222 compares the real-time sensed data frames with the background model to obtain foreground objects. The foreground feature extraction module 1223 extracts corresponding foreground object identification features from the foreground objects. The state determination module 1224 determines whether the foreground object belongs to an abnormal group based at least on the foreground object identification features, and confirms that the foreground object belongs to an abnormal group if the determination is "yes". In the aforementioned processing device 1220, the background acquisition module 1221 provides a background model as a reference benchmark for comparison; and by processing the real-time sensed data frames and the background model in the foreground detection module 1222, the foreground object to be analyzed is further acquired; then, the foreground feature extraction module 1223 extracts foreground object identification features from the foreground object that can assist in making the final judgment, and finally, the state judgment module 1224 determines whether the foreground object belongs to an abnormal group based at least on the foreground object identification features. This facilitates further warnings or other operations, prevents abnormal groups from using the passenger transport device 1100 or avoids reminding abnormal groups who have already started using the passenger transport device 1100 to be extra careful, thereby eliminating potential safety hazards caused by abnormal groups.

[0036] As used throughout this paper, background learning can be achieved through methods such as Gaussian Mixture Model, Codebook Model, Principal Component Analysis (PCA), and Robust PCA. Foreground object detection can be achieved through methods such as inter-frame differencing, morphological operators like erosion and dilation, and temporal, spatial, or spatiotemporal kernel convolution. Feature extraction can include estimation of temporal, spatial, or spatiotemporal gradients; histograms of color, depth, and arbitrary parameters; parametric models (e.g., human or animal skeleton models); and so on. Determining whether a set of features belongs to a specific category can be achieved through distance functions. For example, these distance functions could be p-norm distance (including box metrics and Euclidean distance), eccentric Pearson correlation, etc. Alternatively, the judgment that a set of features belongs to a specific category can be given in a probabilistic manner, for example, based on methods of single or joint Bayesian inference.

[0037] The foregoing embodiments also have several improvements or modifications, some of which will be described below.

[0038] Optionally, the foreground feature extraction module 1223 further includes an abnormal human body model generation submodule 1223a, which is used to define the abnormal human body model based on non-self-learning human body modeling and / or self-learning human body modeling, such as skeletal graph models and / or binary graph models. Skeletal graph models and binary graph models are both relatively mature human body modeling technologies, and well-known skeletal graph models and binary graph models in the prior art can be used to generate the abnormal human body model of interest in this embodiment. Furthermore, the self-learning human body model is updated in real-time or periodically based on the continuous learning and improvement of the monitoring system 1200 during use, or based on the continuous updates of the shared database. For example, as a practical application example, the abnormal human body model can cover the elderly, disabled persons, or persons carrying oversized objects, etc.

[0039] As used throughout this paper, the model can learn from data using deep learning techniques, including but not limited to deep neural networks, convolutional neural networks, recursive neural networks, dictionary learning, and bag-of-visual-words. Autonomous learning capabilities can be achieved through reinforcement learning and inverse reinforcement learning. Furthermore, non-learning modeling techniques can be applied to situations where parametric models (such as skeletal models) are predefined as normal or abnormal, and where regions of the parametric space (such as inappropriate limb positions) are predefined as normal or abnormal. Any of these techniques and models can be temporal, spatial, or spatiotemporally based.

[0040] Optionally, the foreground object identification features extracted by the foreground feature extraction module 1223 include one or more of the color, size, shape, and speed of the human body in the foreground object. The state judgment module 1224 determines whether the foreground object belongs to an abnormal group based on the color and / or size and / or speed of the human body in the foreground object. Further, when the color and / or size and / or speed of the human body in the foreground object falls within the abnormal human body model, the state judgment module 1224 determines that the foreground object belongs to an abnormal group. Those skilled in the art should understand that, in practical applications, based on the teachings of the embodiments disclosed herein, the foreground feature extraction module can also extract other foreground object identification features as actually needed to achieve the purposes of the disclosure herein.

[0041] For example, as a practical application, if a specific part of the human body (e.g., the head) in the foreground object is gray or white, the human body in that foreground object may belong to an elderly person in an abnormal human anatomy model. Similarly, if the vertical dimension (e.g., height) of the human body in the foreground object is too small, the human body in that foreground object may belong to a hunched elderly person or a person with a leg injury in an abnormal human anatomy model. Furthermore, if the vertical or horizontal dimension of the human body in the foreground object is too large, the human body in that foreground object may belong to a person carrying an oversized object. Also, if the speed of the human body in the foreground object is too slow, the human body in that foreground object may belong to an elderly person, a person with a disability, or a person carrying an oversized object, etc. Optionally, combining multiple foreground object identification features in data processing can further improve the accuracy of recognition.

[0042] Optionally, the foreground feature extraction module 1223 further includes an abnormal object model generation submodule 1223b, which is used to define abnormal object models based on non-self-learning object modeling and / or self-learning object modeling. The self-learning object modeling updates the abnormal object model in real-time or periodically based on the continuous learning and improvement of the monitoring system 1200 during use, or on the continuous updates of the shared database. For example, as a practical application example, the abnormal object model can cover objects exceeding the carrying capacity or permitted passage capacity of the passenger transport device 1100, etc.

[0043] Optionally, the foreground object identification features extracted by the foreground feature extraction module 1223 also include the size and / or shape of objects in the foreground object. The condition judgment module determines whether the foreground object belongs to an abnormal group based on the size and / or shape of objects in the foreground object. Further, when the size and / or shape of objects in the foreground object fall into the abnormal object model, the state judgment module 1224 determines that the foreground object belongs to an abnormal group.

[0044] For example, as a practical application, if an object in the foreground is too large or too long, it may fall within the range of an anomalous object model. Similarly, if the shape of an object in the foreground exceeds specifications, it may also fall within the range of an anomalous object model. Optionally, combining multiple foreground object identification features during data processing can further improve the accuracy of recognition.

[0045] Furthermore, when the foreground object belongs to an abnormal group, the foreground feature extraction module 1223 is also used to extract the corresponding foreground object marker features around the abnormal group from the foreground object; the state judgment module 1224 is also used to determine that the foreground object is in an abnormal state based at least on the corresponding foreground object marker features around the abnormal group, and to determine that the foreground object is in an abnormal state if the judgment is "yes". In some practical applications, simply identifying an abnormal group is not enough to trigger a warning behavior. For example, when an elderly person is accompanied by a relative, the possibility of them being in danger on the passenger transport device 1100 is relatively small. Therefore, further analysis can be performed to identify the foreground object marker features around them, and then a judgment can be made on whether the abnormal group is ultimately in an abnormal state.

[0046] For example, as a practical application, the presence of relatives accompanying elderly or disabled people can be identified based on the foreground object features around the abnormal group. If no relatives are present, the abnormal group is considered to be in an abnormal state.

[0047] Optionally, the foreground feature extraction module 1223 also includes a pet model generation submodule 1223c, which is used to define a pet model based on non-self-learning pet modeling and / or self-learning pet modeling. The self-learning pet modeling updates the pet model in real-time or periodically based on the continuous learning and improvement of the monitoring system 1200 during use, or on continuous updates to the shared database. For example, as an example, the original pet model already includes common pets such as cats and dogs. At a certain time, pet pigs become popular in the market, and pet pigs can be automatically incorporated into the pet model through self-learning pet modeling.

[0048] Optionally, the foreground object identification features extracted by the foreground feature extraction module 1223 include the shape and / or size and / or color of the pet in the foreground object, and the state judgment module 1224 determines whether the foreground object belongs to an abnormal group based on the shape and / or size and / or color of the pet in the foreground object. Further, when the size and / or shape of the pet in the foreground object falls within the pet model, the state judgment module 1224 determines that the foreground object belongs to an abnormal group.

[0049] For example, as a practical application, when the size, shape, and color of an object in the foreground object are similar to a pet (e.g., a cat), it will fall into the pet model.

[0050] Optionally, in certain situations, an accurate judgment cannot be made based on a single frame. For example, it may be necessary to analyze the motion trend of the foreground object separately or to analyze it in combination with other features to obtain the required information. In this case, the processing device 1220 can be further improved. The processing device 1220 is configured to further include a trajectory generation module 1225, which is used to generate a trajectory of change of foreground object identification features based on the foreground object identification features extracted by the foreground feature extraction module 1223 from the foreground objects corresponding to multiple consecutive data frames. Correspondingly, the state judgment module 1224, which is capable of analysis and judgment, is further configured to: predict whether the foreground object is about to belong to an abnormal group based on the trajectory of change of foreground object identification features, and determine that the foreground object is about to belong to an abnormal group if the judgment is "yes".

[0051] As used throughout this paper, trajectory generation can be achieved using Bayesian estimation techniques based on feature vectors. Specifically, methods such as Kalman filtering and particle filtering can be employed. These feature vectors can include one or more predefined temporal, spatial, or spatiotemporal features.

[0052] For example, as a practical application, if the presence of a pet cat at the elevator entrance / exit has been identified through the acquired foreground object features, but the trajectory generation module 1225 identifies the pet cat's movement trajectory as moving away from the elevator entrance / exit, then based on the pet cat's changing trajectory, it can be determined that there is no risk of it walking onto the elevator; therefore, no corresponding warning is necessary.

[0053] In addition, to enhance and improve the various effects of the foregoing embodiments in various aspects, several improvements and modifications based on the foregoing embodiments are provided herein.

[0054] Optionally, the status judgment module 1224 is further configured to determine that the foreground object belongs to an abnormal group only when the judgment results of at least two consecutive data frames are both that the foreground object belongs to an abnormal group. With this setting, the foreground object is only determined to belong to an abnormal group after multiple monitoring sessions and obtaining the same judgment result. This further improves the accuracy of the judgment and reduces the false judgment rate.

[0055] Optionally, the imaging sensor and / or depth sensor 1210 senses and acquires data frames within a predetermined time period every predetermined time interval for processing by the processing device 1220. This configuration allows for intermittent detection, achieving a similar level of monitoring effectiveness. For example, during evening hours, fewer users utilize the passenger transport device 1100. In this case, lower-frequency monitoring saves on monitoring costs while still achieving a similar level of detection and security.

[0056] Optionally, since the imaging sensor and / or depth sensor 1210 have more accurate sensing results within a certain area, multiple imaging sensors and / or depth sensors 1210 are arranged to cover the entrance and / or exit and / or the entire length of the passenger transport device 1100 in order to improve the accuracy of detection.

[0057] Optionally, the monitoring system 1200 also includes an alarm unit 1230, which is triggered by the status judgment module 1224 when it determines that the foreground object belongs to an abnormal group. Accordingly, the processing device 1220 is also configured to trigger an output signal to the passenger transport device 1100 and / or the building's elevator maintenance center when the status judgment module 1224 determines that the foreground object belongs to an abnormal group, in order to prevent safety accidents from occurring.

[0058] See Figure 5 Furthermore, a monitoring method for passenger transport devices is provided to be used in conjunction with the monitoring system described in the foregoing embodiments. This monitoring method includes the following steps: S100, a data frame acquisition step: sensing the monitoring area of ​​the passenger transport device to acquire data frames; S200, a background acquisition step: acquiring a background model based on the data frames sensed when the monitoring area is in a normal state; S300, a foreground detection step: comparing the real-time sensed data frames with the background model to obtain foreground objects; S400, a foreground feature extraction step: extracting corresponding foreground object identification features from the foreground objects; and S500, a state judgment step: judging whether the foreground object belongs to an abnormal group based at least on the foreground object identification features, and determining that the foreground object belongs to an abnormal group if the judgment is "yes". This method can accurately and promptly identify whether an abnormal group is using or about to use the passenger transport device, and feed the information back to the monitoring center or maintenance personnel so that they can respond promptly and take actions such as issuing warnings, slowing down the device, or even braking the device according to the actual situation.

[0059] In addition, to improve the practicality or accuracy of the monitoring method in the foregoing embodiments, several improvements or modifications to the various steps therein are also provided.

[0060] Optionally, to facilitate the identification of abnormal human bodies, S400 also includes S410, an abnormal human body model generation sub-step, used to define abnormal human body models based on skeletal graph models and / or binary graph models and / or human self-learning modeling. As an example of practical application, abnormal human body models can cover the elderly, disabled persons, or persons carrying oversized objects, etc. This step mainly provides a reference benchmark for subsequent comparison and judgment, making it relatively easy to identify which objects are abnormal human body models of interest.

[0061] Optionally, in S410, the extracted foreground object identifier features include the color and / or size and / or speed of the human body in the foreground object; in S500, it is determined whether the foreground object belongs to an abnormal group based on the color and / or size and / or speed of the human body in the foreground object. Optionally, when the color and / or size and / or speed of the human body in the foreground object falls within the abnormal human body model, it is determined that the foreground object belongs to an abnormal group. A specific determination method is provided here, namely, identifying whether a foreground object belongs to an abnormal group through some foreground object identifier features that can highlight the human body in the foreground object. These identifier features can be color and / or size and / or speed, or other single features or combinations of features. As an example of practical application, for example, when the color of a specific part of the human body in the foreground object (e.g., the head) is gray or white, the human body in the foreground object may belong to an elderly person in the abnormal human body model. Similarly, when the vertical dimension (e.g., height) of the human body in the foreground object is too small, the human body in the foreground object may belong to a hunched elderly person or a disabled person with a leg injury in the abnormal human body model. For example, if the human figure in a foreground object is too large in the vertical or horizontal direction, the person in the foreground object may be carrying an oversized object. Similarly, if the human figure in a foreground object is moving too slowly, the person may be elderly, disabled, or carrying an oversized object. Optionally, combining multiple foreground object identification features during data processing can further improve the accuracy of recognition.

[0062] Optionally, to facilitate the identification of anomalous objects, S400 also includes S420, an anomalous object model generation sub-step: used to define an anomalous object model based on non-self-learning object modeling and / or self-learning object modeling. As an example of practical application, an anomalous object model may cover objects exceeding the carrying capacity or permitted passage capacity of a passenger transport device, etc. This step mainly provides a reference benchmark for subsequent comparison and judgment, making it relatively easy to identify which objects are anomalous object models of interest.

[0063] Optionally, in S420, the extracted foreground object identification features also include the size and / or shape of objects within the foreground object; in S500, the size and / or shape of objects within the foreground object are used to determine whether the foreground object belongs to an anomalous group. Further, if the size and / or shape of an object within the foreground object falls within the anomalous object model, the foreground object is determined to belong to an anomalous group. A specific determination method is provided here, namely, identifying whether an object belongs to an anomalous group through foreground object identification features that can highlight the foreground object. These features can be size and / or shape, or other single or combined features. As an example of practical application, for instance, when the size of an object in the foreground object is too large or too long, the object may fall within the range of the anomalous object model. Similarly, when the shape of an object in the foreground object exceeds specifications, the object may fall within the range of the anomalous object model. Optionally, combining multiple foreground object identification features for data processing can further improve the accuracy of identification.

[0064] Furthermore, when the foreground object belongs to an abnormal group, in S400, the corresponding foreground object marker features around the abnormal group are extracted from the foreground object; correspondingly, in S500, the foreground object is determined to be in an abnormal state based at least on the corresponding foreground object marker features around the abnormal group, and if the determination is "yes", the foreground object is confirmed to be in an abnormal state. In some practical applications, simply identifying an abnormal group is insufficient to trigger a warning action. For example, when an elderly person is accompanied by a relative, the likelihood of them being in danger on a passenger transport device is relatively low. Therefore, further analysis can be performed to identify the foreground object marker features around them, and then a final judgment can be made as to whether the abnormal group is ultimately in an abnormal state.

[0065] For example, as a practical application, the presence of relatives accompanying elderly or disabled people can be identified based on the foreground object features around the abnormal group. If no relatives are present, the abnormal group is considered to be in an abnormal state.

[0066] Optionally, S400 also includes S430, a pet model generation sub-step, which defines a pet model based on non-self-learning pet modeling and / or self-learning pet modeling. As an example of practical application, the pet model can cover various common pets on the market. This step mainly provides a reference benchmark for subsequent comparison and judgment, making it relatively easy to identify which objects are the pet models of interest.

[0067] Optionally, in S430, the extracted foreground object identifier features include the shape and / or size and / or color of the pet in the foreground object; in S500, it is determined whether the foreground object belongs to an anomalous group based on the shape and / or size and / or color of the pet in the foreground object. Further, if the size and / or shape of the pet in the foreground object falls within the pet model, it is determined that the foreground object belongs to an anomalous group.

[0068] For example, as a practical application, when the size, shape, and color of an object in the foreground object are similar to a pet (e.g., a cat), it will fall into the pet model.

[0069] Optionally, in certain situations, an accurate judgment cannot be made based on a single frame. For example, it may be necessary to analyze the motion trend of the foreground object separately or to analyze it in combination with other features to obtain the required information. In this case, the processing device can be further improved by including a trajectory generation module. Correspondingly, S600 should be added to the monitoring method, the trajectory generation step: generating a trajectory about the change of the foreground object's identification features based on the foreground object's identification features extracted from the foreground object in multiple consecutive data frames in S400. Correspondingly, in S500, based on the trajectory of the change of the foreground object's identification features, it is predicted whether the foreground object is about to belong to an anomalous group, and if the judgment is "yes", it is determined that the foreground object is about to belong to an anomalous group.

[0070] For example, as a practical application, if the presence of a pet cat at the elevator entrance / exit has been identified through the acquired foreground object features, but the trajectory generation module indicates that the cat's movement trajectory is away from the elevator entrance / exit, then based on the cat's changing trajectory, it can be determined that there is no risk of the cat walking onto the elevator; therefore, no warning is necessary.

[0071] In addition, to enhance and improve the various effects of the foregoing embodiments in various aspects, several improvements and modifications based on the foregoing embodiments are provided herein.

[0072] Optionally, in S500, the foreground object is determined to belong to an abnormal group only if the judgment results of at least two consecutive data frames are both positive. This setting ensures that the foreground object is only classified as belonging to an abnormal group after multiple monitoring sessions and obtaining the same judgment result. This further improves the accuracy of the judgment and reduces the false positive rate.

[0073] Optionally, data frames within a predetermined time period are sensed and acquired at regular intervals for processing by the processing device. This setup allows for intermittent monitoring, achieving a similar level of effectiveness. For example, during evening hours, fewer users utilize passenger transport devices. In this case, lower-frequency monitoring saves on monitoring costs while still achieving comparable detection and security results.

[0074] Optionally, it also includes S700, an alarm procedure: triggering alarm unit 1230 upon determining that the foreground object belongs to an abnormal group, to alert the monitoring center or maintenance personnel to respond and eliminate the security hazard.

[0075] See appendix Figure 2 The document illustrates a monitoring system 2200 for a passenger transport device 2100. The monitoring system 2200 includes: an imaging sensor and / or a depth sensor 2210 for sensing a monitoring area of ​​the passenger transport device 2100 to acquire data frames; and a processing device 2220 for processing the data frames to monitor for anomalies within the monitoring area. Specifically, the processing device 2220 is configured to include: a background acquisition module 2221 for acquiring a background model based on data frames sensed when the monitoring area is in a normal state; a foreground detection module 2222 for comparing the real-time sensed data frames with the background model to obtain a foreground object; a foreground feature extraction module 2223 for extracting corresponding foreground object state features from the foreground object; and a state judgment module 2224 for judging whether the foreground object is in an abnormal state based at least on the foreground object state features, and determining that the foreground object is in an abnormal state if the judgment is "yes". In the aforementioned processing device 2220, the background acquisition module 2221 provides a background model as a comparison benchmark; and by processing the real-time sensed data frames and the background model in the foreground detection module 2222, the foreground object to be analyzed is further acquired; then, the foreground feature extraction module 2223 extracts the foreground object state features that can assist in making the final judgment from the foreground object, and finally, the state judgment module 2224 determines whether the foreground object is in an abnormal state based at least on the foreground object state features. This facilitates further warnings or other operations, avoids abnormal states that may cause safety hazards on the passenger transport device 2100, or warns objects that are already in an abnormal state to be extra cautious, thereby eliminating potential safety hazards.

[0076] The foregoing embodiments also have several improvements or modifications, some of which will be described below.

[0077] Optionally, the foreground feature extraction module 2223 includes a scene model generation submodule 2223a, which is used to define a danger zone based on data frames sensed in the normal state of the monitoring area of ​​the passenger transport device 2100 and / or a scene self-learning model. As an example, the danger zone can be pre-specified, such as by the installer or user identifying a specific physical area on a computer. This danger zone is a specific area on the passenger transport device 2100 that may be dangerous under normal circumstances, and may change depending on the actual situation. For example, in one example, the entrance to the passenger transport device 2100 may be defined as a danger zone by the sensed data frames and / or the scene self-learning model. If there is a high density of people in this danger zone, it indicates a potential for problems such as stampedes.

[0078] Optionally, the foreground object state features extracted by the foreground feature extraction module 2223 include one or more of the foreground object's velocity, acceleration, target density, and position. The condition judgment module determines whether the foreground object is in an abnormal state based on one or more of the foreground object's velocity, acceleration, target density, and position. Those skilled in the art should understand that, in practical applications, according to the teachings of the embodiments disclosed herein, the foreground feature extraction module can also extract other required foreground object identification features to achieve the purposes of this disclosure. Further, when the velocity of the foreground object in the extracted foreground object state features exceeds a set velocity threshold, the condition judgment module determines that the foreground object is in an abnormal state; and / or when the acceleration of the foreground object in the extracted foreground object state features exceeds a set acceleration threshold, the condition judgment module determines that the foreground object is in an abnormal state; and / or when the target density of the foreground object in the extracted foreground object state features exceeds a set target density threshold, the condition judgment module determines that the foreground object is in an abnormal state.

[0079] For example, as a practical application, when the speed and / or acceleration of a foreground object within a danger zone is excessively high, it might be interpreted as a collision, thus indicating an abnormal state. Similarly, when the speed and / or acceleration of a foreground object within a danger zone is excessively low, it might be interpreted as crowding or stagnation, again indicating an abnormal state. Furthermore, when the density of foreground objects within a danger zone is excessively high, it might be interpreted as crowding or stagnation, again indicating an abnormal state. Optionally, combining multiple foreground object identification features during data processing can further improve the accuracy of identification.

[0080] Optionally, the foreground feature extraction module 2223 further includes a human abnormal action model generation submodule 2223b, which is used to define the human abnormal action model based on a skeletal graph model and / or a binary graph model and / or human self-learning modeling. Skeletal graph models and binary graph models are both mature human modeling technologies, and well-known skeletal graph models and binary graph models in the prior art can be used to generate the human abnormal action model of interest in this embodiment. Furthermore, human self-learning modeling updates the human abnormal action model in real time or periodically based on the continuous learning and improvement of the monitoring system 2200 during use, or on the continuous updating of the shared database. For example, as a practical application example, the human abnormal action model can cover lying down, both feet off the ground, legs extended upwards, etc.

[0081] As used throughout this article, human motion models can be learned from data using deep learning techniques. These deep learning techniques include, but are not limited to, deep neural networks, convolutional neural networks, recursive neural networks, dictionary learning, bag-of-visual-words, and more. The model can also be constructed using Markov models, hidden Markov models, Markov decision processes, partially observable Markov decision processes (MDPs), Markov decision logic, and probabilistic programming, among others.

[0082] Optionally, the foreground object state features extracted by the foreground feature extraction module 2223 include the actions of the human body in the foreground object, and the condition judgment module determines whether the foreground object is in an abnormal state based on the actions of the human body in the foreground object. Further, when the actions of the human body in the foreground object fall into the abnormal human body action model, the condition judgment module determines that the foreground object is in an abnormal state.

[0083] For example, as a practical application, when a human body in a foreground object is in a pose with both feet off the ground, this pose should be considered an action within the abnormal human motion model. Similarly, when a human body in a foreground object is in a lying-down pose, this pose should also be considered an action within the abnormal human motion model.

[0084] Optionally, in some situations, not all actions performed in the abnormal human movement model in any area will pose a safety hazard. It is necessary to consider the actual location of the action. In this modification, the foreground feature extraction module 2223 also includes a scene model generation submodule 2223a, which is used to define a danger zone based on data frames sensed in the normal state of the monitoring area of ​​the passenger transport device 2100 and / or a scene self-learning model. This danger zone is a specific area on the passenger transport device 2100 where abnormal actions performed by a passenger may lead to danger, and it may change depending on the actual situation. For example, in one example, when a passenger extends their leg beyond the handrail side of the passenger transport device 2100, this leg-extending action can be identified as an abnormal human movement, and the handrail side can be identified as a specific area.

[0085] Optionally, the foreground object state features extracted by the foreground feature extraction module 2223 include the position and movement of the human body in the foreground object. The condition judgment module determines whether the foreground object is in an abnormal state based on the position and movement of the human body in the foreground object. Further, when the position of the human body in the foreground object is in a dangerous area and the movement of the human body in the foreground object falls into the abnormal human movement model, the condition judgment module determines that the foreground object is in an abnormal state.

[0086] As an example of practical application, an abnormal state is determined only when a foreground object is within a danger zone and exhibits an abnormal movement that falls within the abnormal human movement model. For instance, when a foreground object extends its leg on the handrail side, the handrail side is a danger zone, and extending the leg is an abnormal movement, thus determining that the foreground object is in an abnormal state. Similarly, when a foreground object falls on a step, the step is a danger zone, and falling is an abnormal movement, thus determining that the foreground object is in an abnormal state. Furthermore, when a foreground object walks in the opposite direction on a step, the step is a danger zone, and walking in the opposite direction is an abnormal movement, thus determining that the foreground object is in an abnormal state. Optionally, combining multiple foreground object identification features for data processing can further improve the accuracy of recognition.

[0087] Optionally, in certain states, accurate judgment cannot be made based on a single frame of data. For example, it may be necessary to analyze the motion trend of the foreground object separately or to analyze it in combination with other features to obtain the required information. In this case, the processing device 2220 can be further improved. The processing device 2220 is configured to further include a trajectory generation module 2225, which generates a trajectory of changes in the state features of the foreground object based on the state features of the foreground object extracted by the foreground feature extraction module 2223 from the foreground object corresponding to multiple consecutive data frames. Correspondingly, the state judgment module 2224, which is capable of analysis and judgment, is further configured to: predict whether the foreground object is about to be in an abnormal state based on the trajectory of changes in the state features of the foreground object, and determine that the foreground object is about to be in an abnormal state if the judgment is "yes".

[0088] More specifically, the foreground object state features extracted by the foreground feature extraction module 2223 include the foreground object's velocity and / or acceleration and / or target density and / or position. The condition judgment module determines whether the foreground object is about to enter an abnormal state based on the trajectory of the foreground object's velocity and / or acceleration and / or target density changes in the hazardous area. Specifically, when the velocity change trajectory of the foreground object exceeds a preset velocity trajectory threshold within a preset time period, the condition judgment module determines that the foreground object is about to enter an abnormal state; and / or when the acceleration change trajectory of the foreground object exceeds a preset acceleration trajectory threshold within a preset time period, the condition judgment module determines that the foreground object is about to enter an abnormal state; and / or when the target density change trajectory of the foreground object exceeds a preset density trajectory threshold within a preset time period, the condition judgment module determines that the foreground object is about to enter an abnormal state.

[0089] For example, as a practical application, when a large number of passengers have gathered at the entrance of passenger transport unit 2100, but their trajectory shows that the passenger group is gradually dispersing, meaning that although the target density here exceeds the density setting threshold, the target density change trajectory does not exceed the density trajectory setting threshold. This indicates that the congestion situation here may have been greatly alleviated, and therefore, it is no longer necessary to consider it to be in an abnormal state.

[0090] Optionally, the foreground object state features extracted by the foreground feature extraction module 2223 include the actions of the human body within the foreground object. The condition judgment module determines whether the foreground object is about to enter an abnormal state based on the trajectory of the human body's actions within the foreground object. Further, when the trajectory of the human body's actions within the foreground object exceeds a preset threshold within a preset time period, the condition judgment module determines that the foreground object is about to enter an abnormal state.

[0091] For example, as a practical application, when a human in the foreground object gets up on their own after falling, although this action has been identified as an abnormal human movement model, its movement trend indicates that it has overcome the problem and is moving in a positive direction. Therefore, the trajectory of the movement change will not exceed the threshold set by the movement trajectory. At this time, it is no longer necessary to identify it as being in an abnormal state.

[0092] Optionally, in some situations, not all actions performed in the abnormal human motion model in any area will pose a safety hazard. It is necessary to consider the actual location of the action. In this modified version, the foreground object state features extracted by the foreground feature extraction module 2223 also include the position of the human body within the foreground object. The condition judgment module determines whether the foreground object is about to enter an abnormal state based on the position of the human body and the trajectory of its movements. Specifically, when the trajectory of the human body's position changes within a preset time period approaches a danger zone, and when the trajectory of the human body's movements exceeds a preset threshold within a preset time period, the condition judgment module determines that the foreground object is about to enter an abnormal state.

[0093] Optionally, the state judgment module 2224 is further configured to determine that the foreground object is in an abnormal state only when the judgment results of at least two consecutive data frames are both that the foreground object is in an abnormal state. This setting ensures that the foreground object is only classified as belonging to the abnormal group after multiple monitoring sessions and obtaining the same judgment result. This further improves the accuracy of the judgment and reduces the false judgment rate.

[0094] Optionally, the imaging sensor and / or depth sensor 2210 senses and acquires data frames within a predetermined time period at regular intervals for processing by the processing device 2220. This configuration allows for intermittent monitoring, achieving a similar level of effectiveness. For example, during evening hours, fewer users utilize the passenger transport device 2100. In this case, lower-frequency monitoring saves on monitoring costs while still achieving a similar level of detection and security.

[0095] Optionally, since the imaging sensor and / or depth sensor 2210 have more accurate sensing results within a certain area, multiple imaging sensors and / or depth sensors 2210 are arranged to cover the entrance and / or exit and / or the entire length of the passenger transport device 2100 in order to improve the accuracy of detection.

[0096] Optionally, the monitoring system 2200 also includes an alarm unit 2230, which is triggered by the status judgment module 2224 when it determines that the foreground object is in an abnormal state. Accordingly, the processing device 2220 is also configured to trigger an output signal to the passenger transport device 2100 and / or the building's elevator maintenance center when the status judgment module 2224 determines that the foreground object is in an abnormal state, in order to prevent safety accidents from occurring.

[0097] See Figure 6 Furthermore, a monitoring method for passenger transport devices is provided to be used in conjunction with the monitoring system described in the foregoing embodiments. This monitoring method includes the following steps: S100, a data frame acquisition step: sensing the monitoring area of ​​the passenger transport device to acquire data frames; S200, a background acquisition step: acquiring a background model based on the data frames sensed when the monitoring area is in a normal state; S300, a foreground detection step: comparing the real-time sensed data frames with the background model to obtain foreground objects; S400, a foreground feature extraction step: extracting corresponding foreground object state features from the foreground objects; and S500, a state judgment step: judging whether the foreground object is in an abnormal state based at least on the foreground object state features, and determining that the foreground object is in an abnormal state if the judgment is "yes". This method can accurately and promptly identify whether a group in a passenger transport device is in an abnormal state and feed the information back to the monitoring center or maintenance personnel so that they can respond promptly and take actions such as issuing warnings, slowing down the device, or even braking the device according to the actual situation.

[0098] In addition, to improve the practicality or accuracy of the monitoring method in the foregoing embodiments, several improvements or modifications to the various steps therein are also provided.

[0099] Optionally, S400 also includes S410, a scene model generation sub-step: used to define a danger zone based on data frames sensed in the monitoring area of ​​the passenger transport device under normal conditions and / or a scene self-learning model. This danger zone is a specific area on the passenger transport device that may be dangerous under normal circumstances, and may change according to the actual situation.

[0100] Optionally, in S410, the extracted foreground object state features include the velocity and / or acceleration and / or target density of the foreground object. In S500, it is determined whether the foreground object is in an abnormal state based on the velocity and / or acceleration and / or target density of the foreground object in the danger zone. Further, if the velocity of the foreground object in the extracted foreground object state features exceeds a set velocity threshold, it is determined that the foreground object is in an abnormal state; and / or if the acceleration of the foreground object in the extracted foreground object state features exceeds a set acceleration threshold, it is determined that the foreground object is in an abnormal state; and / or if the target density of the foreground object in the extracted foreground object state features exceeds a set target density threshold, it is determined that the foreground object is in an abnormal state.

[0101] For example, as a practical application, when the speed and / or acceleration of a foreground object within a danger zone is excessively high, it might be interpreted as a collision, thus indicating an abnormal state. Similarly, when the speed and / or acceleration of a foreground object within a danger zone is excessively low, it might be interpreted as crowding or stagnation, again indicating an abnormal state. Furthermore, when the density of foreground objects within a danger zone is excessively high, it might be interpreted as crowding or stagnation, again indicating an abnormal state. Optionally, combining multiple foreground object identification features during data processing can further improve the accuracy of identification.

[0102] Optionally, S400 also includes S420, a sub-step for generating abnormal human movement models: used to define abnormal human movement models based on skeletal graph models and / or binary graph models and / or human self-learning modeling. Skeletal graph models and binary graph models are both mature human modeling technologies; therefore, well-known skeletal graph models and binary graph models in the prior art can be used to generate the abnormal human movement models of interest in this embodiment. Furthermore, human self-learning modeling updates the abnormal human movement models in real-time or periodically based on the continuous learning and improvement of the monitoring system during use, or on the continuous updating of a shared database. For example, as a practical application example, the abnormal human movement model can cover lying down, both feet off the ground, legs extended upwards, etc.

[0103] Optionally, in S420, the extracted foreground object state features include the actions of the human body within the foreground object; in S500, the foreground object is determined to be in an abnormal state based on the actions of the human body within the foreground object. Further, if the actions of the human body within the foreground object fall into the abnormal human body action model, the foreground object is determined to be in an abnormal state.

[0104] For example, as a practical application, when a human body in a foreground object is in a pose with both feet off the ground, this pose should be considered an action within the abnormal human motion model. Similarly, when a human body in a foreground object is in a lying-down pose, this pose should also be considered an action within the abnormal human motion model.

[0105] Optionally, in some situations, not all actions performed in the abnormal human movement model will pose a safety hazard. It is necessary to consider the actual location of the action. In this modification, S400 of the monitoring method also includes S430, a scene model generation sub-step: defining a danger zone based on data frames sensed in the normal state of the monitoring area of ​​the passenger transport device and / or a scene self-learning model. This danger zone is a specific area on the passenger transport device where abnormal actions performed by a passenger may lead to danger, and it may change depending on the actual situation. For example, in one example, when a passenger extends their leg beyond the handrail side of the passenger transport device, this leg extension action can be identified as an abnormal human movement, and the handrail side can be identified as a danger zone.

[0106] Optionally, in S430, the extracted foreground object state features include the position and action of the human body in the foreground object. In S500, the foreground object is judged to be in an abnormal state based on the position and action of the human body in the foreground object. Further, if the position of the human body in the foreground object is in a danger zone and the action of the human body in the foreground object falls into the abnormal human action model, the foreground object is judged to be in an abnormal state.

[0107] Optionally, in certain situations, accurate judgments cannot be made based on a single frame of data. For example, it may be necessary to analyze the motion trend of the foreground object separately or to analyze it in combination with other features to obtain the required information. In this case, the processing device can be further improved. The corresponding monitoring method at this time also includes: S600, trajectory generation step: generating a trajectory of changes in the state features of the foreground object based on the state features of the foreground object extracted from the foreground object in multiple consecutive data frames in S400. Correspondingly, in S500, it is predicted whether the foreground object is about to be in an abnormal state based on the trajectory of changes in the state features of the foreground object, and if the judgment is "yes", it is determined that the foreground object is about to be in an abnormal state.

[0108] More specifically, in S400, the extracted foreground object state features include the foreground object's velocity and / or acceleration and / or target density; in S500, it is determined whether the foreground object is about to enter an abnormal state based on the trajectory of the foreground object's velocity and / or acceleration and / or target density changes in the danger zone. Specifically, if the foreground object's velocity change trajectory exceeds a preset velocity trajectory threshold within a preset time period, it is determined that the foreground object is about to enter an abnormal state; and / or if the foreground object's acceleration change trajectory exceeds a preset acceleration trajectory threshold within a preset time period, it is determined that the foreground object is about to enter an abnormal state; and / or if the foreground object's target density change trajectory exceeds a preset density trajectory threshold within a preset time period, it is determined that the foreground object is about to enter an abnormal state.

[0109] For example, as a practical application, when a large number of passengers have gathered at the entrance of a passenger transport facility, but their trajectory shows that the passenger group is gradually dispersing, meaning that although the target density here exceeds the density setting threshold, the target density change trajectory does not exceed the density trajectory setting threshold, this indicates that the congestion situation here may have been significantly alleviated. Therefore, it is no longer necessary to consider it to be in an abnormal state.

[0110] Optionally, in S400, the foreground object state features extracted by the foreground feature extraction module include the actions of the human body in the foreground object; in S500, it is determined whether the foreground object is about to enter an abnormal state based on the trajectory of the human body's actions in the foreground object. Further, if the trajectory of the human body's actions in the foreground object exceeds a preset threshold within a preset time period, it is determined that the foreground object is about to enter an abnormal state.

[0111] For example, as a practical application, when a human in the foreground object gets up on their own after falling, although this action has been identified as an abnormal human movement model, its movement trend indicates that it has overcome the problem and is moving in a positive direction. Therefore, the trajectory of the movement change will not exceed the threshold set by the movement trajectory. At this time, it is no longer necessary to identify it as being in an abnormal state.

[0112] Optionally, in some situations, not all actions performed in the abnormal human motion model in any area will pose a safety hazard. It is necessary to consider the actual location of the action. In this modification, in S400, the foreground feature extraction module extracts the foreground object state features including the position of the human body within the foreground object; in S500, it determines whether the foreground object is about to enter an abnormal state based on the position of the human body and the trajectory of its movements. Specifically, when the trajectory of the human body's position changes within a preset time period approaches a danger zone, and when the trajectory of the human body's movements within the preset time period exceeds a set threshold for the movement trajectory, it is determined that the foreground object is about to enter an abnormal state.

[0113] Optionally, in S500, the foreground object is determined to be in an abnormal state only if the judgment results of at least two consecutive data frames are both in an abnormal state. This setting ensures that the foreground object is only classified as belonging to the abnormal group after multiple monitoring sessions and obtaining the same judgment result. This further improves the accuracy of the judgment and reduces the false positive rate.

[0114] Optionally, data frames within a predetermined time period are sensed and acquired at regular intervals for processing by the processing device. This setup allows for intermittent monitoring, achieving a similar level of effectiveness. For example, during evening hours, fewer users utilize passenger transport devices. In this case, lower-frequency monitoring saves on monitoring costs while still achieving comparable detection and security results.

[0115] Optionally, it also includes S700, an alarm procedure: triggering alarm unit 1230 to operate when it is determined that the foreground object is in an abnormal state. This achieves the prevention of safety accidents.

[0116] See appendix Figure 3Another embodiment of a monitoring system 3200 for a passenger transport device 3100 is also provided. This monitoring system 3200 includes: an imaging sensor and / or a depth sensor 3210 for sensing a monitoring area of ​​the passenger transport device 3100 to obtain a first data frame; a receiving device 3230 for receiving a second data frame from a wearable device; the second data frame is obtained by the wearable device's data sensor sensing the wearable device and its wearer; and a processing device 3220 for analyzing and processing the first and second data frames to monitor whether the operating passenger transport device 3100 and / or the wearable device wearer are in a normal state. In this embodiment, the use of currently popular wearable devices to monitor the passenger transport device 3100 further enhances the targeting and accuracy of the monitoring. For example, a mobile phone is used as an example of a wearable device. Current mobile phones typically have at least sensors such as a gravity sensor, a position sensor, and a camera. Therefore, the situation of the mobile phone wearer can also be obtained through the mobile phone, thereby assisting in determining whether there is an anomaly.

[0117] Optionally, the second data frame includes: a location data frame; and a current state data frame and / or acceleration data; the processing device 3220 determines whether the wearable device wearer is on the passenger transport device 3100 based on the location data frame; and the processing device 3220 determines whether the wearable device wearer is gazing at the wearable device based on the current state data frame; and / or the processing device 3220 determines whether the wearable device wearer's second acceleration exceeds a second acceleration preset threshold based on the acceleration data frame. The determination that the user is gazing at the device can be achieved by using the device's camera and a corresponding algorithm to recognize the face. This algorithm can be a Viola-Jones algorithm followed by head pose estimation based on the symmetry in the detected face. When the head pose is below a very small threshold, it is considered that the user is looking at the device. Various scenarios will be analyzed in detail below.

[0118] Optionally, the processing device 3220 is configured to determine that the wearable device wearer is in an abnormal state when the wearable device wearer is on the passenger transport device 3100 and when the wearable device wearer is looking at the wearable device. For example, it might be determined that the wearer is using their phone in the elevator, which is a relatively dangerous behavior. Therefore, they are considered to be in an abnormal state.

[0119] Optionally, the processing device 3220 is configured to determine that the wearable device wearer is in an abnormal state when the wearable device wearer is on the passenger transport device 3100 and when the wearable device wearer stares at the wearable device for more than a preset period of time. As an example, it would be a detrimental user experience if an abnormal alert were triggered simply by the wearer casually glancing at their phone. Therefore, a time and location factor is further introduced here. Only when the wearer stares at their phone for more than a preset period of time and is relatively far from the escalator's entrance or exit is it determined that the wearer may be using their phone in a dangerous area of ​​the escalator, thus constituting an abnormal state. As the transport device approaches the escalator's entrance or exit, the probability of an accident increases, and the time factor is correspondingly reduced.

[0120] Optionally, the processing device 3220 is configured to: when the wearable device wearer is on the passenger transport device 3100; and the second acceleration of the wearable device wearer exceeds a second acceleration setting threshold; further determine, in conjunction with the first data frame, whether the first acceleration of the passenger transport device 3100 exceeds the first acceleration setting threshold; determine that the passenger transport device 3100 is in an abnormal state when the first acceleration of the passenger transport device 3100 exceeds the first acceleration setting threshold; and / or determine that the wearable device wearer is in an abnormal state when the first acceleration of the passenger transport device 3100 does not exceed the first acceleration setting threshold. As an example, when it is determined that the wearable device and its wearer may be in an acceleration state based on the second acceleration provided by the wearable device exceeding the second acceleration setting threshold, it is further determined whether the first acceleration of the passenger transport device 3100 exceeds the first acceleration setting threshold, thereby determining whether the source of the acceleration state is the wearable device and its wearer or the passenger transport device 3100. Then, it is determined whether the wearable device and its wearer are in an abnormal state.

[0121] Optionally, the receiving device 3230 is used to receive a second data frame group from multiple wearable devices; the second data frame group is acquired by multiple data sensors of the multiple wearable devices sensing the multiple wearable devices and their wearers; and the processing device 3220 is used to analyze and process the first data frame and the second data frame group to monitor whether the operating passenger transport device 3100 and / or the multiple wearable device wearers are in a normal state. Since many abnormal states cannot be identified using a single data sensor, acquiring the second data frame group through multiple data sensors can assist in providing a more accurate judgment.

[0122] See Figure 7To further improve the technical effect of the monitoring system in the aforementioned embodiments, a monitoring method for a passenger transport device is also provided, comprising: S100, a first data frame acquisition step: sensing the monitoring area of ​​the passenger transport device to acquire a first data frame; S200, a second data frame acquisition step: receiving a second data frame from a wearable device; the second data frame is acquired by the data sensor of the wearable device sensing the wearable device and its wearer; and S300, a processing step: analyzing and processing the first data frame and the second data frame to monitor whether the operating passenger transport device and / or the wearable device wearer are in a normal state.

[0123] Optionally, in S200, the second data frame includes: a location data frame; and a current state data frame and / or acceleration data; in S300, the location data frame is used to determine whether the wearable device wearer is on the passenger transport device; and the current state data frame is used to determine whether the wearable device wearer is staring at the wearable device; and / or the acceleration data frame is used to determine whether the wearable device wearer's second acceleration exceeds a second acceleration set threshold.

[0124] It is worth noting that the wearable device wearer must first be determined to be on the passenger transport device based on the location data frame before any other determinations are necessary. Determinations based on the current state data frame and the acceleration data frame can be performed individually to identify the specific anomaly, or they can be combined to arrive at a broader and more accurate assessment.

[0125] As the basis for the above analysis, several examples of judgments are provided below to facilitate understanding.

[0126] Optionally, in S300, the wearable device wearer is in an abnormal state when he / she is on the passenger transport device and staring at the wearable device.

[0127] Optionally, in S300, when the wearable device wearer is on the passenger transport device and when the wearable device wearer stares at the wearable device for more than a preset period of time, it is determined that the wearable device wearer is in an abnormal state.

[0128] Optionally, in S300, when the wearable device wearer is on the passenger transport device and the second acceleration of the wearable device wearer exceeds the second acceleration setting threshold, the first data frame is further used to determine whether the first acceleration of the passenger transport device exceeds the first acceleration setting threshold; when the first acceleration of the passenger transport device exceeds the first acceleration setting threshold, the passenger transport device is determined to be in an abnormal state; and / or when the first acceleration of the passenger transport device does not exceed the first acceleration setting threshold, the wearable device wearer is determined to be in an abnormal state.

[0129] Optionally, in S200, a second data frame group is received from multiple wearable devices; the second data frame group is acquired by multiple data sensors of the multiple wearable devices sensing the multiple wearable devices and their wearers; and in S300, the first data frame and the second data frame group are analyzed and processed to monitor whether the operating passenger transport device and / or the multiple wearable device wearers are in a normal state. Since many abnormal states cannot be identified using a single data sensor, acquiring the second data frame group through multiple data sensors can assist in providing a more accurate judgment.

[0130] See appendix Figure 4 Another embodiment of a monitoring system 4200 for a passenger transport device 4100 is also provided. This monitoring system 4200 includes: a depth sensing sensor 4210 for sensing the monitoring area of ​​the passenger transport device 4100 to obtain a first data frame; an auxiliary sensor group 4240 for sensing the monitoring area of ​​the passenger transport device 4100 to obtain a second data frame group; a preprocessing device 4230-4220 for performing probability distribution processing on the data frames in the second data frame group to obtain a preprocessed second data frame group; and a processing device 4220 for analyzing the first data frame and the preprocessed second data frame group to monitor whether a specific event has occurred in the operating passenger transport device 4100. Optionally, the auxiliary sensor group 4240 includes imaging sensors, audio sensors, pressure sensors, gravity sensors, accelerometers, capacitance sensors, brake beam sensors, etc. Generally, the more types of sensors used, the more accurate the event judgment will be. Correspondingly, the more computational processing is required by the processing device 4220. The monitoring system 4200 in this embodiment makes reasonable use of the auxiliary sensor group 4240. By processing the probability distribution of specific events in the data frames of the acquired second data frame group, it selects the data frames with higher probability distributions. Subsequently, these preprocessed second data frame groups are combined with the first data frames sensed by the depth sensing sensor 4210 for analysis. On the one hand, this eliminates various data redundancies and reduces analysis and processing time; on the other hand, it greatly improves the accuracy of judgment compared to using only the depth sensing sensor 4210.

[0131] Specifically, if the probability of the preprocessed second data frame group in the probability distribution processing for a specific event is greater than the probability threshold for that specific event, then this preprocessed second data frame group can be considered to have a significant impact on the occurrence of the specific event and should be taken into account.

[0132] See Figure 8Correspondingly, an embodiment of a monitoring method for a passenger transport device is also provided, comprising: S100, a first data frame acquisition step, sensing the monitoring area of ​​the passenger transport device to acquire a first data frame; S200, a second data frame group acquisition step, sensing the monitoring area of ​​the passenger transport device to acquire a second data frame group; S300, a preprocessing step, performing probability distribution processing on the data frames in the second data frame group to acquire a preprocessed second data frame group; and S400, a state judgment step, used to analyze and process the first data frame and the preprocessed second data frame group to monitor whether a specific event has occurred in the operating passenger transport device. The monitoring method of this embodiment makes reasonable use of an auxiliary sensor group. By performing probability distribution processing on the data frames in the acquired second data frame group to select data frames with a higher probability distribution, the method then combines these preprocessed second data frame groups with the first data frames sensed by the depth sensor itself for analysis and processing. This eliminates various data redundancies, reducing analysis and processing time; and significantly improves the accuracy of judgment compared to using only a depth sensor.

[0133] Optionally, in S300, if the probability of the preprocessed second data frame group in the probability distribution processing for a specific event is greater than the probability threshold for the specific event, then this preprocessed second data frame group can be considered to have a significant impact on the occurrence of the specific event and should be taken into account.

[0134] Furthermore, a passenger transport device is also provided, which can selectively apply or combine the monitoring systems of the aforementioned embodiments, and control them with corresponding monitoring methods. This can eliminate various potential safety hazards and improve passengers' user experience and trust in the passenger transport device.

[0135] It should be noted that the elements disclosed and described herein (including the flowcharts and block diagrams in the accompanying drawings) refer to logical boundaries between elements. However, in accordance with software or hardware engineering practice, the described elements and their functions can be executed on a machine by a computer-executable medium having a processor capable of executing program instructions stored thereon, which may be as a monolithic software architecture, as an independent software module, or as a module using external programs, code, services, etc., or any combination thereof, and all such execution schemes may fall within the scope of this disclosure.

[0136] While different non-limiting embodiments have specifically described components, embodiments of the invention are not limited to these specific combinations. Some of the components or features from any non-limiting embodiment may be used in combination with features or components from any other non-limiting embodiment.

[0137] Although a particular order of steps has been shown, disclosed and claimed, it should be understood that the steps may be performed, separated or combined in any order, unless otherwise specified, and will still benefit from this disclosure.

[0138] The foregoing description is exemplary and not intended to limit it. Various non-limiting embodiments are disclosed herein; however, those skilled in the art will recognize that various modifications and alterations will fall within the scope of the appended claims in light of the foregoing teachings. Therefore, it will be understood that within the scope of the appended claims, disclosures beyond those specifically disclosed may be implemented. For this reason, the appended claims should be studied to determine their true scope and content.

Claims

1. A monitoring system for a passenger transport device, characterized in that, include: Imaging sensors and / or depth sensors are used to sense the monitoring area of ​​the passenger transport device to obtain data frames; as well as A processing device, configured to perform data processing on the data frame to monitor whether there is an anomaly within the monitoring area, is configured to include: The background acquisition module is used to acquire a background model based on data frames sensed when the monitoring area is in a normal state. The foreground detection module is used to compare the real-time sensed data frames with the background model to obtain the foreground object; A foreground feature extraction module is used to extract corresponding foreground object state features from the foreground object. This foreground feature extraction module further includes a human abnormal action model generation submodule, which is used to define a human abnormal action model based on non-self-learning human modeling and / or self-learning human modeling. The state determination module is used to determine whether the foreground object is in an abnormal state based at least on the state characteristics of the foreground object, and to determine that the foreground object is in an abnormal state if the determination is "yes".

2. The monitoring system as described in claim 1, characterized in that, The foreground feature extraction module includes a scene model generation submodule, which is used to define dangerous areas based on data frames sensed in the normal state of the monitoring area of ​​the passenger transport device and / or scene self-learning models and / or area identifiers presented on the computer by the installer or user.

3. The monitoring system as described in claim 2, characterized in that, The foreground object state features extracted by the foreground feature extraction module include one or more of the foreground object's velocity, acceleration, target density, and position. The state judgment module determines whether the foreground object is in an abnormal state based on one or more of the foreground object's velocity, acceleration, target density, and position.

4. The monitoring system as described in claim 3, characterized in that, When the speed of the foreground object in the extracted foreground object state features exceeds a set speed threshold, the state judgment module determines that the foreground object is in an abnormal state. and / or When the acceleration of the foreground object in the extracted foreground object state features exceeds the acceleration set threshold, the state judgment module determines that the foreground object is in an abnormal state. and / or When the target density of the foreground object in the extracted foreground object state features exceeds the target density set threshold, the state judgment module determines that the foreground object is in an abnormal state.

5. The monitoring system as described in claim 1, characterized in that, The human self-learning modeling uses deep learning technology to obtain abnormal human movement models, including deep neural networks, convolutional neural networks, recurrent neural networks, dictionary learning, or bag-of-words models; and / or the human non-self-learning modeling includes: skeletal graph models and / or binary graph models.

6. The monitoring system as described in claim 1, characterized in that, The human self-learning modeling method obtains abnormal human movement models through Markov model technology, hidden Markov model, Markov decision process, partially observable Markov decision process, Markov decision logic, or probabilistic programming.

7. The monitoring system as described in claim 1, characterized in that, The foreground object state features extracted by the foreground feature extraction module include the actions of the human body in the foreground object. The state judgment module determines whether the foreground object is in an abnormal state based on the actions of the human body in the foreground object.

8. The monitoring system as described in claim 7, characterized in that, When the human body's movement in the foreground object falls into the abnormal human body movement model, the state judgment module determines that the foreground object is in an abnormal state.

9. The monitoring system as described in claim 8, characterized in that, The foreground feature extraction module also includes a scene model generation submodule, which is used to define the danger zone based on data frames sensed in the monitoring area of ​​the passenger transport device under normal conditions and / or a scene self-learning model.

10. The monitoring system as described in claim 9, characterized in that, The foreground object state features extracted by the foreground feature extraction module include the position and action of the human body in the foreground object. The state judgment module determines whether the foreground object is in an abnormal state based on the position and action of the human body in the foreground object.

11. The monitoring system as described in claim 10, characterized in that, When the position of the human body in the foreground object is in the danger zone and the movement of the human body in the foreground object falls into the abnormal human body movement model, the state judgment module determines that the foreground object is in an abnormal state.

12. The monitoring system as described in claim 1, characterized in that, The processing apparatus is configured to further include: The trajectory generation module is used to generate a trajectory of changes in the state features of the foreground objects based on the state features of the foreground objects extracted by the foreground feature extraction module from the foreground objects corresponding to multiple consecutive data frames.

13. The monitoring system as described in claim 12, characterized in that, The state judgment module is further configured to: predict whether the foreground object is about to be in an abnormal state based on the change trajectory of the state characteristics of the foreground object, and determine that the foreground object is about to be in an abnormal state if the judgment is "yes".

14. The monitoring system as described in claim 13, characterized in that, The foreground object state features extracted by the foreground feature extraction module include one or more of the foreground object's velocity, acceleration, and target density. The state judgment module determines whether the foreground object is about to be in an abnormal state based on the trajectory of the foreground object's velocity, acceleration, and target density changes in the danger zone.

15. The monitoring system as described in claim 14, characterized in that, When the velocity change trajectory of the foreground object exceeds a preset threshold within a preset time period, the state judgment module determines that the foreground object is about to be in an abnormal state. and / or When the acceleration change trajectory of the foreground object exceeds the acceleration trajectory set threshold within a preset time period, the state judgment module determines that the foreground object is about to be in an abnormal state. and / or When the target density change trajectory of the foreground object exceeds the density trajectory set threshold within a preset time period, the state judgment module determines that the foreground object is about to be in an abnormal state.

16. The monitoring system as described in claim 13, characterized in that, The foreground object state features extracted by the foreground feature extraction module include the human body's movements in the foreground object. The state judgment module determines whether the foreground object is about to enter an abnormal state based on the changing trajectory of the human body's movements in the foreground object.

17. The monitoring system as described in claim 16, characterized in that, When the trajectory of a human body in the foreground object exceeds a preset threshold within a preset time period, the state judgment module determines that the foreground object is about to be in an abnormal state.

18. The monitoring system as described in claim 16, characterized in that, The foreground object state features extracted by the foreground feature extraction module also include the position of the human body in the foreground object. The state judgment module determines whether the foreground object is about to be in an abnormal state based on the position of the human body in the foreground object and the trajectory of the human body's movements.

19. The monitoring system as described in claim 18, characterized in that, When the trajectory of the human body's position change in the foreground object approaches a danger zone within a preset time period, and when the trajectory of the human body's movement change in the foreground object exceeds a preset threshold for movement trajectory within a preset time period, the state judgment module determines that the foreground object is about to be in an abnormal state.

20. The monitoring system according to any one of claims 1 to 19, characterized in that, The state determination module is further configured to determine that the foreground object is in an abnormal state when the determination results of at least two consecutive data frames are that the foreground object is in an abnormal state.

21. The monitoring system according to any one of claims 1 to 19, characterized in that, The imaging sensor and / or depth sensor sense and acquire data frames within the predetermined time period every predetermined time period for the processing device to perform data processing.

22. The monitoring system according to any one of claims 1 to 19, characterized in that, The imaging sensor and / or depth sensor are multiple and are arranged to cover the entrance and / or exit and / or the entire length of the passenger transport device.

23. The monitoring system according to any one of claims 1 to 19, characterized in that, The monitoring system also includes an alarm unit, which is triggered by the status judgment module when it determines that the foreground object is in an abnormal state.

24. The monitoring system according to any one of claims 1 to 19, characterized in that, The processing device is further configured to trigger an output signal to the passenger transport device and / or the building's elevator maintenance center when the state determination module determines that the foreground object is in an abnormal state, in order to prevent safety accidents from occurring.

25. A method for monitoring a passenger transport device, characterized in that, include: S100, Data frame acquisition step: Sensing the monitoring area of ​​the passenger transport device to acquire data frames; S200, Background acquisition step: Acquire a background model based on data frames sensed when the monitoring area is in a normal state; S300, Foreground detection step: Compare the real-time sensed data frame with the background model to obtain the foreground object; S400, Foreground feature extraction step, extracting corresponding foreground object state features from the foreground object; as well as S500, State determination step: At least based on the state characteristics of the foreground object, determine whether the foreground object is in an abnormal state, and if the determination is "yes", determine that the foreground object is in an abnormal state. S400 further includes S420, a human abnormal action model generation sub-step: used to define human abnormal action models based on human non-self-learning modeling and / or human self-learning modeling.

26. The monitoring method as described in claim 25, characterized in that, The S400 also includes S410, a scene model generation sub-step: for defining a danger zone based on data frames sensed in the monitoring area of ​​the passenger transport device under normal conditions and / or a scene self-learning model and / or an identifier in the area presented on a computer.

27. The monitoring method as described in claim 26, characterized in that, In S410, the extracted foreground object state features include one or more of the foreground object's velocity, acceleration, target density, and position. In S500, it is determined whether the foreground object is in an abnormal state based on one or more of the foreground object's velocity, acceleration, target density, and position.

28. The monitoring method as described in claim 27, characterized in that, When the speed of the foreground object in the extracted foreground object state features exceeds a set speed threshold, the foreground object is determined to be in an abnormal state; and / or When the acceleration of the foreground object in the extracted foreground object state features exceeds the acceleration set threshold, the foreground object is determined to be in an abnormal state. and / or When the target density of the foreground object in the extracted foreground object state features exceeds the target density set threshold, the foreground object is determined to be in an abnormal state.

29. The monitoring method as described in claim 25, characterized in that, The human self-learning modeling uses deep learning technology to obtain abnormal human movement models, including deep neural networks, convolutional neural networks, recurrent neural networks, dictionary learning, or bag-of-words models; and / or the human non-self-learning modeling includes: skeletal graph models and / or binary graph models.

30. The monitoring method as described in claim 25, characterized in that, The human self-learning modeling method obtains abnormal human movement models through Markov model technology, hidden Markov model, Markov decision process, partially observable Markov decision process, Markov decision logic, or probabilistic programming.

31. The monitoring method as described in claim 25, characterized in that, In S420, the extracted foreground object state features include the actions of the human body within the foreground object; In S500, it is determined whether the foreground object is in an abnormal state based on the actions of the human body in the foreground object.

32. The monitoring method as described in claim 31, characterized in that, When the human body's movement in the foreground object falls into the abnormal human body movement model, the foreground object is determined to be in an abnormal state.

33. The monitoring method as described in claim 32, characterized in that, The S400 also includes S430, a scene model generation sub-step: defining a danger zone based on data frames sensed in the monitoring area of ​​the passenger transport device under normal conditions and / or a scene self-learning model.

34. The monitoring method as described in claim 33, characterized in that, In S430, the extracted foreground object state features include the position and action of the human body within the foreground object. In S500, the position and movement of the human body in the foreground object are used to determine whether the foreground object is in an abnormal state.

35. The monitoring method as described in claim 34, characterized in that, When the position of the human body in the foreground object is in the danger zone and the movement of the human body in the foreground object falls into the abnormal human body movement model, the foreground object is determined to be in an abnormal state.

36. The monitoring method as described in claim 25, characterized in that, Also includes: S600, Trajectory generation step: Generate a trajectory about the change of foreground object state features based on the foreground object state features extracted from the foreground objects corresponding to multiple consecutive data frames in S400.

37. The monitoring method as described in claim 36, characterized in that, In S500, based on the change trajectory of the foreground object's state characteristics, it is predicted whether the foreground object is about to be in an abnormal state, and if the judgment is "yes", it is determined that the foreground object is about to be in an abnormal state.

38. The monitoring method as described in claim 37, characterized in that, In S400, the extracted foreground object state features include one or more of the foreground object's velocity, acceleration, and target density; In S500, it is determined whether the foreground object is about to be in an abnormal state based on the trajectory of one or more of the changes in the velocity, acceleration, and target density of the foreground object.

39. The monitoring method as described in claim 38, characterized in that, When the velocity change trajectory of the foreground object exceeds a preset threshold within a preset time period, it is determined that the foreground object is about to be in an abnormal state. and / or When the acceleration change trajectory of the foreground object exceeds the acceleration trajectory set threshold within a preset time period, it is determined that the foreground object is about to be in an abnormal state. and / or When the target density change trajectory of the foreground object exceeds the density trajectory set threshold within a preset time period, it is determined that the foreground object is about to be in an abnormal state.

40. The monitoring method as described in claim 37, characterized in that, In S400, the extracted foreground object state features include the actions of the human body in the foreground object; In S500, it is determined whether the foreground object is about to be in an abnormal state based on the changing trajectory of the human body in the foreground object.

41. The monitoring method as described in claim 40, characterized in that, When the trajectory of a human body in the foreground object exceeds a preset threshold within a preset time period, it is determined that the foreground object is about to be in an abnormal state.

42. The monitoring method as described in claim 40, characterized in that, In S400, the extracted foreground object state features also include the position of the human body in the foreground object; In S500, it is determined whether the foreground object is about to be in an abnormal state based on the position of the human body in the foreground object and the trajectory of the human body's movements.

43. The monitoring method as described in claim 42, characterized in that, When the trajectory of a human body's position change in the foreground object approaches a danger zone within a preset time period, and when the trajectory of a human body's movement change in the foreground object exceeds a preset threshold for movement trajectory within a preset time period, it is determined that the foreground object is about to be in an abnormal state.

44. The monitoring method according to any one of claims 25 to 42, characterized in that, In S500, if the determination result of at least two consecutive data frames is that the foreground object is in an abnormal state, the foreground object is determined to be in an abnormal state.

45. The monitoring method according to any one of claims 25 to 42, characterized in that, Data frames within a predetermined time period are sensed and acquired at regular intervals for processing by a processing device. The processing device is used to process the data frames to monitor for any abnormalities within the monitored area.

46. ​​The monitoring method according to any one of claims 25 to 42, characterized in that, It also includes S700, alarm step: triggering the alarm unit to work when it is determined that the foreground object is in an abnormal state.

47. A passenger transport device, characterized in that, Includes the monitoring system as described in any one of claims 1 to 24.

Images