Elevator detection violation automatic judgment method based on machine vision

Abstract

An elevator detection violation automatic judgment method based on machine vision comprises the steps that video frames of elevator operation in all areas are obtained, and time synchronization is conducted on the video frames so as to construct a visual sequence set with a unified time sequence; and extracting abnormal events of different areas from the visual sequence set, and carrying out structured representation on the abnormal events to generate a local abnormal event set. And performing time sequence analysis on the local abnormal event set to determine the time sequence of the abnormal events in different regions, and constructing a candidate causal chain between the cross-region abnormal events according to the time sequence. And consistency verification is conducted on the candidate causal chains according to preset elevator operation constraints, so that effective causal chains conforming to the violation mechanism are screened out. According to the method, the effective causal chain is subjected to violation fact judgment, so that the violation type, the violation time and the violation region corresponding to the effective causal chain are output, automatic closing judgment of cross-region violation facts is realized, and a complete formation mechanism of various high-risk violation can be accurately reflected.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an automatic method for detecting violations in elevators based on machine vision. Background Technology

[0002] In engineering scenarios involving cross-regional causal chains of violations, elevator violations are often not directly triggered by a single isolated action, but rather constitute a complete violation by multiple abnormal operations or states that occur sequentially in different physical areas.

[0003] Currently, existing technical solutions for violation detection typically use machine vision-based methods to analyze visual information from single areas such as the car interior, landing door area, or hoistway entrance. Abnormal behaviors in each area are detected and judged separately. However, these solutions lack a unified mechanism for expressing the causal sequence and engineering logic connections between different areas. This results in the system only being able to identify several local violation fragments, unable to integrate them into a complete violation event with a clear cause, process, and result. Taking "abnormal landing door opening—personnel entering the hoistway—car not in position" as a typical example, in this process, the abnormal opening of the landing door itself might only be judged as an abnormal state of the door area, personnel entering the door area would be identified as boundary crossing, and the car not being in position would be an independent operational anomaly. Because these behaviors occur in different areas and are spaced out in time, existing technical solutions struggle to logically close the three events into a single violation, making it difficult to accurately reflect the complete formation mechanism and severity of such high-risk violations. Summary of the Invention

[0004] Therefore, it is necessary to propose a machine vision-based automatic violation judgment method for elevator detection to address the aforementioned technical problems.

[0005] The present invention adopts the following technical solution.

[0006] The first aspect of this invention discloses a machine vision-based method for automatically determining elevator violations, the method comprising: Video frames of elevator operation in each area are acquired and the video frames are synchronized in time to construct a unified set of visual sequences. Extract anomalous events from different regions from the visual sequence set and perform structured representation of the anomalous events to generate a local anomalous event set. A time series analysis is performed on the local abnormal event set to determine the temporal order of the abnormal events in different regions, and a candidate causal chain between cross-regional abnormal events is constructed based on the temporal order. The candidate causal chains are subjected to consistency verification based on preset elevator operation constraints in order to filter out valid causal chains that meet the violation mechanism. The valid causal chain is subjected to violation fact determination to output the violation type, violation time and violation area corresponding to the valid causal chain.

[0007] Furthermore, the step of acquiring video frames of elevator operation in each area and synchronizing the video frames in time to construct a unified temporal visual sequence set includes: The video frames are collected at a fixed sampling frame rate by cameras that monitor the operation of elevators in each area, and the local timestamps corresponding to each camera are written into the video frames to output a video frame sequence carrying the local timestamps. The local timestamps corresponding to each camera are converted into relative times to align the start times of the video frame sequences output by different cameras, thus obtaining an initial set of synchronized video frame sequences. Based on the initial set of synchronized video frame sequences, the relative times of each camera are mapped to the same global time axis using a linear model, and the video frames are mapped to the global time axis in chronological order to construct the set of visual sequences.

[0008] Furthermore, the step of extracting anomalous events from different regions of the visual sequence set and performing structured representation of the anomalous events to generate a local anomalous event set includes: Each region is assigned a corresponding state observation, and the image frame corresponding to each region is converted into a state fragment of the corresponding region based on the state observation. Each region type corresponds to a type of state fragment. Calculate the state stability of each type of state segment within a set time window, and segment out anomaly candidate intervals within each region based on the comparison results of the state stability, state scalar, and set threshold. The state stability is used to measure the temporal continuity of the state segment, the state scalar is used to quantify the state segment, and both the state stability and the state scalar are not lower than their respective set thresholds in the anomaly candidate interval.

[0009] Furthermore, the step of extracting anomalous events from different regions of the visual sequence set and performing structured representation of the anomalous events to generate a local anomalous event set also includes: Calculate an event intensity score within the anomaly candidate interval, and classify the event intensity score into different local anomaly event categories. The event intensity score is proportional to the state stability and the state scalar. If the time interval between adjacent abnormal events of the same type in the same region is less than a set time interval, then the adjacent abnormal events are merged into the same local abnormal event.

[0010] Furthermore, the step of performing time-series analysis on the local anomaly event set to determine the temporal order of the anomaly events in different regions, and constructing candidate causal chains between cross-regional anomaly events based on the temporal order, includes: Calculate the center time and duration of each local abnormal event in the set of local abnormal events, and assign a unique number to each local abnormal event to construct a standard segment of event time sequence; The event time sequence standard segments are sorted according to the chronological order of the central moments, and cross-regional candidate association pairs are established between the abnormal events in different regions when the central moment interval of the abnormal events in different regions is within a set time window.

[0011] Furthermore, the step of performing time-series analysis on the local set of anomalous events to determine the temporal order of the anomalous events in different regions, and constructing candidate causal chains between cross-regional anomalous events based on the temporal order, further includes: Using the cross-regional candidate association pairs as edges to be connected, the abnormal events in different regions are spliced ​​together into a candidate causal chain of a set length through the edges to be connected; The local abnormal events in the candidate causal chain are causally assigned according to time sequence to obtain the role tags corresponding to the first event, the last event, and the middle event of the candidate causal chain. The role tags are then checked for consistency according to the preset regional priority to output the candidate causal chain that passes the consistency check and carries the role tags.

[0012] Furthermore, the step of performing consistency checks on the candidate causal chains based on preset elevator operation constraints to filter out valid causal chains that conform to the violation mechanism includes: Based on the role labels of each abnormal event in the abnormal event sequence corresponding to the candidate causal chain and the cross-regional candidate association pairs, a regional sequence corresponding to the candidate causal chain is generated, and the regional sequence is subjected to consistency verification according to the preset regional combination constraints to determine the regional violation penalty value. Extract the abnormal event type sequence from the candidate causal chain that satisfies the region combination constraint, and compare the abnormal event type sequence with the preset event type order template to output the candidate causal chain that matches the event type order template and the type order violation penalty value. Based on the comparison between the time interval between adjacent abnormal events in the chain and the set time interval, the time beat violation penalty value is determined, and based on the time beat violation penalty value, the region violation penalty value and the type order violation penalty value, the valid causal chain and the validity score corresponding to the valid causal chain are output.

[0013] Furthermore, the step of determining the violation facts of the valid causal chain to output the violation type, violation time, and violation area corresponding to the valid causal chain includes: The event regions and start and end time intervals within the effective causal chain are examined to determine the chain-level basic facts corresponding to the effective causal chain. Based on the chain-level basic facts and the corresponding event types within the chain and a preset mapping table, the violation type corresponding to the effective causal chain is generated. Chain-level basic facts of the same violation type with overlapping start and end time intervals or time intervals shorter than a set time interval are merged to output violation facts containing the violation type, violation time, and violation region corresponding to the effective causal chain.

[0014] The second aspect of this invention discloses an automatic elevator violation determination device based on machine vision, used to implement the automatic elevator violation determination method based on machine vision described in any one of the first aspects, the device comprising: The visual sequence construction module is used to acquire video frames of elevator operation in each area and synchronize the video frames in time to construct a set of visual sequences with a unified time sequence. An abnormal event generation module is used to extract abnormal events from different regions of the visual sequence set and perform structured representation of the abnormal events to generate a local abnormal event set. The causal chain construction module is used to perform time series analysis on the local abnormal event set to determine the time sequence of the abnormal events in different regions, and to construct candidate causal chains between cross-regional abnormal events based on the time sequence. The causal chain verification module is used to perform consistency verification on the candidate causal chains according to preset elevator operation constraints, so as to filter out the valid causal chains that meet the violation mechanism. The violation determination module is used to determine the violation facts of the valid causal chain, and output the violation type, violation time and violation area corresponding to the valid causal chain.

[0015] A third aspect of the present invention discloses a terminal, including a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method described in the first aspect.

[0016] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the method described in the first aspect.

[0017] The present invention has the following advantages: (1) This invention synchronously acquires continuous video sequences in the landing area, car door area, and inside the car, maps the video frames of each area to a unified timestamp, constructs a global time reference shared across areas, and forms a set of elevator operation visual sequences covering multiple physical areas. Subsequently, abnormal events are extracted for different physical areas, and each abnormal event is labeled with its area, occurrence time interval, and event type, forming a locally related abnormal event unit. This effectively avoids the situation where abnormalities in the same area are broken into too many fragments, affecting the closure of subsequent causal chains, and lays the foundation for a unified expression mechanism of causal sequence and engineering logic association between different areas.

[0018] (2) This invention performs time sequence analysis on a set of local abnormal events and constructs candidate causal chains for cross-regional abnormal events based on the sequential occurrence of these events across different regions, clarifying the leading, successor, or result role of each abnormal event in the causal chain. Then, based on the engineering logic constraints of elevator operation, the consistency of the regional combinations, time intervals, and sequence of each abnormal event in the causal chain is verified, and valid causal chains conforming to the engineering violation formation mechanism are selected. Finally, each causal chain is confirmed as a complete violation fact, outputting the corresponding violation type, occurrence time interval, and involved region, thus achieving automatic closed-loop determination of cross-regional violation facts and accurately reflecting the complete formation mechanism and severity of various high-risk violations. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an automatic violation detection method for elevators based on machine vision provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the structure of an automatic violation detection device for elevators based on machine vision provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 As shown in one embodiment, a machine vision-based automatic violation detection method for elevators includes the following steps: Step S110: Obtain video frames of elevator operation in each area and synchronize the video frames in time to construct a unified visual sequence set.

[0024] In some embodiments, the automatic violation determination method for elevator detection based on machine vision provided by the present invention includes the following steps in step S110: Step S111: Video frames are collected at a fixed sampling frame rate by monitoring the operation of elevators in each area by various cameras, and the local timestamps corresponding to each camera are written into the video frames to output a video frame sequence carrying the local timestamps.

[0025] Step S112: Convert the local timestamps corresponding to each camera into relative times to align the start times of the video frame sequences output by different cameras, and obtain an initial set of synchronized video frame sequences.

[0026] Step S113: Based on the initial synchronized video frame sequence set, the relative time corresponding to each camera is mapped to the same global time axis through a linear model, and the video frames are mapped to the global time axis in chronological order to construct a visual sequence set.

[0027] In a specific embodiment, the present invention provides an automatic violation determination method for elevator detection based on machine vision, comprising steps 1 to 5: Step 1: Unified temporal acquisition and synchronous mapping of visual information on elevator operation in multiple areas.

[0028] Continuous video sequences are simultaneously acquired in the landing door area, car door area, and inside the car. A unified timestamp mapping is applied to the video frames from each area to construct a global time reference shared across areas, forming a set of elevator operation visual sequences covering multiple physical areas. This includes the following sub-steps: Sub-step 1.1: Construct a video frame sequence.

[0029] Specifically, image frames are acquired at a fixed sampling frame rate for each camera. The local timestamp of the camera output is written into the frame header or bypass log, constructing a traceable association pair between the image frame and the local time to output a set of frame sequences with local timestamps. To avoid the impact of frame rate differences between different cameras on subsequent alignment, the sampling frame rate is required to be within an engineering-feasible range, such as 15-30fps. At the same time, the starting frame index and starting local time of each video stream are recorded.

[0030] Sub-step 1.2: Video frame synchronization processing.

[0031] Specifically, the local time of each camera is converted into a relative timeline to eliminate the influence of different start times for each camera. This makes different regions comparable in terms of starting time from their respective starting points, thus outputting a set of coarsely synchronized frame sequences across regions. The relative time of a video frame on the relative timeline is equal to the local timestamp of that video frame minus the start frame timestamp of the corresponding camera.

[0032] Sub-step 1.3, global time mapping.

[0033] Specifically, a linear mapping is established between global time and each relative time path, ensuring that each relative time path can be mapped to the same global time axis. This mapping process employs a linear model that is easy to implement in engineering and can be updated online, expressed as follows: ; In the formula, For the first Road camera Global time after frame mapping, in seconds; For the first The time scale coefficient of the road camera is used to compensate for the difference in clock speed of the road. The value range is 0.98-1.02, which corresponds to the engineering range of clock drift within ±2%. For the first The time offset of the road camera, in seconds, is used to compensate for the fixed difference between the relative time of the road and the zero point of the global time. The value range is -2s to +2s, which is caused by the difference between installation and startup. For the first Road camera The relative time of a frame, in seconds.

[0034] It should be noted that within the set alignment time window, the same physical event that can be observed across regions is used as the anchor point, such as the moment when the door starts to move, the moment when the door is completely closed, and the moment when the car stops at the station. The event occurrence time of the reference route is taken as the benchmark, and then the time scale coefficient and time offset that can make the anchor point time consistent are obtained for other routes.

[0035] Sub-step 1.4, multi-region visual sequence mapping.

[0036] Specifically, the time of each frame is first mapped to the global timeline, and then a uniform set of sampling points is established on the global timeline. This results in a unified frame rate output of a multi-region synchronously mapped visual sequence set, where each output time point corresponds to one frame or a small segment of neighboring frames for each region. For each global sampling moment, the video frame with the closest temporal match from each camera is selected as the representative frame for that moment. If missing frames exist, pairing adjacent frames for time interpolation is allowed, but the outputs are still identified by the same global sampling moment.

[0037] Step S120: Extract abnormal events from different regions from the visual sequence set and perform structured representation of the abnormal events to generate a local abnormal event set.

[0038] A local anomaly event refers to the smallest verifiable anomaly unit obtained by merging continuous time intervals after a sequence of state fragments has been determined to be an anomaly within a single physical region.

[0039] In some embodiments, the automatic violation determination method for elevator detection based on machine vision provided by the present invention includes the following steps in step S120: Step S121: Set corresponding state observations for each region, and convert the image frames corresponding to each region into state segments of the corresponding region based on the state observations. Each region type corresponds to a type of state segment.

[0040] State observations are engineering-grade, landable measurements that convert image frames into calculable states, used to obtain state scalars or state codes. They are obtained by defining the Region of Interest (ROI) and the target to be detected within a region, and then using visual detection / tracking to obtain the observations. For example: door state observation = door gap width / door edge distance; door intrusion observation = percentage of human / object within the door ROI or the Intersection over Union (IOU).

[0041] Step S122: Calculate the state stability of each type of state segment within a set time window, and segment out the abnormal candidate intervals in each region based on the comparison results of state stability, state scalar and set threshold.

[0042] Among them, state stability is used to measure the temporal continuity of state segments, and state scalar is used to quantify state segments. In the anomaly candidate interval, both state stability and state scalar are not lower than their respective set thresholds.

[0043] In some embodiments, the automatic violation determination method for elevator detection based on machine vision provided by the present invention further includes the following steps in step S120: Step S123: Calculate the event intensity score within the anomaly candidate interval and classify the event intensity score into different local anomaly event categories. The event intensity score is proportional to the state stability and the state scalar.

[0044] Step S124: Determine whether the time interval between adjacent abnormal events of the same abnormal event type in the same region is lower than the set time interval. If so, merge the adjacent abnormal events into the same local abnormal event.

[0045] In a specific embodiment, the present invention provides an automatic elevator violation detection method based on machine vision. Step 2 involves the extraction and structured description of local anomaly events based on region attributes. In the unified temporal visual sequence output in step 1, anomaly events are extracted for different physical regions, and each anomaly event is labeled with its region, occurrence time interval, and event type, forming a locally associatable anomaly event unit. This includes the following sub-steps: Sub-step 2.1, state fragment transition.

[0046] Specifically, a set of engineering-applicable state observations is defined for each region, converting image frames into state segments. To ensure that subsequent anomaly extraction does not rely on cross-regional data sharing, only the state within the region is output, without cross-regional conclusions. Therefore, for each state type, a temporal continuity metric is required to identify the transition from normal to anomaly. The temporal continuity metric uses the state stability within a window, expressed as: ; In the formula, For the region At the point of time The state stability at a given point, with a value ranging from 0 to 1; The stability calculation window length is a constant, ranging from 3 to 15 frames. , They are respectively regions At the point of time , The numerical representation of the state scalar or state code (e.g., door state: closed=0, half open=0.5, open=1; someone=1, no one=0).

[0047] Sub-step 2.2: Segmentation of abnormal candidate intervals.

[0048] Specifically, within each region, anomaly candidate intervals are segmented from the state sequence based on the principle that the state exceeds a threshold and the duration meets the requirements. First, an anomaly triggering criterion function needs to be defined. When the function value equals 1 at a certain moment, the state is considered abnormal at that moment. The prerequisite for the anomaly triggering criterion function value to equal 1 is that the numerical representation of the state scalar or state code is greater than or equal to a set state threshold (a constant, ranging from 0.6 to 0.9, used to suppress edge states) and the state stability of the corresponding region at that moment is greater than or equal to a set stability threshold (a constant, ranging from 0.5 to 0.9, used to suppress short-term jitter). If other conditions exist, the function value is 0. Next, segments with consecutive function values ​​of 1 are merged into candidate intervals, and their duration is calculated. Only intervals whose duration meets a set lower limit are retained. Finally, a set of anomaly candidate intervals is output, with each interval having a corresponding initial anomaly type label.

[0049] Sub-step 2.3, Event Intensity Scoring and Abnormal Event Unit Construction.

[0050] Specifically, for each candidate interval, an event intensity score is calculated within the interval, and the event intensity score is further subdivided into engineering-interpretable event types, such as: abnormal opening of landing doors, door zone intrusion, and abnormal aggregation within the car. The event intensity score is determined using the area exceeding a threshold method; the higher the state scalar and the longer the event duration, the higher the event intensity score, indicating a more significant anomaly.

[0051] Sub-step 2.4: Event merging.

[0052] Specifically, within the same region and for the same event type, if the interval between two adjacent events is less than a set time interval, these two events are merged into one event, and the time boundary and score are recalculated. The merged score is calculated using a weighted average method, with the weight taken from the event duration. After merging, a final set of local abnormal events is output. Duplicate or adjacent events in this set have all been merged, ensuring that events of the same type in a single region are not fragmented into multiple pieces within a short period of time.

[0053] Step S130: Perform time series analysis on the local abnormal event set to determine the time sequence of the abnormal events in different regions, and construct candidate causal chains between cross-regional abnormal events based on the time sequence.

[0054] The candidate causal chain is constructed as follows: First, each local anomalous event is standardized into fragments; then, events from different regions are compared pairwise. If the central time difference falls within an allowable window, cross-regional candidate association pairs are established and weighted; finally, using these "candidate association pairs" as edges, the events are pieced together chronologically into paths of length 2 to 4, which constitute the candidate causal chain. Understandably, candidate causal chains only satisfy the time sequence + window constraint (connectability), and are essentially potentially related; while valid causal chains are chains remaining after passing the engineering logic consistency check in step 4 and achieving a comprehensive score ≥ the threshold, based on the candidate causal chains. Essentially, they are "determinable evidence chains that conform to the violation formation mechanism."

[0055] In some embodiments, the automatic violation determination method for elevator detection based on machine vision provided by the present invention includes the following steps in step S130: Step S131: Calculate the center time and duration of each local abnormal event in the set of local abnormal events, and assign a unique number to each local abnormal event to construct a standard segment of event time sequence.

[0056] The event sequence standard segment transforms each local abnormal event into a unified time sequence representation, facilitating cross-region sorting and edge connection.

[0057] Step S132: Sort the event time sequence standard segments according to the order of the center time, and establish cross-regional candidate association pairs between abnormal events in different regions when the center time interval of abnormal events in different regions is within the set time window.

[0058] In some embodiments, the automatic violation determination method for elevator detection based on machine vision provided by the present invention further includes the following steps in step S130: Step S133: Using cross-regional candidate association pairs as edges to be connected, different regional abnormal events are spliced ​​together into a candidate causal chain of a set length through the edges to be connected.

[0059] Step S134: Perform causal assignment on local abnormal events in the candidate causal chain according to time order to obtain the role tags corresponding to the head event, tail event and middle event of the candidate causal chain respectively, and perform consistency verification on the role tags according to the preset regional priority to output the candidate causal chain that passes the consistency verification and carries the role tags.

[0060] In a specific embodiment, the present invention provides a machine vision-based automatic violation judgment method for elevator detection. Step 3 involves constructing a causal association of cross-regional abnormal events based on time sequence. The local abnormal event set output in step 2 is analyzed in time sequence. Based on the sequential occurrence relationship of abnormal events in different regions, a candidate causal chain of cross-regional abnormal events is constructed, and the leading, successor, or result role of each abnormal event in the causal chain is clarified. This includes the following sub-steps: Sub-step 3.1, standardization of local abnormal events.

[0061] Specifically, to facilitate causal correlation across regions in chronological order, each local anomalous event needs to be converted into a unified time-series standard segment representation. First, the event center time and duration are calculated, and each event is assigned a unique number to ensure the traceability and verifiability of subsequent chain construction. Then, a set of event time-series standard segments containing event numbers, event center times, and event durations is output.

[0062] Sub-step 3.2, cross-regional candidate association.

[0063] Specifically, the set of standard event time series segments is sorted in ascending order of the event midpoint time to obtain a sorted event set. Then, the midpoint times of any two abnormal events in different regions of the set are compared. If the midpoint time of one abnormal event is greater than the midpoint time of the other abnormal event, and the time interval between the two (the difference between their midpoint times) falls within the set engineering allowable window, then a candidate association pair of the two abnormal events is established.

[0064] The allowable window for engineering operations is composed of a minimum time interval threshold (a constant, ranging from 0 to 0.5 seconds, used to exclude accidental concurrent events occurring at the same time) and a maximum time interval threshold (a constant, ranging from 1 to 20 seconds, set according to the elevator's operating cycle and the door operator's operating cycle). If the time interval between the two abnormal events falls within the allowable window, a correlation weight is defined for these two abnormal events. This correlation weight needs to consider both the fact that the closer the events are, the greater the weight, and the greater the event intensity, the greater the weight. A higher correlation weight indicates that the two abnormal events are more likely to constitute a causal adjacent link.

[0065] Sub-step 3.3: Candidate causal chain generation.

[0066] Specifically, candidate correlation pairs are used as connectable edges. Abnormal events are concatenated into candidate causal chains of length 2-4, with a concatenation length not exceeding four segments. These chains represent the cause, sequence, result, and subsequent results. The chain weight of a candidate causal chain is defined as the weighted average of the weights of adjacent edges on the chain, with the weight coefficient being the reciprocal of the interval between corresponding edges to strengthen tight connections. Simultaneously, the time span of the candidate causal chain, specifically the time interval between the center moments of the events at the beginning and end of the chain, is calculated to subsequently determine whether the chain is too long and whether it conforms to the engineering cycle time.

[0067] Sub-step 3.4: Event role labeling for candidate causal chains.

[0068] Specifically, for each candidate causal chain, events are assigned roles according to chronological order: the first event is marked as the initiator, the last event as the result, and the events in between as successors. Simultaneously, consistency is checked against regional priority rules; if inconsistencies are found, the chain's validity is lowered. To provide a quantifiable role consistency score, a role consistency rating is defined, ranging from 0 to 1, with higher ratings indicating greater consistency between the role and regional rules. Finally, the role consistency rating is combined with the aforementioned chain weights to form the final ranking score for the candidate causal chains.

[0069] Step S140: Perform consistency verification on the candidate causal chains according to the preset elevator operation constraints in order to filter out the valid causal chains that meet the violation mechanism.

[0070] In some embodiments, the automatic violation determination method for elevator detection based on machine vision provided by the present invention includes the following steps in step S140: Step S141: Based on the role labels of each abnormal event in the abnormal event sequence corresponding to the candidate causal chain and the cross-regional candidate association pairs, generate the region sequence corresponding to the candidate causal chain, and perform consistency verification on the region sequence according to the preset region combination constraints to determine the region violation penalty value.

[0071] A region sequence maps events in a candidate causal chain into a sequence of region labels in chronological order.

[0072] Step S142: Extract the abnormal event type sequence from the candidate causal chain that satisfies the region combination constraint, and compare the abnormal event type sequence with the preset event type order template to output the candidate causal chain that matches the event type order template and the type order violation penalty value.

[0073] Step S143: Based on the comparison result between the time interval between adjacent abnormal events in the chain and the set time interval, determine the time beat violation penalty value, and output the valid causal chain and the validity score corresponding to the valid causal chain according to the time beat violation penalty value, the region violation penalty value and the type order violation penalty value.

[0074] In a specific embodiment, the present invention provides an automatic elevator violation detection method based on machine vision. Step 4 involves determining the validity of the violation causal chain based on engineering logic constraints. For the candidate violation causal chains formed in step 3, based on the engineering logic constraints of elevator operation, the consistency of the region combination, time interval, and sequence of each abnormal event in the causal chain is verified to filter out valid causal chains that conform to the engineering violation formation mechanism. This includes the following sub-steps: Sub-step 4.1, Region combination consistency check.

[0075] Specifically, for each candidate causal chain, the cross-regional candidate association pairs and role tags of each event in its event sequence are first read to form the chain's regional sequence. Then, the chain's regional sequence is checked for consistency with the project's preset set of allowed regional combinations. To avoid a lack of gradient in subsequent filtering due to using only "yes / no," a regional violation penalty value is defined to characterize the proportion of events within the chain that do not conform to the project's regional role habits. Finally, a regional combination consistency flag is given based on the regional violation penalty value. If the regional violation penalty value does not exceed the set regional penalty threshold (a constant, ranging from 0 to 0.25, indicating that a maximum of 0 to 25% of nodes are allowed to be non-compliant in their regions), the regional combination consistency check is considered passed. If the check passes, the regional combination consistency flag is set to a value of 1; otherwise, it is set to a value of 0.

[0076] Sub-step 4.2, Event type sequence consistency check.

[0077] Specifically, for candidate causal chains that pass the region combination consistency check, a type order check is performed. During the check, an event type sequence is extracted from each chain, and the extracted event type sequence is compared with a preset set of allowed event type orders. Similarly, to support cases where the template does not match perfectly but is locally reasonable, a type order violation penalty value is defined as the proportion of adjacent event pairs that do not meet the allowed sequential relationship, and a type penalty threshold is set (a constant, ranging from 0 to 0.34; for example, a chain of length 3 allows a maximum of 1 imperfect adjacent relationship, and a chain of length 4 allows a maximum of 1 imperfection). Only when the type order violation penalty value is less than or equal to the type penalty threshold is the event type order consistency flag given a value of 1, indicating that the check has passed; otherwise, it is marked with a value of 0, indicating that the check has failed.

[0078] Sub-step 4.3, time interval consistency check.

[0079] Specifically, for candidate causal chains whose event type sequence consistency is marked with a value of 1, a time interval check is performed. During the check, the center time difference of each pair of adjacent events in the chain is taken, and this center time difference is compared with the corresponding allowed time interval range for the current type pair. The degree to which the time interval exceeds the allowed range is defined as a standardized violation (used to characterize the time violation between adjacent events; the closer the value is to 0, the closer it is to the allowed range; exceeding 1 indicates a significant deviation), and the average violation value is calculated for the entire chain. Similarly, a time tick penalty value also needs to be defined. This penalty value is taken as the average of the violation values ​​of the entire chain, ranging from 0 to 2. The larger the value, the less it conforms to the engineering tick. Combined with the set time penalty threshold (range 0.3-0.8), if the time tick penalty value is less than or equal to the time penalty threshold, the time interval consistency is marked with a value of 1, indicating that the check passed; otherwise, it is marked with a value of 0, indicating that the check failed.

[0080] Sub-step 4.4: Output the valid violation causal chain.

[0081] Specifically, for causal chains marked with a time interval consistency value of 1, a comprehensive validity score is calculated. This comprehensive validity score is determined by the aforementioned chain ranking score combined with the regional violation penalty value, the type order violation penalty value, and the time rhythm penalty value. The higher the value, the more consistent it is with the engineering violation formation mechanism. Finally, when the comprehensive validity score is not lower than the set minimum validity threshold (a constant, ranging from 0.2 to 1.5, calibrated through historical engineering cases), the corresponding causal chain is output as a valid causal chain.

[0082] Step S150: Determine the violation facts for the valid causal chain, and output the violation type, violation time, and violation area corresponding to the valid causal chain.

[0083] Violation types are chain-level violation labels that map a valid causal chain. Example 1: Door intrusion violation: Precursor = Door intrusion (car door area) → Continuation = Repeated door opening and closing / door remaining open → Result = Car operation obstructed / passengers stranded. Example 2: Abnormal landing door opening violation: Precursor = Landing door opened before arrival (landing door area) → Continuation = Car not fully stopped / not aligned → Result = Personnel approaching the landing door, creating a risk (door area / landing door area combination).

[0084] In some embodiments, the automatic violation determination method for elevator detection based on machine vision provided by the present invention includes the following steps in step S150: Step S151 involves verifying the event regions and start-end time intervals within a valid causal chain to determine the chain-level basic facts corresponding to the valid causal chain. Based on the chain-level basic facts and the corresponding in-chain event types, and a pre-defined mapping table, a violation type corresponding to the valid causal chain is generated. Chain-level basic facts refer to factual description units formed by merging multiple local abnormal events contained within the causal chain as constituent evidence of the same violation, assuming the valid causal chain has been confirmed. In-chain event types refer to the sequence of abnormal categories of the local abnormal events constituting the same causal chain in chronological order. This sequence fully reflects the abnormal morphological structure of the violation from triggering, evolution, to its result.

[0085] Step S152: Merge chain-level basic facts of the same violation type that have overlapping start and end time intervals or time intervals shorter than a set time interval, so as to output violation facts that include the violation type, violation time and violation area corresponding to the valid causal chain.

[0086] In a specific embodiment, the present invention provides an automatic violation determination method for elevator detection based on machine vision. Step 5 involves automatic determination and output of violation facts based on complete causal chains. The valid violation causal chains confirmed in step 4 are judged as a whole, with each causal chain confirmed as a complete violation fact. The corresponding violation type, occurrence time interval, and involved area are output, achieving automatic closed-loop determination of cross-regional violation facts. This includes the following sub-steps: Sub-step 5.1: Reverse lookup of events within the chain.

[0087] Specifically, for each valid causal chain, based on the known dictionary of local anomalies, the regions and time intervals of each event within the chain are sequentially looked up according to the event number, generating chain-level fact boundaries. The start and end times of the facts are taken as the earliest start and latest end times of all event intervals within the chain, i.e., the regions are taken as a union to ensure complete coverage of the evidence within the chain. Finally, duplicates are removed from the region union to obtain the chain-level basic fact set.

[0088] Sub-step 5.2, chain-level violation type determination.

[0089] Specifically, for the chain-level basic facts obtained in sub-step 5.1, the event type sequence within the corresponding valid causal chain is read, and the read event type sequence is mapped to the winning violation type according to the pre-defined violation type mapping table of the project. Then, to avoid multiple possible types within the same chain, a type confidence score needs to be calculated. This confidence score comprehensively considers the chain validity and the intensity of events within the chain; a higher value indicates a more credible type, and different weights are assigned to different roles (leader, successor, and result have different importance).

[0090] Sub-step 5.3: Merge the violations.

[0091] Specifically, facts with overlapping time intervals or intervals shorter than a set time interval for the same violation type are merged to avoid multiple duplicate violations output by adjacent chains. Furthermore, the merged confidence score is weighted by fact duration to ensure that longer facts are not distorted by shorter facts. After merging, only violations with a merged fact confidence score not lower than a set threshold (a constant ranging from 0.2 to 1.2 seconds, set according to engineering requirements) are output, including the corresponding causal chain, violation type, violation time, and violation region.

[0092] The following describes an automatic elevator violation determination device based on machine vision provided by the present invention. The automatic elevator violation determination device based on machine vision described below and the automatic elevator violation determination method based on machine vision described above can be referred to and correspond to each other.

[0093] like Figure 2 As shown in one embodiment, an automatic violation determination device for elevator detection based on machine vision includes a visual sequence construction module, an abnormal event generation module, a causal chain construction module, a causal chain verification module, and a violation fact determination module.

[0094] The visual sequence construction module is used to acquire video frames of elevator operation in each area and synchronize the video frames in time to build a set of visual sequences with a unified time sequence.

[0095] The abnormal event generation module is used to extract abnormal events from different regions of the visual sequence set and perform structured representation of the abnormal events to generate a local abnormal event set.

[0096] The causal chain construction module is used to perform time series analysis on a local set of anomalous events to determine the temporal order of the anomalous events in different regions, and to construct candidate causal chains between anomalous events across regions based on the temporal order.

[0097] The causal chain verification module is used to perform consistency verification on candidate causal chains based on preset elevator operation constraints, so as to filter out valid causal chains that meet the violation mechanism.

[0098] The violation determination module is used to determine the violation facts of valid causal chains, and output the violation type, violation time and violation area corresponding to the valid causal chain.

[0099] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A machine vision-based automatic violation detection method for elevators, characterized in that, The method includes: Video frames of elevator operation in each area are acquired and the video frames are synchronized in time to construct a unified set of visual sequences. Extract anomalous events from different regions from the visual sequence set and perform structured representation of the anomalous events to generate a local anomalous event set. A time series analysis is performed on the local abnormal event set to determine the temporal order of the abnormal events in different regions, and a candidate causal chain between cross-regional abnormal events is constructed based on the temporal order. The candidate causal chains are subjected to consistency verification based on preset elevator operation constraints in order to filter out valid causal chains that meet the violation mechanism. The valid causal chain is subjected to violation fact determination to output the violation type, violation time and violation area corresponding to the valid causal chain.

2. The automatic violation determination method for elevator detection based on machine vision according to claim 1, characterized in that, The process of acquiring video frames of elevator operation in each area and synchronizing these video frames in time to construct a unified temporal visual sequence set includes: The video frames are collected at a fixed sampling frame rate by cameras that monitor the operation of elevators in each area, and the local timestamps corresponding to each camera are written into the video frames to output a video frame sequence carrying the local timestamps. The local timestamps corresponding to each camera are converted into relative times to align the start times of the video frame sequences output by different cameras, thus obtaining an initial set of synchronized video frame sequences. Based on the initial set of synchronized video frame sequences, the relative times of each camera are mapped to the same global time axis using a linear model, and the video frames are mapped to the global time axis in chronological order to construct the set of visual sequences.

3. The automatic violation determination method for elevator detection based on machine vision according to claim 1, characterized in that, The step of extracting anomalous events from different regions of the visual sequence set and performing structured representation of the anomalous events to generate a local anomalous event set includes: Each region is assigned a corresponding state observation, and the image frame corresponding to each region is converted into a state fragment of the corresponding region based on the state observation. Each region type corresponds to a type of state fragment. Calculate the state stability of each type of state segment within a set time window, and segment out anomaly candidate intervals within each region based on the comparison results of the state stability, state scalar, and set threshold. The state stability is used to measure the temporal continuity of the state segment, the state scalar is used to quantify the state segment, and both the state stability and the state scalar are not lower than their respective set thresholds in the anomaly candidate interval.

4. The automatic violation determination method for elevator detection based on machine vision according to claim 3, characterized in that, The step of extracting anomalous events from different regions of the visual sequence set and performing structured representation of the anomalous events to generate a local anomalous event set further includes: Calculate an event intensity score within the anomaly candidate interval, and classify the event intensity score into different local anomaly event categories. The event intensity score is proportional to the state stability and the state scalar. If the time interval between adjacent abnormal events of the same type in the same region is less than a set time interval, then the adjacent abnormal events are merged into the same local abnormal event.

5. The automatic violation determination method for elevator detection based on machine vision according to claim 1, characterized in that, The step of performing time-series analysis on the local anomaly event set to determine the temporal order of the anomaly events in different regions, and constructing candidate causal chains between cross-regional anomaly events based on the temporal order, includes: Calculate the center time and duration of each local abnormal event in the set of local abnormal events, and assign a unique number to each local abnormal event to construct a standard segment of event time sequence; The event time sequence standard segments are sorted according to the chronological order of the central moments, and cross-regional candidate association pairs are established between the abnormal events in different regions when the central moment interval of the abnormal events in different regions is within a set time window.

6. The automatic violation determination method for elevator detection based on machine vision according to claim 5, characterized in that, The step of performing time-series analysis on the local anomaly event set to determine the temporal order of the anomaly events in different regions, and constructing candidate causal chains between cross-regional anomaly events based on the temporal order, further includes: Using the cross-regional candidate association pairs as edges to be connected, the abnormal events in different regions are spliced ​​together into a candidate causal chain of a set length through the edges to be connected; The local abnormal events in the candidate causal chain are causally assigned according to time sequence to obtain the role tags corresponding to the first event, the last event, and the middle event of the candidate causal chain. The role tags are then checked for consistency according to the preset regional priority to output the candidate causal chain that passes the consistency check and carries the role tags.

7. The automatic violation determination method for elevator detection based on machine vision according to claim 6, characterized in that, The step of performing consistency checks on the candidate causal chains based on preset elevator operation constraints to filter out valid causal chains that meet the violation mechanism includes: Based on the role labels of each abnormal event in the abnormal event sequence corresponding to the candidate causal chain and the cross-regional candidate association pairs, a regional sequence corresponding to the candidate causal chain is generated, and the regional sequence is subjected to consistency verification according to the preset regional combination constraints to determine the regional violation penalty value. Extract the abnormal event type sequence from the candidate causal chain that satisfies the region combination constraint, and compare the abnormal event type sequence with the preset event type order template to output the candidate causal chain that matches the event type order template and the type order violation penalty value. Based on the comparison between the time interval between adjacent abnormal events in the chain and the set time interval, the time beat violation penalty value is determined, and based on the time beat violation penalty value, the region violation penalty value and the type order violation penalty value, the valid causal chain and the validity score corresponding to the valid causal chain are output.

8. The automatic violation determination method for elevator detection based on machine vision according to claim 7, characterized in that, The step of determining the violation facts of the valid causal chain, and outputting the violation type, violation time, and violation area corresponding to the valid causal chain, includes: The event regions and start and end time intervals within the effective causal chain are examined to determine the chain-level basic facts corresponding to the effective causal chain. Based on the chain-level basic facts and the corresponding event types within the chain and a preset mapping table, the violation type corresponding to the effective causal chain is generated. Chain-level basic facts of the same violation type with overlapping start and end time intervals or time intervals shorter than a set time interval are merged to output violation facts containing the violation type, violation time, and violation region corresponding to the effective causal chain.

9. A terminal, comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-8.

Images