Pole number plate missing detection method, device and equipment based on space-time fusion perception and medium

Abstract

The invention discloses a rod number plate missing detection method and device based on space-time fusion perception, equipment and a medium, and the method comprises the steps: obtaining a multi-view image or a time sequence multi-frame image of a rod number plate, and obtaining the position and character string of the rod number plate in each image based on a deep learning model; judging whether the corresponding image is reserved or not according to the depth value of the pole number plate and the depth value of the strut; obtaining a space fusion result of the pole number plate based on the reserved image; defining a hidden state set, an observation state set and a hidden Markov model; and a historical observation sequence of the strut is constructed, the corresponding hidden state set is speculated based on a hidden Markov model, early warning is carried out on abnormal deletion of the pole number plate based on a preset accident degree mechanism and a cumulative likelihood mechanism of a dominant state, and the historical observation sequence is related to the observation state set. The invention belongs to the field of pole number plate detection. According to the invention, the missing pole number plate can be detected.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of pole number plate detection, and in particular to a pole number plate missing detection method and device based on spatiotemporal fusion perception, equipment and medium. BACKGROUND

[0002] In the operation and maintenance of the overhead contact system of a railway, the pole number plate is a key identity of the support, and its integrity is directly related to the equipment management and safety inspection. According to the Operation and Maintenance Rules for Overhead Contact System of High-speed Railway, the missing of the pole number plate belongs to the incomplete identification defects that must be found in time. The current mainstream inspection relies on the 4C / 2C vehicle-mounted vision system to automatically take pictures during the high-speed operation of the train and identify the pole number content through OCR. The existing technology mainly focuses on improving the accuracy of OCR in a single inspection, such as through multi-picture screening or logical correction based on the sequence rules of the pole number.

[0003] However, the above method has two major defects: first, it is based only on two-dimensional images and cannot determine the true three-dimensional space belonging of the pole number plate and the current support, which is easy to misassociate the adjacent pole number in complex scenes such as multiple tracks, resulting in missed detection of the true missing or pollution of historical data; second, it lacks time dimension perception and only determines the missing based on a single undetected, which is difficult to distinguish between the true loss and the instantaneous undetected caused by obstruction, weather, etc., resulting in high false positives or false negatives. Therefore, it is urgent to propose a pole number plate missing detection method based on spatiotemporal fusion perception. SUMMARY

[0004] The present application provides a pole number plate missing detection method, device, equipment and medium based on spatiotemporal fusion perception, which solves the technical problem of high false positive rate of the true state of the pole number plate in the prior art and achieves the technical effect of improving the accuracy of pole number plate detection.

[0005] In a first aspect, the present application provides a pole number plate missing detection method based on spatiotemporal fusion perception, comprising: S11, obtaining multi-view images or time-series multi-frame images of the pole number plate, and obtaining the position and string of the pole number plate in each image based on a deep learning model, wherein the multi-view images or time-series multi-frame images each include a plurality of images; S12, including S121, positioning the support of each image based on a target detection or segmentation algorithm, and determining whether the corresponding image is retained according to the depth value of the pole number plate and the depth value of the support; S122, obtaining the spatial fusion result of the pole number plate based on the retained images, wherein the spatial fusion result includes the existence recognition result of the pole number plate and the string number recognition result; S13, including S131, defining a hidden state set , an observation state set and a hidden Markov model wherein the hidden state set is related to the actual physical state of the pole sign, and the observation state set is related to the spatial fusion result, , is an initial state probability of the pole, is a state transition probability matrix of the pole sign, is an observation probability matrix; S132, constructing a historical observation sequence of the pole wherein is is observation data at the moment, and based on the hidden Markov model speculating the corresponding hidden state set , and based on a preset abnormality mechanism and a cumulative likelihood mechanism of the dominant state, the abnormal absence of the pole sign is warned, and the historical observation sequence is related to the observation state set .

[0006] Further, judging whether the corresponding image is reserved according to the depth value of the pole sign and the depth value of the pole, comprising: determining the absolute value difference between the depth value of the pole sign and the depth value of the corresponding pole; if the absolute value difference is less than or equal to an adaptive threshold value, the corresponding image is reserved, otherwise, the corresponding image is rejected, wherein the adaptive threshold value is related to the depth value of the pole.

[0007] Further, when the reserved image belongs to a multi-view image, based on the reserved image, the spatial fusion result of the pole sign is obtained, comprising: if the pole sign exists in at least one image, the pole sign existing recognition result is judged as existing, otherwise it is judged as not existing; when the pole sign existing recognition result is judged as existing, the corresponding string with the highest occurrence frequency in the image is taken as the string number recognition result; when the pole sign existing recognition result is judged as not existing, the string number recognition result is judged as no string.

[0008] Further, when the reserved image belongs to a time-series multi-frame image, based on the reserved image, the spatial fusion result of the pole sign is obtained, comprising: based on a multi-target tracking algorithm, the pole sign in the image with consecutive frames is associated as a motion track, if the number of frames of the motion track is greater than a preset frame number, the pole sign existing recognition result is judged as existing, otherwise it is judged as not existing; when the pole sign existing recognition result is judged as existing, the corresponding string with the highest occurrence frequency in the image is taken as the string number recognition result; when the pole sign existing recognition result is judged as not existing, the string number recognition result is judged as no string.

[0009] Further, the hidden state set is defined , the set of observation states and the hidden Markov model , comprising: defining the set of hidden states comprising wherein is the actual physical state of the pole sign is present and the visual observation is ideal, is the actual physical state of the pole sign is absent, is the actual physical state of the pole sign is present but the visual observation is limited; the set of observation states comprising wherein is the pole sign is determined to be present in the spatial fusion result, is the pole sign is determined to be absent in the spatial fusion result; defining the initial state probability of the pole sign wherein the initial state of the pole sign is present with a probability the initial state of the pole sign is absent with a probability the initial state of the pole sign is visually observed to be limited with a probability wherein all belong to ; defining the state transition probability matrix of the pole sign wherein, from time to time, the state of the pole sign being present and the visual observation being ideal does not change with a probability ; from time to time, the state of the pole sign changes from being present and the visual observation being ideal to being present but the visual observation being limited with a probability ; from time to time, the state of the pole sign changes from being present and the visual observation being ideal to being absent with a probability ; from time to time, the state of the pole sign changes from being visually observed to be limited to being present and the visual observation being ideal with a probability ; from time to time, the state of the pole sign being present but the visual observation being limited does not change with a probability ; from time to time, the state of the pole sign changes from being present but the visual observation being limited to being absent with a probability ; from time to Probability that the pole number plate is recovered from the missing state to the existing state with ideal visual observation at time t ; from time t to time t, the probability that the pole number plate is recovered from the missing state to the existing state with limited visual observation ; from time t to time t, the probability that the pole number plate is not recovered from the missing state , wherein all belong to ; Define the observation probability matrix , wherein when the actual physical state of the pole number plate is existing, the probability that the pole number plate is determined to exist in the spatial fusion result is ; when the actual physical state of the pole number plate is existing, the probability that the pole number plate is determined to not exist in the spatial fusion result is ; when the actual physical state of the pole number plate is existing but visual observation is limited, the probability that the pole number plate is determined to exist in the spatial fusion result is ; when the actual physical state of the pole number plate is existing but visual observation is limited, the probability that the pole number plate is determined to not exist in the spatial fusion result is ; when the actual physical state of the pole number plate is non-existing, the probability that the pole number plate is determined to exist in the spatial fusion result is ; when the actual physical state of the pole number plate is non-existing, the probability that the pole number plate is determined to not exist in the spatial fusion result is , wherein all belong to .

[0010] Further, the historical observation sequence of the support is constructed , and the corresponding hidden state set is inferred based on a hidden Markov model , including: When the pole number plate is determined to exist in the spatial fusion result, the pole number is matched according to the string digit recognition result and the mileage information; When the pole number plate is determined to not exist in the spatial fusion result, the string digit recognition results of the previous pole number plate and the next pole number plate are extracted, and the string digit recognition result corresponding to the pole number plate is obtained based on linear interpolation, and the pole number is matched according to the string digit recognition result and the mileage information; The string digit recognition results of a plurality of matched supports are sequentially constructed into the historical observation sequence of the support ; When the number of observation results in the historical observation sequence is greater than a preset number, the hidden state posterior probability up to the previous time is determined according to the historical observation sequence; determining a prior state distribution at a current time according to a hidden state posterior probability and a state transition probability matrix determining a prior state distribution at a current time according to a hidden state posterior probability and a state transition probability matrix a probability of non-existence of a pole number plate in a hidden Markov model, including actual physical state non-existence and determination of non-existence of a pole number plate; when the number of observation results in the historical observation sequence is less than or equal to the preset number, then continue to accumulate the number of observation results.

[0011] Further, based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of a dominant state, an abnormal absence of a pole number plate is warned, including: the preset unexpectedness mechanism includes:

[0012] wherein, unexpectedness, a probability of non-existence of a pole number plate in a hidden Markov model; determining a dominant state, including:

[0013] wherein, a dominant state at a current time, an observation state at a time t and before, a hidden state of a system at a time t; a hidden state set of an i-th pole number plate; determining a cumulative likelihood in a visual observation limited state, including:

[0014] wherein, a number of times that a same position is continuously observed as being limited in visual observation and observation results are continuously not detected from a time t back to before; a cumulative likelihood; when the unexpectedness is greater than a preset unexpectedness threshold, the number of observation results is greater than a preset number, and a pole number plate is determined to be absent in a spatial fusion result at a current time, a preliminary warning signal of absence of a pole number plate is generated; when the unexpectedness is less than or equal to a preset unexpectedness threshold, the dominant state is an absence state, the number of observation results is greater than a preset number, and a pole number plate is determined to be absent in a spatial fusion result at a current time, a preliminary warning signal of long-term continuous absence of a pole number plate is generated; when the unexpectedness is less than or equal to a preset unexpectedness threshold, the dominant state is an absence state, the number of observation results is greater than a preset number, and a pole number plate is determined to be absent in a spatial fusion result at a current time, a preliminary warning signal of long-term continuous absence of a pole number plate is generated;​​​​​ for the visual observation limited state, less than a preset level threshold, the number of observation results is greater than a preset number, and the pole number plate does not exist in the spatial fusion result at the current moment, a preliminary warning signal of the pole number plate missing in the complex fluctuation environment is generated; When the number of observation results in the historical observation sequence is less than or equal to the preset number, the number of observation results is continuously accumulated. When the pole number plate exists in the spatial fusion result at the current moment, no signal is generated.

[0015] In a second aspect, the present application provides a pole number plate missing detection device based on spatio-temporal fusion perception, comprising: An image acquisition module is used for S11 to acquire multi-view images or time sequence multi-frame images of the pole number plate, and to obtain the position and string of the pole number plate in each image based on a deep learning model. The multi-view images or time sequence multi-frame images each include a plurality of images. An identification module is used for S12 to include S121 to position the support of each image based on a target detection or segmentation algorithm, and to determine whether the corresponding image is retained according to the depth value of the pole number plate and the depth value of the support. S122, based on the retained image, obtains the spatial fusion result of the pole number plate, wherein the spatial fusion result includes a pole number plate existence recognition result and a string number recognition result. A warning module is used for S13 to include S131 to define a hidden state set , an observation state set and a hidden Markov model , wherein the hidden state set is related to the actual physical state of the pole number plate, the observation state set is related to the spatial fusion result, , is the initial state probability of the support, is the state transition probability matrix of the pole number plate, is the observation probability matrix; S132, a historical observation sequence of the support is constructed , wherein is the observation data at the current moment, and based on the hidden Markov model , the corresponding hidden state set is inferred, and based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state, the abnormal missing of the pole number plate is warned. The historical observation sequence is related to the observation state set .

[0016] In a third aspect, the present application provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; The processor is configured to implement the pole plate missing detection method based on spatio-temporal fusion perception as provided in the first aspect.

[0017] In a fourth aspect, the present application provides a non-transitory computer readable storage medium, when instructions in the non-transitory computer readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to implement the pole plate missing detection method based on spatio-temporal fusion perception as provided in the first aspect.

[0018] The one or more technical solutions provided in the present application have at least the following technical effects or advantages: The present application introduces two perception dimensions of space fusion and time fusion into pole plate missing detection, constructs a double verification and correction framework, and makes up for the limitations of single image recognition method. The present application uses monocular depth estimation algorithm to expand two-dimensional image analysis to three-dimensional space relationship understanding, and improves the attribution accuracy of multiple pole plates in complex scenes. The present application discards the traditional judgment logic based on fixed rules, and proposes a real-time inference framework based on historical prediction. The core of the framework is that the real physical state of the pole plate is modeled as the "hidden state" of the Hidden Markov Model (HMM), and the single inspection result is modeled as the "observation state". At each inspection time, the state confidence is calculated based on the historical observation using the forward algorithm, and the probability of "not detected" at the current time is predicted. The deviation between the predicted probability and the actual observation (i.e. "unexpectedness" and "cumulative likelihood of dominant state") is compared to determine whether an anomaly occurs in real time, rather than waiting for multiple observations before making a judgment. A fixed unexpectedness threshold that can be adaptively linked is introduced, so that the framework can intelligently distinguish between "real missing" and "continuous occlusion" and other different scenarios, while realizing the unification of real-time warning and high accuracy. The present application realizes real-time warning under the premise of ensuring accuracy through the real-time prediction inference framework, and solves the core technical problem of the contradiction between timeliness and accuracy in the inspection business. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 The flowchart of the pole plate missing detection method based on spatio-temporal fusion perception provided by the present application is shown. Figure 2 The schematic diagram of the pole plate provided by the present application is shown. DETAILED DESCRIPTION

[0021] The embodiment of the application solves the technical problem of high false report rate of the real state of the pole number plate in the prior art by providing a pole number plate missing detection method based on spatio-temporal fusion perception.

[0022] The technical solution of the application is as follows to solve the above technical problem: The pole number plate missing detection method based on spatio-temporal fusion perception comprises: S11, acquiring multi-view images or time-series multi-frame images of a pole number plate, and obtaining the position and string of the pole number plate in each image based on a deep learning model, wherein the multi-view images or time-series multi-frame images each comprise a plurality of images; S12, comprising S121, positioning a support of each image based on a target detection or segmentation algorithm, and determining whether the corresponding image is retained according to the depth value of the pole number plate and the depth value of the support; S122, obtaining a spatial fusion result of the pole number plate based on the retained images, wherein the spatial fusion result comprises a pole number plate existence recognition result and a string number recognition result; S13, comprising S131, defining a hidden state set , an observation state set , and a hidden Markov model , wherein the hidden state set is related to the actual physical state of the pole number plate, the observation state set is related to the spatial fusion result, , is an initial state probability of the support, is a state transition probability matrix of the pole number plate, is an observation probability matrix; S132, constructing a historical observation sequence of the support , and inferring the corresponding hidden state set based on the hidden Markov model , and warning about abnormal missing of the pole number plate based on a preset abnormality mechanism and a cumulative likelihood mechanism of a dominant state, wherein the historical observation sequence is related to the observation state set .

[0023] In order to better understand the above technical solution, the above technical solution will be described in detail in combination with the drawings in the specification and the specific embodiments.

[0024] Firstly, the term "and / or" appearing in the present text is only to describe the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present text generally represents an "or" relationship between the front and rear associated objects.

[0025] Automatic inspection is performed, and the inspection content includes confirming whether various marks are complete, wherein the inspection of the pole number plate on the support is one of the important contents of the inspection.

[0026] The application provides a pole number plate missing detection method based on space-time fusion perception, as shown in the formula (I) : Figure 1 S11, a multi-view image or a time sequence multi-frame image of a pole number plate is acquired, and a position of the pole number plate in each image and a string of the pole number plate are obtained based on a deep learning model, and the multi-view image or the time sequence multi-frame image comprises a plurality of images.

[0027] When a train is detected to run to a predetermined effective shooting window of a support column, image information can be acquired in the following one or more ways according to a vehicle-mounted device configuration: Multi-view image acquisition (for a multi-camera system, such as a 4C device): a plurality of single-frame multi-view images of the support column are acquired by triggering a plurality of cameras at different angles to take pictures synchronously, and finally the multi-view images of the support column are collected.

[0028] Time sequence multi-frame image acquisition (for a single-camera system, such as a 2C device): time sequence multi-frame images of the support column are acquired by continuously shooting the support column in a short time by using the camera.

[0029] The acquired multi-view image or time sequence multi-frame image is input into a pre-trained deep learning model, a position of the pole number plate in each image and a string of the pole number plate are obtained by preliminary identification of each image.

[0030] As shown in the formula (II) : Figure 2 Figure 2 For simplicity of the schematic diagram of the support column, 0196 is a string of the pole number plate (which can also be referred to as a support column number).

[0031] S12, comprising S121 and S122: S121, a target detection or segmentation algorithm is used to locate the support column in each image, and whether the corresponding image is retained is determined according to a depth value of the pole number plate and a depth value of the support column.

[0032] Whether the corresponding image is retained is determined according to the depth value of the pole number plate and the depth value of the support column, comprising: determining an absolute value difference between the depth value of the pole number plate and the depth value of the corresponding support column; if the absolute value difference is less than or equal to an adaptive threshold value, the corresponding image is retained, otherwise, the corresponding image is removed, wherein the adaptive threshold value is related to the depth value of the support column.

[0033] The target detection or segmentation algorithm can be used to locate the overhead contact system support column in the image. The adaptive threshold value is dynamically determined according to the average depth of the support column, comprising:

[0034] wherein, is the adaptive threshold value, ​​preset ratio coefficient, depth value of the support, The value range is {0.05, 0.10}.

[0035] If the absolute value difference is less than or equal to the adaptive threshold, it is determined that the pole sign and the support are spatially associated, and is retained.

[0036] Otherwise, it is determined to be a background or a pole sign of another lane, and is rejected.

[0037] S122, based on the retained image, obtaining a spatial fusion result of the pole sign, wherein the spatial fusion result includes a pole sign existence recognition result and a string number recognition result.

[0038]

When the retained image belongs to multi-view images, based on the retained image, obtaining a spatial fusion result of the pole sign

[0039] When the retained image belongs to multi-view images (multi-view images of the support), all retained multi-view images are summarized.

[0040] Then cross-validation is performed, as long as the pole sign is recognized in at least one view, it is preliminarily determined that the pole sign of the support exists in this inspection.

[0041] For example, three of the four camera views (C01, C02, C03, C04) (C01, C03, C04) have detected the pole sign, and it is determined that the pole sign physically exists.

[0042] The string corresponding to the most frequent occurrence in the image is taken as the string number recognition result.

[0043] The string number recognition results of C01 and C04 views are both T101, and the string number recognition result of C03 view is T7O1, so T101 is the string number recognition result of the support.

[0044] The spatial fusion result includes a pole sign existence recognition result and a string number recognition result.

[0045]

When the retained image belongs to time-series multi-frame images, based on the retained image, obtaining a spatial fusion result of the pole sign

[0046] When the reserved image belongs to a time sequence multi-frame image, for the reserved image, the effective detection frame screened in the consecutive image frames can be input into a multi-target tracking algorithm, and the detection frames belonging to the same pole number plate are associated into a motion trajectory. When the stable and continuous frame number of the motion trajectory is greater than or equal to 5 times (the preset frame number can be set by the user, which is not limited here), it is confirmed that the pole number plate exists. The spatial fusion result includes the pole number plate existence recognition result and the string number recognition result.

[0047] S13, comprising S131 and S132: S131, defining a hidden state set , an observation state set and a hidden Markov model , wherein the hidden state set is related to the actual physical state of the pole number plate, the observation state set is related to the spatial fusion result, , is the initial state probability of the support column, is the state transition probability matrix of the pole number plate, is the observation probability matrix.

[0048] defining a hidden state set , an observation state set and a hidden Markov model , comprising: defining a hidden state set comprising , wherein is the actual physical state of the pole number plate is present and the visual observation is ideal (no occlusion, normal light), is the actual physical state of the pole number plate is absent, is the actual physical state of the pole number plate is present but the visual observation is limited (such as vegetation occlusion, strong backlight, etc.).

[0049] Based on the double-state HMM, a probability inference framework based on a three-layer hidden state hidden Markov model (HMM) is revised, which not only distinguishes between "existence" and "absence", but also introduces a "visual limited" state to represent environmental interference, which can further improve the stability of the evaluation.

[0050] an observation state set comprising wherein is determined to exist in the spatial fusion result (i.e., the pole number plate is detected), is determined to not exist in the spatial fusion result (i.e., the pole number plate is not detected).

[0051] Hidden Markov Model contains three sets of parameters, the initial state probability reflects the prior state distribution of the pole.

[0052] define the initial state probability of the pole number plate wherein the probability of the initial state of the pole number plate being present is the probability of the initial state of the pole number plate being missing is the probability of the initial state of the pole number plate being visually observed limited is wherein all belong to .

[0053] The corresponding meanings are that the newly put into operation pole, the pole number plate of the pole has a 99.3% probability of being completely installed and having ideal observation conditions in the initial state, and the pole number plate of the pole has a 0.5% probability of being visually limited and unable to be observed due to environmental obstruction or shooting angle problems in the initial state.

[0054] State transition probability matrix reflects the change rule of the physical state of the pole number plate over time, i.e., the state change probability from time to time.

[0055] define the state transition probability matrix of the pole number plate wherein, from time to time, the probability that the pole number plate exists and the visually observed ideal state does not change is ; from time to time, the probability that the pole number plate changes from the state of existing and being visually observed ideal to the state of existing but being visually observed limited is ; from time to time, the probability that the pole number plate changes from the state of existing and being visually observed ideal to the state of missing is ; from time to time, the probability that the pole number plate recovers from the state of being visually observed limited to the state of existing and being visually observed ideal is ; from time to Probability that the state of the pole number plate does not change from the state of existing but limited visual observation to the state of missing at time ; from time to time Probability that the state of the pole number plate changes from the state of existing but limited visual observation to the state of missing at time ; from time to time Probability that the state of the pole number plate recovers from the state of missing to the state of existing and ideal visual observation at time ; from time to time Probability that the state of the pole number plate recovers from the state of missing to the state of existing but limited visual observation at time ; from time to time Probability that the state of the pole number plate does not recover from the state of missing at time wherein all belong to .

[0056] A time adaptation mechanism can also be added: when the actual inspection cycle deviates from the standard inspection cycle, the state transition probability can be dynamically adjusted:

[0057] wherein, is the state transition probability of the pole number plate for the standard inspection cycle, is the standard inspection cycle, is the actual inspection cycle, is the probability that the state of the pole number plate changes from the state of existing to the state of missing, corresponding to the value of matrix A.

[0058] Define the observation probability matrix wherein when the actual physical state of the pole number plate is existing, the probability that the pole number plate is determined to exist in the spatial fusion result is ; when the actual physical state of the pole number plate is existing, the probability that the pole number plate is determined to not exist in the spatial fusion result is ; when the actual physical state of the pole number plate is existing but limited visual observation, the probability that the pole number plate is determined to exist in the spatial fusion result is ; when the actual physical state of the pole number plate is existing but limited visual observation, the probability that the pole number plate is determined to not exist in the spatial fusion result is ; when the actual physical state of the pole number plate is missing, the probability that the pole number plate is determined to exist in the spatial fusion result is ; when the actual physical state of the pole number plate is missing, the probability that the pole number plate is determined to not exist in the spatial fusion result is wherein all belong to .

[0059] Define the observation probability matrix It can be updated annually based on changes in operation and maintenance data to ensure that it always keeps pace with actual business needs.

[0060] S13, including S131, defines the hidden state set. Observation state set and Hidden Markov Models The hidden state set is related to the actual physical state of the pole number plate, and the observed state set is related to the spatial fusion result. , Let be the initial state probability of the pillar. Let be the state transition probability matrix for the pole number plate. S132, construct the historical observation sequence of the pillar. ,in for Observational data at time points, and based on a hidden Markov model. For the corresponding hidden state set Inferences are made, and warnings are issued for abnormal missing pole numbers based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state. The historical observation sequence and observation state set are used for this purpose. Related.

[0061] Constructing a historical observation sequence for the pillar And based on the Hidden Markov Model For the corresponding hidden state set To make inferences, including: When the pole number plate is determined to exist in the spatial fusion results, the pole number is matched based on the string number recognition result and mileage information.

[0062] When the spatial fusion result determines that the pole number plate exists, the string number recognition result output in step S122 is used as the pole number identifier.

[0063] The precise route location is obtained by combining the GPS or odometer data corresponding to the image; the pre-established route database is queried to find the pole record that is closest to the current mileage and has the same pole number; and the unique pole number of the record is used as the identifier for this observation.

[0064] When the pole number plate is determined to be missing in the spatial fusion result, the string number recognition results of the previous pole number plate and the next pole number plate are extracted, and the string number recognition result corresponding to the pole number plate is obtained based on linear interpolation. Then, the pole number is matched according to the string number recognition result and the mileage information.

[0065] For example, the last pole number plate is T120, and the next pole number plate is T124 (which can be obtained from the past historical data), then the current is T122.

[0066]

Construct the historical observation sequence of the pole according to the string number recognition results of the pole matched several times 。

[0067]

When the number of observation results in the historical observation sequence is greater than the preset number, determine the hidden state probability up to the last time according to the historical observation sequence

[0068]

According to the current time hidden probability prediction value and observation probability matrix B, the observation state probability of the current time is calculated

[0069] That is, the result of whether the pole number plate is detected at the time of not observing t, the prediction probability of the pole number plate not being detected at time t is directly predicted through historical information; can be written as . is the prediction probability that the actual physical state is the existence of the pole number plate and the observation is good at time t, is the prior probability that the actual physical state of the pole number plate is existence, and the space fusion result is determined as non-existence (i.e. model not detected); is the prediction probability that the actual physical state is the existence of the pole number plate but the observation is limited at time t, is the prior probability that the actual physical state of the pole number plate is existence but the observation is limited, and the space fusion result is determined as non-existence (i.e. model not detected); P is the prediction probability that the actual physical state is the missing of the pole number plate at time t, is the prior probability that the actual physical state of the pole number plate is missing, and the space fusion result is determined as non-existence (i.e. model not detected).

[0070] When the number of observation results in the historical observation sequence is less than or equal to the preset number, the number of observation results is continued to be accumulated, and the preset number can be 5 or more.

[0071]

Calculate the unexpectedness and the cumulative likelihood of the dominant state to generate a warning

[0072] Among them, is the unexpectedness, Hidden Markov Model The probability that the pole number plate is missing; 1 represents the certainty that the pole number plate was not detected in actual observation (100% probability after the fact). Hidden Markov Model The probability of observing a pole number plate as "not detected" at the current moment, based on historical predictions, can be simplified as follows: .

[0073] This formula measures the degree of surprise in the deviation between actual observations and historical predictions. The lower the predicted probability but the more unexpected the event, the less consistent the failure to detect is with the historical performance pattern of that pillar.

[0074] Determine the dominant state, including:

[0075] in, The dominant state at the current moment. The observed state at time t and before. for The hidden state of the time system; For the first The hidden state set of each pole number plate.

[0076] In particular, The calculation method is as follows: combining the hidden state prediction probability mentioned above. Given the observation probability matrix B, we can obtain the following using Bayes' theorem. ,in: It is the system's hidden state set. ; To predict the true physical state at time t based on the observed states at time t-1 and earlier, is... The probability, In the state The following observations The probability of (detected or not detected).

[0077] Determining the cumulative likelihood under visually restricted conditions includes:

[0078] in, To trace back from time t, the number of times the same position was continuously under visual observation limitation and the observation result was undetected in the dominant state; To accumulate the likelihood.

[0079] If the unexpectedness exceeds the preset unexpectedness threshold, the number of observation results exceeds the preset number, and the spatial fusion results at the current moment indicate that the pole number plate does not exist, a preliminary warning signal for the missing pole number plate will be generated. When the unexpectedness is less than or equal to the preset unexpectedness threshold, the dominant state is the missing state, the number of observation results is greater than the preset number, and the pole number plate does not exist in the spatial fusion result at the current moment, a preliminary early warning signal of long-term continuous missing of the pole number plate is generated. When the unexpectedness is less than or equal to the preset unexpectedness threshold, the dominant state For the visual observation limited state, is less than the preset level threshold, the number of observation results is greater than the preset number, and the pole number plate does not exist in the spatial fusion result at the current moment, a preliminary early warning signal of missing of the pole number plate in a complex fluctuation environment is generated. When the number of observation results in the historical observation sequence is less than or equal to the preset number, the number of observation results is continuously accumulated. When the pole number plate exists in the spatial fusion result at the current moment, no signal is generated.

[0080] In addition, the maximum unexpectedness derivation: when the observation is "not detected", the unexpectedness generated for the historically 100% stable support is the maximum.

[0081] At this time, the prediction probability reaches the theoretical lower limit, which is equal to the basic missed detection rate B (not detected | exists). According to the probability axiom, it can be obtained: , wherein is the maximum unexpectedness.

[0082] Sensitivity coefficient setting: introduce the sensitivity coefficient k (value 0.9-0.98, recommended 0.95) The preset unexpectedness threshold is:

[0083] , wherein is the preset unexpectedness threshold, B (not detected | exists) is the probability that the spatial fusion result is determined as the pole number plate not existing (i.e. the model is not detected) when the actual physical state of the pole number plate is existing, and the threshold is automatically adjusted with the detection model performance (observation probability matrix B). When the model accuracy improves, the threshold is correspondingly increased, so as to maintain the robustness of prediction.

[0084] In order to convert the abstract algorithm of the application into a reproducible calculation process, a complete numerical example is provided in this section. In the cold start process (T<5), any event is not alarmed, and then (T≥5) the normal alarm process is started. Taking the unexpectedness threshold of 90% and the significance level threshold of 5% as an example. Specific embodiments: 【1】 A support is stably detected in historical inspection, and "not detected" is observed for the first time in a certain inspection. This case analyzes the influence of different historical lengths on early warning triggering.

[0086] Table 1

[0087] This case demonstrates the high timeliness of the real-time alert framework. For a historically stable pillar, no matter the length of history is 5 times, 7 times or 10 times, as long as the first "not detected" appears, the system can immediately trigger an alert at that moment, achieving immediate discovery of the anomaly. 【2】 Scenario description: A certain pillar is continuously unable to be observed from the first inspection due to being in a special section such as a station yard soft cross-over or being continuously severely obstructed by an obstacle along the line.

[0089] Table 2

[0090] This case demonstrates the intelligent recognition ability of the framework for continuous obstruction scenarios. Through historical learning, it identifies the continuous "not detected" as the regular performance of the pillar, effectively filtering false positives. 【3】 Scenario description: A certain pillar has unstable historical observations due to partial obstruction, adverse weather or insufficient detection model generalization ability. At the 12th inspection, "not detected" is observed again. This case analyzes the impact of different degrees of unstable history on the result.

[0092] Table 3

[0093] This case clearly demonstrates adaptability. Whether an alert is triggered or not depends on the deviation of the current "not detected" from its historical performance. The system makes intelligent judgments through three core factors: Historical fluctuation frequency (distinguishing between frequent obstruction and sudden drop): Large fluctuation (suppress false positives): As in scenario 3a, which corresponds to a physically frequent obstruction scenario (high obstruction frequency but not continuous complete obstruction). The system learns from historical observations that "restricted" is the dominant state, reducing the unexpectedness of the current "not detected", considering that not seeing is within expectations, thereby avoiding false positives.

[0094] Small fluctuation (capture anomalies): As in scenario 3c, which corresponds to a pillar with a long-term good state. The recent long-term stability establishes a strong belief in the existing state, and the sudden "not detected" leads to high unexpectedness, and the system determines it as a sudden drop.

[0095] Time interval influence (distinguish between environmental deterioration and sudden drop): Utilize the "short-term memory" feature of the model. Compared with 3c and 3d, the abnormality of scene 3d is concentrated in the near future, which corresponds to short-term environmental deterioration such as bad weather or seasonal light, and the model automatically adjusts the expectation of the current state, reduces the unexpectedness of the current un-detected, and prevents false alarms caused by environmental fluctuations; 3c is continuously detected in the near future, and any newly appearing abnormality will produce high unexpectedness and be judged as a sudden drop.

[0096] Sequence likelihood test (detecting dropouts under interference): For the complex scene of "historical fluctuation, continuous un-detection" (such as 3e), although the first level unexpectedness decision does not trigger, the system performs a deep test: calculates the probability of finding "un-detection" for 4 times in a row under the premise that the dominant state continues to be "restricted" is only 2.56% (lower than the 5% threshold). This means that environmental interference (such as bad weather, tree branches shaking, light changes, etc.) is usually intermittent or random, and it is difficult to achieve continuous complete shielding. The system rejects the "restricted" hypothesis and determines that it is a "real drop under complex fluctuating environment".

[0097] The present application can ensure that the system can distinguish between "instantaneous un-detection" and "real absence", and for a support pillar that has just appeared abnormal in the near future, the system will give a certain tolerance period; and for a long-term stable support pillar, any sudden abnormality will be highly vigilant.

[0098] In summary, the present application introduces spatial fusion and temporal fusion into the detection of pole number plate absence, constructs a dual verification and correction framework, and makes up for the limitations of single image recognition method. The present application uses monocular depth estimation algorithm to extend two-dimensional image analysis to three-dimensional spatial relationship understanding, and improves the attribution accuracy of multiple pole number plates in complex scenes. The present application discards the traditional judgment logic based on fixed rules and proposes a real-time inference framework based on historical prediction. The core of the framework is that the real physical state of the pole number plate is modeled as the "hidden state" of the HMM, and the single inspection result is modeled as the "observation state". At each inspection time, the state confidence is calculated based on the historical observation using the forward algorithm, and the probability of "un-detection" at the current time is predicted. The deviation between the predicted probability and the actual observation (i.e. "unexpectedness" and "cumulative likelihood of dominant state") is compared to determine whether an abnormality occurs in real time, rather than waiting for multiple observations before making a decision. The fixed unexpectedness and significance level thresholds are introduced, which are self-adaptive, so that the framework can intelligently distinguish between "real absence" and "continuous shielding" and other different scenarios, while realizing real-time warning and high accuracy. The present application realizes real-time warning while ensuring accuracy through the real-time prediction and inference framework, and solves the core technical problem of the contradiction between timeliness and accuracy in the inspection business.

[0099] Based on the same inventive concept, this invention provides a pole number missing detection device based on spatiotemporal fusion perception, comprising: The image acquisition module, used by S11, acquires multi-view images or time-series multi-frame images of pole number plates, and obtains the position and string of the pole number plate in each image based on a deep learning model. The multi-view images or time-series multi-frame images each include several images. The recognition module, used in S12, includes S121, locating the pillars in each image based on an object detection or segmentation algorithm, and determining whether the corresponding image should be retained based on the depth value of the pole number and the depth value of the pillar; S122, obtaining the spatial fusion result of the pole number based on the retained image, wherein the spatial fusion result includes the pole number recognition result and the string number recognition result. The early warning module, used in S13, includes S131, which defines the hidden state set. Observation state set and Hidden Markov Models The hidden state set is related to the actual physical state of the pole number plate, and the observed state set is related to the spatial fusion result. , Let be the initial state probability of the pillar. Let be the state transition probability matrix for the pole number plate. S132, construct the historical observation sequence of the pillar. ,in for Observational data at time points, and based on a hidden Markov model. For the corresponding hidden state set Inferences are made, and warnings are issued for abnormal missing pole numbers based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state. The historical observation sequence and observation state set are used for this purpose. Related.

[0100] Based on the same inventive concept, the present invention also provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to execute a pole number missing detection method based on spatiotemporal fusion perception, as described above.

[0101] Based on the same inventive concept, the present invention also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the pole number missing detection method based on spatiotemporal fusion perception as described above.

[0102] Since the electronic device introduced in the embodiment is the electronic device used for implementing the method for processing information in the embodiment of the present application, based on the method for processing information introduced in the embodiment of the present application, those skilled in the art can understand the specific implementation of the electronic device in the embodiment and various changes thereof, so how the electronic device implements the method in the embodiment of the present application is not introduced in detail here. As long as the electronic device used for implementing the method for processing information in the embodiment of the present application is implemented by those skilled in the art, it belongs to the scope of the present application.

[0103] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0104] The present application is described in relation to flowcharts and / or block diagrams that illustrate the methodology, apparatus (system) and computer program product according to embodiments of the present application. It is understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the block or blocks.

[0105] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions means which implement the function specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the block or blocks.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1steps of the functions specified in the block or blocks.

[0107] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the preferred embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to encompass within their scope all such variations and modifications as are included within the scope of the application.

[0108] It is clear that the application can be subject to many modifications and variations without departing from the scope of the application as defined by the appended claims. The application is therefore intended to cover all technical and structural equivalents.

Claims

1. A pole number plate missing detection method based on spatiotemporal fusion perception, characterized in that, The method comprises the following steps: S11, obtaining a multi-view image or a time-series multi-frame image of a pole number plate, and obtaining the position and string of the pole number plate in each image based on a deep learning model, wherein the multi-view image or the time-series multi-frame image comprises a plurality of images; S12, comprising S121, positioning a support of each image based on a target detection or segmentation algorithm, and determining whether to retain the corresponding image according to the depth value of the pole number plate and the depth value of the support; S122, obtaining a spatial fusion result of the pole number plate based on the retained images, wherein the spatial fusion result comprises a pole number plate existence recognition result and a string number recognition result; S13, including S131, defines the hidden state set. Observation state set and Hidden Markov Models The hidden state set is related to the actual physical state of the pole number plate, and the observed state set is related to the spatial fusion result. , Let be the initial state probability of the pillar. Let be the state transition probability matrix for the pole number plate. S132, construct the historical observation sequence of the pillar. ,in for Observational data at time points, and based on a hidden Markov model. For the corresponding hidden state set Inferences are made, and warnings are issued for abnormal missing pole numbers based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state. The historical observation sequence and observation state set are used for this purpose. Related.

2. The method for detecting missing pole number plate based on spatio-temporal fusion perception according to claim 1, wherein, determining whether to retain the corresponding image according to the depth value of the pole number plate and the depth value of the support, comprising: determining the absolute value difference between the depth value of the pole number plate and the depth value of the corresponding support; if the absolute value difference is less than or equal to an adaptive threshold value, the corresponding image is retained, otherwise the corresponding image is discarded, wherein the adaptive threshold value is related to the depth value of the support.

3. The method of claim 2, wherein the method comprises: when the retained image belongs to a multi-view image, obtaining the spatial fusion result of the pole number plate based on the retained images, comprising: if the pole number plate exists in at least one image, the pole number plate existence recognition result is determined to exist, otherwise it is determined to not exist; when the pole number plate existence recognition result is determined to exist, the corresponding string with the highest occurrence frequency in the image is taken as the string number recognition result; when the pole number plate existence recognition result is determined to not exist, the string number recognition result is determined to have no string.

4. The method of claim 2, wherein the method comprises: when the retained image belongs to a time-series multi-frame image, obtaining the spatial fusion result of the pole number plate based on the retained images, comprising: based on a multi-target tracking algorithm, the pole number plate in the images with consecutive frames is associated as a motion trajectory, if the number of frames of the motion trajectory is greater than a preset number, the pole number plate existence recognition result is determined to exist, otherwise it is determined to not exist; when the pole number plate existence recognition result is determined to exist, the corresponding string with the highest occurrence frequency in the image is taken as the string number recognition result; when the pole number plate existence recognition result is determined to not exist, the string number recognition result is determined to have no string.

5. The method for detecting missing pole number plate based on spatio-temporal fusion perception according to claim 1, wherein, defining a set of hidden states , a set of observation states , and a hidden Markov model , comprising: Define the set of hidden states Comprise wherein is the actual physical state of the pole sign is present and visually observed ideal, is the actual physical state of the pole sign is not present, is the actual physical state of the pole sign is present but visually observed limited; set of observation states comprising wherein determining that the number plate is present in the spatial fusion result, determining that the number plate is not present in the spatial fusion result; defining the initial state probabilities of the pole number plate where the initial state of the pole number plate is present with probability the initial state of the pole number plate is missing with probability the initial state of the pole number plate is visually observed with probability where all belong to ; Define the state transition probability matrix of the pole number plate. , among which, from Time's up At what moment is the probability that the pole number plate exists and the ideal visual observation state remains unchanged? ;from Time's up At what moment does the pole number plate change from a state where it exists and is ideally visually observed to a state where it exists but is visually limited? ;from Time's up At what moment does the pole number plate change from a state where it exists and is visually ideal to a state where it is missing? ;from Time's up At what moment does the pole number plate revert from a state of limited visual observation to a state of existence where visual observation is ideal? ;from Time's up The probability that the pole number plate exists but visual observation is limited at any given moment remains unchanged. ;from Time's up The probability that, at any given moment, the pole number plate changes from a state of existence but limited visual observation to a state of absence. ;from Time's up At what moment is the probability that the pole number plate recovers from a missing state to a state where it is present and visually ideal? ;from Time's up At what moment does the pole number plate change from a missing state to a state where it exists but is visually limited? ;from Time's up At any given moment, the probability that the pole number plate will not be restored from its missing state. ,in All belong to ; Defining the observation probability matrix wherein the probability of determining the presence of the number plate in the spatial fusion result is when the actual physical state of the number plate is present; the probability of determining the absence of the number plate in the spatial fusion result is when the actual physical state of the number plate is present; the probability of determining the presence of the number plate in the spatial fusion result is when the actual physical state of the number plate is present but the visual observation is limited; the probability of determining the absence of the number plate in the spatial fusion result is when the actual physical state of the number plate is present but the visual observation is limited; the probability of determining the presence of the number plate in the spatial fusion result is when the actual physical state of the number plate is absent; the probability of determining the absence of the number plate in the spatial fusion result is when the actual physical state of the number plate is absent; wherein all belong to .

6. The method for pole number plate missing detection based on spatio-temporal fusion perception according to claim 1, wherein, Constructing a history sequence of observations for a strut and based on a hidden Markov model to a corresponding set of hidden states making an inference, including: when it is determined in the spatial fusion result that the pole number plate exists, the string number recognition result and the mileage information are used for support number matching; when it is determined in the spatial fusion result that the pole number plate does not exist, the string number recognition results of the previous and next pole number plates are extracted, and based on linear interpolation, the corresponding string number recognition result of the pole number plate is obtained, and the string number recognition result and the mileage information are used for support number matching; The string numeral recognition result of the several matched completed support posts is constructed in sequence to form a historical observation sequence of the support post ; when the number of observation results in the historical observation sequence is greater than a preset number, the hidden state posterior probability up to the previous time is determined according to the historical observation sequence; determining a prior state distribution for a current time instance based on the hidden state posterior probability and the state transition probability matrix ​ determining a probability of a non-existence of a pole number plate in the hidden Markov model based on the prior state distribution and the observation probability matrix a non-existence of an actual physical state and a decision of a non-existence of the pole number plate​ when the number of observation results in the historical observation sequence is less than or equal to the preset number, the number of observation results is continuously accumulated.

7. The method as claimed in claim 5, wherein the method is based on spatio-temporal fusion perception based pole number plate missing detection. based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of a dominant state, an abnormal absence of the pole number plate is warned, comprising: the preset unexpectedness mechanism comprises: wherein, is the unexpectedness, is a hidden Markov model the probability that a middle pole number plate is not present; determining the dominant state, comprising: wherein, is the dominant state at the current time, is the observed state up to time t, is the hidden state of the system at time t; is the hidden state set of the th pole sign. determining the cumulative likelihood in the visual observation limited state, comprising: wherein, is the number of times the same location has been observed with the dominant state persisting as restricted and the observation result consecutively as not detected, looking back from time t forward; is the cumulative likelihood. When the unexpectedness is greater than the preset unexpectedness threshold, the number of observation results is greater than the preset number, and the pole number plate does not exist in the spatial fusion result at the current moment, a preliminary warning signal of pole number plate missing is generated; When the unexpectedness is less than or equal to the preset unexpectedness threshold, the dominant state is the missing state, the number of observation results is greater than the preset number, and the pole number plate does not exist in the spatial fusion result at the current moment, a preliminary warning signal of long-term and continuous missing of the pole number plate is generated; when the unexpectedness is less than or equal to a preset unexpectedness threshold, the dominant state for a visual observation limited state, less than a preset level threshold, the number of observation results is greater than a preset number, and the judgment pole sign does not exist in the spatial fusion result at the current moment, a preliminary early warning signal of missing pole sign in a complex fluctuation environment is generated. When the number of observation results in the historical observation sequence is less than or equal to the preset number, the number of observation results is continuously accumulated; When the pole number plate exists in the spatial fusion result at the current moment, no signal is generated.

8. A pole number plate missing detection device based on spatiotemporal fusion perception, characterized in that, The image acquisition module is configured to acquire, at S11, multi-view images or time-series multi-frame images of the pole number plate, and obtain the position and string of the pole number plate in each image based on a deep learning model. The multi-view images or time-series multi-frame images each include a plurality of images. The recognition module is configured to, at S12, include S121, locate the support of each image based on a target detection or segmentation algorithm, and determine whether the corresponding image is retained according to the depth value of the pole number plate and the depth value of the support. S122, based on the retained images, obtain the spatial fusion result of the pole number plate, wherein the spatial fusion result includes a pole number plate existence recognition result and a string number recognition result. The processor; The early warning module, used in S13, includes S131, which defines the hidden state set. Observation state set and Hidden Markov Models The hidden state set is related to the actual physical state of the pole number plate, and the observed state set is related to the spatial fusion result. , Let be the initial state probability of the pillar. Let be the state transition probability matrix for the pole number plate. S132, construct the historical observation sequence of the pillar. ,in for Observational data at time points, and based on a hidden Markov model. For the corresponding hidden state set Inferences are made, and warnings are issued for abnormal missing pole numbers based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state. The historical observation sequence and observation state set are used for this purpose. Related.

9. An electronic device, comprising: The memory for storing the instructions executable by the processor; The processor is configured to execute to realize the pole number plate missing detection method based on spatio-temporal fusion perception as claimed in any one of claims 1 to 7. When the instructions in the non-transitory computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the pole number plate missing detection method based on spatio-temporal fusion perception as claimed in any one of claims 1 to 7. ​ 10. A non-transitory computer-readable storage medium, comprising: ​

Images