Pole number plate missing detection method and device based on space-time fusion perception, equipment and medium

Abstract

The application discloses a pole number plate missing detection method and device based on space-time fusion perception, equipment and medium, including: acquiring multi-view images or time sequence multi-frame images of the pole number plate, and obtaining the position and string of the pole number plate in each image based on a deep learning model; judging whether the corresponding image is retained according to the depth value of the pole number plate and the depth value of the support column; obtaining the space fusion result of the pole number plate based on the retained image; defining a hidden state set, an observation state set and a hidden Markov model; constructing a historical observation sequence of the support column, and speculating the corresponding hidden state set based on the hidden Markov model, and warning the abnormal missing of the pole number plate based on a preset accident degree mechanism and a cumulative likelihood degree mechanism of a dominant state, the historical observation sequence being related to the observation state set. The application belongs to the field of pole number plate detection. The application can detect the missing pole number plate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of pole number plate detection, and in particular to a pole number plate missing detection method and device based on spatiotemporal fusion perception, equipment and medium. BACKGROUND

[0002] In the operation and maintenance of the overhead contact system of a railway, the pole number plate is a key identity of the support, and its integrity is directly related to the equipment management and safety inspection. According to the "High-speed Railway Overhead Contact System Operation and Maintenance Rules", the missing of the pole number plate belongs to the incomplete identification defect that must be found in time. The current mainstream inspection relies on 4C / 2C and other vehicle-mounted vision systems to automatically take pictures during high-speed train operation and identify the pole number content through OCR. The existing technology mainly focuses on improving the accuracy of OCR in a single inspection, such as through multi-picture screening or logical correction based on the sequence rules of the pole number.

[0003] However, the above method has two major defects: first, it is based only on two-dimensional images and cannot determine the true three-dimensional space belonging of the pole number plate and the current support, which easily misassociates the adjacent pole number in complex scenes such as multiple tracks, leading to missed detection of the true missing or pollution of historical data; second, it lacks time dimension perception and only determines the missing based on a single undetected result, which makes it difficult to distinguish between real loss and instantaneous undetected caused by obstruction, weather, etc., resulting in high false positives or false negatives. Therefore, it is urgent to propose a pole number plate missing detection method based on spatiotemporal fusion perception. SUMMARY

[0004] The present application provides a pole number plate missing detection method, device, equipment and medium based on spatiotemporal fusion perception, which solves the technical problem of high false positive rate of the true state of the pole number plate in the prior art and achieves the technical effect of improving the accuracy of pole number plate detection.

[0005] In a first aspect, the present application provides a pole number plate missing detection method based on spatiotemporal fusion perception, comprising:

[0006] S11, obtaining multi-view images or time-series multi-frame images of the pole number plate, and obtaining the position and string of the pole number plate in each image based on a deep learning model, the multi-view images or time-series multi-frame images each including a plurality of images;

[0007] S12, including S121, positioning the support of each image based on a target detection or segmentation algorithm, and determining whether the corresponding image is retained according to the depth value of the pole number plate and the depth value of the support; S122, obtaining a spatial fusion result of the pole number plate based on the retained images, wherein the spatial fusion result includes a pole number plate existence recognition result and a string number recognition result;

[0008] S13, including S131, defining a hidden state set , an observation state set and Hidden Markov Models The hidden state set is related to the actual physical state of the pole number plate, and the observed state set is related to the spatial fusion result. , Let be the initial state probability of the pillar. Let be the state transition probability matrix for the pole number plate. S132, construct the historical observation sequence of the pillar. ,in for Observational data at time points, and based on a hidden Markov model. For the corresponding hidden state set Inferences are made, and warnings are issued for abnormal missing pole numbers based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state. The historical observation sequence and observation state set are used for this purpose. Related.

[0009] Furthermore, the determination of whether to retain the corresponding image is based on the depth value of the pole number plate and the depth value of the post, including:

[0010] Determine the absolute difference between the depth value of the pole number plate and the depth value of the corresponding support post;

[0011] If the absolute difference is less than or equal to the adaptive threshold, the corresponding image is retained; otherwise, the corresponding image is discarded. The adaptive threshold is related to the depth value of the support.

[0012] Furthermore, when the retained image is a multi-view image, the spatial fusion result of the pole number plate is obtained based on the retained image, including:

[0013] If at least one image contains a pole number plate, the pole number plate is identified as present; otherwise, it is identified as not present.

[0014] When the pole number plate is identified as existing, the string that appears most frequently in the image is used as the string number recognition result; when the pole number plate is identified as not existing, the string number recognition result is determined to be non-existent.

[0015] Furthermore, when the retained images belong to a temporal multi-frame image sequence, the spatial fusion result of the pole number plate is obtained based on the retained images, including:

[0016] Based on a multi-target tracking algorithm, pole numbers in images with consecutive frames are associated with motion tracks. If the number of consecutive stable frames of the motion track is greater than a preset number, the pole number is identified as existing; otherwise, it is identified as not existing.

[0017] When the pole number plate existence recognition result is determined as existing, the string corresponding to the most frequent occurrence in the image is taken as the string number recognition result; when the pole number plate existence recognition result is determined as non-existing, the string number recognition result is determined as no string.

[0018] Further, define a hidden state set , an observation state set and a hidden Markov model , comprising:

[0019] define a hidden state set comprising , wherein is the actual physical state of the pole number plate is existing and the visual observation is ideal, is the actual physical state of the pole number plate is non-existing, is the actual physical state of the pole number plate is existing but the visual observation is limited;

[0020] define an observation state set comprising , wherein is determined that the pole number plate exists in the spatial fusion result, is determined that the pole number plate does not exist in the spatial fusion result;

[0021] define the initial state probability of the pole number plate , wherein the initial state of the pole number plate is existing with a probability , the initial state of the pole number plate is missing with a probability , and the initial state of the pole number plate is visual observation limited with a probability , wherein all belong to ;

[0022] define the state transition probability matrix of the pole number plate , wherein, from time to time, the state of the pole number plate existing and the visual observation being ideal does not change with a probability ; from time to time, the state of the pole number plate changes from existing and the visual observation being ideal to existing but the visual observation being limited with a probability ; from time to time, the state of the pole number plate changes from existing and the visual observation being ideal to missing with a probability ; from time to time, the state of the pole number plate changes from visual observation limited to existing and the visual observation being ideal with a probability ; from the time instant the probability that the state of the presence of the pole number plate but with limited visual observation does not change from the time instant the time instant the probability that the state of the presence of the pole number plate but with limited visual observation changes to the state of absence from the time instant the time instant the probability that the state of the absence of the pole number plate recovers to the state of presence and ideal visual observation from the time instant the time instant the probability that the state of the absence of the pole number plate recovers to the state of presence but with limited visual observation from the time instant the time instant the probability that the state of the absence of the pole number plate does not recover wherein all belong to ;

[0023] define the observation probability matrix wherein the probability that the pole number plate is determined to be present in the spatial fusion result is when the actual physical state of the pole number plate is present; the probability that the pole number plate is determined to be absent in the spatial fusion result is when the actual physical state of the pole number plate is present but with limited visual observation; the probability that the pole number plate is determined to be present in the spatial fusion result is when the actual physical state of the pole number plate is present but with limited visual observation; the probability that the pole number plate is determined to be absent in the spatial fusion result is when the actual physical state of the pole number plate is absent; the probability that the pole number plate is determined to be present in the spatial fusion result is when the actual physical state of the pole number plate is absent; the probability that the pole number plate is determined to be absent in the spatial fusion result is wherein all belong to .

[0024] further, construct the historical observation sequence of the support and based on the hidden Markov model infer the corresponding hidden state set , including:

[0025] when the pole number plate is determined to be present in the spatial fusion result, match the pole number according to the string number recognition result and the mileage information;

[0026] When the pole number plate is determined to be missing in the spatial fusion result, the string number recognition results of the previous pole number plate and the next pole number plate are extracted, and the string number recognition result corresponding to the pole number plate is obtained based on linear interpolation. The pole number is matched according to the string number recognition result and mileage information.

[0027] The string digit recognition results that match several pillars are used to construct a historical observation sequence of the pillars in sequence. ;

[0028] When the number of observations in the historical observation sequence is greater than the preset number, the posterior probability of the hidden state up to the previous moment is determined based on the historical observation sequence.

[0029] Based on the hidden state posterior probability and the state transition probability matrix Determine the prior state distribution at the current moment;

[0030] Based on the prior state distribution and the observation probability matrix Determine the Hidden Markov Model The probability that the pole number plate does not exist includes both the actual physical absence of the pole number plate and the determination that the pole number plate does not exist;

[0031] If the number of observations in the historical observation sequence is less than or equal to the preset number, the number of observations will continue to be accumulated.

[0032] Furthermore, based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state, an early warning is issued for abnormal missing pole numbers, including:

[0033] The default surprise mechanism includes:

[0034]

[0035] in, For the sake of surprise, Hidden Markov Model The probability that the center pole number plate does not exist;

[0036] Determine the dominant state, including:

[0037]

[0038] in, The dominant state at the current moment. The observed state at time t and before. for The hidden state of the time system; For the first The hidden state set of each pole number plate;

[0039] determining the cumulative likelihood in the visual observation limited state, comprising:

[0040]

[0041] wherein, is the number of times that the same position is continuously observed as the visual observation limited state and the observation result is continuously not detected from the time t back; is the cumulative likelihood;

[0042] when the unexpectedness is greater than the preset unexpectedness threshold, the number of observation results is greater than the preset number, and the pole number plate does not exist in the spatial fusion result of the current time, a preliminary warning signal of the missing pole number plate is generated;

[0043] when the unexpectedness is less than or equal to the preset unexpectedness threshold, the dominant state is the missing state, the number of observation results is greater than the preset number, and the pole number plate does not exist in the spatial fusion result of the current time, a preliminary warning signal of the long-term missing pole number plate is generated;

[0044] when the unexpectedness is less than or equal to the preset unexpectedness threshold, the dominant state is the visual observation limited state, is less than the preset level threshold, the number of observation results is greater than the preset number, and the pole number plate does not exist in the spatial fusion result of the current time, a preliminary warning signal of the missing pole number plate in the complex fluctuation environment is generated;

[0045] when the number of observation results in the historical observation sequence is less than or equal to the preset number, the number of observation results is continuously accumulated;

[0046] when the pole number plate exists in the spatial fusion result of the current time, no signal is generated.

[0047] In a second aspect, the present application provides a pole number plate missing detection device based on spatio-temporal fusion perception, comprising:

[0048] an image acquisition module, configured to acquire multi-view images or time sequence multi-frame images of the pole number plate, and obtain the position and string of the pole number plate in each image based on a deep learning model, wherein the multi-view images or time sequence multi-frame images each include a plurality of images;

[0049] an identification module, configured to position the support of each image based on a target detection or segmentation algorithm, and determine whether the corresponding image is retained according to the depth value of the pole number plate and the depth value of the support; and obtain the spatial fusion result of the pole number plate based on the retained images, wherein the spatial fusion result includes a pole number plate existence recognition result and a string number recognition result;

[0050] The early warning module, used in S13, includes S131, which defines the hidden state set. Observation state set and Hidden Markov Models The hidden state set is related to the actual physical state of the pole number plate, and the observed state set is related to the spatial fusion result. , Let be the initial state probability of the pillar. Let be the state transition probability matrix for the pole number plate. S132, construct the historical observation sequence of the pillar. ,in for Observational data at time points, and based on a hidden Markov model. For the corresponding hidden state set Inferences are made, and warnings are issued for abnormal missing pole numbers based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state. The historical observation sequence and observation state set are used for this purpose. Related.

[0051] Thirdly, the present invention provides an electronic device, comprising:

[0052] processor;

[0053] Memory used to store the processor's executable instructions;

[0054] The processor is configured to execute a pole number missing detection method based on spatiotemporal fusion perception as provided in the first aspect.

[0055] Fourthly, the present invention provides a non-transitory computer-readable storage medium, wherein when the instructions in the non-transitory computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to execute the pole number missing detection method based on spatiotemporal fusion perception as provided in the first aspect.

[0056] One or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0057] The application introduces two perception dimensions of space fusion and time fusion into the pole number plate missing detection, constructs a double verification and correction framework, and makes up for the limitations of single image recognition method. The application uses monocular depth estimation algorithm, expands two-dimensional image analysis to three-dimensional space relationship understanding, and improves the attribution accuracy of multiple pole number plates in complex scenes. The application discards the traditional judgment logic based on fixed rules, and proposes a real-time inference framework based on historical prediction. The core of the framework is that the real physical state of the pole number plate is modeled as the "hidden state" of the Hidden Markov Model (HMM), and the single inspection result is modeled as the "observation state". At each inspection time, the state confidence is calculated based on the historical observation by using the forward algorithm, and the probability of "not detected" at the current time is predicted. The deviation between the predicted probability and the actual observation (i.e. "unexpectedness" and "cumulative likelihood of dominant state") is compared to determine whether an abnormality occurs in real time, rather than waiting for multiple observations before making a judgment. A fixed unexpectedness threshold that can be adaptively linked is introduced, so that the framework can intelligently distinguish between "real missing" and "continuous occlusion" and other different scenarios, while realizing the unification of real-time warning and high accuracy. The application realizes real-time warning under the premise of ensuring accuracy through the real-time prediction and inference framework, and solves the core technical problem of the contradiction between timeliness and accuracy in the inspection business. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0059] Figure 1 A flowchart of the pole number plate missing detection method based on spatio-temporal fusion perception provided by the application is shown in the figure.

[0060] Figure 2 A schematic diagram of the pole number plate provided by the application is shown in the figure. DETAILED DESCRIPTION

[0061] The embodiment of the application provides a pole number plate missing detection method based on spatio-temporal fusion perception, which solves the technical problem of high false alarm rate of the real state of the pole number plate in the prior art.

[0062] The technical solution of the application is to solve the above technical problems, and the general idea is as follows:

[0063] The pole number plate missing detection method based on space-time fusion perception comprises the following steps: S11, acquiring multi-view images or time sequence multi-frame images of a pole number plate, and obtaining the position and string of the pole number plate in each image based on a deep learning model, wherein the multi-view images or time sequence multi-frame images each comprise a plurality of images; S12, comprising S121, positioning a support of each image based on a target detection or segmentation algorithm, and judging whether the corresponding image is retained according to the depth value of the pole number plate and the depth value of the support; S122, obtaining a space fusion result of the pole number plate based on the retained images, wherein the space fusion result comprises a pole number plate existence recognition result and a string number recognition result; S13, comprising S131, defining a hidden state set , an observation state set and a hidden Markov model , wherein the hidden state set is related to the actual physical state of the pole number plate, the observation state set is related to the space fusion result, , is an initial state probability of the support, is a state transition probability matrix of the pole number plate, is an observation probability matrix; S132, constructing a historical observation sequence of the support and inferring the corresponding hidden state set based on the hidden Markov model , and warning the abnormal missing of the pole number plate based on a preset abnormality mechanism and a cumulative likelihood mechanism of a dominant state, wherein the historical observation sequence is related to the observation state set .

[0064] In order to better understand the above technical solutions, the above technical solutions will be described in detail in combination with the drawings of the specification and specific embodiments.

[0065] Firstly, the term "and / or" appearing in the present text is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present text generally represents an "or" relationship between the front and rear associated objects.

[0066] Automatic inspection is performed, and the inspection content comprises confirming whether various marks are complete, wherein the inspection of the pole number plate on the support is one of the important contents of the inspection.

[0067] The present application provides a pole number plate missing detection method based on space-time fusion perception as shown in Figure 1 , comprising steps S11-S13:

[0068] S11. Obtain multi-view images or time-series multi-frame images of pole number plates, and obtain the position and string of the pole number plate in each image based on a deep learning model. Each multi-view image or time-series multi-frame image includes several images.

[0069] When the detection train reaches the designated valid shooting window of the support, image information can be acquired through one or more of the following methods, depending on the onboard equipment configuration:

[0070] Multi-view image acquisition (for multi-camera systems, such as 4C devices): Trigger multiple cameras at different angles to take pictures simultaneously, acquire multiple single-frame and multi-view images of the pillar, and finally combine them into a multi-view image of the pillar.

[0071] Temporal multi-frame image acquisition (for single-camera systems, such as 2C devices): Acquire temporal multi-frame images of the pillar by continuously shooting with the camera in a short period of time.

[0072] The acquired multi-view images or time-series multi-frame images are input into a pre-trained deep learning model to perform preliminary recognition on each image, thereby obtaining the position of the pole number plate in each image and the string of the pole number plate.

[0073] like Figure 2 As shown, Figure 2 To simplify the diagram of the support post, 0196 is the string of the post number plate (also known as the post number).

[0074] S12, including S121 and S122:

[0075] S121, Based on object detection or segmentation algorithms, locate the pillars in each image, and determine whether to retain the corresponding image based on the depth value of the pole number and the depth value of the pillar.

[0076] The process involves determining whether to retain the corresponding image based on the depth values ​​of the pole number plate and the corresponding post, including: determining the absolute difference between the depth values ​​of the pole number plate and the corresponding post; if the absolute difference is less than or equal to an adaptive threshold, the corresponding image is retained; otherwise, the corresponding image is discarded. The adaptive threshold is related to the depth value of the post.

[0077] The catenary supports can be located in an image using object detection or segmentation algorithms. The adaptive threshold is dynamically determined based on the average depth of the support, including:

[0078]

[0079] in, For adaptive threshold, This is a preset proportional coefficient. The depth value of the support. The value range is {0.05, 0.10}.

[0080] If the absolute value difference is less than or equal to the adaptive threshold, it is determined that the pole sign and the support are spatially associated, and is retained.

[0081] Otherwise, it is determined to be background or a pole sign of another lane, and is rejected.

[0082] In S122, based on the retained image, a spatial fusion result of the pole sign is obtained, wherein the spatial fusion result includes a pole sign existence recognition result and a string number recognition result.

[0083]

When the retained image belongs to multi-view images, based on the retained image, a spatial fusion result of the pole sign is obtained

[0084] If the pole sign exists in at least one image, the pole sign existence recognition result is determined to exist, otherwise it is determined to not exist; when the pole sign existence recognition result is determined to exist, the string corresponding to the most frequent occurrence in the image is taken as the string number recognition result; when the pole sign existence recognition result is determined to not exist, the string number recognition result is determined to be no string.

[0085] When the retained image belongs to multi-view images (multi-view images of the support), all retained multi-view images are summarized.

[0086] Then cross-validation is performed, as long as the pole sign is recognized in at least one view, it is preliminarily determined that the pole sign of the support exists in this inspection.

[0087] For example, three of the four camera views (C01, C02, C03, C04) (C01, C03, C04) have detected the pole sign, and it is determined that the pole sign physically exists.

[0088] The string corresponding to the most frequent occurrence in the image is taken as the string number recognition result.

[0089] The string number recognition results of C01 and C04 views are both T101, and the string number recognition result of C03 view is T7O1, so T101 is the string number recognition result of the support.

[0090] The spatial fusion result includes the pole sign existence recognition result and the string number recognition result.

[0091]

When the retained image belongs to time-series multi-frame images, based on the retained image, a spatial fusion result of the pole sign is obtained

[0092] Based on the multi-target tracking algorithm, the pole number plate in the image with consecutive frames is associated as a motion trajectory. If the number of stable and continuous frames of the motion trajectory is greater than the preset frame number, it is determined that the pole number plate exists in the recognition result. Otherwise, it is determined that the pole number plate does not exist. When the pole number plate existence recognition result is determined to exist, the string corresponding to the most frequent occurrence in the image is taken as the string number recognition result. When the pole number plate existence recognition result is determined to not exist, the string number recognition result is determined to be no string.

[0093] When the reserved image belongs to a time sequence multi-frame image, for the reserved image, the effective detection frame screened in the consecutive image frames can be input into the multi-target tracking algorithm, and the detection frames belonging to the same pole number plate are associated into a motion trajectory. When the stable and continuous frame number of the motion trajectory is greater than or equal to 5 times (the preset frame number can be set by the user, which is not limited here), it is confirmed that the pole number plate exists. The spatial fusion result includes the pole number plate existence recognition result and the string number recognition result.

[0094] S13, including S131 and S132:

[0095] S131, defining a hidden state set , an observation state set and a hidden Markov model , wherein the hidden state set is related to the actual physical state of the pole number plate, the observation state set is related to the spatial fusion result, , is the initial state probability of the support column, is the state transition probability matrix of the pole number plate, is the observation probability matrix.

[0096] defining a hidden state set , an observation state set and a hidden Markov model , including:

[0097] defining a hidden state set including , wherein is the actual physical state of the pole number plate is present and the visual observation is ideal (no occlusion, normal light), is the actual physical state of the pole number plate is not present, is the actual physical state of the pole number plate is present but the visual observation is limited (such as vegetation occlusion, strong backlight, etc.).

[0098] Based on the double-state HMM, a probability inference framework based on a three-layer hidden state hidden Markov model (HMM) is revised. This framework not only distinguishes between "existence" and "absence", but also introduces a "visual limitation" state to represent environmental interference, which can further improve the stability of the evaluation.

[0099] set of observation states comprising wherein for determining the presence of the pole number plate in the spatial fusion result (i.e. detecting the pole number plate), for determining the absence of the pole number plate in the spatial fusion result (i.e. not detecting the pole number plate).

[0100] hidden Markov model contains three sets of parameters, initial state probability reflects the prior state distribution of the pole.

[0101] defining the initial state probability of the pole number plate wherein the probability of the initial state of the pole number plate being present is the probability of the initial state of the pole number plate being missing is the probability of the initial state of the pole number plate being visually observed as limited is wherein all belong to .

[0102] The corresponding meanings are that the newly put into operation pole has a 99.3% probability of being a complete installation with ideal observation conditions and a 0.5% probability of being visually limited and unobservable due to environmental obstruction or shooting angle problems in the initial state of the pole number plate of the pole.

[0103] state transition probability matrix reflects the change rule of the physical state of the pole number plate over time, i.e. the state change probability from time to time.

[0104] defining the state transition probability matrix of the pole number plate wherein, from time to time, the probability of the state of the pole number plate being present and visually observed as ideal not changing is from time to time, the probability of the state of the pole number plate changing from being present and visually observed as ideal to being present but visually observed as limited is from time to time, the probability of the state of the pole number plate changing from being present and visually observed as ideal to being missing is from time to time, the probability of the state of the pole number plate changing from being visually observed as limited to being present and visually observed as ideal is from t t, the probability that the state of the pole number plate does not change from the state of existing but limited visual observation ; from t t, the probability that the state of the pole number plate changes from the state of existing but limited visual observation to the state of missing ; from t t, the probability that the state of the pole number plate recovers from the state of missing to the state of existing and ideal visual observation ; from t t, the probability that the state of the pole number plate recovers from the state of missing to the state of existing but limited visual observation ; from t t, the probability that the state of the pole number plate does not recover from the state of missing wherein all belong to .

[0105] A time adaptation mechanism can also be added: when the actual inspection cycle deviates from the standard inspection cycle, the state transition probability can be dynamically adjusted:

[0106]

[0107] wherein, is the state transition probability of the pole number plate for the standard inspection cycle, is the standard inspection cycle, is the actual inspection cycle, is the probability that the state of the pole number plate changes from existing to missing, corresponding to the value of matrix A.

[0108] Define the observation probability matrix wherein when the actual physical state of the pole number plate is existing, the probability that the pole number plate is determined to exist in the spatial fusion result is ; when the actual physical state of the pole number plate is existing, the probability that the pole number plate is determined to not exist in the spatial fusion result is ; when the actual physical state of the pole number plate is existing but limited visual observation, the probability that the pole number plate is determined to exist in the spatial fusion result is ; when the actual physical state of the pole number plate is existing but limited visual observation, the probability that the pole number plate is determined to not exist in the spatial fusion result is ; when the actual physical state of the pole number plate is not existing, the probability that the pole number plate is determined to exist in the spatial fusion result is ; when the actual physical state of the pole number plate is not existing, the probability that the pole number plate is determined to not exist in the spatial fusion result is wherein All belong to .

[0109] Define the observation probability matrix It can be updated annually based on changes in operation and maintenance data to ensure that it always keeps pace with actual business needs.

[0110] S13, including S131, defines the hidden state set. Observation state set and Hidden Markov Models The hidden state set is related to the actual physical state of the pole number plate, and the observed state set is related to the spatial fusion result. , Let be the initial state probability of the pillar. Let be the state transition probability matrix for the pole number plate. S132, construct the historical observation sequence of the pillar. ,in for Observational data at time points, and based on a hidden Markov model. For the corresponding hidden state set Inferences are made, and warnings are issued for abnormal missing pole numbers based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state. The historical observation sequence and observation state set are used for this purpose. Related.

[0111] Constructing a historical observation sequence for the pillar And based on the Hidden Markov Model For the corresponding hidden state set To make inferences, including:

[0112] When the pole number plate is determined to exist in the spatial fusion results, the pole number is matched based on the string number recognition result and mileage information.

[0113] When the pole number plate is determined to exist in the spatial fusion result, the string number recognition result output in step S122 is used as the pole number identifier.

[0114] The precise route location is obtained by combining the GPS or odometer data corresponding to the image; the pre-established route database is queried to find the pole record that is closest to the current mileage and has the same pole number; and the unique pole number of the record is used as the identifier for this observation.

[0115] When the pole number plate is determined to be missing in the spatial fusion result, the string number recognition results of the previous pole number plate and the next pole number plate are extracted, and the string number recognition result corresponding to the pole number plate is obtained based on linear interpolation. Then, the pole number is matched according to the string number recognition result and the mileage information.

[0116] For example, the last pole number is T120, and the next pole number is T124 (which can be obtained from the past historical data), then the current pole number is T122.

[0117]

The string number recognition result of the support column is constructed in sequence according to the number of matches .

[0118] According to the matched support column number, the observation result archives of the support column number from the first time to the last time are extracted from the historical database to form a historical observation sequence . The observation result of this time is appended to the end of the sequence to form a complete full-cycle observation sequence , wherein represents the observation state at time i, and the value range is .

[0119]

When the number of observation results in the historical observation sequence is greater than the preset number, the hidden state probability up to the last time is determined according to the historical observation sequence

[0120] The forward algorithm is used to calculate and normalize to obtain the hidden state posterior probability up to the last time , which is the confidence degree of the existence and absence of the state. Wherein is the hidden state of the system at time t-1, and the value range is ; is the observation state at the corresponding time, is all observation data at the first t-1 times, is the hidden state posterior probability;

[0121] In particular, at time t=1, = , wherein is the hidden state of the system at time t=1, and the value range is ; is the observation state at the corresponding time, and the value range is ; is the initial probability of state , is the probability of observing under the condition of state , is the sum of the joint probability of all hidden states at this time, = is the normalized hidden state posterior probability at time t=1. When The probability that the real physical state of the license plate at time t-1 is present is ( = |o1...o t-1 ), which is based on the historical observation belief that the license plate is present at time t-1.

[0122]

According to the current time hidden probability prediction value and observation probability matrix B, the observation state probability of the current time is calculated

[0123] According to the prior state distribution and the observation probability matrix , the probability of the license plate "not detected" in the hidden Markov model is determined, which includes the actual physical state of non-existence and the judgment of non-existence of the license plate, including: , where not detected means that the license plate is not detected, limited means that the visual observation is limited, missing means that the license plate is missing, and present means that the license plate is present, , that is, the above , and the rest is the same.

[0124] That is, whether the license plate is detected at time t without observation, the prediction probability of the license plate not detected at time t is directly predicted through historical information; it can be simply written as . is the prediction probability that the actual physical state at time t is the license plate present and the observation is good, is the prior probability that the actual physical state of the license plate is present, and the space fusion result is judged as non-existent (i.e. the model is not detected); is the prediction probability that the actual physical state at time t is the license plate present but the observation is limited, is the prior probability that the actual physical state of the license plate is present but the observation is limited, and the space fusion result is judged as non-existent (i.e. the model is not detected);

[0125] P is the prediction probability that the actual physical state at time t is the license plate missing, is the prior probability that the actual physical state of the license plate is missing, and the space fusion result is judged as non-existent (i.e. the model is not detected).

[0126] When the number of observation results in the historical observation sequence is less than or equal to the preset number, the number of observation results is continued to be accumulated, and the preset number can be 5 or more.

[0127]

Calculate the unexpectedness and the cumulative likelihood of the dominant state to generate a warning

[0128] Based on the preset unexpectedness mechanism and the cumulative likelihood mechanism of the dominant state, the abnormal absence of the license plate is warned, including:

[0129] The default surprise mechanism includes:

[0130]

[0131] in, For the sake of surprise, Hidden Markov Model The probability that the pole number plate does not exist; 1 represents the certainty that the pole number plate is not detected in actual observation (100% probability after the fact). Hidden Markov Model The probability of observing a pole number plate as "not detected" at the current moment, based on historical predictions, can be abbreviated as: .

[0132] This formula measures the degree of surprise in the deviation between actual observations and historical predictions. The lower the predicted probability but the more unexpected the event, the less consistent the failure to detect is with the historical performance pattern of that pillar.

[0133] Determine the dominant state, including:

[0134]

[0135] in, The dominant state at the current moment. The observed state at time t and before. for The hidden state of the time system; For the first The hidden state set of each pole number plate.

[0136] In particular, The calculation method is as follows: combine the hidden state prediction probability mentioned above. Given the observation probability matrix B, we can obtain the following using Bayes' theorem. ,in: It is the system's hidden state set. ; To predict the true physical state at time t based on the observed states at time t-1 and earlier, is... The probability, In the state The following observations The probability of (detected or not detected).

[0137] Determining the cumulative likelihood under visually restricted conditions includes:

[0138]

[0139] in, The number of times that the same position is continuously in the dominant state and the observation result is continuously not detected from the time t; The accumulated likelihood.

[0140] When the unexpectedness is greater than the preset unexpectedness threshold, the number of observation results is greater than the preset number, and the pole number plate is determined to be absent in the spatial fusion result at the current time, a preliminary early warning signal of pole number plate absence is generated.

[0141] When the unexpectedness is less than or equal to the preset unexpectedness threshold, the dominant state is the absence state, the number of observation results is greater than the preset number, and the pole number plate is determined to be absent in the spatial fusion result at the current time, a preliminary early warning signal of long-term continuous absence of the pole number plate is generated.

[0142] When the unexpectedness is less than or equal to the preset unexpectedness threshold, the dominant state The visual observation is limited, the number of observation results is greater than the preset number, and the pole number plate is determined to be absent in the spatial fusion result at the current time, a preliminary early warning signal of pole number plate absence in a complex fluctuation environment is generated.

[0143] When the number of observation results in the historical observation sequence is less than or equal to the preset number, the number of observation results is continuously accumulated.

[0144] When the pole number plate is determined to exist in the spatial fusion result at the current time, no signal is generated.

[0145] In addition, the maximum unexpectedness derivation: for a 100% stable support in history, the unexpectedness generated when "not detected" is observed is the maximum.

[0146] At this time, the prediction probability reaches the theoretical lower limit, which is equal to the basic missed detection rate B (not detected | exists). According to the probability axiom, it can be obtained that: wherein is the maximum unexpectedness.

[0147] Sensitivity coefficient setting: introduce the sensitivity coefficient k (value 0.9-0.98, recommended 0.95)

[0148] The preset unexpectedness threshold is:

[0149]

[0150] wherein, To preset the unexpectedness threshold, B (not detected | present) is the probability that the actual physical state of the pole sign is present, but the space fusion result determines that the pole sign does not exist (that is, the model is not detected), and the threshold is automatically adjusted according to the detection model performance (observation probability matrix B). When the model accuracy improves, the threshold is correspondingly increased, and the prediction robustness is maintained.

[0151] To convert the abstract algorithm of the application into a reproducible calculation process, this section provides a complete numerical example. In the cold start process (T<5), any event is not alarmed, and then (T≥5) the normal alarm process is started. Taking the unexpectedness threshold of 90% and the significance level threshold of 5% as an example. Specific embodiments: 【1】

[0154] A certain support is stably detected in historical inspection, and "not detected" is observed for the first time in a certain inspection. This case analyzes the influence of different historical lengths on early warning triggering.

[0155] Table 1

[0156]

[0157] This case shows the high timeliness of the real-time early warning framework. For a historically stable support, as long as "not detected" appears for the first time, the system can trigger early warning at that moment, realizing the immediate discovery of abnormalities. 【2】

[0159] Scene description: A support is continuously unable to be observed from the first inspection due to being in a special section such as a station field soft crossbar or being continuously and severely blocked by an obstacle along the line.

[0160] Table 2

[0161]

[0162] This case shows the intelligent identification ability of the framework for the continuous blocking scene. Through historical learning, the continuous "not detected" is recognized as the regular performance of the support, thereby effectively filtering false alarms. 【3】

[0164] Scene description: A support is continuously unable to be observed from the first inspection due to being in a special section such as a station field soft crossbar or being continuously and severely blocked by an obstacle along the line.

[0165] Table 3

[0166]

[0167] This case clearly demonstrates the adaptability. The triggering of the alert does not depend on this "non-detection" itself, but on the degree of deviation of this "non-detection" from its historical performance. The system makes intelligent judgments through three core factors:

[0168] Historical fluctuation frequency (distinguish between frequent occlusion and sudden loss):

[0169] Large fluctuation (suppress false positives): As in scenario 3a, it corresponds to a frequent occlusion scenario in physics (high occlusion frequency but not continuous complete occlusion). The system learns from historical observations that "restricted" is the dominant state, reduces the unexpectedness of the current non-detection, and considers that it is expected to be invisible, thereby avoiding false positives.

[0170] Small fluctuation (capture anomalies): As in scenario 3c, it corresponds to a long-term stable support in physics. The recent long-term stability establishes a strong belief in the existence of the state, and the sudden non-detection leads to high unexpectedness, and the system determines that it is a sudden loss.

[0171] Time interval effect (distinguish between environmental deterioration and sudden loss):

[0172] Use the "short-term memory" feature of the model. Comparing 3c and 3d, the anomaly in scenario 3d is concentrated in the recent past, which corresponds to short-term environmental deterioration such as bad weather or seasonal lighting in physics. The model automatically adjusts the expectation of the current state, reducing the unexpectedness of the current non-detection, thereby preventing false positives caused by environmental fluctuations; while 3c has been continuously detected in the recent past, any new anomaly will result in high unexpectedness and be determined as a sudden loss.

[0173] Sequence likelihood test (recognize loss under interference):

[0174] For the complex scenario of "historical fluctuation, continuous non-detection" (such as 3e), although the first-level unexpectedness decision does not trigger, the system performs a deep test: calculates the probability of finding "non-detection" for 4 times in a row under the premise that the dominant state continues to be "restricted" is only 2.56% (lower than the 5% threshold). This means in physics that environmental interference (such as bad weather, tree branches swaying, changes in backlight, etc.) usually has intermittent or random nature, and it is difficult to achieve continuous complete occlusion. The system rejects the "restricted" hypothesis and determines that it is a "real loss under complex fluctuating environment".

[0175] The invention can ensure that the system can distinguish between "instantaneous non-detection" and "real loss", and for a support that has just experienced an anomaly recently, the system will give a certain tolerance period; while for a long-term stable support, any sudden anomaly will be highly vigilant.

[0176] In summary, the present application introduces two perception dimensions of spatial fusion and temporal fusion into the pole number plate missing detection, constructs a double verification and correction framework, and makes up for the limitations of single image recognition method. The present application uses monocular depth estimation algorithm, expands two-dimensional image analysis to three-dimensional space relationship understanding, and improves the attribution accuracy of multiple pole number plates in complex scenes. The present application discards the traditional judgment logic based on fixed rules, and proposes a real-time inference framework based on historical prediction. The core of the framework is that the real physical state of the pole number plate is modeled as the "hidden state" of the HMM, and the single inspection result is modeled as the "observation state". At each inspection time, the state confidence is calculated based on the historical observation by using the forward algorithm, and the probability of "not detected" at the current time is predicted. The deviation between the predicted probability and the actual observation (i.e. "unexpectedness" and "cumulative likelihood of dominant state") is compared to make a real-time judgment of whether an abnormality occurs, instead of waiting for multiple observations before making a judgment. The fixed unexpectedness and prominence level thresholds are introduced, which can be adaptively linked, so that the framework can intelligently distinguish between "real missing" and "continuous occlusion" and other different scenarios, while realizing the unification of real-time warning and high accuracy. The present application realizes real-time warning under the premise of ensuring accuracy through the real-time prediction inference framework, and solves the core technical problem of the contradiction between timeliness and accuracy in the inspection business.

[0177] Based on the same inventive concept, the present application provides a pole number plate missing detection device based on spatio-temporal fusion perception, comprising:

[0178] An image acquisition module is configured to acquire multi-view images or time-series multi-frame images of the pole number plate in S11, and obtain the position and string of the pole number plate in each image based on a deep learning model. The multi-view images or time-series multi-frame images each include a plurality of images.

[0179] An identification module is configured to identify the pole number plate in S12, including S121, positioning the support of each image based on a target detection or segmentation algorithm, and determining whether the corresponding image is retained according to the depth value of the pole number plate and the depth value of the support. S122, based on the retained image, obtaining the spatial fusion result of the pole number plate, wherein the spatial fusion result includes the pole number plate existence recognition result and the string number recognition result.

[0180] A warning module is configured to warn in S13, including S131, defining a hidden state set , an observation state set , and a hidden Markov model , wherein the hidden state set is related to the actual physical state of the pole number plate, and the observation state set is related to the spatial fusion result, , is the initial state probability of the support, is the state transition probability matrix of the pole number plate, is an observation matrix; S132, a history observation sequence of the support is constructed wherein is observation data at the moment, and based on a hidden Markov model a corresponding hidden state set is inferred, and an abnormal absence of the pole number plate is warned based on a preset abnormality degree mechanism and a cumulative likelihood degree mechanism of a dominant state, the history observation sequence is related to the observation state set .

[0181] Based on the same inventive concept, the present application also provides an electronic device, comprising:

[0182] a processor;

[0183] a memory for storing processor-executable instructions;

[0184] wherein the processor is configured to execute to implement the pole number plate absence detection method based on spatio-temporal fusion perception as provided in the foregoing.

[0185] Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the pole number plate absence detection method based on spatio-temporal fusion perception as provided in the foregoing.

[0186] Since the electronic device introduced in the present embodiment is the electronic device used to implement the information processing method in the present embodiment, the specific implementation of the electronic device and its various forms of changes can be understood by those skilled in the art based on the information processing method introduced in the present embodiment, so the implementation of the electronic device in the method of the present embodiment will not be introduced in detail. As long as the electronic device used to implement the information processing method in the present embodiment is implemented by those skilled in the art, it belongs to the scope of the present application.

[0187] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media containing computer usable program code (including but not limited to disk storage, CD-ROM, optical storage, etc.).

[0188] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0189] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0190] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0191] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0192] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for detecting missing pole numbers based on spatiotemporal fusion perception, characterized in that, include: S11, acquire multi-view images or time-series multi-frame images of pole number plates, and obtain the position and string of the pole number plate in each image based on a deep learning model. The multi-view images or time-series multi-frame images each include several images. S12 includes S121, locating the pillars in each image based on an object detection or segmentation algorithm, and determining whether the corresponding image should be retained based on the depth value of the pole number and the depth value of the pillar; S122, obtaining the spatial fusion result of the pole number based on the retained image, wherein the spatial fusion result includes the pole number recognition result and the string number recognition result. S13, including S131, defines the hidden state set. Observation state set and Hidden Markov Models The hidden state set is related to the actual physical state of the pole number plate, and the observed state set is related to the spatial fusion result. , Let be the initial state probability of the pillar. Let be the state transition probability matrix for the pole number plate. S132, construct the historical observation sequence of the pillar. ,in for Observational data at time points, and based on a hidden Markov model. For the corresponding hidden state set Inferences are made, and warnings are issued for abnormal missing pole numbers based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state. The historical observation sequence and observation state set are used for this purpose. Relatedly, this includes: when the spatial fusion results indicate the existence of pole numbers, matching the pole numbers based on the string number recognition results and mileage information; when the spatial fusion results indicate the absence of pole numbers, extracting the string number recognition results of the previous and next pole numbers, and obtaining the corresponding string number recognition results based on linear interpolation, and matching the pole numbers based on the string number recognition results and mileage information; and constructing a historical observation sequence of the poles by sequentially using the string number recognition results of several matched poles. When the number of observations in the historical observation sequence exceeds a preset number, the posterior probability of the hidden state up to the previous time step is determined based on the historical observation sequence; then, based on the posterior probability of the hidden state and the state transition probability matrix... Determine the prior state distribution at the current moment; based on the prior state distribution and the observation probability matrix... Determine the Hidden Markov Model The probability that the pole number plate does not exist includes both the actual physical absence and the determination that the pole number plate does not exist; when the number of observation results in the historical observation sequence is less than or equal to a preset number, the number of observation results continues to be accumulated, including: The default surprise mechanism includes: in, For the sake of surprise, Hidden Markov Model The probability that the center pole number plate does not exist; Determine the dominant state, including: in, The dominant state at the current moment. The observed state at time t and before. for The hidden state of the time system; For the first The hidden state set of each pole number plate The posterior probability of the hidden state; Determining the cumulative likelihood under visually restricted conditions includes: in, To trace back from time t, the number of times the same position was continuously under visual observation limitation and the observation result was undetected in the dominant state; To accumulate the likelihood, the probability of determining that a pole number plate does not exist in the spatial fusion result is: ; If the unexpectedness exceeds the preset unexpectedness threshold, the number of observation results exceeds the preset number, and the spatial fusion results at the current moment indicate that the pole number plate does not exist, then a preliminary warning signal for the missing pole number plate will be generated. If the unexpectedness is less than or equal to the preset unexpectedness threshold, the dominant state is the missing state, the number of observation results is greater than the preset number, and the pole number is determined to be missing in the spatial fusion results at the current moment, then a preliminary warning signal for the long-term continuous missing pole number is generated. When the unexpectedness is less than or equal to the preset unexpectedness threshold, the dominant state is... For visual observation restricted state, If the number of observations is less than the preset threshold, the number of observations is greater than the preset number, and the pole number is determined to be missing in the spatial fusion results at the current moment, then a preliminary warning signal for the missing pole number under complex fluctuation environment is generated. If the number of observations in the historical observation sequence is less than or equal to the preset number, the number of observations will continue to be accumulated. If the pole number plate is determined to exist in the spatial fusion results at the current moment, no signal generation will be performed.

2. The pole number missing detection method based on spatiotemporal fusion perception as described in claim 1, characterized in that, Determine whether to retain the corresponding image based on the depth value of the pole number plate and the depth value of the support post, including: Determine the absolute difference between the depth value of the pole number plate and the depth value of the corresponding support post; If the absolute difference is less than or equal to the adaptive threshold, the corresponding image is retained; otherwise, the corresponding image is discarded. The adaptive threshold is related to the depth value of the support.

3. The pole number missing detection method based on spatiotemporal fusion perception as described in claim 2, characterized in that, When the retained image is a multi-view image, the spatial fusion result of the pole number plate is obtained based on the retained image, including: If at least one image contains a pole number plate, the pole number plate is identified as present; otherwise, it is identified as not present. When the pole number plate is identified as existing, the string that appears most frequently in the image is used as the string number recognition result; when the pole number plate is identified as not existing, the string number recognition result is determined to be non-existent.

4. The pole number missing detection method based on spatiotemporal fusion perception as described in claim 2, characterized in that, When the retained image belongs to a multi-frame temporal sequence, the spatial fusion result of the pole number plate is obtained based on the retained image, including: Based on a multi-target tracking algorithm, pole numbers in images with consecutive frames are associated with motion tracks. If the number of consecutive stable frames of the motion track is greater than a preset number, the pole number is identified as existing; otherwise, it is identified as not existing. When the pole number plate is identified as existing, the string that appears most frequently in the image is used as the string number recognition result; when the pole number plate is identified as not existing, the string number recognition result is determined to be non-existent.

5. The pole number missing detection method based on spatiotemporal fusion perception as described in claim 1, characterized in that, Define hidden state set Observation state set and Hidden Markov Models ,include: Define hidden state set include ,in The actual physical state of the pole number plate exists and the visual observation is ideal. The actual physical state of the pole number plate is that it does not exist. The actual physical state of the pole number plate exists, but visual observation is limited; Observation state set include ,in To determine the presence of pole number plates in the spatial fusion results, The pole number plate was determined to be non-existent in the spatial fusion results; Define the initial state probability of the pole number plate. The initial state of the pole number plate is the probability of its existence. The initial state of the pole number plate is the probability of being missing. The initial state of the pole number plate has a probability of being visually restricted. ,in All belong to ; Define the state transition probability matrix of the pole number plate. , among which, from Time's up At what moment is the probability that the pole number plate exists and the ideal visual observation state remains unchanged? ;from Time's up At what moment does the pole number plate change from a state where it exists and is ideally visually observed to a state where it exists but is visually limited? ;from Time's up At what moment does the pole number plate change from a state where it exists and is visually ideal to a state where it is missing? ;from Time's up At what moment does the pole number plate revert from a state of limited visual observation to a state of existence where visual observation is ideal? ;from Time's up The probability that the pole number plate exists but visual observation is limited at any given moment remains unchanged. ;from Time's up The probability that, at any given moment, the pole number plate changes from a state of existence but limited visual observation to a state of absence. ;from Time's up At what moment is the probability that the pole number plate recovers from a missing state to a state where it is present and visually ideal? ;from Time's up At what moment does the pole number plate change from a missing state to a state where it exists but is visually limited? ;from Time's up At any given moment, the probability that the pole number plate will not be restored from its missing state. ,in All belong to ; Define the observation probability matrix When the actual physical state of the pole number plate is as described above, the probability of determining the pole number plate as existing in the spatial fusion result is as follows: When the actual physical state of the pole number plate is that it exists, the probability of determining that the pole number plate does not exist in the spatial fusion result is: When the actual physical state of the pole number plate is that it exists but visual observation is limited, the probability of determining the existence of the pole number plate in the spatial fusion result is: When the actual physical state of the pole number plate is that it exists but visual observation is limited, the probability of determining that the pole number plate does not exist in the spatial fusion result is: When the actual physical state of the pole number plate is non-existent, the probability of determining that the pole number plate exists in the spatial fusion result is: When the actual physical state of the pole number plate is non-existent, the probability of determining that the pole number plate does not exist in the spatial fusion result is: ,in All belong to .

6. A pole number plate missing detection device based on spatiotemporal fusion perception, characterized in that, The pole number missing detection method based on spatiotemporal fusion perception, applied to any one of claims 1-5, includes: The image acquisition module, used by S11, acquires multi-view images or time-series multi-frame images of pole number plates, and obtains the position and string of the pole number plate in each image based on a deep learning model. The multi-view images or time-series multi-frame images each include several images. The recognition module, used in S12, includes S121, locating the pillars in each image based on an object detection or segmentation algorithm, and determining whether the corresponding image should be retained based on the depth value of the pole number and the depth value of the pillar; S122, obtaining the spatial fusion result of the pole number based on the retained image, wherein the spatial fusion result includes the pole number recognition result and the string number recognition result. The early warning module, used in S13, includes S131, which defines the hidden state set. Observation state set and Hidden Markov Models The hidden state set is related to the actual physical state of the pole number plate, and the observed state set is related to the spatial fusion result. , Let be the initial state probability of the pillar. Let be the state transition probability matrix for the pole number plate. S132, construct the historical observation sequence of the pillar. ,in for Observational data at time points, and based on a hidden Markov model. For the corresponding hidden state set Inferences are made, and warnings are issued for abnormal missing pole numbers based on a preset unexpectedness mechanism and a cumulative likelihood mechanism of the dominant state. The historical observation sequence and observation state set are used for this purpose. Related.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the pole number missing detection method based on spatiotemporal fusion perception as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the non-transitory computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the pole number missing detection method based on spatiotemporal fusion perception as described in any one of claims 1 to 5.

Images