Machine learning solution for matching vehicle exits to unmatched entry events

Abstract

An edge device generates an exit event for a vehicle leaving a parking facility. The edge device determines whether the exit event matches an entry event. In response to the decision that the exit event does not match an entry event, the edge device inputs an image of the vehicle into a supervised machine learning model and receives an exit feature vector as output from the model. The edge device reads an entry feature vector corresponding to an unmatched entry event. An unmatched entry event is an entry event for a vehicle with an unknown vehicle identifier. The edge device inputs the exit feature vector and the entry feature vector into an unsupervised machine learning model and receives a matching score for each entry feature vector as output from the model. Based on the matching score, the edge device matches the exit event with one of the unmatched entry events.

Description

Technical Field

[0001] Technical Field The present disclosure generally relates to the field of machine learning, and more specifically, to machine learning techniques for vehicle identification and enforcement.

Background Art

[0002] Background Parking facilities utilize gates to manage the use of their space and often require vehicles to pass through the gates in response to one or more of entry and exit. As parking facilities migrate to more automated systems and implement seamless vehicle entry and exit, they may require little or no intervention from human operators and vehicle users. While this migration can improve efficiency, it can pose challenges in identifying vehicles entering or exiting the space. For example, physical space limitations in a parking facility may only allow for cameras facing the front of the vehicle, and environmental factors such as glare, low light levels, or weather conditions like snow and mud can further obscure the vehicles and their identification information. Additionally, automated parking management can result in encouraging malicious actors to break the rules within parking facilities with little human oversight. Considering that continuous monitoring of such a wide range of potential activities requires a vast amount of computing power, it is not practical to equip parking facilities with numerous cameras and computer vision to detect malicious behavior from an implementation and computational efficiency perspective.

Summary of the Invention

Means for Solving the Problems

[0003] Summary This specification discloses a system and method for improving vehicle identification in a parking facility, particularly for improving the matching of vehicle entry to vehicle exit, by using machine learning techniques. Entry or exit events may be generated in response to a vehicle entering or exiting a parking facility. Machine learning techniques may be applied to generate feature vectors describing the vehicles involved in such events. Upon vehicle exit, in response to a determination that the generated exit event does not match an entry event, an additional machine learning process may be invoked to compare the entry feature vector and the exit feature vector to determine a match.

[0004] In one embodiment, an edge device generates an exit event for a vehicle leaving a parking facility. The edge device determines whether the exit event matches an entry event. In response to the determination that the exit event does not match an entry event, the edge device inputs an image of the vehicle into a model (e.g., a supervised machine learning model) and receives an exit feature vector as output from the model. The edge device reads an entry feature vector corresponding to an unmatched entry event. An unmatched entry event is an entry event for a vehicle with an unknown vehicle identifier. The edge device inputs the exit feature vector and the entry feature vector into a second model (e.g., an unsupervised machine learning model) and receives a matching score for each entry feature vector as output from the model. Based on the matching score, the edge device matches the exit event with one of the unmatched entry events.

[0005] Systems and methods for improving enforcement capabilities within parking facilities using computer vision and machine learning techniques while addressing the aforementioned inefficiencies are also disclosed herein. In some embodiments, when a particular sensor flags a potential violation, a decision on whether a violation has occurred is triggered, and in response to these sensors flagging this possibility, a machine learning process is activated to confirm that a violation has indeed occurred. Furthermore, the use of machine learning to generate a vehicle fingerprint associated with a malicious actor may be limited to scenarios where vehicles cannot be identified otherwise. Further techniques for conducting enforcement in a computationally viable manner are discussed in detail below.

[0006] In one embodiment, an edge device detects violations caused by a vehicle by using sensors installed within the parking facility. The edge device generates a vehicle fingerprint by inputting an image of the vehicle into a model (e.g., a supervised machine learning model) and receiving a feature vector of the vehicle as output from the model. In various parking facilities, the edge device uses the vehicle fingerprint to monitor vehicle entry and triggers corrective actions in response to the detection of vehicle entry in the parking facility. [Brief explanation of the drawing]

[0007] Brief explanation of the drawing The embodiments disclosed have advantages and features that will become more readily apparent from the detailed description, the appended claims, and the appended figures (or drawings). A brief introduction to the figures is given below.

[0008] [Figure 1] Figure 1 illustrates one embodiment of a system environment for determining gate status using edge devices and a parking control server.

[0009] [Figure 2]Figure 2 illustrates one embodiment of an exemplary module operated by an edge device.

[0010] [Figure 3] Figure 3 illustrates an exemplary embodiment of a module operated by a parking control server.

[0011] [Figure 4] Figure 4 is a block diagram illustrating the components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller).

[0012] [Figure 5] Figure 5 illustrates one embodiment of an exemplary process for matching exit events with entry events.

[0013] [Figure 6] Figure 6 illustrates one embodiment of an exemplary process for detecting and responding to violations caused by a vehicle.

[0014] [Figure 7A] Figures 7A to 7C illustrate exemplary embodiments of the vicinity of the parking facility and the movable gate. [Figure 7B] Figures 7A to 7C illustrate exemplary embodiments of the vicinity of the parking facility and the movable gate. [Figure 7C] Figures 7A to 7C illustrate exemplary embodiments of the vicinity of the parking facility and the movable gate. [Modes for carrying out the invention]

[0015] Detailed explanation The figures and the following description relate to preferred embodiments for illustrative purposes only. It should be noted from the following discussion that alternative embodiments of the structures and methods disclosed herein will be readily recognizable as viable alternatives that can be adopted without departing from the principles of the claimed ones.

[0016] Herein, several embodiments are referenced in detail, and these embodiments are illustrated in the accompanying figures. It should be noted that whenever possible, similar or identical reference numerals may be used in the figures to indicate similar or identical functionality. The figures depict embodiments of the disclosed system (or method) for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be adopted without departing from the principles described herein.

[0017] (Configuration overview) Figure 1 illustrates one embodiment of a system environment for seamless parking gate operation using edge devices and a parking control server. As depicted in Figure 1, the environment 100 includes an edge device 110, a camera 112, a gate 114, a data tunnel 116, a sensor 118, a network 120, and a parking control server 130. Only one of each feature of the environment is depicted, but this is for convenience only, and any number of each feature may exist. Where singular articles are used to refer to these features (e.g., "camera 112"), scenarios in which multiple of those features are referred to are also within the scope of disclosure (e.g., a reference to "camera 112" may mean that multiple cameras are involved).

[0018] Edge device 110 uses camera 112 to detect vehicles approaching gate 114. In response to detecting such a vehicle, edge device 110 performs various operations (e.g., lifting the gate, updating a profile associated with the vehicle, etc.) that are described in further detail below, at least with reference to FIG. 2. Camera 112 can include any number of cameras that capture images and / or videos of the vehicle from one or more angles (e.g., from behind the vehicle, from the front of the vehicle, from the side of the vehicle, etc.). Camera 112 can be in a fixed position or can be movable (e.g., along a track or line) to capture images and / or videos from different angles. When the term "image" is used, this can be an independent image or a frame of a video. When the term "video" is used, this can include a plurality of images (e.g., frames of a video), and the plurality of images can form a sequence that together forms a video.

[0019] The gate 114 can be any object that blocks entry into and / or exit from a facility (e.g., a parking facility) until it is moved. For example, the gate 114 could be a pole that blocks entry or exit by being erected parallel to the ground, and rises perpendicular to the ground to allow vehicles to pass. In another embodiment, the gate 114 could be a pole or a group of poles that block vehicle access until they are lowered to a position coplanar with the ground. Any form of blocking of vehicle entry / exit is movable to remove the blockage and is within the context of the gate 114. In some embodiments, there is no physical gate that blocks traffic from entering or exiting the facility. Rather, in such embodiments, the gate 114 as referred to herein is a logical boundary between the inside and outside of the facility, and all embodiments disclosed herein that refer to moving the gate equally refer to scenarios in which the gate is not moved, but other processing occurs when entry and exit coincide (e.g., recording that a vehicle has left the facility). Furthermore, gate 114 may be any general gate that does not communicate directly with edge device 110. Instead, edge device 110 may communicate directly with components that are separate from the gate but are arranged in association with the gate, and these components are configured, by arrangement, to move the gate.

[0020] Edge device 110 optionally communicates information associated with a detected vehicle to parking control server 130 across network 120 using data tunnel 116. Data tunnel 116 can be any tunneling mechanism such as a virtual private network (VPN). Network 120 can be any mode of communication including cellular tower communication, Internet communication, WiFi, WLAN, etc. The information provided can include an image of the detected vehicle. Additionally or alternatively, the information provided can include information extracted from or otherwise obtained based on an image of the detected vehicle (as further described below with respect to, for example, FIG. 2). Transmitting the extracted information rather than the underlying image can result in bandwidth throughput efficiency that enables real-time or near real-time movement of gate 114 by avoiding the need to transmit high data volume images.

[0021] In some embodiments, edge device 110 can apply computer vision to determine environmental factors around a vehicle. As used herein, the term environmental factors can refer to features that affect traffic flow in the vicinity of gate 114 such as street traffic blocking an exit from a facility, the orientation of vehicles in an image relative to each other, etc. In one embodiment, when instructing the movable gate to move, edge device 110 applies parameters based on the determined environmental factors (e.g., there is a vehicle in front of the vehicle attempting to exit and thus blocking the exit, so gate 114 waits to open even though an entry to an exit has been matched).

[0022] The parking control server 130 receives information from the edge device 110 and performs actions based on that reception. These actions may include storing the information, updating the profile, reading information related to that information, and communicating to the edge device 110 to return any additional information to respond. The parking control server 130 may control the sides of parking equipment, such as status lights above the parking gate. The operation of the parking control server 130 is described in more detail below with reference to at least Figure 3.

[0023] Figure 2 shows one embodiment of an exemplary module operated by an edge device. As depicted in Figure 2, the edge device 110 includes an entry detection module 212, an exit detection module 214, a vehicle recognition module 216, an event matching module 218, a matching resolution module 220, a violation detection module 222, a fingerprint generation module 224, an entry monitoring module 226, and a corrective action module 228. The modules described with respect to the edge device 110 are illustrative only, and fewer or additional modules may be used to accomplish the activities disclosed herein. Furthermore, while the modules of the edge device 110 typically reside within the edge device 110, in various embodiments they may instead reside, partially or entirely, within the parking control server 130 (for example, if images, rather than data from images, are sent to the parking control server 130 for processing). In some embodiments, the modules and functions of the edge device 110 may be implemented, whole or partially, within the sensor 118.

[0024] The entry detection module 212 detects and stores entry events. An entry event represents a vehicle approaching the parking facility from the entry side and, in some embodiments, entering the parking facility through an entry gate. The entry detection module 212 may detect entry events by using the camera 112 to capture a series of images over time. In some embodiments, the camera 112 may continuously capture images or capture images when certain conditions (e.g., motion is detected, or any other heuristic such as during a certain time of day) are met. In some embodiments, the edge device 110 may continuously receive images from the camera 112 and determine whether the images contain a vehicle, in which case the entry detection module 212 may perform processing on the images containing the vehicle and discard other images. In some embodiments, the entry detection module 212 may command the camera 112 to transmit only images containing vehicles and perform processing on those received images. At the point where each camera 112 faces either the gate or an area near the gate (e.g., entry side only, exit side only, or both), the captured images are associated with a movable gate or logical boundary (e.g., gate 114). Each image may have a timestamp and / or sequence number. The entry detection module 212 may associate all images containing the motion of a given vehicle from the time the vehicle enters the image to the time the vehicle exits the image (e.g., during the time the vehicle approaches the gate and then drives through or passes through the gate). In some embodiments, the entry detection module 212 may separate the portion of the image containing the vehicle and exclude the portion of the image that does not contain the vehicle (e.g., background, environment, other vehicles). For example, the entry detection module 212 may place a boundary polygon around the portion of the image containing the largest vehicle in the frame. From the image containing the vehicle, the entry detection module 212 may further separate the portion of the image containing vehicle identifiers such as license plates, or place a boundary polygon around the portion of the image.

[0025] The entry detection module 212 may determine a dataset corresponding to a vehicle from an image featuring the vehicle. The dataset may include parameters describing the vehicle's attributes and vehicle identifier. Parameters describing the vehicle's attributes may include both the vehicle's identification attributes and directional attributes. Identification attributes may include any information that can be derived from an image describing the vehicle, such as manufacturer, model, color, type (e.g., sedan vs. sport utility vehicle), height, length, bumper style, number of windows, door handle type, and any other descriptive features of the vehicle. Directional attributes may refer to absolute direction (e.g., base direction) or relative direction (e.g., the vehicle's direction relative to the entry gate, and / or the vehicle's direction relative to the assigned direction of the lane blocked by the entry gate (e.g., if different gates are used for entry and exit lanes, and the vehicle is approaching the gate from an entrance to a parking facility via an exit lane, the direction is indicated as being opposite to the intended direction of the lane)). Directional attributes can also be determined for camera imaging access, thus indicating whether the vehicle is moving toward or away from the camera. In one embodiment, a single machine learning model is used to generate the entire dataset of both parameters and vehicle identifiers. In another embodiment, a first machine learning model is used to determine the parameters, and a different second machine learning model is used to determine the vehicle identifier.

[0026] In this two-model approach, the entry detection module 212 inputs an image featuring a vehicle into the first machine learning model and determines parameters by receiving parameters describing the vehicle's attributes as output from the first machine learning model. In one embodiment, the output of the first machine learning model may be more granular and may include the number of objects in the image (e.g., the number of vehicles), the type of object in the image (e.g., vehicle type information or vehicle-specific identification attribute information), the result score (e.g., confidence level for each object classification), and bounding boxes (e.g., subsegments of the image for downstream processing, such as license plates, for use by the second machine learning model).

[0027] A first machine learning model can be trained to output vehicle identification attributes using exemplary data having images of vehicles labeled with one or more candidate identification attributes. For example, various images from a camera facing a gate may be manually labeled by a user, and each of the various images may show the vehicle's manufacturer, model, color, type, and other attributes mentioned above. The first machine learning model may be a supervised model trained using the exemplary data to predict those attributes for new images.

[0028] A first machine learning model can be trained to output vehicle direction attributes using exemplary data, and / or data from which the entry detection module 212 can determine some or all of the direction attributes. The exemplary data may show the motion of a vehicle toward one or more gates across a series of sequential frames, may be annotated with lane type (e.g., entry lane vs. exit lane) and / or gate type (e.g., exit gate vs. entry gate), and may be labeled with direction between two or more frames (e.g., towards entry gate, away from entry gate, towards exit gate, away from exit gate). The lane type may be derived from environmental factors (e.g., the model may be trained, through sufficient exemplary data, to recognize that the direction beyond a gate showing a blue sky is the exit direction, and the direction toward a halogen light is the entry direction). From this training, the first machine learning model may output a direction based directly on learned motion for gate type and / or lane type, or it may output a lane type and / or gate type and a marker of directional movement, from which the entry detection module 212 may apply heuristics to determine directional attributes (e.g., moving towards the entry gate, moving away from the entry gate, moving towards the exit gate, moving away from the exit gate). That is, a direction vector may be output along with the gate type and / or lane type (environmental factors, which may include other information such as lighting and sky information, may be output along with the direction vector), and the direction vector may be used to determine directional attributes along with environmental factors.

[0029] Since vehicles are tracked as they move, it is advantageous to determine direction attributes along with identification attributes. However, determining direction and identification attributes in a single step can lead to false positives. However, separate models can be used for identification attribute detection and direction attribute detection, hence the three-model approach (two models are used in what is called the "first machine learning model" above, and each of these separate models is trained separately for each task using its own training data).

[0030] Continuing with the two-model approach, the entry detection module 212 determines the vehicle identifier by inputting an image featuring the depiction of the vehicle's license plate into a second machine learning model. That is, rather than using optical character recognition (OCR), a second machine learning model may be used to decode the vehicle's license plate into the vehicle's vehicle identifier. OCR methods are often inaccurate for license plate detection due to the complexity of license plates, which often use various fonts (e.g., cursive vs. script), as well as the problems of backgrounds filled with complex photographs, different colors, and lighting. Furthermore, various types of license plates often contain non-generalizable slogans, making them difficult to read accurately. Even a minor inaccuracy in an OCR read, such as the determination of a single character or geographic identifier being off, can result in the inability to effectively identify the vehicle.

[0031] For this purpose, a second machine learning model may be trained to identify and output both the geographical nomenclature and the string of characters of a vehicle identifier (for example, directly or using a confidence score exceeding a threshold applied by the entry detection module 212). As used herein, the term “geographical nomenclature” may refer to a form of identification that identifies the jurisdiction that issued the license plate. That is, in the United States, individual states issue license plates, and the geographical identifier will identify that state. In some jurisdictions, national license plates are issued, in which case the geographical identifier is the country identifier. A geographical identifier may identify more than one jurisdiction (for example, in the European Union (EU), some license plates identify both the EU and the member state that issued the license plate, and the geographical identifier may identify both of those locations or only the member state). The term “string of characters” may refer to a unique symbol issued by a jurisdiction to uniquely identify a vehicle, such as “license plate number” (which may include numbers, letters, and symbols). In other words, for each given jurisdiction, the string is unique to any other string issued by that given jurisdiction. In some embodiments, a vehicle's license plate number may include a string in which the characters are written both vertically (e.g., read from top to bottom) and horizontally (e.g., read from left to right). The term “license plate identifier” may refer to a combination of a geographical name and a license plate number.

[0032] To train a second machine learning model, training examples of license plate images are used, in which case the training examples are labeled. In one embodiment, the training examples are labeled with both the geographical jurisdiction and the characters depicted in the image. Characters may be labeled individually (for example, by labeling the segments of an image containing the segments), or the entire image may be labeled with each character or combination thereof present. In the case of strings containing both vertical and horizontal characters, the strings may be labeled in a standardized format such as left-to-right, top-to-bottom rules (e.g., license plate).

number

number

[0033] In one embodiment, training instances may be labeled only by geographical jurisdiction, and a second machine learning model predicts the geographical jurisdiction for a new image of a license plate. Following this prediction, a third machine learning model may be selected from a group of candidate machine learning models, each corresponding to a different geographical jurisdiction and trained to predict the characters of a string from training instances specific to that particular geographical jurisdiction, and the selected third machine learning model may be chosen based on the predicted geographical jurisdiction. The third machine learning model is applied to the image or segment containing each character and therefore can yield predictions from training instances specific to that jurisdiction.

[0034] In any case, the training examples can demonstrate implementations under any number of conditions, including low-light conditions, dirty license plate conditions where letters are partially or completely obscured, license plate frame conditions where geographic identifiers (e.g., the word "New York") are partially or completely obscured, and conditions where license plate covers make it difficult to directly read the letters. Advantageously, by using machine learning to predict geographical naming conventions and character strings, accuracy is improved compared to OCR, as the second machine learning model can accurately predict the contents of the license plate even when partial obscuration occurs or lighting conditions make it difficult to read the letters.

[0035] In a one-model approach, the manner in which the first and second machine learning models are trained will be applied to a single model, rather than distinguishing what is learned between the two models. This will have the advantage of providing the model with all inputs as a single dataset, but it may also have the disadvantage of a less specialized model with a noisier output. Furthermore, training one large model to perform all of this functionality requires intensive data and time. The large model may be slower and have lower output quality than using two separate models. The two-model approach also allows for "fail-fast" processing, meaning that a vehicle can be detected and processed based on that detection even before other activities (e.g., license plate reading) are completed.

[0036] Regardless of the modeling method used, in one embodiment, the entry detection module 212 may determine from the vehicle's orientation attributes whether the vehicle's orientation attributes match the function of the entry gate, and thus confirm that the vehicle has performed an entry event. That is, the entry detection module 212 determines that the vehicle has used, or is using, an entry lane as opposed to an exit lane. In some embodiments, the entry detection module 212 may move the gate to allow entry into the facility blocked by the gate (or, if the gate is a logical boundary, record that the vehicle has entered the facility without needing to move the gate).

[0037] In some embodiments, the entry detection module 212 may determine a feature vector, an "entry feature vector," corresponding to an entry event. To generate the entry feature vector, the entry detection module 212 inputs a depiction of a vehicle into a supervised machine learning model. The depiction of a vehicle may include an image containing the vehicle, for example, as captured by a camera 112. In some embodiments, the depiction of a vehicle may include only isolated portions of an image containing the vehicle. In some embodiments, the depiction of a vehicle may include other data, such as data from a dataset. The supervised machine learning model outputs the entry feature vector. The entry feature vector may include multiple embeddings, each embedding derived from one or more dimensions of the depiction of the vehicle. The supervised machine learning model may be trained to output the feature vector. In some embodiments, the supervised machine learning model may be trained so that feature vectors corresponding to different vehicles have the maximum distance from each other in the feature space. For example, the supervised machine learning model may be trained so that feature vectors are penalized based on the angular margin between one feature vector and another, with smaller angular margins resulting in larger penalties. This training results from greater distances between feature vectors.

[0038] In some embodiments, a supervised machine learning model may be a multitask model, such as a multitask neural network, which has branches, each trained to determine different parameters. The structure of a multitask model has a set of shared layers and multiple branched task-specialized layers, where each branch in the branched task-specialized layer corresponds to a task. The tasks are related within a domain, meaning that each task determines parameters that can be determined based on a highly overlapping information space. For example, when determining the entry feature vector of a vehicle, different tasks may predict the vehicle's license plate, manufacturer and model, etc. Thus, once trained, the shared layers perform each task and generate useful information to output each of these predictions. Eleven or more embedded shared layers may be used to generate feature vectors.

[0039] Although the model used by the entry detection module 212 to generate entry feature vectors is described as a supervised machine learning model, supervised machine learning models are merely illustrative. The entry detection module 212 may use other types of models to generate entry feature vectors. For example, the entry detection module 212 may use a classification model (e.g., logistic regression, decision tree, random forest, or naive Bayesian model) to classify vehicles in entry events.

[0040] As will be discussed later with respect to the event matching module 218 and the matching resolution module 220, the process of matching entry events with exit events (e.g., representations of vehicles exiting a parking facility) may not necessarily require the entry detection module 212 to generate feature vectors. The event matching module 218 may match entry events with exit events without using feature vectors. For example, if the vehicles are known vehicles, the event matching module 218 may match entry events with exit events based solely on the vehicle's vehicle identifier. Alternatively, in another example, the event matching module 218 may match entry events with exit events based on a dataset of entry and exit events, for example, based on the vehicle type, model, and color. However, in response to the event matching module 218 failing to find a match between entry and exit events, the matching resolution module 220 may attempt to match entry and exit events using feature vectors. The matching resolution module 220 may request feature vectors from the entry detection module 212. Thus, in some embodiments, to avoid generating feature vectors when they may not necessarily be used in the matching process, the entry detection module 212 may suspend generating feature vectors in response to vehicle entry detection and instead generate entry feature vectors in response to receiving requests from the matching resolution module 220. This technique conserves computing resources (e.g., processing power, memory) by first attempting less computationally expensive means to match entry and exit events before generating feature vectors.

[0041] The entry detection module 212 may store entry events corresponding to a vehicle in the entry data database 358 of the parking control server 130. An entry event corresponding to a vehicle includes a data set corresponding to the vehicle (e.g., parameters and vehicle identifier), and in some embodiments, an entry feature vector, an image featuring the vehicle, a timestamp corresponding to the entry (e.g., a timestamp and / or sequence number of the image), and the parking facility into which the vehicle entered. In one embodiment, the entry detection module 212 may store the entry event in the edge device 110.

[0042] The exit detection module 214 operates in a similar manner to the entry detection module 212, in that machine learning is applied in a similar manner to detect exit events. That is, when it is detected that a vehicle is approaching gate 114 to exit the facility, the same dataset and / or feature vectors that were determined when the vehicle performed an entry action are performed for the exit action. When an exit action is detected (for example, when it is determined that the vehicle has directional attributes that match approaching a gate designated for use as an exit), the exit detection module 214 determines that an exit event may have occurred (and other activities may also be performed, for example, generating and storing data structures or feature vectors (e.g., into the exit data database 360) as described for entry events). In some embodiments, the exit detection module 214 may determine feature vectors in response to the edge device 110 determining that the exit event does not match an entry event.

[0043] The vehicle recognition module 216 determines whether a vehicle is a known vehicle. A known vehicle is one that has a profile stored in the profile database 356. The vehicle recognition module 216 may read a vehicle identifier (e.g., license plate) from an entry event associated with the vehicle (e.g., stored in the entry data database 358). The vehicle recognition module 216 may use the vehicle identifier as an index to search the profile database 356. In response to finding an entry in the profile database 356 corresponding to the vehicle identifier, the vehicle recognition module 216 determines that the vehicle is known. In response to a vehicle entering or exiting a parking facility, the vehicle recognition module 216 may determine whether a vehicle is a known vehicle and therefore may update the respective entry data database 358 or exit data database 360 ​​using the vehicle identifier or using an indication that the vehicle is known and has a profile in the profile database 356.

[0044] The event matching module 218, in response to the exit detection module 214 detecting an exit event, determines whether a match exists between the detected exit event and the entry event. That is, the event matching module 218 determines whether the vehicle corresponding to the entry event is the same as the vehicle corresponding to the exit event. In some embodiments, the event matching process may be as simple as determining whether the vehicle corresponding to the exit event is known and matching the exit event with the entry event corresponding to the known vehicle. The event matching module 218 determines whether the vehicle corresponding to the exit event is known by using the vehicle recognition module 216, which relies on a vehicle identifier (e.g., license plate) to search the profile database 356 for a vehicle profile. In response to the determination that the vehicle corresponding to the exit event is a known vehicle, the event matching module 218 may use the vehicle identifier to search either the entry data database 358 or the profile database 356 to determine whether there is a record of the known vehicle entering the parking facility. In response to the detection of a known vehicle entry event, the event matching module 218 matches the exit event with the entry event.

[0045] However, license plate reading is not perfect even when using the second machine learning model described. Factors such as low image quality, low frame rate, lighting conditions (e.g., glare, low light), debris, dirt, or weather-related conditions (e.g., snow, ice, rain, mud) can obscure license plate information and make it difficult to read license plates. Thus, it may be impossible for the vehicle recognition module 216 to determine whether a vehicle is known based on the vehicle identifier, and as a result, the event matching module 218 may not be able to match exit events with entry events using only the vehicle identifier.

[0046] In some embodiments, the event matching module 218 matches exit events with entry events by comparing information in the exit event dataset with information in the entry event dataset from the entry event set. The event matching module 218 determines a match between an exit event and an entry event from the set of entry events for which a heuristic is satisfied. For example, the event matching module 218 may determine that an exit event matches an entry event if the license plate number and geographic name match. Since license plate numbers are not unique identifiers and can be duplicated as long as geographic names are unique, if the exit and entry events match between license plate numbers but not between geographic names, the event matching module 218 will not match the exit event with the entry event. As previously mentioned, license plate reading is not perfect, so matching may not be found by the event matching module 218 using only the vehicle identifier. For this purpose, matching may be determined based on other identifying information from the exit and entry event datasets, such as identifying partial matches of geographical name and / or other vehicle attributes to match, such as manufacturer, model, and color. Any heuristics can be programmed to determine whether a match has occurred.

[0047] The event matching module 218 may filter entry events for comparison with exit events. For example, the event matching module 218 may compare exit events with only entry events that do not match, entry events associated with the same parking facility, or entry events with a timestamp within a threshold time window (e.g., within a 24-hour time window). The event matching module 218 may filter entry events so that the set of entry events includes events associated with vehicles of the same type (e.g., car or truck), color, or model as the vehicle associated with the entry event.

[0048] In response to the detection of a match, the event matching module 218 may instruct the parking control server 130 to indicate in the profile database 356, the entry data database 358, or the exit data database 360 ​​that a vehicle has exited the facility. For example, the event matching module 218 may instruct the parking control server 130 to delete the vehicle's entry and exit events, or to archive them in separate databases. In some embodiments, in response to the detection of a match, the event matching module 218 may raise the gate 114 (for example, if the gate 114 is a physical gate rather than a logical boundary), thus allowing the vehicle to exit the facility.

[0049] In response to not finding a match, the event matching module 218 may expand the set of entry events that an exit event may match and retry the matching process. For example, the event matching module 218 may expand the set of entry events to include entry events associated with parking facilities beyond the parking facility associated with the exit event, such as parking facilities within a threshold distance from the parking facility associated with the exit event. In another example, the event matching module 218 may expand the time window to which entry events are associated to include, for example, entry events that occurred within one month rather than within one day.

[0050] In some embodiments, in response to the failure to detect a match between an exit event and an entry event, the event matching module 218 may refer to the matching resolution module 220.

[0051] The matching resolution module 220 resolves the matching between exit events and unmatched entry events. Unmatched entry events are entry events for vehicles for which the entry detection module 212 was unable to identify the vehicle identifier. The matching resolution module 220 may determine the exit feature vectors corresponding to the exit events (e.g., by the exit detection module 214) or read them (e.g., from the exit data database 360). The matching resolution module 220 may determine the set of entry feature vectors corresponding to the set of unmatched entry events (e.g., by the entry detection module 212) or read them (e.g., from the entry data database 358). The matching resolution module 220 may input the sets of exit feature vectors and entry feature vectors into an unsupervised machine learning model.

[0052] An unsupervised machine learning model may output a matching score for each entry feature vector. The matching score may represent how well an entry event matches an exit event, with better matches having higher matching scores. In these embodiments, the matching resolution module 220 may match exit events with entry events based on the matching scores. For example, the matching resolution module 220 may automatically match the entry event with the highest matching score with the exit event. In other embodiments, the matching resolution module 220 may compare the match scores to a threshold score. In response that the highest match score exceeds the threshold score, the matching resolution module 220 may determine that the entry event with the highest match score matches the exit event. In response that the match score does not exceed the threshold score, the matching resolution module 220 may determine that there is no match for the exit event. In some embodiments, the matching resolution module 220 may compare the difference between the two highest match scores to a threshold difference and match the entry event with the highest match score to the exit event in response only to differences that exceed the threshold difference. Therefore, if the top two entry events match well with the exit event (for example, with a match score within a threshold difference from each other), the matching resolution module 220 may determine that there is no match for the exit event. In other embodiments, the matching resolution module 220 may, for display purposes, provide a subset of entry events for the administrator to manually select a match for the exit event.

[0053] In some embodiments, the matching resolution module 220 can resolve unmatched entry events without waiting for matching exit events. To do so, the matching resolution module 220 can match unmatched entry events with previous entry events, where the previous entry events correspond to known vehicles. The matching resolution module 220 can determine or read entry feature vectors corresponding to unmatched entry events and determine or read sets of entry feature vectors corresponding to previous entry events (e.g., from entry data database 358). The matching resolution module 220 can input the entry feature vectors corresponding to unmatched entry events and the sets of entry feature vectors corresponding to previous entry events into an unsupervised machine learning model. The unsupervised machine learning model can output a matching score for each entry feature vector corresponding to a previous entry event.

[0054] The matching model used by the matching resolution module 220 to resolve the matching between exit events and unmatched entry events is described as an unsupervised machine learning model, but this is only illustrative. The matching resolution module 220 may use other types of models to generate the entry feature vectors. For example, the matching resolution module 220 may use a mathematical model that uses cosine similarity to calculate the similarity between the exit feature vector and the entry feature vector.

[0055] The matching resolution module 220 may select a set of previous entry events. The matching resolution module 220 may select entry events detected by the entry detection module 212 within a time window, such as a window of the past three days. The matching resolution module 220 may select entry events that occurred at the same parking facility as the unmatched entry event. The matching resolution module 220 may select entry events that have vehicles of the same type (e.g., truck, SUV, sedan), model, or color as the vehicle corresponding to the unmatched entry event. In some embodiments, the matching resolution module 220 may start by selecting a smaller set of previous entry events with a higher probability of matching (e.g., entry events that occurred at the same parking facility in the past three days), and in response to failing to resolve a match between the unmatched entry event and the selected set of previous entry events, it may iteratively select larger and larger sets of previous entry events (e.g., entry events that occurred within the past month at parking facilities within 20 miles of the parking facility associated with the unmatched entry event) to retry the matching process. The matching resolution module 220 may use metrics such as retention to further inform the selection of a set of previous entry events. For example, if retention (e.g., the percentage of vehicles returning to the same parking space) is 80% in 11 months, the matching resolution module 220 may select a set of previous entry events as entry events that occurred at the same parking space within the last month. However, if retention is 30% in 1 month, the matching resolution module 220 may select a set of previous events as entry events that occurred in a group of parking spaces within the last month (e.g., within the same zip code, within a threshold distance) instead of the same parking space. By using an iterative search process to check sets of previous events that are more likely to match before expanding to a larger set of previous events and checking, the matching resolution module 220 may save time and computational resources (e.g., processing power, storage, etc.).

[0056] In some embodiments, in response to the event matching module 218 or the matching resolution module 220 matching an exit event with an entry event, the edge device 110 may update the parking control server's profile database 356 with any or all events, datasets, or feature vectors describing the vehicle. If the vehicle does not have a profile in the profile database 356, the edge device 110 may request the parking control server 130 to create a profile for the vehicle. If the vehicle has an existing profile in the profile database 356, the edge device 110 may request the parking control server 130 to update the profile with new information (events, datasets, feature vectors) corresponding to the vehicle. In some embodiments, the edge device 110 may update the entry data database 358 and exit data database 360 ​​to reflect the matching between the exit and entry events (e.g., deleting an entry or indicating that the events matched).

[0057] In response to the event matching module 218 or the matching resolution module 220 failing to detect a match, the edge device 110 or the parking control server 130 may provide a message to the user for displaying the vehicle corresponding to the exit event. The message may include an instruction that the user's vehicle could not be matched and / or a request to the user to manually enter vehicle information (e.g., license plate information) or create a profile. The parking equipment may display the message on a screen, for example, a screen located at the exit gate.

[0058] The violation detection module 222 detects violations caused by vehicles and triggers corrective actions in response to the detection of such vehicle entry. Violations may be violations of rules associated with the parking facility. A non-exclusive set of examples of violations may include damaging the gates of the parking facility (e.g., hitting or breaking through the entry or exit gates), damaging other vehicles within the parking facility, entering the parking facility without a profile associated with the vehicle, speeding within the parking facility, occupying two or more parking spaces, parking outside a parking space, or remaining within the parking facility for an extended period (e.g., overnight, beyond closing time, or for an excessively long period). In some embodiments, the violation detection module 222 may detect violations caused by users of the parking facility, both users associated with vehicles and users not associated with vehicles. User-initiated violations may include, for example, damaging a vehicle, trespassing on a vehicle, or theft of a vehicle.

[0059] The violation detection module 222 may detect violations based on sensor data. Sensor data may include data from camera 112, sensor 118 mounted on gate 114, parking sensors, acoustic sensors, speedometer, or any other type of sensor in the parking facility. Parking sensors detect when a vehicle is in a parking space. Examples of parking sensors include magnetometers, ultrasonic sensors, or optical sensors. The violation detection module 222 may use different sensors for different types of violations. For example, the violation detection module 222 may use sensor 118 to detect whether the gate has been moved from one of its operating states (e.g., open, closed) to a slightly open state, which may indicate that a vehicle has hit the gate. In another example, the violation detection module 222 may use acoustic sensors to detect when a vehicle has entered (e.g., by detecting glass shattering sounds or car alarms).

[0060] In some embodiments, the violation detection module 222 may use a combination of multiple sensors to detect violations. For example, the violation detection module 222 may use a combination of camera 112 and parking sensors to determine whether a vehicle is in two or more parking spaces. The violation detection module 222 may detect a violation in response to two or more parking sensors for two or more adjacent parking spaces detecting that a parking space has transitioned from an empty state (e.g., no vehicle detected) to an occupied state (e.g., a vehicle detected) within a threshold time. The violation detection module 222 may use data from camera 112 to determine whether the two instances of parking sensors for adjacent parking spaces included a parking sensor that detected two or more separate vehicles that had parked simultaneously, or a parking sensor that detected a single vehicle occupying multiple parking spaces. In another example, the violation detection module 222 may use an acoustic sensor to detect glass shattering sounds and car alarms, and use camera 112 to confirm a violation involving a user entering a vehicle.

[0061] In some embodiments, the violation detection module may use a mobile camera system. A non-exclusive set of examples of mobile camera systems includes cameras on wheels (e.g., on a vehicle), cameras configured to move along wires or beams running across the ceiling, and / or drone cameras. The violation detection module 222 may instruct the mobile camera system to navigate to the location of the violation. For example, the violation detection module 222 may instruct the mobile camera system to navigate to a viewpoint including the adjacent parking space described above, capture an image of the adjacent parking space, and determine whether a vehicle is occupying the adjacent parking space. In some embodiments, the violation detection module 222 may instruct the mobile camera system to navigate to the location of the violation in response to sensor data from another sensor that detects the violation (e.g., a parking sensor). In some embodiments, the violation detection module 222 may instruct the mobile camera system to move periodically through the parking facility while scanning for violations. To detect violations, a mobile camera system can be more efficient than a system with many stationary cameras because it reduces the resources required to maintain the cameras (e.g., powering the cameras while the parking facility is open) by deploying cameras throughout the entire parking facility. Furthermore, by triggering the navigation of the mobile camera system in response to the detection of specific sensor data, the processing of fuel, energy, and images from the mobile camera system is minimized to only the scenario in which a potential violation is first detected, thereby improving efficiency.

[0062] In some embodiments, the violation detection module 222 may log the violation to the violation database 362 along with other information associated with the violation (e.g., a timestamp).

[0063] The fingerprint generation module 224 generates a vehicle fingerprint in response to the detection of a violation. The vehicle fingerprint of a violating vehicle may include a feature vector corresponding to the vehicle, a "violation feature vector." The fingerprint may include other information associated with the vehicle, such as a vehicle identifier or various vehicle parameters. The fingerprint generation module 224 generates a vehicle fingerprint by inputting a depiction of the vehicle into a model (e.g., a supervised machine learning model). The depiction of the vehicle may include an image containing the vehicle, such as one captured by camera 112. The model may be a supervised machine learning model or any other model described with respect to the intrusion detection module 212, and therefore may be trained as described with respect to the intrusion detection module 212. As output from the model, the fingerprint generation module 224 receives a violation feature vector describing the vehicle involved in the detected violation. The violation feature vector may include multiple embeddings, each embedding derived from one or more dimensions of the depiction of the vehicle. In some embodiments, the fingerprint generation module 224 adds the violation feature vector to a violation database, such as violation database 362. In some embodiments, the fingerprint generation module 224 generates a vehicle fingerprint without detecting a violation.

[0064] In embodiments where the violation detection module 222 detects violations caused by a user, the fingerprint generation module 224 may determine the vehicle associated with the user and generate a vehicle fingerprint of the user's vehicle. To do so, the fingerprint generation module 224 may read the violation timestamp from the violation database 362. The fingerprint generation module 224 may access sensor data (e.g., RFID reader of a locked pedestrian door to the parking facility, camera 112) within a threshold time window near the violation timestamp. Using the sensors, the fingerprint generation module 224 may determine how the user entered the parking facility. In response to the determination that the user entered the parking facility through an RFID-enabled pedestrian door, the fingerprint generation module 224 may access logs associated with the pedestrian door and access a set of user authentication information from which the user gained access to the parking facility. User authentication information may include user information such as user profile information, through which the fingerprint generation module 224 may obtain a vehicle identifier associated with the user. In response to a user's decision that a vehicle has entered a parking facility, the fingerprint generation module 224 may retrieve vehicle information stored in an entry log associated with the vehicle. Such embodiments are further described with reference to Figure 7A.

[0065] In some embodiments, the fingerprint generation module 224 determines whether a vehicle is unknown and generates a vehicle fingerprint in response to the vehicle being unknown. A vehicle may be determined to be unknown by the fingerprint generation module 224 in response to the determination that the vehicle does not exist in the profile database 356, or if the vehicle's vehicle identifier (e.g., geographic name and license plate number) is not recognized. To determine whether a vehicle is unknown, the fingerprint generation module 224 may extract a vehicle identifier from the vehicle using a model similar to the supervised machine learning model described with respect to the entry detection module 212. The fingerprint generation module 224 may search the profile database 356 using the vehicle identifier as an index. In response to the determination that the vehicle is known, the fingerprint generation module 224 may use an existing feature vector of the vehicle (e.g., an entry or exit feature vector stored in the profile database 356) as a violation feature vector for the vehicle fingerprint.

[0066] The entry monitoring module 226 monitors the entry of a vehicle associated with a violation into one of several parking facilities. For each parking facility, the entry monitoring module 226 may receive a dataset and / or entry feature vectors corresponding to the vehicle entering the parking facility from the entry detection module 212. The entry monitoring module 226 may compare the vehicle fingerprint stored in the violation database with the vehicle's entry feature vectors. In some embodiments, the entry monitoring module 226 may input a set of entry feature vectors and violation feature vectors (e.g., from the vehicle fingerprint) into a model and receive a match score for each violation feature vector as output from the model. The model may be similar to the unsupervised machine learning model in the matching resolution module 220. Similar to the matching resolution module 220, the entry monitoring module 226 may match the entry feature vectors to violation feature vectors from the set of violation feature vectors based on the matching score.

[0067] The corrective action module 228 triggers a corrective action in response to the entry monitoring module 226 detecting the entry of a vehicle associated with a violation. Examples of corrective actions include issuing a violation (e.g., a parking ticket or other violation notice), contacting the parking facility manager, contacting an external agency (e.g., law enforcement), deploying an exit or entry blocking device to prevent the vehicle from moving within the parking facility (e.g., a metal bar, a tire breaker, closing or preventing a gate from opening), displaying a message to the user associated with the vehicle, or otherwise requesting action from the user (e.g., email, text, or push notification). Exemplary corrective actions are shown in Figure 7C.

[0068] In some embodiments, the corrective action module 228 may trigger different corrective actions for different types of violations. Thus, the corrective action module 228 may determine the type of violation and send corrective commands that result in a corrective action based on the violation type. For example, for a violation of entering a parking facility without a profile associated with the vehicle, the corrective action module 228 may trigger an action prompting the vehicle user to enter profile details (e.g., contact information, license plate number). In another example, for a violation of occupying multiple parking spaces, the corrective action module 228 may trigger a corrective action that assigns the vehicle to use multiple parking spaces. For a violation of damaging a gate, the corrective action module 228 may trigger a corrective action that contacts the parking facility manager. In some embodiments, the corrective action module 228 may trigger different corrective actions depending on the parking facility. The corrective action module 228 may store corrective action preferences for different parking facilities in, for example, the parking facility preference storage device 364 of the parking control server 130. In some embodiments, the corrective action module 228 may trigger multiple corrective actions. For example, the corrective action module 228 may trigger two corrective actions at once. Additionally or alternatively, the corrective action module 228 may trigger a first corrective action and wait for a threshold time window before canceling or triggering a second corrective action. For example, the corrective action module may issue a message to the user and wait 10 minutes before contacting law enforcement. In response that the user has resolved the issue within the threshold time window, the corrective action module 228 may cancel the second corrective action. In response that the user has not resolved the issue within the threshold time window, the corrective action module 228 may trigger the second corrective action.

[0069] The corrective action module 228 may remove a vehicle from the violation database. The corrective action module 228 may remove a vehicle from the violation database in response to a request from the parking facility manager or in response to the vehicle user taking a corrective action (e.g., creating a profile, addressing a violation notice, etc.).

[0070] Figure 3 illustrates one embodiment of an exemplary module operated by a parking control server. As depicted in Figure 3, the parking control server 130 includes a vehicle identification module 332, a vehicle orientation module 334, a parameter determination model training module 336, a license plate model training module 338, an event reading module 340, a model database 352, a profile database 356, a training example database 354, an entry data database 358, an exit data database 360, a violation database 362, and a parking equipment preference storage device 364. The modules and databases depicted in Figure 3 are merely illustrative, and fewer or more modules and / or databases may be used to accomplish the activities disclosed herein. Furthermore, although the modules and databases are depicted within the parking control server 130, they may be distributed entirely or partially to edge devices 110, which may, entirely or partially, perform any activities described with respect to the parking control server 130. Furthermore, modules and databases can be maintained separately from any entities depicted in Figure 1 (for example, the decision model training module 336 and the license plate training module 338 can be stored completely offline from the parking control server 130, or in separate entities).

[0071] The vehicle identification module 332 identifies a vehicle using a first machine learning model described with respect to the entry detection module 212. Specifically, the vehicle identification module 332 accesses the first machine learning model from the model database 352, applies input images and / or any other data to the machine learning model, and receives vehicle parameters from there. The vehicle identification module 332 operates in a scenario where the images are transmitted to the parking control server 130 for processing rather than being processed by the edge device 110. Similarly, the vehicle orientation module 334 determines the orientation of a vehicle in an image captured at the edge device 110 by the camera 112, in the same manner as described above with respect to the entry detection module 212, except that it uses images and / or other data received at the parking control server 130 as input rather than being processed by the edge device 110.

[0072] The parameter determination model training module 336 trains a first machine learning model to predict vehicle parameters in the manner described above with respect to the entry detection module 212. The parameter determination model training module may also train the first machine learning model to predict the direction of the vehicle. The parameter determination model training module may access training examples from the training example database 354 and store the models in the model database 352. Similarly, the license plate model training module 338 may train a second machine learning model using training examples stored in the training example database 354 and store the trained models in the model database 352.

[0073] The event reading module 340 receives a command from the event matching module 218 to read entry data from the entry data database 358 that matches the detected exit data, and returns to the event matching module 218 a decision on whether at least partially matching data and / or a match was found. The event reading module 340 stores the exit data in the exit data database 360 ​​as needed.

[0074] The profile database 356 stores profile data about the encountered vehicle. For example, identification information and / or license plate information may be used to index the profile database 356. When a vehicle enters and exits the facility, the profile database 356 may capture a profile for each vehicle, storing those entry and exit events. The profile may indicate the vehicle's owner and / or driver, and may indicate contact information about those users. The event reading module 340 may read the contact information when an event is detected and initiate communication with the user (e.g., a welcome message to the parking facility or other information related to how to use the facility).

[0075] Figure 4 is a block diagram illustrating the components of an exemplary machine capable of reading instructions from a machine-readable medium and executing them within a processor (or controller). Specifically, Figure 4 shows a schematic representation of a machine in an exemplary form of a computer system 400, in which program code (e.g., software) can be executed to cause the machine to perform one or more of the methodologies discussed herein. The program code may consist of instructions 424 that can be executed by one or more processors 402. In alternative embodiments, the machine may operate as a standalone device or be connected to other machines (e.g., networked). In a networked deployment, the machine may operate within the capacity of a server machine or client machine in a server / client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0076] A machine can be a computing system capable of executing instructions 424 (sequentially or otherwise) that define the actions to be taken by that machine. Furthermore, although only a single machine is illustrated, the term “machine” should also be understood to include any set of machines that execute instructions 424 individually or together to implement one or more of the methodologies discussed herein.

[0077] An exemplary computer system 400 includes one or more processors 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), a field-programmable gate array (FPGA)), main memory 404, and static memory 406, which are configured to communicate with each other via a bus 408. The computer system 400 may further include a visual display interface 410. The visual interface may include software drivers that enable (or provide) the rendering of a user interface on a screen, either directly or indirectly. The visual interface 410 may interact with a touch-enabled screen. The computer system 400 may also include an input device 412 (e.g., a keyboard, a mouse), a cursor control device 414, a storage unit 416, a signal generating device 418 (e.g., a microphone and / or speaker), and a network interface device 420, which are also configured to communicate via the bus 408.

[0078] The memory unit 416 includes a machine-readable medium 422 (e.g., a magnetic disk or solid-state memory) in which instructions 424 (e.g., software) embodying any one or more of the methodologies or functions described herein are stored. The instructions 424 (e.g., software) may also reside, in whole or at least in part, in main memory 404 or in processor 402 (e.g., in the processor's cache memory) during execution.

[0079] Figure 5 illustrates one embodiment of an exemplary process for matching exit events with entry events. Alternative embodiments may include more, fewer, or different steps than those shown in Figure 5, and the steps may be performed in a different order than those shown in Figure 5. Process 500 operates using one or more processors (e.g., the edge device 110 and / or the processor 402 of the parking control server 130) that execute instructions (e.g., instruction 424) causing one or more modules to perform their respective operations.

[0080] Process 500 begins with the edge device 110 generating an exit event 510 representing a vehicle exiting the parking facility (for example, using the exit detection module 214). The edge device 110 may use the camera 112 to detect the exit event, capture a series of images over time, determine a dataset corresponding to the vehicle in the exit event, and store the exit event along with the dataset corresponding to the vehicle, images featuring the vehicle, or other data.

[0081] The edge device 110 determines whether an exit event matches an entry event that also represents a vehicle (for example, using the event matching module 218). To make this determination, the edge device compares the dataset stored with the exit events with the dataset stored with the entry events. The edge device determines a match between the exit and entry events if a heuristic is satisfied, for example, if the vehicle identifiers in each dataset match, or if the combination of characteristics matches.

[0082] In response to the decision 520 that an exit event does not match an entry event, the edge device 110 inputs a vehicle description into a supervised machine learning model 530 and receives an exit feature vector (for example, using the exit detection module 214) as output from the supervised machine learning model 540. The vehicle description may include an image containing the vehicle, for example, as captured by camera 112. The exit feature vector may include an embedding derived from the dimensions of the vehicle description.

[0083] The edge device 110 reads a set of entry feature vectors 550. The edge device 110 may pre-calculate the entry feature vectors, for example, at the time of entry. Alternatively, the edge device 110 may calculate the entry feature vectors in response to a decision 520. In some embodiments, the set of entry feature vectors includes a set of entry feature vectors associated with unmatched entry events, where unmatched entry events are entry events that do not match with exit events.

[0084] The edge device 110 inputs a set of exit and entry feature vectors into an unsupervised machine learning model 560 and receives a set of matching scores as output from the unsupervised machine learning model 570, including a matching score for each entry feature vector in the set of entry feature vectors. The edge device 110 matches exit events with one or more unmatched entry events based on the matching scores 580. The edge device 110 may automatically match exit events with entry events, for example, by automatically matching an exit event with the entry event having the highest matching score. In some embodiments, the edge device 110 may provide a subset of entry events for display purposes, allowing an administrator to manually select matches for exit events. For example, the edge device 110 may provide entry events with the three highest matching scores.

[0085] Figure 6 illustrates one embodiment of an exemplary process for detecting and responding to violations caused by a vehicle. Alternative embodiments may include more, fewer, or different steps than those illustrated in Figure 6, and the steps may be performed in a different order than those shown in Figure 6. Process 600 operates using one or more processors (e.g., processor 402 of edge device 110 and / or parking control server 130) that execute instructions (e.g., instruction 424) causing one or more modules to perform their respective operations.

[0086] Process 600 begins with the edge device 110 detecting a violation caused by a vehicle in the parking facility 610. The edge device 110 may detect violations based on sensor data, such as data from a camera 112, a sensor 118 mounted on the gate 114, a parking sensor, or any other type of sensor in the parking facility (for example, using a violation detection module 222). The edge device 110 may detect violations by using two or more sensors in combination.

[0087] The edge device 110 inputs a vehicle description into a supervised machine learning model 630 and receives a violation feature vector of the vehicle as output from the supervised machine learning model 640, thereby generating a vehicle fingerprint corresponding to the vehicle that caused the violation (for example, using a fingerprint generation module 224) 620. The vehicle description may include an image containing the vehicle, for example, captured by a camera 112. The violation feature vector may include embeddings derived from the dimensions of the vehicle description.

[0088] The edge device 110 monitors vehicle entry into one of several parking spaces (for example, using an entry monitoring module 226). The edge device 110 may monitor vehicle entry by comparing a vehicle entry feature vector (for example, determined by an entry detection module 212) with a set of violation feature vectors.

[0089] In response to the detection of a vehicle entering a given facility among the parking facilities, the edge device 110 triggers a corrective action 660. The edge device 110 may trigger a corrective action based on the type of violation.

[0090] Figures 7A–7C depict exemplary embodiments of a parking facility and a movable gate. As depicted in Figure 7A, the parking facility 700 includes a set of parking spaces 702 in which a vehicle 705 (e.g., an automobile) can park. The parking facility 700 includes sensors such as a parking sensor 715 and a camera 112. The parking sensor 715 may be located within the parking space 702 to detect when a vehicle 705 is present. As depicted on the left side of the parking facility 700, the parking facility 700 includes a gate 114. The lower gate 114 allows a vehicle 705 to enter the parking facility 700 from the street 720 via an entry lane 740, and the upper gate 114 allows a vehicle 705 to exit the parking facility 700 via an exit lane 735.

[0091] The parking facility 700 may include a pedestrian door 710 that allows pedestrians to enter, for example, from a sidewalk 730. In response to the edge device 110 receiving a set of user authentication information from the user, the pedestrian door may be locked and RFID may be activated so that the user can enter through the pedestrian door. Exemplary user authentication information may include user personal information, contact information, account information, and vehicle information (e.g., manufacturer, model, color, license plate).

[0092] Figure 7A also depicts a violating vehicle 706. The edge device 110 may determine that vehicle 706 is a violating vehicle due to the manner in which the vehicle is parked, where the vehicle is referring to two parking spaces 702 instead of one parking space 702. In response to the detection of a violation, the edge device 110 may trigger a corrective action that assigns the vehicle to use multiple parking spaces. In response to detecting several violations, the edge device 110 may trigger a corrective action that deploys an exit blocking device (e.g., gate 114) to prevent the vehicle 705 (or 706) from moving out of the parking facility 700.

[0093] Figures 7B and 7C depict an embodiment of a parking facility 700 in which a two-gate system is implemented in the entry lane 740. The two-gate system includes a first gate 113, to which a camera 112 is directed, and a second gate 115. Between the first gate 113 and the second gate 115 is a secondary zone 745. The secondary zone 745 includes access to the exit lane 735 (e.g., by crossing a dashed line). Figure 7B shows the operation of the two-gate system in response to a non-violating vehicle (e.g., vehicle 705) attempting to enter the parking facility. In Figure 7B, in response to detecting vehicle 705 at the first gate 113, the edge server 110 may open the first gate 113, allowing vehicle 705 to enter the secondary zone 745. While in the secondary zone 745, camera 112 may capture an image of vehicle 705. In response to a determination (for example, through the entry monitoring module 226) that vehicle 705 is not a violating vehicle, the edge device 110 may open the second gate 115, allowing vehicle 705 to enter the parking facility 700. Figure 7C illustrates the operation of the two-gate system in response to a violating vehicle (for example, vehicle 706) attempting to enter the parking facility. In Figure 7C, in response to the detection of violating vehicle 706 at the first gate 113, the edge server 110 may open the first gate 113, allowing violating vehicle 706 to enter the secondary zone 745. While in the secondary zone 745, camera 112 may capture an image of violating vehicle 706. In response to a determination (for example, through the entry monitoring module 226) that vehicle 706 is a violating vehicle, instead of opening the second gate 115 as the edge device 110 did with vehicle 705, the edge device 110 may trigger a corrective action. For example, as a corrective action, the edge device 110 may provide the user of the offending vehicle 706 with a message prompting the user to route the offending vehicle 706 to the exit lane 735 for display at the second gate 115.

[0094] Additional configuration considerations

[0095] Throughout this specification, multiple instances may implement a component, operation, or structure described as a single instance. While individual operations of one or more methods are illustrated and described as separate operations, one or more of these operations may be performed in parallel, and nothing requires the operations to be performed in the order they are illustrated. Structures and functions presented as separate components in exemplary configurations may be implemented as combined structures or components. Similarly, structures and functions presented as single components may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of this specification.

[0096] Embodiments described herein include logic or several components, modules, or mechanisms. Modules can consist of either software modules (e.g., code embodied on a machine-readable medium and processor-executable code) or hardware modules. Hardware modules are tangible units capable of performing certain operations and can be configured or arranged in certain ways. In exemplary embodiments, one or more computer systems (e.g., standalone, client, or server computer systems) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as hardware modules that operate to perform certain operations as described herein.

[0097] In various embodiments, hardware modules may be mechanically or electronically implemented. For example, a hardware module is a tangible component that may comprise a dedicated network or logic permanently configured to perform a certain operation (e.g., as a special processor such as a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC)). A hardware module may also comprise a programmable logic or network temporarily configured by software to perform a certain operation (e.g., contained within a general-purpose processor or other programmable processor). It should be understood that the decision to mechanically implement a hardware module within a dedicated and permanently configured network or a temporarily configured network (e.g., configured by software) may depend on cost and time considerations.

[0098] The implementation of certain operations may be distributed among one or more processors and deployed across several machines, rather than residing within a single machine. In some exemplary embodiments, one or more processors or processor implementation modules may be located within a single geographical location (e.g., a home environment, an office environment, or a server farm). In other exemplary embodiments, one or more processors or processor implementation modules may be distributed across several geographical locations.

[0099] Several parts of this specification are presented in terms of algorithms or symbolic representations of operations relating to data stored as bits or binary digital signals in machine memory (e.g., computer memory). These algorithms or symbolic representations are examples of techniques used by those skilled in the field of data processing to convey the importance of their operations to others skilled in the art. As used herein, “algorithm” is a self-consistent sequence of operations or similar operations that lead to a desired result. In this context, algorithms and operations involve the physical manipulation of physical quantities. Usually, but not always, such quantities may take the form of electrical, magnetic, or optical signals that can be stored, accessed, transferred, combined, compared, or otherwise manipulated by machines. Sometimes, mainly for reasons of general use, it is convenient to refer to such signals using words such as “data,” “content,” “bit,” “value,” “element,” “symbol,” “character,” “term,” “number,” “numeral,” or equivalents. However, these words are merely labels for convenience and should be associated with appropriate physical quantities.

[0100] Unless otherwise specifically stated, discussions herein using words such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” or equivalents may refer to the actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as a physical (e.g., electronic, magnetic, or optical) quantity in one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0101] Upon careful reading of this disclosure, those skilled in the art will understand that, through the principles disclosed herein, further additional applications to systems and processes for seamless entry and exit to parking facilities blocked by movable gates will be illustrated and described, but the embodiments disclosed herein will be understood to be alternative structures and functional designs. Therefore, it should be understood that we are not limited to specific embodiments and the precise organization and components disclosed herein. It will be apparent to those skilled in the art that various modifications, alterations, and variations may be made in the arrangement, operation, and details of the methods and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

Claims

1. A method, wherein the said method is This triggers an exit event representing a vehicle leaving the parking facility, To determine whether the aforementioned exit event matches an entry event that also represents the aforementioned vehicle, In response to the decision that the exit event does not match the entry event, Inputting the description of the aforementioned vehicle into a supervised machine learning model, The output from the supervised machine learning model is to receive an exit feature vector comprising multiple embeddings, each derived from the dimensions of the vehicle's description. The method involves reading a set of entry feature vectors, where the entry feature vectors are generated using a given vehicle represented by a given entry event of a group of unmatched entry events, and each of the unmatched entry events is associated with a vehicle that is not paired with a corresponding exit event. Inputting the set of exit feature vectors and entry feature vectors into an unsupervised machine learning model, The output from the unsupervised machine learning model is to receive a plurality of matching scores, each of which is a matching score for each of the entry feature vectors in the set of entry feature vectors, and Based on the aforementioned multiple matching scores, the exit event is matched to one or more of the previously unmatched entry events. To do Methods that include...

2. The method according to claim 1, wherein the exit event comprises one or more images of the vehicle exiting the parking facility.

3. Inputting the aforementioned description of the vehicle into the supervised machine learning model is, To separate a first image portion containing the vehicle from one or more of the aforementioned images, To remove a second image portion from the one or more images that does not contain the vehicle. The method according to claim 2, including the method described in claim 2.

4. The method according to claim 1, further comprising matching the exit event to the entry event in response to a determination that the exit event matches to an entry event that also represents the vehicle.

5. The method according to claim 1, wherein determining that the exit event matches an entry event that also represents the vehicle includes comparing the vehicle identifier of the vehicle with a set of vehicle identifiers.

6. Comparing the vehicle identifier of the aforementioned vehicle with the set of vehicle identifiers means Inputting the license plate identifier and the set of license plate identifiers corresponding to the vehicle into a machine learning model, The output from the machine learning model is to receive a plurality of matching scores, each of which has a matching score for each license plate identifier in the set of license plate identifiers. The method according to claim 5, including the method described in claim 5.

7. The method according to claim 5, wherein the license plate identifier consists of vertical and horizontal characters.

8. An entry event is generated representing a vehicle entering the aforementioned parking facility, To determine whether the aforementioned vehicle is within the known candidate set of vehicles, In response to the determination that the vehicle is not in the known candidate set of vehicles, the approach feature vector corresponding to the vehicle is read out. The method according to claim 1, further comprising:

9. The method according to claim 8, wherein the set of candidate known vehicles comprises vehicles having profiles stored in a profile database.

10. The method according to claim 1, further comprising providing a user interface for display, the user interface comprising one or more matches between the exit event and the one or more unmatched entry events.

11. The method according to claim 1, wherein matching the exit event with one or more of the unmatched entry events includes automatically matching the exit event with the unmatched entry event having the highest matching score.

12. The method according to claim 1, wherein the matching score represents how well the entry feature vector matches the exit feature vector.

13. A non-temporary computer-readable medium having memory on which instructions are encoded, This triggers an exit event representing a vehicle leaving the parking facility, To determine whether the aforementioned exit event matches an entry event that also represents the aforementioned vehicle, In response to the decision that the exit event does not match the entry event, Inputting the description of the aforementioned vehicle into a supervised machine learning model, The output from the supervised machine learning model is to receive an exit feature vector comprising multiple embeddings, each derived from the dimensions of the vehicle's description. The method involves reading a set of entry feature vectors, where the entry feature vectors are generated using a given vehicle represented by a given entry event of a group of unmatched entry events, and each of the unmatched entry events is associated with a vehicle that is not paired with a corresponding exit event. Inputting the set of exit feature vectors and entry feature vectors into an unsupervised machine learning model, The output from the unsupervised machine learning model is to receive a plurality of matching scores, each of which is a matching score for each of the entry feature vectors in the set of entry feature vectors, and Based on the aforementioned multiple matching scores, the exit event is matched to one or more of the previously unmatched entry events. To do A non-temporary computer-readable medium containing instructions for performing a certain action.

14. The non-temporary computer-readable medium according to claim 13, wherein the exit event comprises one or more images of the vehicle exiting the parking facility.

15. The instruction for inputting the description of the vehicle into the supervised machine learning model is: To separate a first image portion containing the vehicle from one or more of the aforementioned images, To remove a second image portion from the one or more images that does not contain the vehicle. A non-temporary computer-readable medium according to claim 14, including instructions for performing the following.

16. The non-temporary computer-readable medium according to claim 13, further comprising an instruction to match the exit event to the entry event in response to a determination that the exit event matches to an entry event which also represents the vehicle.

17. The non-temporary computer-readable medium according to claim 13, wherein the instruction for determining that the exit event matches an entry event representing the vehicle also includes an instruction for comparing the vehicle identifier of the vehicle with a set of vehicle identifiers.

18. The instruction for comparing the vehicle identifier of the vehicle with a set of vehicle identifiers is: Inputting the license plate identifier and the set of license plate identifiers corresponding to the vehicle into a machine learning model, The output from the machine learning model is to receive a plurality of matching scores, each of which has a matching score for each license plate identifier in the set of license plate identifiers. A non-temporary computer-readable medium according to claim 17, including instructions for performing the following.

19. The non-temporary computer-readable medium according to claim 17, wherein the license plate identifier consists of vertical and horizontal characters.

20. A system, wherein the system is On top of that is memory with the instructions encoded, Equipped with one or more processors, When the one or more processors execute the instruction, This triggers an exit event representing a vehicle leaving the parking facility, To determine whether the aforementioned exit event matches an entry event that also represents the aforementioned vehicle, In response to the decision that the exit event does not match the entry event, Inputting the description of the aforementioned vehicle into a supervised machine learning model, The output from the supervised machine learning model is to receive an exit feature vector comprising multiple embeddings, each derived from the dimensions of the vehicle's description. The method involves reading a set of entry feature vectors, where the entry feature vectors are generated using a given vehicle represented by a given entry event of a group of unmatched entry events, and each of the unmatched entry events is associated with a vehicle that is not paired with a corresponding exit event. Inputting the set of exit feature vectors and entry feature vectors into an unsupervised machine learning model, The output from the unsupervised machine learning model is to receive a plurality of matching scores, each of which is a matching score for each of the entry feature vectors in the set of entry feature vectors, and Based on the aforementioned multiple matching scores, the exit event is matched to one or more of the previously unmatched entry events. To do A system that can perform actions including those mentioned above.

21. A method, wherein the said method is Using input from one or more sensors installed in one of several parking facilities, the system detects violations caused by vehicles. In response to detecting the aforementioned violation, The method involves inputting a description of the vehicle into a supervised machine learning model, wherein the description is derived from one or more images of the vehicle captured in the parking facility, and The vehicle's feature vector is received as output from the supervised machine learning model, wherein the feature vector comprises a plurality of embeddings, each describing the dimensions of the vehicle. This generates vehicle fingerprints, Using the vehicle fingerprint, the vehicle's entry into each of the multiple parking facilities is monitored. In response to detecting the entry of the vehicle into a given parking facility among the plurality of parking facilities, a corrective action is triggered. Methods that include...

22. The method according to claim 21, wherein the input includes an instruction that two or more adjacent parking spaces within the parking facility have transitioned from an empty state to an occupied state within a threshold time relative to each other.

23. Detecting the aforementioned violation means In response to detecting the aforementioned instruction, the movable camera system is instructed to navigate to a viewpoint that includes the two or more adjacent parking spaces and to capture images of one or more of the two or more adjacent parking spaces. To determine whether the vehicle occupies two or more adjacent parking spaces, To detect a violation response to the determination that the vehicle is occupying two or more adjacent parking spaces. The method according to claim 22, including the method described in claim 22.

24. The method according to claim 21, wherein generating the vehicle fingerprint is performed in response to the detection that the vehicle's license plate is not recognized.

25. The method of claim 24, wherein monitoring the entry of the vehicle includes monitoring the license plate of the vehicle if the license plate of the vehicle is recognized.

26. Inputting the aforementioned description of the vehicle into the supervised machine learning model is, To separate a first image portion containing the vehicle from one or more of the aforementioned images, To remove a second image portion from the one or more images that does not contain the vehicle. The method according to claim 21, including the method described in claim 21.

27. Triggering the aforementioned corrective action is To determine the type of violation, Transmitting a corrective command that brings about the corrective action based on the aforementioned violation type. The method according to claim 21, including the method described in claim 21.

28. The method according to claim 27, wherein the corrective command includes a command to raise a barrier device that prevents the vehicle from moving within the parking facility.

29. The method according to claim 27, wherein the corrective command includes a command for initiating a communication session with a law enforcement entity.

30. To determine whether the aforementioned vehicle is within the known candidate set of vehicles, In response to the determination that the vehicle is not among the candidate set of known vehicles, a vehicle fingerprint corresponding to the vehicle is generated. The method according to claim 21, further comprising:

31. A non-temporary computer-readable medium having memory on which instructions are encoded, Using input from one or more sensors installed in one of several parking facilities, the system detects violations caused by vehicles. In response to detecting the aforementioned violation, The method involves inputting a description of the vehicle into a supervised machine learning model, wherein the description is derived from one or more images of the vehicle captured in the parking facility, and The vehicle's feature vector is received as output from the supervised machine learning model, wherein the feature vector comprises a plurality of embeddings, each describing the dimensions of the vehicle. This generates vehicle fingerprints, Using the vehicle fingerprint, the vehicle's entry into each of the multiple parking facilities is monitored. In response to detecting the entry of the vehicle into a given parking facility among the plurality of parking facilities, a corrective action is triggered. A non-temporary computer-readable medium containing instructions for performing a certain action.

32. The non-temporary computer-readable medium according to claim 31, wherein the input includes an instruction that two or more adjacent parking spaces within the parking facility have transitioned from an empty state to an occupied state within a threshold time relative to each other.

33. The order for detecting the aforementioned violation is: In response to detecting the aforementioned instruction, the movable camera system is instructed to navigate to a viewpoint that includes the two or more adjacent parking spaces and to capture images of one or more of the two or more adjacent parking spaces. To determine whether the vehicle occupies two or more adjacent parking spaces, The system detects a violation in response to a determination that the vehicle is occupying two or more adjacent parking spaces. A non-temporary computer-readable medium according to claim 32, including instructions for performing the following.

34. The non-temporary computer-readable medium according to claim 31, wherein generating the vehicle fingerprint is performed in response to the detection that the vehicle's license plate is not recognized.

35. The non-temporary computer-readable medium according to claim 34, wherein monitoring the entry of the vehicle includes monitoring the license plate of the vehicle if the license plate of the vehicle is recognized.

36. Inputting the aforementioned description of the vehicle into the supervised machine learning model is, To separate a first image portion containing the vehicle from one or more of the aforementioned images, To remove a second image portion from the one or more images that does not contain the vehicle. A non-temporary computer-readable medium according to claim 31, including the following:

37. The instruction for triggering the corrective action is: To determine the type of violation, Transmitting a corrective command that brings about the corrective action based on the aforementioned violation type. A non-temporary computer-readable medium according to claim 31, including instructions for performing the following.

38. The non-temporary computer-readable medium according to claim 37, wherein the corrective command includes a command to raise a barrier device that prevents the vehicle from moving within the parking facility.

39. The non-temporary computer-readable medium according to claim 37, wherein the corrective command includes a command for initiating a communication session with a law enforcement entity.

40. A system, wherein the system is On top of that is memory with the instructions encoded, One or more processors and Equipped with, When the one or more processors execute the instruction, Using input from one or more sensors installed in one of several parking facilities, the system detects violations caused by vehicles. In response to detecting the aforementioned violation, The method involves inputting a description of the vehicle into a supervised machine learning model, wherein the description is derived from one or more images of the vehicle captured in the parking facility, and The vehicle's feature vector is received as output from the supervised machine learning model, wherein the feature vector comprises a plurality of embeddings, each describing the dimensions of the vehicle. This generates vehicle fingerprints, Using the vehicle fingerprint, the vehicle's entry into each of the multiple parking facilities is monitored. In response to the detection of a vehicle entering a given parking facility among the plurality of parking facilities, a corrective action is triggered. A system that can perform actions including those mentioned above.

Images