Vehicle parking contravention detection

Abstract

A method of detecting vehicle parking contraventions using a fixed camera system having an RGB and an IR sensor array, the arrays having a common field-of-view encompassing parking locations to be monitored. The sensor arrays capture a sequence of time spaced still images which are wirelessly transmitted to an administration service 5 over a transmission network 2. After transmission, it is determined from the images whether a vehicle is parked in said parking location. A trained machine and deep learning model analyse the images and identify properties and / or actions of the vehicle, using these to determine whether the vehicle is parked in contravention of predefined parking rules. A parking contravention procedure is initiated in response to a contravention, the procedure including obtaining, from the images, a vehicle registration number, and creating an evidentiary data bundle including the captured images. A composite image may be included in the evidentiary bundle, the image being generated by fusing an RGB and an IR image. The time spacing of the sequence of images may be greater than one second.

Description

Technical Field The present disclosure relates to vehicle detection and in particular to the detection of vehicles present in certain controlled and / or prohibited parking locations. Background Controlled and prohibited parking locations (referred to below simply as “controlled locations”) are ubiquitous and include locations within car parks where payment is required, company car parks available only to employees, and other designated areas which parking is restricted or forbidden. Conventionally, parking in such locations was monitored and controlled by human operatives, e.g. parking wardens, parking attendants / officers and police officers. Whilst this remains common, automated and semi-automated systems are now being used. For example, mobile and fixed cameras can be used to detect the presence of vehicles in controlled parking locations, with captured images being analysed using AN PR (Automatic Number Plate Recognition) technology to determine vehicles’ registration numbers (Reg. Nos). Whilst mobile, vehicle-mounted cameras are useful in some circumstances, their use is limited and of course a vehicle parked without permission or illegally will only be detected should the monitoring vehicle pass the corresponding controlled location. The same issue arises of course when parking attendants or other human operatives are used. The use of human operatives additionally limits the monitoring scope and time window such that round-the-clock monitoring is impractical in most cases. Fixed camera systems can often be a preferred option. Consider for example a controlled car park. Cameras may be used to detect the entry time of a vehicle into a car park and the exit time. Assuming that a driver has paid for his or her stay, the amount paid and corresponding duration may be compared to the actual duration as determined by the camera system. Any overstay may result in the issue of a PCN (Parking Charge Notice). In some cases such systems are integrated with barrier controlled entry and exits, for example to allow a barrier to be raised when a vehicle approaches the exit barrier and the system determines that the driver has paid the correct amount. Whilst the described fixed camera system works well for its intended use, it is not necessarily suited to detecting other types of parking contraventions in controlled locations. Such locations might include, for example, disabled parking spaces, spaces where parking is prohibited to avoid obstruction, no-parking zones, and drop-off / pick-up zones. Even within a given car park many such restricted location might be present dispersed around the car park. To monitor all such locations would require the installation of many cameras. Moreover, power and data cabling would need to be installed to connect the cameras to the back-end system. This is challenging and expensive particularly when it is required to retro-fit such a system to existing infrastructure. AN PR software may be run on a camera by an integrated processor, which through OCR (Optical Character Recognition) technology extracts the vehicle registration number as a part of a photographic evidence record. However, whilst reducing the volume of transmitted data, such an approach would likely increase the power consumption of the camera making it impractical to use battery power (e.g. rechargeable batteries coupled to a solar charging system). Summary To tackle the challenges and to reduce the complexity, cost of installation and coverage of contraventions beyond merely overstay, one might consider capturing still images with a low energy consumption camera and send transmit those images to the back-end system for further contravention analysis. Embodiments of the present invention aim to provide a low-cost and easy to install camera system for parking contravention detection. According to an aspect of the present invention there is provided of detecting vehicle parking contraventions using a fixed camera system having an RGB sensor array and an IR sensor array, the arrays having a substantially common field-of-view encompassing at least one parking location to be monitored. The method comprises, at the fixed camera system, using one or both of the sensor arrays to capture a sequence of time spaced still images, and transmitting the captured still images from the fixed camera system to an administration service over a transmission network comprising at least in part a wireless network. At the administration service, the service receives the transmitted still images, determines from one or more of the received still images that a vehicle is parked in said parking location, and uses a trained machine learning model to analyse one or more of the still image and to identify a property or properties of the parked vehicle and / or an action or actions related to the parked vehicle. The service further uses the identified property or properties and I or action or actions to determine whether or not the vehicle is parked in contravention of a predefined parking rule or rules and, if yes, initiates a parking contravention procedure including obtaining from one or more of the still images a vehicle registration number for the vehicle and creating an evidentiary data bundle including one or more of the received still images. Brief Description of the Drawings Figure 1 illustrates schematically an exemplary system and service for monitoring controlled parking locations; Figure 2 illustrates an exemplary process flow for monitoring controlled parking locations; Figure 3 illustrates a photo bundle that is created during monitoring of a controlled parking location; and Figure 4 : Illustrates multiple gated image processing modules such as tone correction (T), contrast balance (C), sharpening (S), defogging (DF), gamma correction (G), white balancing (W B), and the identity operation (I), referred to as GbIP , that individually enhance images, which are then combined through the weights predicted by the gate to combine their output. Detailed Description A camera system is described here (referred to as an “eCam”) which automates the detection of a subset of parking contravention types, many of which were previously only detectable manually by parking officers (POs) and the like. The eCam characteristics may include: • Built on low-cost hardware. • Still photo images, captured, processed and bundled together. • Low entry barrier, i.e. offers easy deployment, non-intrusive for existing car park infrastructure. • Powered by renewable energy, i.e. solar technology, bypassing the need of costly cabling. • Simple edge processing to reduce power consumption. Most of the contravention analysis is shifted from the camera and conducted in the back-end system hosted on the cloud. • 4G mobile network connectivity to remove the need for network cabling. • Captured photos are sent to the cloud for contravention processing and record keeping. In its simplest and cheapest implementation the eCam is a digital camera that is able to capture both RGB and IR still images at intervals of time, e.g. periodically, and transmit these wirelessly over the mobile to a cloud service with which the eCam is registered. The capturing and transmission of images, and possibly the selection of image type, IR, RGB or RAW, may controlled by local conditions detected by the camera of may be controlled by the cloud service. Figure 1 illustrates schematically a parking control system I service which includes a number of eCams 1 fixed at specific locations associated with respective controlled or prohibited parking locations. These locations may be at car parks or on-street parking sites. The eCams are configured to connect to and transmit I receive data over a 4G network 2. Of course any other suitable connection network may be used including WiFi, Bluetooth™, 5G, 6G etc. In some cases a wired network may be used, e.g. Ethernet. Servers 3 of a cloud computing service 4 are used to implement cloud software as will be described below. The servers 3 are coupled to the 4G network 2 via Internet connections. As such, signalling and data is routed between the cloud service and the eCams via the 4G network and the Internet. Figure 1 also illustrates an administrator terminal 5 that is owned and operated by an administrator of the parking control system I service. Considering now the cloud service that is facilitated by the cloud computing service 4 and the cloud software, a custom training process is created that involves integrating a GDIP within the SAM architecture to help regularize SAM features to detect vehicles in low light settings and adverse condition. The GDIP here will be used as regularizer, which will prevent over fitting, and help the segmentation mode (SAM) adapt to variable situations. Here the SAM architecture is based on the transformer neural network. The model is trained using a large number (e.g. hundreds of thousands) of images - the “training sets” - relevant to the parking contravention that is to be detected, hese images consist of RGB images and IR images, as well as RAW images, where RAW images correspond to data directly output from an image sensor with little or no post-processing. The images are chosen to correspond to a range of light and weather conditions, e.g. dusk, night, rain, mist etc. Use of infrared spectrum images is beneficial in low light and poor contrasting conditions. RAW images on the other hand perform better in low light conditions, whilst RGB images provide textural information which is important for object detection but perform poorly in low light conditions. Recent published work (Chen et al April 2023 ) suggests that RAW images are better than their RGB counterpart in low lighting conditions and yet still provide good textural information. The images may also include “fused images” which are produced by combining data from one or more of the RGB, IR and RAW images. In dark conditions RAW and infrared may be a better choice than RGB and IR, so given such conditions we can take an image based on RAW and IR, then fuse them together to train the model. Fused images might be generated, in a very simplistic example, by superimposing the images to be fused on top of one another and summing the overlapping pixel data. Images of the training sets may be obtained by capturing still images from an eCam installed at a parking location to be controlled. Each set may contain a series of images relating to a particular event, for example a sequence of still images corresponding to entry of a vehicle into a controlled location, parking at the location, and exit from the location. Each of these sets may be classified, e.g. by manual inspection, to tag the set as a parking contravention or not a parking contravention. Images may be enhanced, prior to commencement of testing and inference in a live situation, using a Gated Differentiable Image Processing (GDIP) process. GDIP is a known process that adaptively enhances adverse images for object detection by performing image enhancements based on the context of the image. Once trained, the model is verified using a test data sets, each of which is tagged as a parking contravention or not a parking contravention. If the output of the model for the test data sets matches the a priori tag, the model is considered as validated. If the results from the testing are poor, then either more images or a diverse set of images are utilised along with fine tuning to improve test scores after training the model. Consider for example the case where the parking contravention to be monitored is one of parking in a loading bay for an excessive time during which no loading is performed. The training data sets may include numerous examples (image sequences) of vehicles parked with no loading and examples (other image sequences) of vehicles parked with loading. In some cases the model may be trained to determine a contravention from single images rather than sets of images. A use case for analysing such single images may be, for example, to determine whether a parked vehicle is an emergency services vehicle such as an ambulance. In some cases the cloud service may comprise multiple trained models and the results combined with appropriate logic to determine whether or not a parking contravention has occurred. For example, one model may be trained to detect loading whilst another model may be trained to determine whether or not a parked vehicle is an emergency vehicle. In some other cases a single model may be trained to perform both classifications. In some other cases separate models may be trained for respective image types, e.g. RGB, IR, RAW and or fused. Depending on the type of image collected at the monitored parking location the appropriate model is selected. In some cases classification may be improved by defining within an image frame a geozone, such that classification is only applied within the geo-zone. A geo-zone may be defined, for example, around a loading bay visible within a viewed scene. Multiple geozones may be defined within a given image frame. Once the model(s) is(are) trained the cloud service is configured to communicate with the multiple installed eCams and register those eCams. Transmissions from eCams are associated with respective identifiers, e.g. a mobile number associated with an eCam’s SIM card. This allows the cloud service to identify the controlled parking locations associated with the uploaded data. As already noted the eCams are preferably simple and low-cost components. In one scenario each eCam is configured to capture a still image at regular intervals, e.g. every 1 minute, and upload the captured images to the cloud service. This may be an RGB image or an IR image dependent upon the visibility conditions at the camera’s locations. In some cases the eCam may capture both an RGB and an IR image at regular intervals and upload both to the cloud service. Processing at the eCam is minimal. The eCam may also capture and transmit RAW images if required. The cloud service may be configured to receive a sequence or stream of still images from an eCam and analyse these. The service may for example compare successive still image to determine when a vehicle has entered a controlled parking location, i.e. defined by a geo-zone. This analysis may perform such detection by merely comparing pixel data of the successive images. Similarly the cloud service may determine when a vehicle has parked in the controlled location for a period in excess of an allowed “grace period”, e.g. more than 5 or 10 minutes, and when the vehicle has left the controlled location. The cloud service collects and stores the sequence of still images for the entire duration of the vehicle’s stay. This represents a photograph bundle for the event. The bundle may comprise for example RGB, IR and / or RAW images. Images are pre-processed in the same or a similar manner to that applied to the images of the training and test set, e.g. using GDIP. The cloud service then applies the (pre-processed) still images of the photograph bundle, or a selected subset thereof, to the trained model for classification (or to multiple trained models if these are provided). The model provides as an output a determination as to whether or not a parking contravention has occurred. If so, an AN PR process may be run across one or more of the still images to identify a vehicle number (registration number) of the vehicle in question. The registration number may be included with the photograph bundle, together with the location of the monitored parking location. The photograph bundle and contravention details are sent to a contravention management system for management, appeals and debt recovery. The photograph bundle and added contravention details can be considered to be an “evidentiary data bundle” such as might be used later, e.g. in a court of law, to evidence the contravention. It is noted that PCNs will typically only be issued after a manual review of the photograph bundle and contravention details. However, a simple visual inspection of RGB and / or IR images may in certain circumstances be insufficient from the point of view of the human operator. The fused images referred to above may be more appropriate for this purpose. Regardless of whether or not the classification model is trained using fused images, and consequently whether or not fused images are provided as input to the trained model to classify a new event, a fused image or images may be generated and added to the photograph bundle to facilitate or aid manual verification. The fused image may also provide an image for inclusion with a PCN sent to a vehicle owner or registered keeper. In some cases, rather than capturing and transmitting a continuous sequence of still images, an eCam may incorporate some motion detection features, the eCam being configured to capture and transmit still images only after a relevant motion has been detected, i.e. movement of an object into the field of view of the camera (or into a geozone defined a priori and saved to the eCam). The eCam may cease capturing and transmitting still images when the object is detected as moving out of the location. In some cases capturing and transmission of still images may be controlled by the cloud service based on an analysis of uploaded still images. Where additional processing power and power is available at an eCam, certain processing tasks may be devolved from the cloud service to the eCam. For example, Figure 2 illustrates at a high level a process flow for operating a service and for using collected data for parking contravention management. This assumes that an eCam has been installed at a suitable location such that the field-of-view of the camera includes the location to be controlled, and that the eCam is provided with radar functionality to detect movement. It further assumes that the specific location has been identified as a geozone within the field-of-view. This might involve an administrator using the administrator terminal 5 to connect with the eCam, either directly or via the cloud computing service, to define the geo-zone within the field of view, e.g. using a pointer such as a mouse to identify a boundary around the location within the field-of-view and uploading this data to the eCam. Alternatively some image processing software may be applied automatically to perform this task, e.g. the software may identify a geo-zone around a controlled or prohibited parking location that is marked out in a car park with yellow lines. Preferably the field-of-view of the camera encompasses a region surrounding the parking location being monitored such that vehicles arriving at and leaving the location can be detected immediately prior to entry and immediately upon exit. Data defining the geozone is stored locally at the eCam. It will be appreciated that steps described as being performed at the eCam (camera software) are performed using a processor or 5 processors and memory of the eCam operating on image (pixel) data provided by an image sensor or sensors of the eCam. Figure 2 identifies the process steps and indicates whether these are performed at the eCam or at a backend system (in this case software running on a cloud service). In more 10 detail these steps include: # Event Description Source 1 Vehicle (Reg. No. ABC 123) Vehicle drives into the defined geozone Camera software 2 Moving object detection The radar in the camera detects a moving object inside the geo-zone Camera software 3 Object classification Detect that the moving object is a vehicle Camera software 4 Take photo every N seconds until vehicle stops The camera will take a photo when a moving vehicle object is detected. Thereafter is takes a photo every N seconds until vehicle stops. N is a configurable variable but may be, for example 10 seconds. The Photos are sent to the cloud over the 4G network, with camera identifier, date / time stamp and maybe a geo-zone identifier. The photo(s) are of RGB type (during day light) or IR (during dusk / night). Camera software 5 Vehicle stop validation photo The cloud software analyses the last photo date / time stamp for the specific vehicle. If photo date / time stamp is longer than N sec, send command to camera to take a photo. The photo is of RGB type (during day light) or IR (during dusk / night). Cloud software 5a Vehicle stop validation photo (optional) The camera takes an additional photo, RGB or IR, as instructed and sends to the cloud over the 4G network, with date / time stamp and geo-zone identifier. Camera software 6 Grace period photo The geo-zone is associated with a contravention type. Each contravention time has a grace period, i.e. a time window during which a PCN will not be issued e.g. 2-5 minutes. Once the grace period has elapsed following detection of a vehicle object stopping in the geozone, a command is sent to the camera to take 2 further photos: 1. Infrared photo, and 2. RGB photo Cloud software 7. Take both RGB and IR photos The camera will take both IR and RGB photos. The photos are sent to the cloud over the 4G network, with date / time stamp and geo-zone identifier. Camera Software 8 Vehicle start move photo A photo is taken when the vehicle starts moving. The photo is of RGB type (during day light) or IR (during dusk / night). The photo is sent to the cloud, with date / time stamp and geozone identifier. Camera software 8 Create photo bundle Once all photos have been received, a photo bundle ("package") is created containing the following data: 1. Moving vehicle object detected within geo-zone. 2. A series of photos N sec apart until vehicle has stopped (RGB orIR). 3. Vehicle stop validation photos (IRorRGB). 4. After grace period photos (IR + RGB). 5. Vehicle starts to move photo (RGBorIR). Corresponding RGB + IR photos (4) are merged into one enhanced photo. Cloud software 9 Contravention detection Processing of the photo bundle and geo-zone data to determine if contravention has occurred. There are 4 types of contraventions: Cloud software 1. "Vehicle-in-a-box". Issued based on the vehicle being within the geo-fence after the grace-period. 2. "1" + sub-object classification. Taxi, emergency vehicle and truck etc. 3. "1" + action classification. Loading, unloading, pick-up and passengers etc. 4. "1" + sub-object classification + action classification. Taxi pick-ups passengers. The above contravention types are covered in "Annex 3 -Contraventons". 10 Contravention management The photo bundle and contravention details are sent to a contravention management system for management, appeals and debt recovery. The eCam solution will feed data into contravention management and hence this does not form part of the eCam solution. Cloud software Any one or more of the captured images may be supplemented with a RAW image. For example, at step 7, RGB, IR and RAW images may be captured. Similarly at step 5 for 5 the “Stope” and “Grace” period captured images. Whilst the contravention management procedure is not considered here in detail, it will involve performing ANPR on the combined RGB and IR image. It may also involve performing image recognition operations on that image including, for example, detecting 10 vehicle type (emergency vehicle, taxi, bus, private vehicle etc). The procedure may apply various predefined rules to determine whether or not a PCN or other notice should be sent to the owner or operator of a vehicle recognised in the image and identified using the detected number plate. 15 The cloud service may be linked with other services and systems to provide enhanced monitoring. For example the service may have access to vehicle registration detailed to determine a vehicle type, e.g. an electric I non-electric vehicle. The service, together with such other services and systems where required, may allow the embodiments described above to monitor the following parking contraventions: Causing obstructions Drop Off Point In front of garages No Parking No Parking On Verges No parking out of hours No Roadway No Stopping Not Paying By Phone With appropriately trained models, the following contraventions may also be detected: Ambulance Parking Commercial Vehicle Commercial vehicle outside hours Customer Parking Only Electric Cars Loading bay Loading / Unloading Only Motorcycle Bay Motorcycles in motorcycle bays only No Disabled permit No HGV or Trailers or Coaches Not parked within bay markings Not using electric charging It will be appreciated that these lists are not meant to be inclusive or exclusive and merely indicate the various types of contraventions that may be detected. An exemplary evidentiary bundle is illustrated in Figure 3 and which is generated using data provided by the eCam and provided to the contravention management system. Figure 4 illustrates an exemplary image processing procedure used to enhance the quality of an input image, for example an image captured using the RGB sensor array of the eCam. Each GDIP block contains multiple gated image processing modules that are 5 configured to enhance the image by Tone, Contrast, Sharpening, Defogging, Gamma, and White balance: by training a model to determine the weights we can determine how much and to what extent of each these processes are applied to the final image. In front of the GDIP block a Vision Encoder is provided to transform captured images 10 into a feature embedding format to be used by each GDIP block. The Vision Encoder uses five trained convolutional (neural network) layers followed by an average pooling function. Within each GDIP block there is a linear layer that outputs two sets of data; (1) parameters that are required for that particular image processing block, and (2) a scaler number required by the corresponding gate. 15 It will be appreciated by the skilled person that various modification may be made to the above described embodiments without departing from the scope of the present invention.

Claims

1. A method of detecting vehicle parking contraventions using a fixed camera system having an RGB sensor array and an IR sensor array, the arrays having a substantially common field-of-view encompassing at least one parking location to be monitored, the method comprising: at the fixed camera systemusing one or both of the sensor arrays to capture a sequence of time spaced still images,transmitting the captured still images from the fixed camera system to an administration service over a transmission network comprising at least in part a wireless network,at the administration servicereceiving the transmitted still images,determining from one or more of the received still images that a vehicle is parked in said parking location,using a trained machine and deep learning model to analyse one or more of the still images and to identify a property or properties of the parked vehicle and / or an action or actions related to the parked vehicle, andusing the identified property or properties and I or action or actions to determine whether or not the vehicle is parked in contravention of a predefined parking rule or rules and, if yes, initiating a parking contravention procedure including obtaining from one or more of the still images a vehicle registration number for the vehicle and creating an evidentiary data bundle including one or more of the received still images.

2. A method according to claim 1 and comprising, at the administrative service, generating a composite image by fusing an RGB image and an IR image captured by the fixed camera system and including the fused image in the evidentiary bundle.

3. A method according to claim 1 or 2, wherein the sequence of time spaced still images have a time spacing of greater than 1 second.

4. A method according to any one of the preceding claims and comprising detecting, by the fixed camera, entry of a vehicle into the parking location and, in response,performing said steps of capturing a sequence of time spaced still images and transmitting the captured still images to the administration service in response to said detecting.

5. A method according to claim 4, wherein the detecting is performed by analysing data provided by one or both of the sensor arrays or using data provided by a radar of the camera system.

6. A method according to any one of claims 1 to 3, wherein the camera system is configured to capture a continuous sequence of time spaced still images regardless of the entry or not of a vehicle in the parking location.

7. A method according to any one of the preceding claims, wherein said property or properties is or are selected from:whether or not the vehicle is an emergency vehicle;whether or not the vehicle is a taxi;whether or not the vehicle is a car or a van or a lorry.

8. A method according to any one of the preceding claims, wherein said action or actions is or are selected from:whether or not the vehicle is causing an obstruction;whether or not the vehicle is being loaded or unloaded;whether or not the vehicle is being connected to and / or disconnected from an electric charging point;whether or not the vehicle is properly parked within a marked bay.

9. A method according to any one of the preceding claims, wherein said wireless network is provided by a Public Land Mobile Network, for example a 5G network.

10. A method according to any one of the preceding claims, wherein said step of creating an evidentiary data bundle comprises including in the bundle one or more images or fused images enhanced using a feature encoder using a plurality of trained convolutional neural network layers followed, optionally, by an average pooling function.

Images