An unmanned driving test monitoring system

Abstract

The present invention relates to a kind of unmanned driving test monitoring system, including a server and a monitoring terminal integrated with a camera, a positioning function and capable of connecting to a communication network; the monitoring terminal is configured to collect monitoring videos of unmanned vehicles, detect vehicle line-crossing violations on images in the monitoring video, and send the violation information to the server for storage; at the same time, the monitoring video is parsed, a positioning timestamp and a location stamp are added, and the system is pushed to the server after encoding and compression; the server includes a streaming media server and an information management server; the streaming media server is configured to receive, store, relay and play videos at fixed points; the information management server is used for the storage and retrieval of violation information and video information, and the opening and closing of verification monitoring are verified. The system has easy deployment, real-time observation, high concurrency, and can automatically detect vehicle line-crossing violations, reducing the workload of referees.

Description

Technical Field

[0001] The present disclosure relates to unmanned driving testing, and in particular to an unmanned driving testing monitoring system. Background Art

[0002] To facilitate repeatable testing of autonomous vehicles, a large number of specialized autonomous driving test sites have been built. Existing autonomous driving test sites utilize full Wi-Fi coverage, differential GPS base stations, and various roadside cameras and sensors to monitor the progress of autonomous driving tests. During autonomous driving tests, the Ministry of Industry and Information Technology (MIIT) requires continuous monitoring of test progress and the recording and storage of external vehicle video surveillance footage to ensure compliance, safety, and data support for autonomous driving testing and evaluation. Therefore, all test sites in my country implement comprehensive monitoring, including surveillance and communication equipment. For example, the National Intelligent Connected Vehicle (Shanghai) Pilot Demonstration Zone in Jiading, Shanghai, can simulate nearly 30 real-world urban road traffic scenarios. These include various driving scenarios, such as urban tunnels, shaded roads, urban cross / T-junctions, and roundabouts. These scenarios also encompass a variety of everyday driving scenarios, including roadside parking, underground parking lots, and gas station entrances. In terms of communications, the test site is equipped with one 4G base station, one vehicle positioning base station, two communication base stations for vehicle-road collaborative communications, 16 long-range Radio Frequency Identification (RFID) systems, four roadside units (RUUs) for vehicle-road collaborative communication, and six intelligent traffic lights. To collect vehicle road test data, the test site uses the Beidou positioning system for centimeter-level positioning, and numerous communication base stations provide full Wi-Fi coverage. For real-time monitoring of vehicle driving status, the test site is equipped with 40 cameras of various types.

[0003] However, in addition to unmanned driving tests in professional test sites, unmanned driving tests also include unmanned driving competitions in real traffic environments, such as the "Intelligent Vehicle Future Challenge" (IVFC) in China. The facilities in professional test sites are not convenient to migrate and deploy to real traffic environments. Taking the 12th "China Intelligent Vehicle Future Challenge" as an example, the competition requires that within the prescribed time of 90 minutes, participating vehicles must strictly abide by road signs and markings while ensuring safe and autonomous driving, and complete multiple simulated taxi pick-up and drop-off services at designated locations. The competition venue is the "nine-square grid" urban road in Changshu Development Zone, and its map is as follows: Figure 1The test site features a variety of road scenarios, including urban driving, road construction, traffic control and guidance, mixed pedestrian and vehicle traffic, traffic congestion, realistic overtaking, merging, and meeting vehicles, and speed limits, creating complex traffic conditions. To determine violations, the organizers of this competition primarily employed a "human wave" monitoring deployment. Violation tracing primarily involved reviewing surveillance videos based on the referee's paper records. This process was cumbersome, inefficient, and prone to human error, compromising competition fairness.

[0004] Therefore, it is difficult to apply the detection system used in professional unmanned driving test sites to real traffic environments. The existing monitoring system solutions are difficult to quickly achieve large-scale deployment in complex real traffic environments. If targeted system development is carried out for each layer of real urban traffic environment, there will also be huge investments and possible long development periods. Summary of the Invention

[0005] In view of the above-mentioned prior art, the present invention aims to provide an unmanned driving test monitoring system to at least solve the following problems:

[0006] (1) Convenience and ease of deployment: The monitoring system can conduct intensive multi-angle monitoring of the competition area, including road monitoring, self-monitoring of participating vehicles, vehicle following monitoring, and monitoring from the perspective of social vehicles. In addition, the venue of the competition is unknown each year. In order to quickly realize the monitoring deployment of the competition venue, the monitoring system needs to be convenient and easy to deploy so that it can be quickly set up in various competition venues. Among them, the monitoring terminal needs to be able to operate continuously for 120 minutes to meet the continuous monitoring of a single game, and the monitoring terminal needs to be relocated at any time according to the requirements of the competition.

[0007] (2) Observability: To ensure the fairness of the competition and meet the needs of on-site viewers, the organizers need to broadcast the monitoring video data in real time. Therefore, the monitoring system needs to be observable.

[0008] (3) High concurrency: When multiple monitoring channels are enabled at the same time, the monitoring system server needs to ensure the availability of the system.

[0009] (4) Lane crossing detection: During the race, referees need to constantly monitor vehicle behavior. Especially when performing lane crossing detection, referees need to pay close attention to the relationship between the vehicle and the lane line, which can easily lead to fatigue. Therefore, the monitoring terminal needs to be able to detect lane crossings to assist referees.

[0010] In order to solve the above technical problems, the present invention proposes an unmanned driving test monitoring system, which includes a monitoring terminal and a server. The monitoring terminal integrates a camera and positioning function and can be connected to a communication network; the monitoring terminal is configured to monitor video acquisition during unmanned driving tests, detect vehicle line violations in the images in the monitoring video, and send the violation information to the server for storage; at the same time, the monitoring video is parsed, a positioning timestamp and location stamp are added, and the video is compressed and pushed to the server; the server includes an information management server and a streaming media server that supports high concurrency; the streaming media server is configured for video reception, storage, broadcasting and fixed-point playback; the information management server is used for storage and retrieval of violation information and video information, and verification of the opening and closing of video monitoring.

[0011] In the above technical solution, monitoring terminals, such as mobile phones and tablets, are compact, flexible, and have long battery life, making them convenient for deployment on referee vehicles for vehicle-following monitoring or roadside monitoring. Because they include cameras, positioning capabilities, and connectivity to communication networks, they can transmit captured video in real time, allowing users to log in to the server and view the video in real time. Their vehicle crossing violation detection function improves video monitoring quality, reduces referee workload, and alleviates fatigue. Furthermore, using the monitoring terminal to detect vehicle crossing violations reduces server pressure, allows for timely identification and recording of violations during competition, and fully utilizes existing hardware and communication environments. Streaming server module: During competitions, the monitoring system needs to live stream the competition, allowing players to stream the content using a variety of transmission protocols. After the competition, users can retrieve and replay the video. The information management module stores captured video information, enabling video retrieval. The storage of information related to detected vehicle crossing violations enables retrieval and query of violation information.

[0012] As an improvement to the above technical solution, the monitoring video is encoded and compressed and pushed to the server to solve the problem that the original data collected has a high resolution and is directly transmitted, which is slow and consumes a lot of data. The following steps are included:

[0013] Set the acquired push stream address as the push stream channel; obtain the encoding format;

[0014] After the captured YUV format image is converted into RGB format image, it is encoded and compressed according to the encoding format, and then pushed to the server through the push channel.

[0015] As an improvement to the above technical solution, the timestamp and location stamp are used to further confirm the time and location of the accident during review of the accident appraisal.

[0016] Obtain GPS time and GPS location coordinates and represent them with bitmap codes. To prevent the timestamp and address stamp from covering up all the information on the screen, obtain the bitmap codes in sequence. If the code value is "1", calculate its corresponding position in the captured video image and modify the Y component of the position to integrate the timestamp or address stamp into the captured video image.

[0017] In the above technical solution, the vehicle crossing violation detection includes the following steps:

[0018] The lane lines in the video image are segmented using a preset lane line segmentation model, and the lane line segmentation results are converted into lane line equations;

[0019] Based on the lane line equation, perform inverse perspective transformation to obtain the distance between the two lane lines;

[0020] A preset target detection model is used to identify vehicle targets in video images for detection, and Kalman filtering is used to track and estimate vehicle targets;

[0021] Based on the distance between the vehicle and the lane line, it is determined whether to mark the lane line. If marking the lane line is required, a prompt message is generated.

[0022] In one embodiment of the above technical solution, when the ratio of the distance between the vehicle and the left and right lane lines of the driving road is less than 1:5 in the vehicle's driving direction, a line crossing mark is performed, and the time and location of the line crossing behavior are recorded in the mark.

[0023] As an improvement to the above technical solution, the preset target detection model adopts the MobileNetV3-Large_SSDlite model to achieve vehicle detection and lane line segmentation on low-configuration, low-computing mobile terminals;

[0024] The default lane segmentation model uses the MobileNetV3-Large_DeepLabv3+ model;

[0025] After training, the MobileNetV3-Large_SSDlite model and the MobileNetV3-Large_DeepLabv3+ model are converted into tflite format and transplanted to the monitoring terminal.

[0026] As an improvement to the above technical solution, the lane line segmentation result is converted into a lane line equation to perform real-time inverse perspective transformation, overcoming the problem of unclear relationship between vehicles and lane lines due to perspective effect, where the distance between distant vehicles and lane lines in the image becomes increasingly closer. This includes:

[0027] The lane line segmentation results are downsampled, Hough transform is performed, and similar straight line fitting is performed to merge to obtain the lane line equation.

[0028] As an improvement to the above technical solution, before using Kalman filtering to track and estimate the vehicle target, the following steps are included:

[0029] The vehicle coordinates are grouped based on the principle of the shortest distance between frames. After the vehicle targets are subjected to inverse perspective transformation and rotation transformation, the position of the vehicle relative to the referee car is obtained. The position of the referee car is calculated by the monitoring terminal based on the starting position of the referee car.

[0030] Based on the referee car's position and the vehicle's position relative to the referee car, the vehicle's coordinates within the lane are calculated as the vehicle's observed coordinates, which are then used as input for tracking estimation using the Kalman filter. By using the Kalman filter to track and estimate the vehicle target, the problem of inaccurate vehicle observed coordinates is overcome.

[0031] As an improvement to the above technical solution, the monitoring video acquisition includes:

[0032] Apply for monitoring based on the universal unique identification code generated by the monitoring terminal and the information required for monitoring obtained;

[0033] After receiving authorization from the server, the monitoring terminal can choose to start or end monitoring. When selecting Start, the monitoring terminal begins to collect monitoring video. By applying, verification of video monitoring can be enabled to prevent interference from non-game monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 , a schematic diagram of the monitoring area for unmanned driving test in the background technology;

[0036] Figure 2 、 one A schematic diagram of the system architecture of an unmanned driving test monitoring system in an embodiment;

[0037] Figure 3 、 one A schematic diagram of the hardware configuration of an unmanned driving test monitoring system in an embodiment;

[0038] Figure 4 、 oneA functional structure diagram of an IVFC monitoring application in an embodiment;

[0039] Figure 5a 、 one A schematic diagram of a bitmap display of number 2 in an embodiment;

[0040] Figure 5b 、 one A schematic diagram of a bitmap encoding of the number 2 in an embodiment;

[0041] Figure 6 、 one A schematic diagram of a process for collecting on-site video data in an embodiment;

[0042] Figure 7 、 one A schematic diagram of the perspective effect of an image in a coordinate system according to an embodiment;

[0043] Figure 8 、 one A schematic diagram of the relationship between the world coordinate system and the image coordinate system in an embodiment;

[0044] Figure 9 、 one A schematic diagram of lane line rotation and translation in an embodiment;

[0045] Figure 10a 、 one A schematic diagram of an embodiment before inverse perspective transformation;

[0046] Figure 10b 、 one A schematic diagram of an embodiment after inverse perspective transformation;

[0047] Figure 11 、 one Example diagram of a Kalman filter model in an embodiment;

[0048] Figure 12 、 one A schematic diagram of the relative relationship between a vehicle and a lane line in an embodiment;

[0049] Figure 13a 、 one A schematic diagram of a real scene of urban road testing in an embodiment;

[0050] Figure 13b 、 one Schematic diagram of the results of urban road test line pressure in an embodiment;

[0051] Figure 14a 、 one A schematic diagram of a real scene of urban road testing in an embodiment;

[0052] Figure 14b 、 one Schematic diagram of the results of the urban road test line in an embodiment. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] In one embodiment of the method of the present invention, the unmanned driving test monitoring system is deployed Figure 1 The system architecture of the unmanned driving test monitoring system is shown in the following figure. Figure 2 As shown, it includes a monitoring terminal, a server and a display platform. Among them, the monitoring terminal integrates a camera, a positioning function and can be connected to a communication network, including a referee car monitoring terminal placed in the referee car to realize vehicle-following perspective monitoring, a road monitoring terminal placed on the roadside to realize road monitoring, a participating vehicle monitoring terminal placed in the participating vehicle, and a referee handheld terminal. The monitoring terminal can be a mobile phone, a tablet or other customized mobile terminal. The communication network is for transmitting collected video images or sending and receiving information without distance restrictions. The server includes a tower server that can deploy an information management system and perform configurations such as streaming media protocols. The display platform is used to set up an LCD splicing display wall and project information such as game monitoring videos, on-site maps, and dispatch consoles in the user terminal through a matrix switcher.

[0055] Based on the above system, hardware facilities such as Figure 3 shown.

[0056] At the command center for the unmanned test, a router was installed to connect to the network. Sixteen tablet computers were connected to the wireless router and connected to the server via the internet to obtain monitoring information. The tablet computers were connected to a matrix switcher via high-definition data cables, which in turn connected the matrix switcher to an LCD video wall, enabling visualization of the competition within the command center. VLC Media Player software was installed on the tablet computers for video monitoring and viewing.

[0057] Build a streaming media server and information management server in the server host.

[0058] The streaming media server was built using Nginx, enabling video data reception, forwarding, and storage. When setting up the Nginx streaming media server, the required libraries for Nginx development were installed, including the libpcre3, libpcre3-dev, openssl, libssl-dev, pcre, and zlib libraries. To enable video data transmission, the server was configured with streaming media protocols including HLS (HTTP Live Streaming), RTMP (Real-Time Messaging Protocol), and HTTP-FLV (HTTP Flash Video live stream).

[0059] The information management server is built using the lightweight framework SpringBoot and MySQL database, including a video information management module and a monitoring information query module.

[0060] The video information management module is configured to implement functions such as monitoring terminal information storage, video information generation, monitoring application processing, video start verification, video stop verification, and violation information storage. The monitoring information query module is configured for functions such as violation retrieval, video retrieval, and user login. In the monitoring application processing, monitoring terminal information storage, and video information generation, after the information management server receives the monitoring application initiated by the monitoring terminal, it verifies the information filled in it. After the verification is passed, the monitoring terminal information is stored, and a video unique ID is generated based on the monitoring terminal information and the current time. The video information is then stored and the streaming address is returned. If the monitoring verification fails, an application failure message is returned, and the process ends. In the video start verification (or video stop verification) function, the information management server verifies the video start request (or video stop request) after receiving the streaming information from the streaming media server. If the monitoring terminal information is consistent with the video information, the verification is passed, and the video start (or stop) time is stored. After that, a success status code 200 is returned to end the process. If the verification fails, a status code 400 is returned to end the process. When performing information retrieval, the information management server first verifies the user's identity. Once the verification is successful, it matches the violation information with the video information based on the retrieval information and returns the matching result. During the user login process, when the information management server receives the user's login request, it first verifies the user's account and password, and then generates a unique user ID based on the account and password information for subsequent user operations. In the storage of violation information, after receiving the violation information pushed by the monitoring terminal, the information management server first verifies the identity of the monitoring terminal, confirms that the monitoring terminal corresponds to the video information, and then stores the violation information; if there is no match, the process ends.

[0061] The monitoring terminal is an Android phone with an integrated camera, GPS, and 4G network connection. Video data is collected using the mobile monitoring terminal, and the monitoring data is transmitted through the established 4G network base station. The IVFC (Intelligent Vehicle Future Challenge) monitoring application is installed in the mobile monitoring terminal. The IVFC monitoring application obtains the hardware usage permissions of the current mobile device, including camera, positioning, video recording, audio recording, etc. In the IVFC monitoring application, the Android Camera object, Surface object, and JavaCV plug-in are used to implement the collection, preview, encoding, and streaming of video data.

[0062] The IVFC monitoring application can achieve the following functions: Figure 4 As shown:

[0063] (1) Marking timestamp and location stamp

[0064] For video surveillance data, real-time dynamic GPS time and GPS location coordinates are integrated into the captured video images. This allows for further confirmation of the time and location of the accident through the marked timestamps and location during review for accident identification and other purposes.

[0065] The IVFC monitoring application obtains image data from the camera session pipeline, which is encoded in YUV format. After obtaining the pixel points of the video data, the raw video data is time-stamped and location-stamped frame by frame.

[0066] When marking timestamps and location stamps, include:

[0067] (1) The acquired GPS time and GPS location coordinates are made into a bitmap and represented by bitmap coding. When setting the size of the bitmap, the time or location represented by it can be fully displayed in the captured video image. The bitmap coding is about the bitmap corresponding to the numbers "1" to "9", "-", ":", and ".", such as Figure 5a As shown, this is the bitmap of the number "2". Figure 5b It is the bitmap code corresponding to the bitmap.

[0068] (2) One data point in the bitmap corresponds to one pixel in the image. The bitmap codes are obtained sequentially. If the code is "1", the corresponding pixel position in the captured video image is calculated based on the positions of the set time stamp and location stamp.

[0069] (3) To prevent the timestamp from covering up all the information on the screen, only the Y component of the pixel at the position corresponding to the captured video is modified to 1, thereby integrating the bitmap code into the video image.

[0070] (2) Data collection and processing

[0071] Fill in the information required to initiate monitoring on the installed monitoring terminal, such as vehicle / lane information, verification information, and information about the monitored object. The monitored object can be the road information where the monitoring terminal is located, or the license plate number of the vehicle being tested. The monitoring application generates a universally unique identifier (UUID) for the monitoring terminal based on the information filled in, and submits the UUID generated in the monitoring application and the information filled in by the referee to the server to apply for monitoring. After the server allows it, the application will automatically enter the monitoring page; the referee can choose to start or end monitoring on the monitoring page; after the monitoring ends, the application will return to the information filling page again, and the monitoring terminal can start another monitoring.

[0072] After the monitoring application is approved, the on-site video data will be collected and the processing flow is as follows: Figure 6 As shown in the figure, the captured video data is timestamped and located on the image based on the GPS time, longitude, and latitude. Because the raw data has a high resolution, directly transmitting it would be slow and consume data bandwidth. Therefore, the captured data is encoded and then streamed.

[0073] One specific implementation for encoding and streaming is to use FFmpeg within the JavaCV library to implement image encoding and streaming. Table 1 lists the parameters for audio sampling frequency, audio transmission channel, video sampling frame rate, audio and video encapsulation format, and video encoding format.

[0074] Table 1

[0075]

[0076] After the monitoring application is approved, the streaming address will be obtained and set as the streaming channel. For the collected video images, the FrameRecorder module in FFmpeg is used to encapsulate, encode, stream, and record and save the audio, video, and images: the obtained streaming address is set as the streaming channel, the YUV format image in the video is converted to RGB format, and the interface FFmpegFrameRecorder (frame streamer) is used to encode the obtained audio and video data. The encoded data is then compressed and sent to the server through the streaming channel.

[0077] For the collected video images, the monitoring terminal detects the images in the video for vehicle crossing the line violation and sends the violation information to the server for storage, thereby improving the quality of video monitoring and reducing the workload of referees, alleviating the referee's fatigue in watching the car.

[0078] In lane crossing violation detection, vehicle target detection is used to determine the relationship between the vehicle and the lane markings. This method uses a pre-defined lightweight neural network to address the limited computing resources of the monitoring terminal. The MobileNetV3-Large_SSDlite model is used as the vehicle target detection model, and the MobileNetV3-Large_DeepLabv3+ model is used as the pre-defined lane segmentation model. Both trained models are converted to tflite format and ported to the mobile phone.

[0079] The datasets used for vehicle target detection model training include the BDD100K dataset and the "China Intelligent Vehicle Future Challenge" dataset. The BDD100K dataset comes from the AI Lab (BAIR) at the University of California, Berkeley. This dataset was collected from various locations across the United States and is based on the perspectives of vehicles driving on real roads. This dataset includes scenes from different times, climates, and road conditions, which increases the robustness of the vehicle target detection model. It contains approximately 100,000 images and 100,000 high-definition videos, with a variety of annotation types, including the required vehicle tags. The "China Intelligent Vehicle Future Challenge" dataset is based on the video data from the referee's perspective of the following vehicle in the competition. This data was collected on urban roads in the Changshu "Nine-square Grid" competition, and also includes a small number of private vehicles on the road, which helps to make the model more applicable.

[0080] To reduce training time for this project, 5,000 images were selected from the BDD 100K dataset as the vehicle detection dataset, with 3,000 images in the training set and 1,000 images in the validation and test sets. Labels and images in the BDD 100K dataset are occasionally missing. Therefore, before filtering the dataset, the images in the dataset were filtered and matched to their labels to identify the JPEG files and their corresponding JSON files. The dataset filtering and modification process was as follows: First, the JSON annotation file was converted to XML format, and non-classified "car" labels and small vehicle labels were filtered out. The XML file was then converted to a CSV table format containing the image name, size, horizontal and vertical coordinates of the objects to be detected, and the object type. The CSV files were converted to TFRecord files, resulting in the following files: train.csv, validation.csv, test.csv, train.tfrecord, validation.tfrecord, and test.tfrecord. A new configuration file, label_map.pbtxt, was created and a JSON-formatted item with an id of 1 and a name of "car" was added to enable TensorFlow to recognize the label file.

[0081] We extracted 1940 videos from the "China Smart Car Future Challenge" at a frame rate of 5 seconds each. We then labeled them using LabelImg in PyCharm, Python 3.6. Vehicle labeling was based on human visual perception. Vehicles with an occlusion greater than 70% or that failed to accurately display features were not labeled. Sanitation vehicles, buses, large passenger vehicles, and trucks were not labeled; only small vehicles were labeled. The corresponding XML file for the labeled images contained information about the image and the names of the detected objects. Finally, we converted the XML dataset file to CSV format, and then to TFRecord format, using the created label_map.pbtxt file.

[0082] The MobileNetV3-Large_SSDlite model was trained and tested using the processed BDD100K dataset. The training environment is shown in Table 2. During training, the learning rate (learning_rate) was set to 0.01, the batch size (batch_size) was set to 16, the number of training steps was determined based on convergence, and a validation step was set every 500 steps. When the learning rate (learning_rate) was modified to 0.005, 0.001, 0.0001, and 0.0003, the validation steps were set to 9,000, 12,000, 20,000, and 30,000.

[0083] Table 2

[0084]

[0085] The training results were tested on the BDD100K vehicle test set and the "China Smart Car Future Challenge" dataset. The results were compared to select the appropriate training model. The trained MobileNetV3-Large_SSDlite model was converted to PD format. This format was then converted to TFlite format using the TensorFlow Lite interface and metadata was converted. Finally, the object detection model was ported to the IVFC monitoring application. Before being used in autonomous driving tests, real-world testing was performed on a mobile phone.

[0086] The accuracy rate (Accurancy) commonly used in target detection is used as the evaluation index, and its calculation formula is:

[0087]

[0088] Where: a vehicle is correctly detected as TP; a false detection (a non-vehicle object is detected as a vehicle) is counted as FP; a missed detection (a vehicle object is detected as a non-vehicle) is counted as FN; and a non-vehicle object is detected as a non-vehicle object is counted as TN. In addition, in the vehicle object detection results, if its intersection over union (IoU) is greater than or equal to 0.5, it is considered a correct detection and is classified as TP; if the IoU is less than 0.5, it is classified as FP. The calculation formula for IoU is:

[0089]

[0090] Where: A represents the area of the correct detection box, and B represents the area of the detection box of the same object model.

[0091] After each training session, the model's loss on the dataset was calculated. After multiple training sessions, it was found that the loss decreased with each training session. Furthermore, the model was able to effectively detect small vehicles in the distance even in severe weather, with strong light interference, and in low light conditions.

[0092] Lane segmentation based on the vehicle-following perspective is a key step in identifying lane violations. When training the lane segmentation model based on the vehicle-following perspective, we used Baidu's Apollo Scape Lane Segmentation dataset. This dataset implements pixel-level road annotation, including four types of lane markings, sidewalks, and driving direction.

[0093] From this dataset, 10,000 images with clear lane line structures are selected as the training dataset. The processing of this dataset includes:

[0094] First, the dataset was preprocessed by cropping, and the images with a width of 3381 and a height of 2710 were cropped to images with a width of 3381 and a height of 1110. That is, the areas in the upper part of the image that do not contain the segmentation target, such as a large number of non-road data such as buildings and sky, were deleted to save training costs.

[0095] Secondly, the color labeled image is converted into the grayscale labeled image required for model training, and the value represented by the grayscale label and its corresponding category are determined.

[0096] Afterwards, the grayscale annotation image is converted, and the grayscale values are converted into trainIds. Only two labels are left in the grayscale image, lane lines and background. 0 is used to represent background, and 1 is used to represent lane lines.

[0097] Next, the cropped original image and the processed annotated image are proportionally compressed. Compression helps save training time and accelerate model convergence. Furthermore, with limited computing resources, the compressed image width and height are 1127 and 370 pixels, respectively, which can save significant computing space and increase the batch size.

[0098] Then, the processed original images and grayscale images are divided into training set, test set and validation set, and the amount of image data contained in each data set is 6000, 2000 and 2000 respectively.

[0099] Finally, the labeled result graph is converted to tfrecord format.

[0100] When marking lane lines in the "China Smart Car Future Challenge" dataset, the 1,940 competition images were first converted into mask images, and then these mask images were annotated using Labelme software, using 0 to represent the background and 1 to represent the lane line.

[0101] The processed Apollo Scape Lane Segmentation dataset was divided into training, test, and validation sets. The amount of data contained in each set and the number of categories in the dataset were recorded. Next, pixel counts were performed and weights were assigned to account for imbalanced data. Only the network architecture was used during training, without pretrained parameters. The batch size for training was 16, the image crop size was 513×513, the base learning rate was 0.001, and the decay rate was 0.998. During training, the learning rate was adjusted based on the convergence of the training results: 0.001, 0.0005, 0.0003, 0.0001, 0.0003, 0.0006, and 0.01, for a total of 80,000 steps.

[0102] The trained MobileNetV3-Large_DeepLabv3+ model was tested on the Apollo Scape LaneSegmentation test set and the "China Smart Car Future Challenge" dataset. When evaluating the lane segmentation results, IoU was used as the evaluation metric, and its calculation method was the same as (2).

[0103] After each training, the loss value of the MobileNetV3-Large_DeepLabv3+ model in the data set is calculated. After multiple trainings, it can be found that the loss value decreases with the number of trainings. From the application results of the MobileNetV3-Large_DeepLabv3+ model, it can be seen that for the vehicle currently traveling in the lane, the lane lines on both sides of the vehicle can be detected, and the lane line segmentation results are good. Since the referee car is always following behind the participating vehicles, the only lane lines that need to be detected are the lane lines on both sides of the lane where the referee car is located, so this result can meet the requirements of the present invention.

[0104] Convert the trained lane segmentation model into tflite format and then deploy it on a mobile phone. This enables the mobile phone to detect vehicles and separate lanes, providing data support for lane violation detection.

[0105] When training the above two models, you can use the Tensorflow Research Models deep learning interface to reduce the complexity of the training process.

[0106] In one embodiment, the trained model was ported to a Redmi 10X Android phone for vehicle detection testing. The average detection time per frame was approximately 120ms, effectively detecting the vehicle ahead. A lane segmentation test was also conducted, with a frame segmentation time of approximately 130ms, effectively detecting the lane markings on both sides of the current lane.

[0107] In the vehicle detection results, the center of its lower bounding box is selected as the vehicle coordinates to prevent the detection result from being overly dependent on the size of the detection box. Vehicle coordinates are grouped based on the principle of closest distance between frames. The lane segmentation results are converted into lane equations using the Hough transform. In the monitoring terminal image, due to the perspective effect, the distance between distant vehicles and lane lines in the image becomes increasingly closer. By performing an inverse perspective transformation on the vehicle and lane lines, a clearer relationship between the vehicle and lane lines is obtained. The vehicle's coordinates within the lane lines are then calculated based on the referee vehicle's own position and its relative position to the referee vehicle, serving as the vehicle's observed coordinates. A Kalman filter is then used to track and estimate the vehicle target to address inaccuracies in the observed vehicle coordinates. Finally, lane crossing is determined based on the relative distance between the vehicle and lane line, and the referee is notified.

[0108] The inverse perspective transformation is achieved through camera self-calibration. In this method, the direction of the referee car is aligned with the direction of the lane lines, the camera optical axis is aligned with the direction of the vehicle's travel, and the camera is self-calibrated using the characteristic that the lane lines are parallel to the camera's central axis.

[0109] Before performing the inverse perspective transform, since the lane segmentation result is a collection of pixel points, the result is not expressed as a straight line equation. Therefore, the segmentation result needs to be further transformed. The Hough transform converts the expression of a straight line in a rectangular coordinate system into the (r, θ) coordinate system. Therefore, all straight lines passing through a fixed point are represented as a point set in the (r, θ) coordinate system, and the line connecting this point set is a sine curve. In the (r, θ) coordinate system, the intersection of two sine curves represents the straight line determined by the two fixed points. Because the lane line area after lane line segmentation has a certain width, the lane line detection result is downsampled, and then the Hough transform is performed. Similar lines are merged to obtain the lane line equation.

[0110] The perspective effect in the image is as follows Figure 7 As shown, the perspective effect can be converted into the following formula:

[0111]

[0112]

[0113] Where: the earth coordinate system is (OXYZ), the image coordinate system is (oxyZ), X and Z are the coordinates of point P, x and y are the coordinates of point p, point p is the projection of point P on the image plane, parameter c represents the distance from the image plane to 0, and the Z axis is the optical axis of the camera.

[0114] In the geodetic coordinate system (OXYZ), assuming that the plane where the lane line is located is XZ, its bird's-eye view is as follows: Figure 8 As shown. In the figure, the coordinate system (OXZ) is the ground, and the lane plane is the construction plane, which is perpendicular to the XZ plane. Rotate (OXZ) around the Y axis by an angle of θ0 to make it parallel to the lane plane, and obtain the coordinate system (OX′Z′). Then translate the coordinate system (OX′Z′) by a distance of Z0 to coincide with the lane plane. According to the rotation and translation operations, the following formula can be obtained from the above perspective effect formula:

[0115]

[0116]

[0117] In the above method, the mapping relationship between the lane line and the image plane is converted into three unknown parameters: θ, Z and c.

[0118] Lane lines such as Figure 9 To apply this principle, first convert the lane lines in the image coordinate system {A} to the coordinate system {B}. The X axes of the two coordinate systems are the same, the Y axes are parallel, and the distance between the coordinate points is the image height h. Rotate the points in the lane line coordinate system {B} 90 degrees counterclockwise and then translate them to the right by h. The resulting lane line equation is the following formula:

[0119] y=kx+b (7)

[0120] At this time, the lane line equation is a diagonal line. The above method needs to be used to restore its parallelism and map the equally spaced line segments on the lane line to equal distances.

[0121] First, according to formulas (5) and (6) and the inverse perspective transformation, the following formulas (8) and (9) can be obtained.

[0122]

[0123]

[0124] Since all points on a straight line are projected onto the same straight line, substituting formula (7) into formula (9) yields formula (10). Since Y′ is a constant after the inverse perspective transformation, let this constant be K.

[0125]

[0126] According to formula (10), we can calculate the values of θ, c and Z. That is:

[0127] tanθ=-kc=bZ=Kcosθ (11)

[0128] Since k and b can be derived from the lane line equation, that is, θ and c can be determined, we only need to set the value of K to obtain the value of Z in the transformation equation. The value of K can be set according to actual needs. Then, based on the distance between the two lane lines after the inverse perspective transformation, the pixel distance in the image plane is converted to the actual distance. The results before and after the inverse perspective transformation in one embodiment are as follows: Figure 10a 、 Figure 10b shown.

[0129] Since the Kalman filter algorithm is small in size and fast in speed, it is very suitable for embedded devices. The present invention uses the Kalman filter to track and estimate the position of the vehicle. The Kalman filter uses the state of the previous moment to make the best estimate of the state of the object at the next moment. The Kalman filter calculation formula is (11) to (15), where is the prior state estimate of the nth step, the covariance of the prior estimation error K n is the Kalman gain, the Kalman gain update formula is (13), and the state update equation is (14). is the posterior estimate of the nth step. The posterior estimate is the weighted value of the prior estimate and the observed value. The covariance P of the posterior estimate error is n , whose equation is formula (13). Calculating formulas (11) and (12) to obtain the prior estimate at time n With prior covariance The Kalman gain, the posterior estimate, and the covariance of the posterior estimate error are calculated based on the prior covariance. Then, the calculated values of formulas (13), (14), and (15) are used as the next prior state and substituted into (11) and (12) for iteration.

[0130]

[0131]

[0132]

[0133]

[0134]

[0135] Before tracking the vehicle, the present invention uses the middle of the lane marking as the starting position of the referee car and continuously tracks the racing vehicle within the lane. If no vehicle or lane marking is detected for 10 consecutive frames, the starting position is reset. The displacement of the referee car relative to the starting position is then calculated based on the linear acceleration in the mobile phone.

[0136] When tracking a vehicle:

[0137] First, the vehicle detection results are grouped based on the principle of the shortest distance between frames. Then, the vehicle target is subjected to inverse perspective transformation and rotation transformation to obtain the position of the vehicle relative to the referee car. Among them, the rotation angle in the rotation matrix is based on the current lane line θ n The difference from the starting position θ0 is obtained.

[0138] Then, the vehicle observation coordinates in the ground coordinate system are calculated based on the referee car's own position and the vehicle's position relative to the referee car.

[0139] Finally, the vehicle's state vector X is expressed as x, y represent the vehicle coordinates, Indicates the speed of the vehicle in both directions. The observation state Z is expressed as [x Z y Z ] T , x Z 、y Z Represents the observed coordinate value of the vehicle. The state equation of the vehicle is formula (16), and the observation equation is formula (17). Where F is the state transfer matrix, H is the observation matrix, and w n With v n The Gaussian noise in the estimation process and the measurement process are respectively set, and the values of the process noise matrices Q and R are set according to experience. The example diagram of the above Kalman filter model is as follows Figure 11 shown.

[0140] X n =FX n-1 +w n-1 w n-1 ~N(0,Q) (16)

[0141] Z n =HX n +v n vn~N(0,R) (17)

[0142] in:

[0143]

[0144] In the above process, the present invention obtains the relative position of the measured vehicle and the lane line in the ground coordinate system through inverse perspective transformation and Kalman filtering. Then, the distance from the vehicle to the lane lines on both sides is calculated by formulas (18) and (19), that is, the distance from the point to the straight line is calculated, as follows: Figure 12 In the above embodiment, when the ratio of the distance between the vehicle and the left and right lanes of the road is less than 1:5 in the direction of vehicle travel, it is determined that the vehicle has crossed the line, and the referee is reminded. At the same time, the result is stored in the server to facilitate post-match review by the referee.

[0145] ay+bx+c=0 (18)

[0146]

[0147] To verify the effectiveness of the above solution, we conducted tests and validation using a dataset from the "China Smart Car Future Challenge" race monitoring competition. In this dataset, the referee car continuously tracks the vehicle under test at a certain distance. First, we extracted 120 minutes of road driving from the referee car's perspective, totaling 530 lane crossings. The time at which the vehicle crossed the lane was recorded, and the marked time was compared with the test result time. Because lane crossing is a continuous process, and lane changes are relatively slow, any difference between the recorded and marked time within 3 seconds is considered a correct detection.

[0148] To further validate the effectiveness of this solution, experiments were conducted on both urban and town roads. In the town road experiment, two vehicles were used to simulate a follow-up monitoring scenario in a safe area with clear lane markings and no pedestrians. A mobile phone equipped with the IVFC monitoring app was placed in the referee vehicle to identify vehicles crossing the lane. The urban road experiment lasted 85 minutes and included 500 lane crossing scenarios; the town road experiment lasted 325 minutes and included 2000 lane crossing scenarios.

[0149] The experimental quantitative results are shown in Table 4. Figure 13a 、 Figure 14a They are real pictures of cities and towns respectively. Figure 13b 、 Figure 14b for Figure 13a 、 Figure 14a Corresponding pressure line reminder.

[0150] Table 4

[0151]

[0152] Vehicle crossing line discrimination is a classification model. Therefore, the present invention uses accuracy to describe the detection effect, and uses missed detection rate and false detection rate to further illustrate the detection effect. The missed detection rate and false detection rate are calculated using formula (20) and formula (21), where missed detection is counted as FN, correct crossing line detection is counted as TP, false detection is counted as FP, and correct non-crossing line detection is counted as TN. For the quantitative results of the above data sets shown in Table 4, the accuracy rate is above 75%, which can achieve the purpose of reminding referees to perform crossing line discrimination.

[0153]

[0154]

[0155] This method has poor lane line detection performance on construction sections with severe lane line pollution, but performs well on sections with clear lane lines.

[0156] In summary, in the above-mentioned embodiments, the present invention utilizes the test site dispatching and command center as a platform, leverages the public telecommunications network to construct a test site monitoring network, and deploys a large number of Android phones as mobile roadside and onboard information collection terminals for an unmanned vehicle monitoring system. Vehicle crossing behavior detection is also incorporated into the monitoring terminals to improve monitoring quality. The proposed unmanned vehicle test monitoring system is easy to deploy and, compared to the current large number of fixed roadside monitoring devices and Wi-Fi coverage methods installed at unmanned test sites, is flexible, convenient, and cost-effective. The IVFC monitoring application, based on Android terminals, utilizes Android cameras, JavaCV, GPS, YUV, and bitmap technologies to implement data collection and processing, as well as synchronized timestamps and location stamps. A server based on Nginx, Spring Boot, and MySQL is designed and implemented. By configuring the rtmp and http modules within Nginx, a streaming media server with video transmission, broadcasting, storage, and fixed-point video playback is implemented. The information management server is implemented using Spring Boot, MySQL, HTML, CSS, and JavaScript. By building a high-availability cluster, it can also meet the real-time transmission, storage, broadcasting, and playback requirements for multiple video channels. While providing data support for unmanned driving evaluation, the system significantly reduces the workload of referees. Using the preset MobileNetV3_SSD and MobileNetV3_DeepLabv3+ models, vehicle detection and lane line separation can be performed on a monitoring terminal with limited computing power. Kalman filtering is used to complete the tracking estimation of the tested vehicle. Then, based on the relative position of the tested vehicle and the lane line in the road coordinate system, lane crossing judgment is made. This can effectively assist referees in improving the efficiency and quality of judging illegal lane crossing behaviors. The system can also incorporate more automatic detection functions for violations such as running red lights, speeding, and collisions, which can further reduce the workload of referees and reduce the number of referees.

[0157] The system proposed by the present invention can also be applied to professional unmanned driving tests to replace existing monitoring systems. The automatic violation detection function of the present invention can also be applied to traffic management systems.

[0158] In the present invention, the preset MobileNetV3_SSD and MobileNetV3_DeepLabv3+ models can be replaced by other lightweight neural network models. Through the description of the above implementation methods, those skilled in the art can clearly understand that the present disclosure can be implemented by means of software plus necessary general-purpose hardware. Of course, it can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present disclosure, software program implementation is a better implementation method in most cases.

[0159] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification and without departing from the scope of protection of the claims of the present invention, may also devise various forms, all of which fall within the scope of protection of the present invention.

Claims

1. An unmanned driving test monitoring system, comprising a monitoring terminal and a server, characterized in that: The monitoring terminal integrates a camera, positioning functions and is capable of connecting to a communication network; The monitoring terminal is configured to collect monitoring videos during unmanned driving tests, detect vehicle lane violations in images in the monitoring videos, and send violation information to the server for storage; At the same time, the monitoring video is parsed, and after marking the timestamp and location stamp, it is encoded and compressed and pushed to the server; wherein, the marking of the timestamp and location stamp includes: obtaining the GPS time and GPS location coordinates, and expressing them with bitmap codes; obtaining the bitmap codes in sequence, and if the code value is "1", calculating its corresponding position in the collected video image, and modifying the Y component of the position, so as to fuse the timestamp or address stamp into the collected video image; the vehicle lane crossing violation detection includes the following steps: using a preset lane line segmentation model to segment the lane lines in the video image, and converting the lane line segmentation result into a lane line equation; based on the lane line equation, performing an inverse perspective transformation to obtain the distance between the two lane lines; using a preset target detection model to identify the vehicle target in the video image for detection, and using Kalman filtering to track and estimate the vehicle target; judging whether to perform lane crossing marking based on the distance between the vehicle and the lane line, and generating a prompt message if the lane crossing marking is performed; The server includes an information management server and a streaming media server supporting high concurrency; The streaming media server is configured to receive, store, broadcast and play videos at designated locations; The information management server includes a database configured to store and retrieve violation information and video information, and verify the opening and closing of video monitoring.

2. The system according to claim 1, wherein: The monitoring terminal is placed in the referee car to realize vehicle-following perspective monitoring, or is placed on the roadside to realize road monitoring.

3. The system according to claim 1, wherein: The process of encoding and compressing the monitoring video and pushing it to the server includes the following steps: Set the obtained streaming address as the streaming channel; Get the encoding format; After the captured YUV format image is converted into RGB format image, it is encoded and compressed according to the encoding format, and then pushed to the server through the push channel.

4. The system according to claim 1, wherein: The marking of the lane line according to the distance between the vehicle and the lane line includes: In the direction of vehicle travel, when the ratio of the distance between the vehicle and the lane lines on the left and right sides of the road is less than 1:5, a line crossing mark is performed, and the time and location of the line crossing behavior are recorded in the mark.

5. The system according to claim 1, wherein: The preset target detection model adopts the MobileNetV3-Large_SSDlite model; The default lane segmentation model uses the MobileNetV3-Large_ DeepLabv3+ model; After training, the MobileNetV3-Large_SSDlite model and the MobileNetV3-Large_ DeepLabv3+ model are converted into tflite format and ported to the monitoring terminal.

6. The system according to claim 1, wherein: The converting of the lane line segmentation result into a lane line equation includes: The lane line segmentation results are downsampled, Hough transform is performed, and similar straight line fitting is performed to merge to obtain the lane line equation.

7. The system according to claim 1, wherein: Before using Kalman filtering to track and estimate vehicle targets, the following steps are required: The vehicle coordinates are grouped based on the principle of the shortest distance between frames. After the vehicle targets are subjected to inverse perspective transformation and rotation transformation, the position of the vehicle relative to the referee car is obtained. The position of the referee car is calculated by the monitoring terminal based on the starting position of the referee car. According to the position of the referee car and the position of the vehicle relative to the referee car, the position of the vehicle in the lane line is calculated as the vehicle observation position, and is used as the input of the Kalman filter for tracking estimation.

8. The system according to claim 1, wherein: The monitoring video collection includes: Apply for monitoring based on the universal unique identification code generated by the monitoring terminal and the information required for monitoring obtained; After receiving the server authorization, the monitoring terminal can choose to start monitoring or end monitoring; when choosing to start monitoring, the monitoring terminal begins to collect monitoring video.

Images