V2i car networking image data transmission system and method based on blind area prediction

Abstract

The application discloses a V2I vehicle networking image data transmission system and method based on blind area prediction, which comprises a target detection and tracking module arranged at a vehicle end, a blind area prediction module and a data scheduling module arranged at an edge server end; the target detection and tracking module is used for acquiring a target object image in front of the vehicle, detecting the target object image, identifying and acquiring a boundary box of the target object and performing target tracking; the blind area prediction module is used for performing blind area prediction and obtaining a future boundary box trajectory prediction position as a blind area prediction value; and the data scheduling module is used for performing data scheduling according to the blind area prediction value, uploading image data obtained by a vehicle sensor to the edge server and issuing the image data to a required vehicle. Compared with the prior art, the application utilizes blind area prediction to perform early scheduling of early data, so that data transmission of the intelligent networked vehicle is more timely, stable and reliable, and important information blocked by a visual blind area is effectively supplemented.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent networked vehicle data transmission, in particular to a V2I vehicle networking image data transmission technology based on blind area prediction. BACKGROUND

[0002] Intelligent networked vehicle image data transmission mainly transmits and shares image data obtained by camera sensors through vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) data transmission. Compared with V2V, which transmits data captured by the vehicle's own sensors to other nearby vehicles, V2I mainly transmits data between roadside edge devices and vehicles. The vehicle can use the powerful computing power of the edge device to process data at high speed by transmitting its own data to the edge device. At the same time, the data obtained by the sensors on the edge device can also be shared with the vehicle. After receiving the processed data, the vehicle can supplement the information of its visual blind area, so that the driver and the vehicle have a better driving environment.

[0003] V2I data transmission technology can increase the perception range of the vehicle, discover blind area information blocked by obstacles, supplement the driver's field of view, and improve the road traffic safety of the vehicle. However, the amount of data obtained by sensors such as cameras at each moment is very large, and the traffic situation changes rapidly, making these sensor data have a very high timeliness. The large amount of data transmission will occupy a large amount of bandwidth resources, which will greatly affect the real-time performance of data transmission. At the same time, the urgency and importance of different road information are not the same. For example, for some blind area blocking information, including pedestrians and bicycles on the roadside, the data transmission priority should be higher. In addition, different vehicles may obtain the same data, and repeated transmission of data will further exacerbate the waste of bandwidth resources. SUMMARY

[0004] The purpose of the present application is to provide a V2I vehicle networking image data transmission system based on blind area prediction, which realizes scheduling request based on blind area prediction, schedules data in advance, and can timely transmit data in urgent need.

[0005] The present application utilizes the following technical solutions:

[0006] A V2I vehicle networking image data transmission system based on blind area prediction, the system comprising a target detection and tracking module, a blind area prediction module arranged on a vehicle side, and a data scheduling module arranged on an edge server side.

[0007] The target detection and tracking module is configured to acquire target object images in front of the vehicle through a vehicle-mounted camera, detect the target object images, identify and acquire a bounding box of the target object, perform target tracking, record a sequence of position trajectories of the bounding box of the same target object in a past period of time as an input of the blind area prediction module;

[0008] The blind area prediction module is configured to input the sequence of position trajectories of the bounding box of the same target object in the past period of time as an input of a graph neural network to perform blind area prediction, obtain a predicted position of a future bounding box trajectory as a blind area prediction value, and initiate a data scheduling request at the predicted position of the bounding box trajectory and send the data scheduling request to the data scheduling module located at an edge server end.

[0009] The data scheduling module is configured to perform data scheduling according to the blind area prediction value, upload image data acquired by the vehicle to the edge server, and deliver the image data to a vehicle in need.

[0010] 1. A V2I vehicle networking image data transmission method based on blind area prediction, characterized in that the method comprises the following steps:

[0011] Acquiring target object images in front of the vehicle, detecting the target object images, identifying and acquiring a bounding box of the target object, performing target tracking, and recording a sequence of position trajectories of the bounding box of the same target object in a past period of time as an input of blind area prediction.

[0012] Inputting the sequence of position trajectories of the bounding box of the same target object in the past period of time as an input of a graph neural network to perform blind area prediction, obtaining a predicted position of a future bounding box trajectory as a blind area prediction value, and initiating a data scheduling request at the predicted position of the bounding box trajectory.

[0013] Performing data scheduling according to the blind area prediction value, uploading vehicle data to an edge server, and delivering the vehicle data to a vehicle in need.

[0014] Compared with the prior art, the V2I vehicle networking image data transmission system and method based on blind area prediction have the following technical advantages:

[0015] 1) Blind area prediction is used to perform early data scheduling, so that the data transmission of the intelligent networked vehicle is more timely, stable and reliable, and important information blocked by the visual blind area is effectively supplemented.

[0016] 2. Different vehicles are allocated with different data transmission rates at different times to realize adaptive rate data transmission.

[0017] 3. Image data of repeated targets is filtered out to ensure that all target contents are possessed while the data amount is minimized, data redundancy is sufficiently reduced, network load is greatly reduced, and transmission is more efficient. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A V2I vehicle networking image data transmission system module diagram based on blind area prediction.

[0019] Figure 2 A target detection and tracking module flow chart.

[0020] Figure 3 A blind area prediction module flow chart.

[0021] Figure 4 A data scheduling module flow chart. DETAILED DESCRIPTION

[0022] The technical solutions of the present application will be described in further detail below in conjunction with the drawings and specific embodiments.

[0023] As shown in Figure 1 A V2I vehicle networking image data transmission system module diagram based on blind area prediction. The system includes a target detection and tracking module 100 arranged at the vehicle end, a blind area prediction module 200, and a data scheduling module 300 arranged at the edge server end.

[0024] The target detection and tracking module 100 is used to obtain the target object image in front of the vehicle through the vehicle-mounted camera, detect the target object image, identify and obtain the bounding box (2D) of the target object, perform target tracking, and obtain the bounding box position trajectory of each target object in the past period of time. The bounding box position trajectory sequence of the same target object in the past period of time is recorded in sequence as the input of the blind area prediction module. The target image in front of the vehicle is obtained through the vehicle-mounted camera.

[0025] The blind area prediction module 200 is used to input the bounding box position trajectory sequence of the same target object in the past period of time as the input of the graph neural network to perform blind area prediction, obtain the future bounding box trajectory prediction position, take the future bounding box trajectory prediction position as the blind area prediction value (including size and position), start a data scheduling request at the bounding box trajectory prediction position, and send the data scheduling request to the data scheduling module 300 at the edge end.

[0026] The data scheduling module 300 is used to perform data scheduling according to the blind area prediction value, upload the image data obtained by the vehicle sensor that meets the requirements to the edge server, perform content selection, filter out duplicate data, and distribute the data to the required vehicle.

[0027] As shown in Figure 2 A target detection and tracking module flow chart. The specific steps are described as follows:

[0028] Step 1.1: Initialize parameters. The target object image detection technology used in this module is the YOLO-V4 image detection algorithm, which borrows from the residual network structure to form a deeper network layer and multi-scale detection. The target tracking technology used is the DeepSort multi-object tracking algorithm, which adopts a new association method to improve the accuracy of tracking objects that have been occluded for a long time. Set the detector and tracker, filter coefficient 0.7, neural network layer number 420, detection category "car", association distance 0.4, etc.

[0029] Step 1.2: Acquire camera data;

[0030] Step 1.3: Input the image data into the YOLO-V4 image detection model, obtain the network output layer, and obtain the filtered detection category and 2D bounding box based on the filtering coefficients;

[0031] Step 1.4: Extract features, that is, extract the image features of all target objects in the current frame based on the range of the 2D bounding box, and save the image features;

[0032] Step 1.5, Tracking Prediction: Based on the tracking target created in the previous 3 frames, use Kalman filtering to predict its position in the current frame. Kalman filtering is an algorithm that uses the state equations of a linear system to make an optimal estimate of the system state using system input and output observation data;

[0033] Step 1.6, Tracking Update and Feature Matching: Perform cascaded feature matching on the features of the tracked target in the previous 3 frames and the image features of all target objects in the current frame from Step 1.4. If a match is found with all features in the previous 3 frames, update the tracked target or create a new tracked target. Because matching with features from the previous 3 frames is required, the tracked target will not be created until the 4th frame.

[0034] Step 1.7: Draw the 2D bounding box, that is, draw the 2D bounding box of the tracking target;

[0035] Step 1.8: Store 2D bounding boxes: Store the obtained tracking targets and their 2D bounding boxes. For each tracking target, store the data of the latest 8 consecutive frames of 2D bounding boxes. Since it takes 8 frames to predict 1 frame, blind zone prediction in this step starts from the 9th frame.

[0036] like Figure 3 The diagram shows the blind spot prediction module process, which includes the following steps:

[0037] Step 2.1: Initialize parameters. This module uses the Social-STGCNN graph neural network model. Social-STGCNN mainly consists of a spatiotemporal graph convolutional neural network (ST-GCNN) and a temporal extrapolator convolutional neural network (TXP-CNN), which can effectively model social interactions and predict future trajectories. Set the weights of the trained model, the input sequence length is 8 frames, the prediction sequence length is 1 frame, and the threshold P = 524.

[0038] Step 2.2: Obtain blind zone prediction data: Define the blind zone as the 2D bounding box for target detection and tracking, with the blind zone size equal to the size of the 2D bounding box. For each time t, the 2D bounding box of each target object in each frame of the target image is denoted as... Where h represents the 2D bounding box of the detected blind zone, Let H represent the input, H be the set of bounding boxes, t be a certain time, and T be the set of times. Indicates input, These represent the x and y coordinates of the top-left pixel coordinates of the h-th blind zone bounding box in the image at time t, respectively. This represents the x and y coordinates of the lower right corner; the corresponding blind spot prediction value is denoted as... Indicates the blind zone prediction value. Let x and y represent the x and y coordinates of the top-left pixel coordinates of the h-th blind zone bounding box at time t+1, the next time interval after t. These represent the x-coordinate and y-coordinate of the lower right corner, respectively.

[0039] Step 2.4: Use the processed data as input to the graph neural network, i.e., input it into the Social-STGCNN model;

[0040] Step 2.5: Obtain the output of the Social-STGCNN model and calculate the predicted value. In this invention, it is assumed that the 2D bounding box follows a binary Gaussian distribution, and the bounding box expression is as follows:

[0041]

[0042]

[0043] Where N represents a bivariate Gaussian distribution, Represents the mean of the distribution. Represents the variance of the distribution. Represents the correlation coefficient;

[0044] Then, the output value of the neural network is calculated as the predicted value based on the binary Gaussian distribution.

[0045] Step 2.6: Send a scheduling request based on the blind zone prediction value, including: calculating the blind zone size of each target object at this time, as shown in the following expression:

[0046]

[0047] if This indicates that this vehicle requires data scheduling; set the scheduling information k for vehicle i. i =1, where i represents the vehicle, which sends a data scheduling request and its location information to the edge server. Here, P is the blind spot prediction threshold, set to P=526.

[0048] This invention models the data upload and download process. Here, set N represents vehicles, set T represents time, set L represents available transmission rate, and set E... t Let represent all targets detected by all vehicles at time t. Vehicle i represents the vehicle that needs data, and vehicle j represents the vehicle that provides data to vehicle i. i represents the vehicle number that needs data, and j represents the vehicle number that provides data. i,j∈N. As shown in Table 1, it is a variable description table.

[0049] Table 1

[0050]

[0051] The vehicle data upload restrictions are expressed as follows:

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] The vehicle data distribution restriction conditions are expressed as follows:

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] During the upload process, formula (1) ensures that each vehicle can only use one data rate to upload data to the edge server at any given time. Formula (2) limits the proportion of data transmission time between the vehicle and the edge server. Formulas (3) to (5) represent variables.

[0065] During the distribution process, formula (6) means ensuring that each object e m ∈E t It must be detected by at least one vehicle. Formula (7) indicates that the downloaded data must be selected from the uploaded data at the same time. Formulas (8) to (11) represent variables.

[0066] The objective of this invention is to minimize the amount of data received by the vehicle from the edge server within each time period. The objective function expression is as follows:

[0067]

[0068] like Figure 4 The diagram shown is a flowchart of the data scheduling module, which includes the following steps:

[0069] Step 3.1: Obtain the scheduling request, and get the scheduling information k for each vehicle from Step 2.6. i And the location information of each vehicle.

[0070] Step 3.2: Configure vehicle i to require data upload from vehicle j. Iterate through each car, according to formula (4) and the k of each car. i and location information, settings The value of .

[0071] Step 3.3: Calculate the data requirement for each vehicle j and sort them:

[0072] Statistical data requirement for each vehicle (s) j,l,t The expression is as follows:

[0073]

[0074] in, This represents the probability of the edge server receiving data when vehicle j uploads data at a rate l at time t. Its value is obtained from the piecewise function of distance and receiving probability at different rates obtained in the experiment.

[0075] Next, the data requirement for each vehicle will be s. j,l,t Sort by largest to smallest;

[0076] Step 3.4: Allocate upload data rate for each vehicle: based on the data demand (s) of each vehicle. j,l,tSort the data and assign the index (j,l) to the specified values ​​in order. if This indicates that the data for vehicle j is uploaded to the edge server at time t at a rate l. To avoid uploading, calculate the reception rate of data content from vehicle j within a time slot t. The expression is as follows:

[0077]

[0078] Step 3.5: The edge server receives vehicle data, if... Then the data of vehicle j will be uploaded to the edge server at time t at a rate l;

[0079] Step 3.6: Select all data from vehicle j required by vehicle i using a greedy algorithm: First, count the number of target objects captured by each vehicle i and sort them from largest to smallest; then, take out the data from vehicle j in order. If none of the targets of vehicle j are in the selection set S, add the data of vehicle j to the selection set and set... Data from vehicle j is transmitted from the edge server to vehicle i at time t at rate l; conversely, all targets of vehicle j are in the selection set S, and the settings are... No data uploaded;

[0080] Step 3.7: Allocate a rate for the distributed data: Similar to steps 3.3 and 3.4, calculate the data demand s for each vehicle. j,l,t The expression is as follows:

[0081]

[0082] in, This represents the probability that vehicle i will receive data when vehicle j uploads data at rate l at time t.

[0083] Based on the data requirement s for each vehicle j,l,t Sort the data and assign the index (j,l) to the specified values ​​in order. Assign values, and then calculate the reception rate of vehicle i for the data content of vehicle j within one time slot t, as shown in the following expression:

[0084]

[0085] Step 3.8: Edge server distributes vehicle data: Similar to step 3.6, a greedy algorithm is used to select all the data from vehicle j needed by vehicle i: First, the number of target objects captured by each vehicle j is counted and sorted in descending order; then, the data from vehicle j is retrieved sequentially. If a target of vehicle j is not in the selection set S, then the target of vehicle j is added to the selection set S. Data from vehicle j is sent from the edge server to vehicle i at time t at rate l; conversely, if all targets of vehicle j are in the selection set S, then... No data will be distributed.

[0086] This invention is primarily aimed at intelligent connected vehicles, thereby effectively avoiding various traffic accidents caused by blind spots, reducing personal injury and vehicle property damage, and improving road traffic safety.

[0087] The above description is merely a preferred embodiment of the present invention and is intended only to illustrate the technical concept and features of the present invention. It is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A V2I (Vehicle-to-Everything) image data transmission system based on blind spot prediction, characterized in that, The system includes a target detection and tracking module and a blind spot prediction module installed on the vehicle side, as well as a data scheduling module installed on the edge server side; The target detection and tracking module is used to acquire images of target objects in front of the vehicle through the vehicle-mounted camera, detect the target object images, identify and acquire the bounding box of the target object, track the target, and record the bounding box position trajectory sequence of the same target object over a period of time as input to the blind spot prediction module. The blind spot prediction module is used to use the sequence of bounding box position trajectories of the same target object over a period of time as input to the graph neural network to predict the blind spot, obtain the future bounding box trajectory prediction position as the blind spot prediction value, and initiate a data scheduling request at the bounding box trajectory prediction position and send it to the data scheduling module located on the edge server. The data scheduling module is used to send a data scheduling request and the vehicle's location information to the edge server based on the blind spot prediction value, perform data scheduling for the vehicle, upload the image data acquired by the vehicle to the edge server, and distribute it to the vehicles that need it; the data scheduling module specifically includes the following processing: Obtain a dispatch request, which includes dispatch information for each vehicle. And the location information of each vehicle; Set up vehicles Vehicle needed Data upload flag ( ); Statistics on the data requirements for each vehicle The expression is as follows: ； in, Indicates vehicle exist Time in rate The probability of the edge server receiving data during data upload. This indicates the vehicle number for which data is required. Indicates the vehicle number that provided the data; Next, the data requirements for each vehicle will be determined. Sort by largest to smallest; Data upload rate allocation: based on the data demand of each vehicle. Sort by index in order. Give ,if Indicates vehicle Data in Time in rate Uploaded to the edge server. To avoid uploading, calculation is performed within one time slot. Inside the vehicle Data content reception rate The expression is as follows: ； if Then the vehicle Blind spot prediction data in Time in rate Uploaded to the edge server; statistics for each vehicle. The number of captured target objects is sorted from largest to smallest; then, vehicles are retrieved in sequence. Blind spot prediction data, such as those not included in the selected set In the middle, upload blind spot prediction data; otherwise, do not upload blind spot prediction data. Data transmission rate allocation: based on the data demand of each vehicle. Sort by index in order. Give Assign values, then calculate the vehicle. In a time slot Inside the vehicle The data reception rate is expressed as follows: ； Statistics for each vehicle The number of captured target objects is sorted from largest to smallest; vehicles are retrieved in sequence. Data, such as vehicles None of the targets were in the selection set. In the middle, the vehicle Add to selection set, settings , will the vehicle Data in Time in rate Issued to vehicles from the edge server Conversely, vehicles All objectives are in the selection set. In the settings No data will be distributed.

2. The V2I vehicle-to-everything (V2I) image data transmission system based on blind spot prediction as described in claim 1, characterized in that, The target detection and tracking module specifically includes the following processes: Acquire camera data; Input image data into the YOLO-V4 image detection model, obtain the network output layer, and obtain the filtered detection categories and 2D bounding boxes based on the filtering coefficients; Based on the bounding box range, extract the image features of all target objects in the current frame and save the image features; Based on the tracking target created in the previous 3 frames, use Kalman filtering to predict its position in the current frame; Perform concatenated feature matching on the features of the tracked target in the previous 3 frames and the image features of all target objects in the current frame. If a match is found with all 3 previous frames, update the tracked target or create a new tracked target. Draw the bounding box of the tracked target; Store the tracked target and its bounding box.

3. The V2I vehicle-to-everything (V2I) image data transmission system based on blind spot prediction as described in claim 1, characterized in that, The blind spot prediction module specifically includes the following processing: The blind zone is defined as the bounding box for target detection and tracking at each time step. The bounding box of each target object in each frame of the target image is denoted as . ,in, The bounding box represents the detected blind zone. For the bounding box set, At a certain moment, A collection of moments, Indicates input, They represent in The first time in the image at time 1 The x and y coordinates of the top-left pixel coordinates of the blind zone bounding box, and the corresponding... The x and y coordinates of the lower right corner are represented by the blind zone prediction value, denoted as . , Indicates the blind zone prediction value. They represent in The next moment At that moment, the The x and y coordinates of the top-left pixel coordinates of the blind zone bounding box These represent the x and y coordinates of the lower right corner, respectively, enabling the acquisition of blind zone prediction data; The blind spot prediction data is used as the input to the graph neural network, which is then fed into the Social-STGCNN model. Obtain the output of the Social-STGCNN model, calculate the predicted value, and use the output value of the neural network as the predicted value based on the binary Gaussian distribution. ; Calculate the first Time of the first The predicted value of the blind zone size is The expression is as follows: ； if If this vehicle requires data scheduling, it sends a data scheduling request and its location information to the edge server based on the blind spot prediction value. This represents the blind zone prediction threshold.

4. A V2I vehicle-to-everything (V2I) image data transmission method based on blind spot prediction, characterized in that, The method includes the following steps: The system acquires images of target objects in front of the vehicle, detects and identifies the bounding boxes of the target objects, tracks the targets, and records the bounding box position trajectory sequence of the same target object over a period of time as input for blind spot prediction. The sequence of bounding box position trajectories of the same target object over a period of time is used as input to the graph neural network for blind zone prediction. The predicted position of the future bounding box trajectory is used as the blind zone prediction value, and a data scheduling request is initiated at the predicted position of the bounding box trajectory. Data scheduling is performed based on blind spot prediction values, uploading vehicle data to the edge server and distributing it to the required vehicles; the data scheduling further includes the following steps: Obtain a dispatch request, which includes dispatch information for each vehicle. And the location information of each vehicle; Set up vehicles Vehicle needed Data upload flag ; Statistics on the data requirements for each vehicle The expression is as follows: ； in, Indicates vehicle exist Time in rate The probability of the edge server receiving data during data upload; Next, the data requirements for each vehicle will be determined. Sort by largest to smallest; Data upload rate allocation: based on the data demand of each vehicle. Sort by index in order. Give ,if Indicates vehicle Data in Time in rate Uploaded to the edge server. To avoid uploading, calculation is performed within one time slot. Inside the vehicle Data content reception rate The expression is as follows: ； if Then the vehicle Blind spot prediction data in Time in rate Uploaded to the edge server; statistics for each vehicle. The number of captured target objects is sorted from largest to smallest; then, vehicles are retrieved in sequence. Blind spot prediction data, such as those not included in the selected set In the middle, upload blind spot prediction data; otherwise, do not upload blind spot prediction data. Data transmission rate allocation: based on the data demand of each vehicle. Sort by index in order. Give Assign values, then calculate the vehicle. In a time slot Inside the vehicle The data reception rate is expressed as follows: ； Statistics for each vehicle The number of captured target objects is sorted from largest to smallest; vehicles are retrieved in sequence. Data, Such as vehicles None of the targets were in the selection set. In the middle, the vehicle Add to selection set, settings , will the vehicle Data in Time in rate Issued to vehicles from the edge server Conversely, vehicles All objectives are in the selection set. In the settings No data will be distributed.

5. The V2I vehicle-to-everything (V2I) image data transmission method based on blind spot prediction as described in claim 4, characterized in that, The steps of detecting, recognizing, and acquiring the bounding box of the target object image and performing target tracking further include the following steps: Input image data into the YOLO-V4 image detection model, obtain the network output layer, and obtain the filtered detection categories and 2D bounding boxes based on the filtering coefficients; Based on the bounding box range, extract the image features of all target objects in the current frame and save the image features; Based on the tracking target created in the previous 3 frames, use Kalman filtering to predict its position in the current frame; Perform concatenated feature matching on the features of the tracked target in the previous 3 frames and the image features of all target objects in the current frame. If a match is found with all 3 previous frames, update the tracked target or create a new tracked target. Draw the bounding box of the tracked target; Store the tracked target and its bounding box.

6. The V2I vehicle-to-everything (V2I) image data transmission method based on blind spot prediction as described in claim 4, characterized in that, The blind zone prediction further includes the following steps: The blind zone is defined as the bounding box for target detection and tracking at each time step. The bounding box of each target object in each frame of the target image is denoted as . ,in, The bounding box represents the detected blind zone. Indicates the input, and the blind zone prediction value is denoted as... , This represents the blind zone prediction value, enabling the acquisition of blind zone prediction data. The blind spot prediction data is used as the input to the graph neural network, which is then fed into the Social-STGCNN model. Obtain the output of the Social-STGCNN model, calculate the predicted value, and use the output value of the neural network as the predicted value based on the binary Gaussian distribution. ; Calculate the blind zone prediction value at this time. The expression is as follows: ； if This indicates that the vehicle needs data scheduling. Based on the blind spot prediction value, a data scheduling request and the vehicle's location information are sent to the edge server.

Images