Video configuration method for in-vehicle edge network

Through Bayesian online learning and Gaussian model video configuration methods, the on-board video configuration is dynamically optimized, and the problems of environmental dependence and cost in the existing technology are solved, more efficient video analysis is achieved, and road safety is enhanced.

CN114828048BActive Publication Date: 2024-11-26XIDIAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210405535.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-18
Publication Date
2024-11-26
Estimated Expiration
2042-04-18

AI Technical Summary

Technical Problem

The existing on-board video configuration method is only applicable to fixed environments, relying on the assumptions of deep learning models and communication links, resulting in inability to adapt when network conditions change, affecting video analysis accuracy and delay performance, increasing video configuration costs and threats Road safety.

Method used

Using Bayesian online learning and Gaussian model, through the interaction between client vehicles and edge computing nodes, the video configuration is dynamically selected, and iterative optimization is performed according to the delay and accuracy of the current environment. The selection of the optimal configuration does not depend on the gold configuration. Reduce costs and improve video analysis accuracy and delay.

Benefits of technology

It expands the applicable environment for video configuration, reduces the cost of video configuration, improves the accuracy and delay performance of video analysis, and enhances road safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114828048B_ABST
    Figure CN114828048B_ABST
Patent Text Reader

Abstract

The present invention discloses a video configuration method for an in-vehicle edge network, which mainly solves the problems of high video configuration cost and fixed use environment of the existing in-vehicle video configuration method. The scheme is: 1) constructing an in-vehicle edge network; 2) the client vehicle user collects video data through the in-vehicle camera to generate the video to be divided; 3) the client vehicle user establishes a communication link with the computing node; 4) applying Bayesian online learning to obtain the delay and accuracy of the video to be analyzed under different video configurations, and applying the Gaussian model to measure the similarity between the current configuration and the configurations obtained by learning the delay and accuracy, and obtaining the optimal configuration according to the similar performance of similar configurations and the frequency of configurations appearing in the potential video configuration set; 5) repeating 4) regularly according to the changes in the video content to obtain a new optimal video configuration. The present invention has low configuration cost, wide applicable environment, safe road maintenance, and can be used for autonomous driving vehicles to maintain road safety.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and further relates to a video configuration method that can be used for autonomous driving vehicles to maintain road safety. Background Art

[0002] Currently, autonomous vehicles and advanced driver assistance systems rely heavily on onboard sensors to identify and assess potential hazards and take necessary actions, which poses a challenge to the accuracy and real-time nature of sensor data analysis. The development of wireless networks provides a way to do this. With the development of modern communication technologies, connected cars are closer to becoming a reality than ever before. Cellular vehicle wireless communications and dedicated short-range communications have facilitated large-scale vehicle communications due to significant improvements in bandwidth, latency, and reliability. In addition, new regulations provide a beneficial legal background for vehicles on public roads, paving the way for enhanced information perception of autonomous and assisted driving vehicles, thereby improving road safety.

[0003] At the same time, with the development of machine learning, powerful servers deployed near the road can run computer-intensive neural networks, which provides convenience for real-time inference of video analysis for vehicle users. In particular, video analysis based on deep learning is the key to future services. However, the implementation system of this video analysis requires low end-to-end latency and accurate inference. The selection of video configuration parameters will have an important impact on its performance indicators.

[0004] The study of video configuration parameter selection has received much attention and research. For example, Xukan Ran et al. developed a mobile deep learning framework for edge video analysis in "DeepDecision: A Mobile Deep Learning Framework for Edge Video Analytics". It establishes an optimization problem based on the accuracy requirements, frame rate requirements, energy consumption costs, and network status of the video to be analyzed. It selects the frame rate of the video, the video encoding rate, and the deep learning model parameters to optimize the accuracy of video analysis and the achievable frame rate. However, this method assumes that the accuracy and latency of the available deep learning model and the wireless link are known in advance, and has certain data or platform dependencies, so it is only applicable to a fixed environment. For another example, Sun Yat-sen University disclosed a "low-latency real-time video analysis method based on edge computing" in its patent document with application number: 202011638574.3. This method first analyzes the delay data collection from the device to the edge server, and then dynamically configures the parameters for uploading the video stream according to the video content to balance the accuracy, resource consumption and latency. At the same time, dynamic profiling execution decision-making is performed: through dynamic calculation of profiling, it is determined whether profiling needs to be re-performed in each time slot, and finally the results generated by profiling and the image difference comparison are used to jointly decide whether to upload the image to the server. However, in this method, since the impact of configuration selection on video analysis is measured by the theoretically optimal golden configuration, the cost is expensive, and as the network conditions change over time, the performance of video analysis corresponding to the golden configuration may not be optimal, thereby affecting the accuracy and latency performance of video analysis, and thus posing a threat to road safety. Summary of the invention

[0005] The purpose of the present invention is to address the shortcomings of the above-mentioned video configuration method and propose a video configuration selection method for a vehicle-mounted edge network to reduce the cost of video configuration, maintain road safety, improve travel efficiency, and expand the applicable environment.

[0006] To achieve the above object, the technical solution of the present invention includes the following:

[0007] 1) Build an in-vehicle edge network:

[0008] The self-driving car or assisted driving car equipped with on-board sensors is used as the client vehicle user, the server deployed near the base station is used as the edge computing node, and the neural network model is deployed at the edge computing node;

[0009] Use M client vehicle users to form a client vehicle user set: V = {v1, v2, ..., v j ,...,vM}, where v j represents the jth client vehicle user, j ranges from 1 to M, M ≥ 2;

[0010] The vehicle edge network is composed of a client vehicle user set V and an edge computing node. In the vehicle edge network, the client vehicle user v j The communication between vehicles and between vehicles and base stations is established through the C-V2X wireless interface;

[0011] 2) The client vehicle user collects video data through the vehicle camera to generate the video to be analyzed;

[0012] 3) The client vehicle user establishes a communication link with the computing node;

[0013] 4) Perform video configuration for the video to be analyzed:

[0014] 4a) Determine the expected frame rate λ of the client vehicle user according to user needs * ;

[0015] 4b) Randomly select an initial configuration x from all available configurations t =(n t ,q t ), where n t Represents the neural network model input size, q t represents the compression percentage selected for the transmission of video frames in each time slot;

[0016] 4c) The client vehicle user selects the configuration x t The video is compressed and uploaded to the edge computing node through the communication link established with the computing node for video analysis to obtain the observation accuracy C(x t ) and the transmission delay of video uploading to the edge computing node, and the video analysis delay;

[0017] 4d) The obtained transmission delay and video analysis delay are processed by model to obtain the observed frame rate λ(x t ), and through this λ(x t ) and C(x t ) These two parameters generate the reward information y for the t-th time slot observation t ;

[0018] 4e) Configure the selected video to x t Add to the selected video configuration set and observe the reward information y t Added to the observation reward collection;

[0019] 4f) Configure x according to the selected video t With the observed reward yt Calculate the mean u of the Gaussian model t (x) and variance k t (x,x), and based on these two parameters, calculate the confidence interval upper bound algorithm function value corresponding to all optional video configurations;

[0020] 4g) taking the video configuration corresponding to the maximum value of the confidence interval upper bound algorithm function value corresponding to all optional video configurations as a potential video configuration, and adding it to the potential video configuration set;

[0021] 4h) Counting the frequency of occurrence of each video configuration in the potential video configuration set;

[0022] 4i) Select the optimal video configuration for the next time slot according to the mean value of the Gaussian model and the frequency of each configuration appearing in the potential video configuration set:

[0023]

[0024] Where χ is the set of all optional video configurations, u t (x) is the mean of the Gaussian model, fre(x') is the frequency of configuration x appearing in the potential video configuration set, and γ is the weight factor of the configuration history frequency;

[0025] 4j) Repeat 4c)-4i) until the set number of iterations is reached to obtain the optimal video configuration;

[0026] 5) Repeat 4) regularly according to the changes in the video content to obtain a new optimal video configuration.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] First, the usage environment has been expanded.

[0029] Most of the existing vehicle video configuration methods make assumptions about the deep learning model and communication link used. Since the accuracy, processing delay and communication link transmission delay of the assumed deep learning model are all known, they are only applicable to a fixed environment. The present invention uses the Bayesian online learning method to first learn the delay and accuracy of different video configurations in the current environment, and then uses the Gaussian model to measure the similarity between the current configuration and the configurations that obtain the delay and accuracy through learning, and selects the optimal configuration based on this, without being limited to a fixed environment, that is, expanding the use environment.

[0030] Second, reduce video configuration costs and improve video analysis accuracy and latency.

[0031] In actual situations, due to the time-varying network conditions, factors that affect the accuracy of video analysis, such as target size, angle, and moving speed, are time-varying. Therefore, the relationship between configuration rewards is difficult to accurately describe and is time-varying. The video analysis performance corresponding to the golden configuration is not necessarily optimal. The existing vehicle-mounted video configuration methods mostly use the video analysis performance corresponding to the golden configuration as a standard for measuring the accuracy and delay of video analysis, which will result in the selected configuration not being the optimal configuration, thereby affecting the accuracy and delay of video analysis; at the same time, the video configuration cost brought by the golden configuration is relatively high. Since the present invention uses the Bayesian online learning method to iteratively select the optimal configuration, and does not rely on the modeling relationship between configuration rewards and the golden configuration, it can reduce the video configuration cost, improve the video analysis accuracy and delay, and thus maintain road safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0033] Figure 1 It is a flow chart for realizing the present invention;

[0034] Figure 2 A sub-flow chart for performing video configuration in the present invention;

[0035] Figure 3 It is a simulation diagram of the present invention. DETAILED DESCRIPTION

[0036] This example mainly aims at the accuracy and latency issues of video analysis in vehicle edge networks, and proposes a video configuration method. This method overcomes the problem that existing vehicle video configuration methods are only applicable to a fixed environment and have data or platform dependency issues. It uses Gaussian process modeling to model the potential correlation between configurations, and combines Bayesian online learning to iteratively select the optimal configuration, maximizing the accuracy of video analysis under latency constraints and maintaining road safety.

[0037] Reference Figure 1 The video configuration method of this example is implemented as follows:

[0038] Step 1. Build the vehicle edge network.

[0039] An autonomous vehicle or assisted driving vehicle equipped with on-board sensors is used as a client vehicle user, a server deployed near a base station is used as an edge computing node, and a neural network model is deployed at the edge computing node to analyze the video.

[0040] Use M client vehicle users to form a client vehicle user set: V = {v1, v2, ..., v j ,...,v M}, where v j represents the jth client vehicle user, j ranges from 1 to M, M ≥ 2;

[0041] The vehicle edge network is composed of a client vehicle user set V and an edge computing node. In the vehicle edge network, the client vehicle user v j Communications between vehicles and between vehicles and base stations are established through the C-V2X wireless interface.

[0042] Step 2. Generate the video to be analyzed.

[0043] Client vehicle users perceive road congestion and the distribution information of other vehicles on the road through on-board cameras, and generate video data for subsequent video analysis, so that client vehicle users can understand road conditions and the distribution information of other vehicles, make correct driving decisions, and maintain road safety.

[0044] Step 3. The client vehicle user establishes a communication link with the computing node.

[0045] In the vehicle edge network, K V2V links are established between client vehicle users, and M V2I links are established between client vehicle users and edge computing nodes. The K V2V links and the M V2I links share an orthogonal spectrum sub-band with a fixed transmission power. It is assumed that the channel fading is approximately the same within a spectrum sub-band, and the channel fading on different spectrum sub-bands is independent. The signal-to-noise ratio of the mth V2I link channel is as follows:

[0046]

[0047] Where m ranges from 1 to M, k represents the kth V2V link, and its value ranges from 1 to K. is the transmission power of the mth V2I link, represents the transmission power of the kth V2V link interfering with the mth V2I link, g m,B [m] is the channel gain of the mth V2I link, g k,B [m] is the channel gain of the interference link generated by the kth V2V link to the mth V2I link, σ is the channel noise power of the mth V2I link, and ρ k [m] is 1, which means the kth V2V link interferes with the mth V2I link. k [m] is 0, which means that the V2V link does not interfere with the mth V2I link.

[0048] Step 4. Perform video configuration on the video to be analyzed.

[0049] Reference Figure 2 , the specific implementation of this step is as follows:

[0050] 4.1) Determine the expected frame rate λ of the client vehicle user according to user needs * ;

[0051] 4.2) Randomly select an initial configuration x from all available configurations t =(n t ,q t ), where n t Represents the input size of each time slot neural network model, q t represents the compression percentage selected for the transmission of video frames in each time slot;

[0052] The neural network model input size n of the initial configuration of this embodiment t Randomly selected from the set {128,192,256,320,384,448,512,576}, the compression percentage q selected for video frame transmission t Randomly select from the set {25,50,75,100};

[0053] 4.3) The client vehicle user selects the configuration x t The video is compressed and uploaded to the edge computing node through the communication link established with the computing node for video analysis to obtain the observation accuracy C(x t ) and the transmission delay t of video upload to edge computing node trans (x t ), video analysis delay t edge (x t );

[0054] 4.4) For the obtained transmission delay t trans (x t ), video analysis delay t edge (x t ) and t offload (x t ) is processed by the model to obtain the observed frame rate λ(x t ), and through this λ(x t ) and C(x t ) These two parameters generate the reward information y for the t-th time slot observation t :

[0055]

[0056] y t =C t (xt )-α|λ t (x t )-λ * |=f(x t )+ε t

[0057] Where c is the delay intervention factor, which is used to punish the delay that makes the client vehicle users increasingly dissatisfied, t offload (x t ) = t trans (x t )+t edge (x t ), ε t is the reward observation noise, which obeys a Gaussian distribution with a mean of 0; α represents the weight factor of the frame rate, f(x t ) is the configuration x t The reward information obtained without noise; λ * represents the expected frame rate of the client vehicle user, when observing the reward information y t Introducing λ * Avoid choosing video configurations that sacrifice analysis accuracy excessively in exchange for high frame rates;

[0058] 4.5) Configure the selected video to x t Add to the selected video configuration set and observe the reward information y t Added to the observation reward collection;

[0059] 4.6) Configure x according to the selected video t With the observed reward y t Calculate the mean u of the Gaussian model t (x) and variance k t (x,x):

[0060] u t (x) = k t (x) Τ (K t +σ 2 I) -1 Y t

[0061] k t (x,x)=R(x t ,x)-k t (x) Τ (K t +σ 2 I) -1 k t (x)

[0062] Among them, k t(x)=[R(x1,x),...,R(x t ,x)] represents the video configuration x to be evaluated and the selected configurations x1,...,x t The Gaussian kernel function value of 1*t vector, R(x t ,x) indicates the selected configuration x t The Gaussian kernel function value between the configuration x to be evaluated, and its size represents the selected configuration x t The degree of similarity to the configuration x to be evaluated;

[0063] K t =[R(x m ,x n )],R(x m ,x n ) represents the Gaussian kernel function value of the configuration of the mth time slot and the configuration of the nth time slot, m,n=1,2,...,t;

[0064] I represents the m*n identity matrix, σ represents the weight factor of the identity matrix I, Y t Reward y t gather;

[0065] 4.7) According to the mean u of the Gaussian model t (x) and variance k t (x,x), calculate the value of the confidence interval upper bound algorithm function g(x) corresponding to all optional video configurations:

[0066]

[0067] Among them, β t The trade-off factor between exploring high-reward behaviors and selecting the configuration with the highest known historical reward is a factor whose value will affect the time to find the optimal configuration. In this embodiment, it is taken as, but not limited to, β t =log((55*π*t) / 6);

[0068] 4.8) Take the video configuration corresponding to the maximum value of the confidence interval upper bound algorithm function value corresponding to all optional video configurations as the potential video configuration, and add it to the potential video configuration set;

[0069] 4.9) Count the frequency of each video configuration in the potential video configuration set, and select the optimal video configuration for the next time slot based on the frequency:

[0070]

[0071] in A collection of all optional video configurations, u t(x) is the mean of the Gaussian model, fre(x') is the frequency of configuration x appearing in the potential video configuration set, and γ is the weight factor of the configuration history frequency;

[0072] 4.10) Repeat 4.3)-4.9) until the set number of iterations is reached and the optimal video configuration is obtained.

[0073] Step 5. Repeat step 4 regularly according to the changes in the video content to obtain a new optimal video configuration.

[0074] Since network conditions vary with time, factors that affect video analysis accuracy, such as target size, angle, and moving speed, also vary with time. Therefore, video configuration needs to be performed regularly to obtain a new optimal video configuration.

[0075] The effect of the present invention is further described below in conjunction with simulation:

[0076] 1. Simulation Environment

[0077] The hardware platform of the simulation experiment of the present invention is: the processor is Intel (R) Core (TM) i7-9700 CPU, the main frequency is 3.00 GHz, and the memory is 16.0 GB.

[0078] The software platforms for the simulation experiment of the present invention are: Windows 10 operating system and PyCharm 2020.

[0079] The specific parameters of the simulation experiment are shown in Table 1:

[0080] Table 1: Simulation experiment parameters

[0081]

[0082] 2. Simulation experiment content and result analysis:

[0083] In the above simulation environment, the present invention and the two existing methods GP-UCB and RX-MAX are used to select the optimal configuration from the pre-given configuration set χ. The results are as follows: Figure 3 Among them, the vertical axis overall Regret represents the cumulative Regret of T iterations, which means the difference between the cumulative reward of the configuration selected in T iterations and the cumulative reward of the optimal configuration in T iterations, specifically expressed as: R T Expressed as overall Regret, f(x * ) represents the reward information corresponding to the optimal configuration, f(x t ) represents the reward information of the configuration selected in each time slot; the horizontal axis is the number of iterations.

[0084] The existing GP-UCB method is the video configuration method proposed by Apostolos Galanopoulos et al. in "Bayesian Online Learning for MEC Object Recognition Systems. GLOBECOM 2020-2020IEEE GlobalCommunications Conference, 2020"

[0085] The existing RX-MAX method is a method of randomly selecting video configurations.

[0086] from Figure 3 It can be seen that the curve of the present invention converges first, and the overall Regret after convergence is the smallest, indicating that the present invention can find the optimal configuration first and reduce the configuration cost.

[0087] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A video configuration method for an in-vehicle edge network, characterized in that: These include: 1) Build an in-vehicle edge network: The self-driving car or assisted driving car equipped with on-board sensors is used as the client vehicle user, the server deployed near the base station is used as the edge computing node, and the neural network model is deployed at the edge computing node; Use M client vehicle users to form a client vehicle user set: V = {v1, v2, ..., v j ,...,v M }, where v j represents the jth client vehicle user, j ranges from 1 to M, M ≥ 2; The vehicle edge network is composed of a client vehicle user set V and an edge computing node. In the vehicle edge network, the client vehicle user v j The communication between vehicles and between vehicles and base stations is established through the C-V2X wireless interface; 2) The client vehicle user collects video data through the vehicle camera to generate the video to be analyzed; 3) The client vehicle user establishes a communication link with the computing node; 4) Perform video configuration for the video to be analyzed: 4a) Determine the expected frame rate λ of the client vehicle user according to user needs * ; 4b) Randomly select an initial configuration x from all available configurations t =(n t ,q t ), where n t Represents the neural network model input size, q t represents the compression percentage selected for the transmission of video frames in each time slot; 4c) The client vehicle user selects the configuration x t The video is compressed and uploaded to the edge computing node through the communication link established with the computing node for video analysis to obtain the observation accuracy C(x t ) and the transmission delay of video uploading to the edge computing node, and the video analysis delay; 4d) The obtained transmission delay and video analysis delay are processed by model to obtain the observed frame rate λ(x t ), and through this λ(x t ) and C(x t ) These two parameters generate the reward information y for the t-th time slot observation t ; The obtained transmission delay and video analysis delay are processed by model, which is the sum of the transmission delay of video uploading to the edge computing node and the video analysis delay t offload (x t ) Find the inverse and get the configuration x t The achievable frame rate λ(x t ): Among them, c is the delay intervention factor; y t =C t (x t )-a|l t (x t )-l * |=f(x t )+e t Among them, ε t is the observation noise of the reward, which follows a Gaussian distribution with a mean of 0; α represents the weight factor of the frame rate, f(x t ) is the configuration x t The reward information obtained without noise, λ * represents the desired frame rate of the client vehicle user; 4e) Configure the selected video to x t Add to the selected video configuration set and observe the reward information y t Added to the observation reward collection; 4f) Configure x according to the selected video t With the observed reward y t Calculate the mean u of the Gaussian model t (x) and variance k t (x,x), and based on these two parameters, calculate the confidence interval upper bound algorithm function value g(x) corresponding to all optional video configurations, as follows: u t (x)=k t (x) Τ (K t +σ 2 I) -1 Y t k t (x,x)=R(x t ,x)-k t (x) Τ (K t +σ 2 I) -1 k t (x) Among them, k t (x)=[R(x1,x),...,R(x t ,x)] represents the video configuration x to be evaluated and the selected configurations x1,...,x t The Gaussian kernel function value of 1*t vector, R(x t ,x) indicates the selected configuration x t The Gaussian kernel function value between the configuration x to be evaluated; K t =[R(x m ,x n )],R(x m ,x n ) represents the Gaussian kernel function value of the configuration of the mth time slot and the configuration of the nth time slot, m,n=1,2,...,t; I represents the m*n identity matrix, σ represents the weight factor of the identity matrix I, Y t Reward y t gather; Among them, β t The trade-off between exploring high-reward behaviors and selecting the configuration with the highest historically known reward; 4g) taking the video configuration corresponding to the maximum value of the confidence interval upper bound algorithm function value corresponding to all optional video configurations as a potential video configuration, and adding it to the potential video configuration set; 4h) Counting the frequency of occurrence of each video configuration in the potential video configuration set; 4i) Select the optimal video configuration for the next time slot according to the mean value of the Gaussian model and the frequency of each configuration appearing in the potential video configuration set: in A collection of all optional video configurations, u t (x) is the mean of the Gaussian model, fre(x') is the frequency of configuration x appearing in the potential video configuration set, and γ is the weight factor of the configuration history frequency; 4j) Repeat 4c)-4i) until the set number of iterations is reached to obtain the optimal video configuration; 5) Repeat 4) regularly according to the changes in the video content to obtain a new optimal video configuration.