Time-varying environment perception video analysis task unloading method based on region of interest

Abstract

The invention discloses a time-varying environment perception video analysis task unloading method based on a region of interest, which comprises the following steps: after capturing video data and obtaining a video frame, generating an initial bounding box through an edge server model, and determining a packing size and an unloading threshold value of the region of interest by a local terminal decision module; according to the initial bounding box, obtaining the region of interest and an accumulated offset and a normalized correlation value of the region of interest, judging whether to unload the region of interest to an edge server or not based on an unloading threshold value, and combining and packaging the region of interest to be unloaded, thereby realizing real-time processing of high-definition video stream data of a small target; the method has the advantages that the dependency relationship between the environment and the system performance is analyzed in combination with the isomerism of end-side resources, and the method has better analysis performance for high-definition videos containing small targets.

Description

Technical Field

[0001] The present invention relates to the field of edge video analysis, and in particular to a method for offloading video analysis tasks based on time-varying environment perception of a region of interest. Background Art

[0002] With the rapid development of information technology, the generation and application of video data have exploded. Video analysis plays a vital role in many fields, such as smart security and traffic monitoring. However, there are still challenges in meeting the real-time analysis needs of small target tracking and detection in high-definition videos.

[0003] When tracking and detecting small targets, traditional video frame-level offloading is difficult to capture their moving trajectory during the target's motion changes, due to the small number of pixels, unclear features, or easy interference from complex backgrounds, resulting in detection failure. Secondly, when processing high-definition video data in the edge-end architecture, a large amount of video data needs to be transmitted to the edge server for processing and analysis, which will cause huge pressure on the edge server's transmission calculation and easily cause data processing delays, thus failing to meet the real-time analysis and rapid response requirements in actual applications. The traditional edge-end collaborative video frame-level offloading method cannot meet the real-time and accuracy requirements of high-definition video analysis with small targets.

[0004] In addition, the current edge-end collaborative architecture is highly dependent on network bandwidth, and video analysis tasks are mainly processed by edge devices with high computing power. Video stream analysis is highly computationally intensive and has extremely high requirements for network bandwidth in terms of transmission. At the same time, this form of edge-end architecture fails to fully utilize the heterogeneous resources of local terminals and edge servers, resulting in resource waste.

[0005] In summary, designing a low network dependency and edge-to-edge collaborative analysis method for high-definition videos with small objects is an urgent problem to be solved.

[0006] Prior art 1

[0007] A system based on on-device object tracking and edge-assisted analysis Hanyao et al. Edge-assisted Online On-device Object Detection for Real-time Video Analytics[C].IEEE INFOCOM 2021-IEEE Conference on Computer Communications,2021:1-10.

[0008] Disadvantages of the second prior art

[0009] 1. Transmitting video frames between the terminal and the edge requires certain computing and network bandwidth resources, and may still cause certain transmission delays.

[0010] 2. It will depend on the initial unloading threshold setting, affecting the subsequent online learning time.

[0011] Prior art 2

[0012] Considering video frame-level offloading in unknown and time-varying environments A. Galanopoulos et al. AutoML for Video Analytics with Edge Computing [C]. IEEE INFOCOM 2021-IEEE Conference on Computer Communications, 2021: 1-10.

[0013] Disadvantages of the second prior art

[0014] 1. For the offloading of frame-level analysis tasks for high-definition videos, there is still a high demand for network bandwidth resources;

[0015] 2. Only using the DNN model deployed on the edge server to process video analysis tasks cannot fully utilize the resources of the local terminal. Summary of the invention

[0016] In view of the defects of the prior art, the present invention provides a method for offloading time-varying environment-aware video analysis tasks based on regions of interest. After capturing video data and obtaining video frames, an initial bounding box is generated through an edge server model, and a local terminal decision module determines the packing size of the region of interest and an offloading threshold. The region of interest and its cumulative offset and normalized correlation value are obtained according to the initial bounding box. Based on the offloading threshold, it is determined whether to offload the region of interest to the edge server and the regions of interest to be offloaded are combined and packaged, thereby realizing real-time processing of high-definition video stream data of small targets.

[0017] In order to achieve the above invention objectives, the present invention discloses a method for offloading time-varying environment-aware video analysis tasks based on regions of interest, which specifically includes the following steps:

[0018] Step 1: The video acquisition device captures video data and obtains video frames through the decoder, and then generates the initial bounding box and the area of ​​interest to be tracked through the model deployed on the edge server;

[0019] Step 2: construct a decision module on the local terminal to determine the packing size of the region of interest, the cumulative pixel offset threshold of the region of interest, and the normalized correlation threshold of the region of interest;

[0020] Step 3: Use the target tracking module deployed on the terminal device to track and calculate the cumulative pixel offset value and normalized correlation value of the area of ​​interest in real time;

[0021] Step 4: The local terminal determines whether to offload the area of ​​interest to the edge server through Socket communication based on the offloading threshold;

[0022] Step 5: The local terminal adaptively combines and packages the interested regions to be uninstalled, and performs uninstallation according to the uninstallation judgment result;

[0023] Step 6: The local terminal receives the inference result from the edge server and updates the final inference result of the local terminal.

[0024] Furthermore, the video data described in step 1 comes from a real camera and a virtual camera, and the camera does not move during the acquisition process.

[0025] Furthermore, the decision module described in step 2 makes decisions on the setting of the packing size of the region of interest, the cumulative pixel offset threshold of the region of interest, and the normalized correlation threshold based on the perception of the time-varying environment, considering the current state of the system, and ultimately balances the processing delay and inference accuracy.

[0026] Furthermore, the implementation of the decision module of the local terminal in step 2 combines the confidence upper bound method, Bayesian optimization, and Gaussian process based on the time series kernel function, and its specific implementation is as follows:

[0027] First, the time is decomposed into multiple rounds t∈{1,2,...,T} with fixed time sequence. The decision module determines the adaptive parameter settings and unloading thresholds for the set of regions of interest in each round of video frames by observing the environmental context, and interacts with the environment to obtain the optimal decision result.

[0028] The main elements of a decision-making problem include actions, environmental context, rewards, and reward predictions;

[0029] Action: The decision action includes the ROI packing size, the ROI cumulative offset threshold, and the normalized cross-correlation threshold, which are defined as:

[0030] a t =(z t ,θ t ,n t ) (1)

[0031] Among them, z t Indicates the combined packing size of the region of interest suitable for the current round, θ t Indicates the cumulative offset threshold setting for the current round, n t represents the normalized cross-correlation threshold of the current round, θ t and nt They will be used together as the uninstall threshold for subsequent determination of whether to uninstall.

[0032] Environmental context: Environmental context includes the system’s processing latency and inference accuracy, which is defined as:

[0033] c t =(D t-1 ,A t-1 ) (2)

[0034] Among them, D t-1 represents the processing delay of the previous round, A t-1 Indicates the reasoning accuracy of the previous round.

[0035] Reward: In each round, the local terminal takes action a t The set of regions of interest generated by the video frame data is packaged, combined and unloaded. After completing the processing of the current round, the reward function is calculated to evaluate the current strategy. The reward function is defined as:

[0036] y t =1 / D t +α·A t (3)

[0037] Among them, α is the weight parameter.

[0038] Reward prediction: In the initial stage of decision making, the prior knowledge of the environment is unknown, and the relationship between the action-context pair and the reward in each round is unknown, so Bayesian optimization is used to predict the reward function. The reward prediction function under Bayesian optimization is defined as:

[0039] y t =f(x t )+∈ t (4)

[0040] Among them, x t =(a t ,c t ) represents the action context of d in each round, represents random noise, i.e., the difference between the actual reward and the observed reward. According to the Gaussian process, the mean function of the prior distribution of the reward prediction function is:

[0041]

[0042] Its covariance function is:

[0043]

[0044] Where x and x′ are elements in the action-context pair set X. Here, the initial prior distribution of the reward function f(·) is set to

[0045] For the update of the prior distribution of the reward prediction function, according to the Sherman-Morrison-Woodbury formula, the reward prediction function Y t =[y1,...,y t ] T The posterior distribution of still obeys the Gaussian process, then the mean of the posterior distribution of each round is:

[0046] μ t (x) = k t (x) T (K t +σ 2 1) -1 Y t (7)

[0047] The variance is:

[0048]

[0049] Among them, the kernel function K t =[k x (x,x′)],k t (x) = [k x (x1,x),...,k x (x t ,x)] T ,x,x′∈X.

[0050] In order to solve the problem that the historical “stale” data leads to the decline in the update effect of the reward function, we first x The time dimension is introduced into the environment context pair, and the timing parameter t is introduced into the environment context pair. Secondly, in the square exponential kernel function k s Based on this, the Ornstein-Uhlenbeck time covariance function k is introduced o , specifically:

[0051]

[0052] Among them, λ controls the rate of decrease of the importance of historical data samples. The larger λ is, the faster the importance of historical data samples decreases, and the new data samples will be more important. l represents the length scale, which determines the degree of mutual influence between x and x′.

[0053] If the interval of experience time series information is uniform, then The kernel matrix is in, for Hardman plot; in, u t with U t represents the reward prediction y t The corresponding data sample weight. The posterior distribution of is as follows:

[0054]

[0055] For the exploration-exploitation balance problem, the confidence upper bound method is used, and the actions of each round are expressed as:

[0056]

[0057] Where D is the action space; β t is a custom constant used to control the weight ratio between exploration kernel utilization; mean Indicates the utilization value of each action in the action space; variance Indicates the importance of exploration to obtain more information in action selection.

[0058] Furthermore, the target tracking module described in step 3 will update the region of interest to be tracked obtained in step 1, and the target tracking module calculates the cumulative offset as the cumulative coordinate offset between the result box and the tracking box of each region of interest, and calculates the normalized correlation value as the calculation of the pixel similarity of each region of interest in the previous frame and the current frame.

[0059] Further, the step 4 of judging whether to offload the area of ​​interest to the edge server through Socket communication based on the offloading threshold is:

[0060] Cumulative pixel offset > cumulative pixel offset threshold or normalized correlation value < normalized correlation threshold → offload to edge server for inference;

[0061] Cumulative pixel offset ≤ cumulative pixel offset threshold and normalized correlation value ≥ normalized correlation threshold → offload to local terminal;

[0062] Furthermore, the adaptive combined packaging of the interest areas to be unloaded described in step 5 is to combine and pack according to the packing size output by the decision module. Its implementation combines the two-dimensional packing method and the Guillotine algorithm. The basic process is the placement of elements in the set of interest areas to be unloaded, the segmentation of blank rectangles, and the maintenance of the remaining rectangle list.

[0063] First, use the packing size z output by the decision module t Initialize the remaining rectangle list S.

[0064] For the placement of element r in the set of interest areas to be unloaded, first find the remaining rectangle s in S that can hold r and put it into the set S. r middle.

[0065] For the segmentation of the blank rectangle, the blank rectangle is segmented in the direction with smaller height and width to obtain s′ and s″.

[0066] To maintain the remaining rectangle list, the remaining rectangle set {s′, s″} excluding the combined position s is added to S.

[0067] Furthermore, the final reasoning result of updating the local terminal described in step 6 is judged based on whether there is an inference box from the edge server. If there is an inference box, the final reasoning result is a composite result of the terminal box and the inference box; if there is no inference box, the final reasoning result is the target tracking result box of the local terminal.

[0068] Compared with the prior art, the advantages of the present invention are:

[0069] 1. Based on reinforcement learning methods, it is able to learn unknown situations from historical data, and by considering environmental context information, the algorithm can make better decisions.

[0070] 2. Analyze the dependency between the utilization environment and system performance, and capture the time-varying characteristics in a dynamic time-varying environment to obtain a better offloading decision strategy.

[0071] 3. It can reduce dependence on network resources and utilize heterogeneous resources on the edge, so it has lower performance requirements for local terminal devices and has good universal deployment and adaptability to network fluctuations. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a structural schematic diagram of a method for offloading time-varying environment-aware video analysis tasks based on a region of interest according to an embodiment of the present invention.

[0073] Figure 2 It is a schematic diagram of an application implementation method of a time-varying environment-aware video analysis task offloading method based on a region of interest according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0075] like Figure 1As shown in the figure, a method for offloading time-varying environment-aware video analysis tasks based on regions of interest is provided. According to the system structure implemented by the method, the system consists of a local terminal device and an edge server. First, the camera collects video stream data and obtains video frames through a decoder. The first frame is uploaded to the edge server to generate an initialized tracking bounding box and a set of regions of interest to be tracked, and the result is sent back to the local terminal. The local terminal performs OpticalFlow tracking and calculates the cumulative offset and normalized correlation value of the region of interest at the same time. The decision module determines the packaging size and offloading threshold of the region of interest, and then makes an offloading judgment based on the decision result and performs adaptive combination packaging of the region of interest.

[0076] like Figure 2 As shown, the system structure based on the method of the present invention will be actually deployed, wherein the local terminal uses Raspberry Pi 4B and the edge server uses Jetson Xavier.

[0077] The specific steps for implementing this method are as follows:

[0078] Step 1: The local video acquisition device captures the video stream data and obtains the video frame; the local terminal device can obtain the video frame by video file reading, virtual camera reading, real camera reading, etc. The video file reading formats include avi, mp4, etc., the virtual camera reading refers to the video frame data in the camera implemented by software, and the real camera reading refers to the real-time reading of the video frame data obtained by the camera hardware.

[0079] Then, the video frame is uploaded to the edge server through Socket communication to initialize the bounding box and the set of regions of interest to be tracked.

[0080] Step 2: Build a decision module on the local terminal to determine the packing size of the region of interest, the cumulative pixel offset threshold of the region of interest, and the normalized mutual correlation threshold of the region of interest; the decision module makes decisions on the setting of the packing size of the region of interest, the cumulative pixel offset threshold of the region of interest, and the normalized mutual correlation threshold based on time-varying environmental perception and considering the current state of the system, and finally balances the processing delay and inference accuracy.

[0081] The implementation of the decision module in step 2 combines the confidence upper bound method, Bayesian optimization, and Gaussian process based on time series kernel function. The specific implementation is as follows:

[0082] First, the time is decomposed into multiple rounds t∈{1,2,...,T} with fixed time sequence. The decision module determines the adaptive parameter settings and unloading thresholds for the set of regions of interest in each round of video frames by observing the environmental context, and interacts with the environment to obtain the optimal decision result.

[0083] The main elements of a decision-making problem include actions, environmental context, rewards, and reward predictions;

[0084] Action: The decision action includes the ROI packing size, the ROI cumulative offset threshold, and the normalized cross-correlation threshold, which are defined as:

[0085] a t =(z t ,θ t ,n t ) (1)

[0086] Among them, z t Indicates the combined packing size of the region of interest suitable for the current round, θ t Indicates the cumulative offset threshold setting for the current round, n t represents the normalized cross-correlation threshold of the current round, θ t and n t They will be used together as the uninstall threshold for subsequent determination of whether to uninstall.

[0087] Environmental context: Environmental context includes the system’s processing latency and inference accuracy, which is defined as:

[0088] c t =(D t-1 ,A t-1 ) (2)

[0089] Among them, D t-1 represents the processing delay of the previous round, A t-1 Indicates the reasoning accuracy of the previous round.

[0090] Reward: In each round, the local terminal takes action a t The set of regions of interest generated by the video frame data is packaged, combined and unloaded. After completing the processing of the current round, the reward function is calculated to evaluate the current strategy. The reward function is defined as:

[0091] y t =1 / D t +α·A t (3)

[0092] Among them, α is the weight parameter.

[0093] Reward prediction: In the initial stage of decision making, the prior knowledge of the environment is unknown, and the relationship between the action-context pair and the reward in each round is unknown, so Bayesian optimization is used to predict the reward function. The reward prediction function under Bayesian optimization is defined as:

[0094] y t =f(x t )+∈ t (4)

[0095] Among them, x t =(a t ,c t ) represents the action context of d in each round, represents random noise, i.e., the difference between the actual reward and the observed reward. According to the Gaussian process, the mean function of the prior distribution of the reward prediction function is:

[0096]

[0097] Its covariance function is:

[0098]

[0099] Where x and x′ are elements in the action-context pair set X. Here, the initial prior distribution of the reward function f(·) is set to

[0100] For the update of the prior distribution of the reward prediction function, according to the Sherman-Morrison-Woodbury formula, the reward prediction function Y t =[y1,...,y t ] T The posterior distribution of still follows the Gaussian process, so the mean of the posterior distribution of each round is

[0101] μ t (x) = k t (x) T (K t +σ 2 1) -1 Y t (7)

[0102] The variance is

[0103]

[0104] Among them, the kernel function K t =[k x (x,x′)],k t (x) = [k x (x1,x),...,k x (x t ,x)],x,x′∈X.

[0105] In order to solve the problem that the historical “stale” data leads to the decline in the update effect of the reward function, we first x The time dimension is introduced into the environment context pair, and the timing parameter t is introduced into the environment context pair. Secondly, in the square exponential kernel function k sBased on this, the Ornstein-Uhlenbeck time covariance function k is introduced o , specifically:

[0106]

[0107] Among them, λ controls the rate of decrease of the importance of historical data samples. The larger λ is, the faster the importance of historical data samples decreases, and the new data samples will be more important. l represents the length scale, which determines the degree of mutual influence between x and x′.

[0108] If the interval of experience time series information is uniform, then The kernel matrix is in, for Hardman plot; in, u t with U t represents the reward prediction y t The corresponding data sample weight. The posterior distribution of is as follows:

[0109]

[0110] For the exploration-exploitation balance problem, the confidence upper bound method is used, and the actions of each round are expressed as:

[0111]

[0112] Where D is the action space; β t is a custom constant used to control the weight ratio between exploration kernel utilization; mean Indicates the utilization value of each action in the action space; variance Indicates the importance of exploration to obtain more information in action selection.

[0113] Step 3: The local terminal updates the set of interest areas to be tracked obtained in step 1 and calculates the cumulative offset d of each interest area in the set. f,t and the normalized cross-correlation value m f,t .

[0114] Step 4: The local terminal determines whether to offload the area of ​​interest to the edge server through Socket communication based on the offloading threshold:

[0115] d f,t >θ t or m f,t <n t → Offload reasoning to edge servers;

[0116] d f,t ≤θ t And m f,t ≥n t →Uninstall to the local terminal;

[0117] Step 5: The local terminal performs adaptive combined packaging according to the packing size results of the interested regions output by the decision module. The implementation combines the two-dimensional packing method and the Guillotine algorithm. The basic process is the placement of elements in the interested region set to be unloaded, the segmentation of blank rectangles, and the maintenance of the remaining rectangle list.

[0118] First, use the packing size z output by the decision module t Initialize the remaining rectangle list S.

[0119] For the placement of element r in the set of interest areas to be unloaded, first find the remaining rectangle s in S that can hold r and put it into the set S. r middle.

[0120] For the segmentation of the blank rectangle, the blank rectangle is segmented in the direction with smaller height and width to obtain s′ and s″.

[0121] To maintain the remaining rectangle list, the remaining rectangle set {s′, s″} excluding the combined position s is added to S.

[0122] Step 6: The local terminal receives the inference result from the edge server and updates the final inference result. The judgment is made based on whether there is an inference frame from the edge server. If there is an inference frame, the final inference result is the composite result of the terminal frame and the inference frame; if there is no inference frame, the final inference result is the target tracking result frame of the local terminal.

[0123] The method according to the present invention described above can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, a method for offloading a time-varying environment-aware video analysis task based on a region of interest described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the processing shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the processing shown herein.

[0124] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A time-varying environment-aware video analysis task offloading method based on regions of interest, characterized in that: The following steps are involved: Step 1: The video acquisition device captures video data and obtains video frames through the decoder, and then generates the initial bounding box and the area of ​​interest to be tracked through the model deployed on the edge server; Step 2: construct a decision module on the local terminal to determine the packing size of the region of interest, the cumulative pixel offset threshold of the region of interest, and the normalized correlation threshold of the region of interest; Step 3: Use the target tracking module deployed on the terminal device to track and calculate the cumulative pixel offset value and normalized correlation value of the area of ​​interest in real time; Step 4: The local terminal determines whether to offload the area of ​​interest to the edge server through Socket communication based on the offloading threshold; Step 5: The local terminal adaptively combines and packages the interested regions to be uninstalled, and performs uninstallation according to the uninstallation judgment result; Step 6: The local terminal receives the inference result from the edge server and updates the final inference result of the local terminal.

2. The method for offloading time-varying environment-aware video analysis tasks based on regions of interest according to claim 1 is characterized in that: The video data described in step 1 comes from a real camera and a virtual camera, and the camera does not move during the acquisition process.

3. The method for offloading time-varying environment-aware video analysis tasks based on regions of interest according to claim 1, characterized in that: The decision module described in step 2 makes decisions on the setting of the ROI packing size, the ROI cumulative pixel offset threshold, and the normalized correlation threshold based on the perception of the time-varying environment and considering the current state of the system, and ultimately balances the processing delay and inference accuracy.

4. The method for offloading time-varying environment-aware video analysis tasks based on regions of interest according to claim 3 is characterized in that: The implementation of the decision module of the local terminal in step 2 combines the confidence upper bound method, Bayesian optimization, and Gaussian process based on the time series kernel function. The specific implementation is as follows: First, the time is decomposed into multiple rounds t∈{1,2,...,T} with fixed time sequence. The decision module determines the adaptive parameter settings and unloading thresholds for the set of regions of interest in each round of video frames by observing the environmental context, and interacts with the environment to obtain the optimal decision result. The main elements of a decision-making problem include actions, environmental context, rewards, and reward predictions; Action: The decision action includes the ROI packing size, the ROI cumulative offset threshold, and the normalized cross-correlation threshold, which are defined as: a t =(z t ,θ t ,n t ) (1) Among them, z t Indicates the combined packing size of the region of interest suitable for the current round, θ t Indicates the cumulative offset threshold setting for the current round, n t represents the normalized cross-correlation threshold of the current round, θ t and n t They will be used together as the uninstall threshold for subsequent determination of whether to uninstall; Environmental context: Environmental context includes the system’s processing latency and inference accuracy, which is defined as: c t =(D t-1 ,A t-1 ) (2) Among them, D t-1 represents the processing delay of the previous round, A t-1 Indicates the reasoning accuracy of the previous round; Reward: In each round, the local terminal takes action a t The set of regions of interest generated by the video frame data is packaged, combined and unloaded. After completing the processing of the current round, the reward function is calculated to evaluate the current strategy. The reward function is defined as: y t =1 / D t +α·A t (3) Among them, α is the weight parameter; Reward prediction: In the initial stage of decision making, the prior knowledge of the environment is unknown, and the relationship between the action-context pair and the reward in each round is unknown, so Bayesian optimization is used to predict the reward function; the reward prediction function under Bayesian optimization is defined as: y t =f(x t )+∈ t (4) Among them, x t =(a t ,c t ) represents the action context of d in each round, represents random noise, that is, the difference between the actual reward and the observed reward; according to the Gaussian process, the mean function of the prior distribution of the reward prediction function is: Its covariance function is: Where x and x′ are elements in the action-context pair set X; here, the initial prior distribution of the reward function f(·) is set to For the update of the prior distribution of the reward prediction function, according to the Sherman-Morrison-Woodbury formula, the reward prediction function The posterior distribution of still obeys the Gaussian process, then the mean of the posterior distribution of each round is: The variance is: Among them, the kernel function First, in the kernel function k x The time dimension is introduced into the environment context pair, and the timing parameter t is introduced into the environment context pair. Secondly, in the square exponential kernel function k s Based on this, the Ornstein-Uhlenbeck time covariance function k is introduced o , specifically: Among them, λ controls the rate of decrease of the importance of historical data samples. The larger the λ is, the faster the importance of historical data samples decreases, and the new data samples will be more important. l represents the length scale, which determines the degree of mutual influence between x and x′. Since the time series information interval is uniform, The kernel matrix is in, for Hardman plot; in, u t with U t represents the reward prediction y t The weight of the corresponding data sample; so for The posterior distribution of is as follows: For the exploration-exploitation balance problem, the confidence upper bound method is used, and the actions of each round are expressed as: Where D is the action space; β t is a custom constant used to control the weight ratio between exploration kernel utilization; mean Indicates the utilization value of each action in the action space; variance Indicates the importance of exploration to obtain more information in action selection.

5. The method for offloading time-varying environment-aware video analysis tasks based on regions of interest according to claim 1, characterized in that: The target tracking module described in step 3 will update the region of interest to be tracked obtained in step 1. The target tracking module calculates the cumulative offset as the cumulative coordinate offset between the result box and the tracking box of each region of interest, and calculates the normalized correlation value as the calculation of the pixel similarity between each region of interest in the previous frame and the current frame.

6. The method for offloading tasks of time-varying environment-aware video analysis based on regions of interest according to claim 1, characterized in that: The method described in step 4 for determining whether to offload the area of ​​interest to the edge server through Socket communication based on the offloading threshold is: Cumulative pixel offset > cumulative pixel offset threshold or normalized correlation value < normalized correlation threshold → offload to edge server for inference; Cumulative pixel offset ≤ cumulative pixel offset threshold and normalized correlation value ≥ normalized correlation threshold → offload to the local terminal.

7. The method for offloading time-varying environment-aware video analysis tasks based on regions of interest according to claim 1, characterized in that: The adaptive combined packing of the interest areas to be unloaded described in step 5 is to combine and pack according to the packing size output by the decision module. Its implementation combines the two-dimensional packing method and the Guillotine algorithm. The basic process is the placement of elements in the interest area set to be unloaded, the segmentation of blank rectangles, and the maintenance of the remaining rectangle list; First, use the packing size z output by the decision module t Initialize the remaining rectangle list S; For the placement of element r in the set of interest areas to be unloaded, first find the remaining rectangle s in S that can hold r and put it into the set S. r middle; For the segmentation of the blank rectangle, the blank rectangle is segmented in the direction with smaller height and width to obtain s′ and s″; To maintain the remaining rectangle list, the remaining rectangle set {s′, s″} excluding the combined position s is added to S.

8. The method for offloading time-varying environment-aware video analysis tasks based on regions of interest according to claim 1, characterized in that: The final reasoning result of updating the local terminal in step 6 is determined based on whether there is a reasoning box from the edge server. If there is a reasoning box, the final reasoning result is a composite result of the terminal box and the reasoning box. If there is no inference box, the final inference result is the target tracking result box of the local terminal.

Images