Video Analysis Method Based on Edge Assistance and RoI Extraction and Autonomous Driving System

Abstract

This invention discloses a video analysis method and autonomous driving system based on edge-assisted and RoI extraction, belonging to the field of video analysis. The method includes: inputting the current video frame into a feature extractor to obtain a feature map; dividing the feature map into four feature map patches along the horizontal and vertical directions; cropping a feature map patch from the central region; obtaining a heatmap of the feature map patch for each object category using a CAM (Camera-Assisted Model), setting larger pixels to 1 and other pixels to 0 to obtain a mask image; extracting the white areas within the mask image to obtain a series of bounding boxes; extracting the corresponding parts of each bounding box from the feature map patch to obtain a series of RoI (Royalty Area) boxes; unloading each RoI box to an edge node, where a DNN (Dual Neural Network) model performs the target visual task analysis of the current video frame. This invention can accurately extract RoI boxes closely related to the video analysis task and unload the task to the edge node in the form of RoI boxes, reducing data transmission while ensuring accuracy.

Description

Technical Field

[0001] This invention belongs to the field of video analysis, and more specifically, relates to a video analysis method and an autonomous driving system based on edge-assisted and RoI extraction. Background Technology

[0002] Autonomous driving can reduce traffic congestion and improve driving safety. Its core technologies can be summarized as perception, planning, and control. Among these, high-precision environmental perception is the foundation of autonomous driving decision-making, especially in complex road conditions and hazardous weather conditions. However, limited by the sensing range and accuracy of onboard sensors, most off-the-shelf vehicles can only achieve Level 2 autonomous driving.

[0003] High-definition video streams generated by vehicle-mounted cameras possess rich semantic information, which can help improve the accuracy of computer vision. However, existing deep neural networks (DNNs) typically use low-resolution images of 224×224 or 640×640 as input. Directly downsampling high-resolution (3840×2160) video frames as input to DNN models can significantly impact detection accuracy, while increasing the input size of DNNs leads to high processing latency, failing to meet real-time requirements.

[0004] Edge nodes, such as roadside units, possess certain computing resources that can help alleviate the computational burden on vehicles. When a vehicle is within the coverage area of ​​an edge node, some computational tasks can be offloaded to that node. However, vehicles in motion are characterized by high-speed movement, and unstable vehicle-to-everything (V2X) connections can lead to data packet loss. Therefore, when network bandwidth is insufficient while the amount of data transmitted is large, it can result in untimely updates to inference results, potentially causing driving safety issues.

[0005] To improve video analytics performance during autonomous driving, existing methods propose dividing the video stream captured by the vehicle's high-definition camera into equal parts and then offloading them as blocks. This offloading method can improve the accuracy of video analytics to some extent, but it may lead to target segmentation, affecting the final accuracy of the video analytics. In some special cases, the segmented blocks may not contain the target object; offloading such blocks to edge nodes would result in the transmission of useless data. Other methods propose predicting which region in the video frame the target object might appear in, directly extracting the corresponding RoI (Region of Interest) boxes from that region, and then offloading them as RoI boxes. Both of these offloading methods can reduce data transmission volume and improve real-time performance to some extent, but they cannot predict newly appearing targets and cannot guarantee the accuracy of video analytics. Summary of the Invention

[0006] To address the shortcomings and improvement needs of existing technologies, this invention provides a video analysis method and autonomous driving system based on edge-assisted and RoI extraction. The aim is to accurately extract RoI boxes closely related to the target video analysis task from the feature map of video frames, and offload the video analysis task to edge nodes in the form of RoI boxes. This reduces the amount of data transmission while ensuring the accuracy of the video analysis task, thereby achieving high-accuracy video task analysis even under poor network conditions.

[0007] To achieve the above objectives, according to one aspect of the present invention, a video analysis method based on edge-assisted and RoI extraction is provided, comprising the following steps:

[0008] (S1) Input the current video frame from the video stream generated by the vehicle camera into the trained feature extractor to extract the feature map of the current video frame;

[0009] (S2) Divide the feature map into four non-overlapping feature map blocks along the horizontal and vertical directions, and cut out a feature map block in the central region of the original feature map to obtain five feature map blocks of the feature map;

[0010] (S3) Obtain the heatmap M of the feature patch with respect to each object category k through class activation mapping. k And in each heatmap, the values ​​greater than the preset threshold σ will be... m The pixels of the feature map are set to 1, and the rest of the pixels are set to 0, so as to obtain the mask map of each feature map block with respect to each object category k.

[0011] (S4) Extract the white area in the mask image to obtain a series of bounding boxes, and extract the corresponding part of each bounding box in the feature map block to obtain a series of RoI boxes;

[0012] (S5) Unload each RoI box to the edge node, and the DNN model in the edge node completes the target visual task analysis of the current video frame.

[0013] Furthermore, between steps (S2) and (S3) is the removal of feature blocks located in the upper left and upper right regions.

[0014] Furthermore, the proportion of feature patches located in the upper left and upper right regions is inversely correlated with the focal length of the vehicle camera.

[0015] Furthermore, in step (S5), before unloading each RoI box to the edge node, the following is also included:

[0016] For any feature map block B within the current video frame, if the interval between the video frame to which the previously unloaded feature map block B belonged and the current video frame is less than the current transmission frame interval T... BIf the condition is met, the RoI bounding boxes extracted from feature map B are discarded directly; otherwise, the RoI bounding boxes extracted from feature map B are retained.

[0017] Furthermore, in step (S5), after unloading each RoI box to the edge node, the following steps are also included:

[0018] For any feature map block B in the current video frame, if no target object is detected after the extracted RoI boxes are unloaded to the edge nodes, and the current transmission frequency of feature map block B is greater than the lower bound of the transmission frequency, then the transmission frame interval of feature map block B is increased.

[0019] Furthermore, the video analysis method based on edge-assisted and RoI extraction provided by the present invention further includes: for any feature map block B in the current video frame, if the RoI box extracted from it is unloaded to an edge node, and a target is detected in at least one RoI box, then the transmission frame interval T of feature map block B is set. B The initial transmission frame interval is set so that its transmission frequency is equal to the frame rate of the video stream.

[0020] Furthermore, in step (S5), the following optimization problem is solved. Determine the downsampling rate r of each RoI box unloaded to the edge node. i :

[0021]

[0022] stC1:

[0023] C2:

[0024] C3: i ≥r min

[0025] Where t represents the current time, r i This represents the downsampling rate assigned to the i-th RoI box. This indicates that the i-th RoI box is assigned a downsampling rate r. i The size of the image after that, B t This indicates the currently available bandwidth resources of the edge node. This represents the GPU resources required for the DNN model to analyze the i-th RoI box in the edge node. It is the maximum GPU utilization currently available at the edge node, r min Indicates the lower bound of the downsampling rate; u t This represents the utility value of the current video frame. N represents the total number of RoI boxes that were uninstalled; This indicates that the i-th RoI box is assigned a downsampling rate r.i The accuracy of the analysis is then input into the DNN model. This represents the computing resources required for edge analysis of the i-th RoI box.

[0026] Furthermore, the feature extractor is a convolutional neural network, in which the number of pooling layers is less than a preset first threshold, and the number of network layers in its backbone network is less than a preset second threshold.

[0027] Furthermore, the feature extractor is ResNet18.

[0028] According to another aspect of the present invention, a video analysis device based on edge-assisted and RoI extraction is provided, comprising: a computer-readable storage medium and a processor;

[0029] A computer-readable storage medium contains a computer program;

[0030] The processor is used to read a computer program from a computer-readable storage medium and execute the video analysis method based on edge-assisted and RoI extraction provided by this invention.

[0031] According to another aspect of the present invention, an autonomous driving system is provided, comprising: an in-vehicle camera, and a video analysis device based on edge-assisted and RoI extraction provided by the present invention.

[0032] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0033] (1) The video analysis method based on edge assistance and RoI extraction provided by the present invention uses the image structure information of the video stream generated by the vehicle camera to crop the feature map of the video frame into 5 feature map blocks, and performs CAM localization on each feature map block to significantly improve the localization effect of the target object. Then, the RoI box is extracted based on the CAM localization result, so that the RoI box closely related to the video analysis task can be accurately extracted. Finally, the video analysis task is unloaded to the edge node in the form of RoI box, which can effectively reduce the amount of data transmitted while ensuring the accuracy of the video analysis task, and ensure the real-time update of the video analysis results. Even under poor network conditions, the video analysis task can be accurately completed.

[0034] (2) In the preferred embodiment of the video analysis method based on edge assistance and RoI extraction provided by the present invention, before RoI box extraction, feature blocks located in the upper left and upper right regions are removed. Based on the image structure information of the high-definition video stream generated by the vehicle camera, the removed feature blocks are the background and do not contain target objects. After removal, the computational amount of RoI box extraction can be reduced, and the transmission of invalid RoI boxes can be reduced, thereby further reducing the amount of data transmission.

[0035] (3) The video analysis method based on edge assistance and RoI extraction provided by the present invention, in its preferred embodiment, adjusts the feature blocks located in the upper left and upper right regions according to the focal length of the vehicle camera, thereby adapting to the image structure information and minimizing the transmission of invalid RoI boxes without removing valid RoI boxes.

[0036] (4) The video analysis method based on edge assistance and RoI extraction provided by the present invention, in its preferred embodiment, will adaptively adjust the transmission frequency of each feature block based on the case that the feature block contains the target object, thereby further reducing the transmission of invalid RoI boxes.

[0037] (5) The video analysis method based on edge assistance and RoI extraction provided by the present invention, in its preferred embodiment, under the conditions of available bandwidth and edge resource constraints, maximizes the utility function to obtain the optimal downsampling rate allocated to different RoI boxes, thereby maximizing bandwidth and edge resource utilization while ensuring the high accuracy of the DNN model analysis results.

[0038] (6) In the preferred embodiment of the video analysis method based on edge assistance and RoI extraction provided by the present invention, the number of pooling layers in the feature extractor is small (less than the preset first threshold), which can avoid affecting the positioning function of CAM, and the number of network layers of its backbone network is low (less than the preset second threshold), thereby ensuring that the extracted features have high sensitivity to location information. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of a video analysis method based on edge assistance and RoI extraction provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of feature block clipping provided in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the RoI box extraction process provided in an embodiment of the present invention; (a) is the original video frame, (b) is the heat map obtained after positioning by CAM, (c) is the corresponding mask map, (d) is the extracted RoI box, (e) is the effective RoI box obtained after removing feature blocks located in the upper left and upper right regions, and (f) is the video analysis result obtained after unloading the effective RoI box shown in (e) to the edge node;

[0042] Figure 4The images show a comparison of the final results obtained based on different video analysis methods. (a) is the result of directly downsampling the video frames and inputting them into the DNN detector, and (b) is the final result obtained based on the video analysis method provided in this embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0044] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0045] To reduce the amount of data transmitted and ensure high-accuracy video analysis even under poor network conditions, this invention provides a video analysis method and autonomous driving system based on edge-assisted and RoI extraction. The overall idea is to accurately extract RoI boxes closely related to the target video analysis task from the feature map of the video frame generated by the vehicle-mounted camera, based on the image structure information of the video frame. The video analysis task is then offloaded to the edge node in the form of RoI boxes. This reduces the amount of data transmitted while ensuring the accuracy of the video analysis task, thus enabling high-accuracy video task analysis even under poor network conditions.

[0046] To accurately extract RoI boxes relevant to video analysis tasks, this invention proposes a RoI extraction method based on class activation mapping (CAM) for high-definition video streams generated by vehicle-mounted cameras. The RoI boxes are then offloaded to the edge for multi-task processing. By extracting RoI boxes from high-resolution video frames, the accuracy of computer vision tasks can be significantly improved, and the amount of data transmitted to edge nodes can be greatly reduced.

[0047] The following is an example.

[0048] Example 1:

[0049] A video analysis method based on edge-assisted and RoI extraction, such as Figure 1 As shown, it includes the following steps:

[0050] (S1) Input the current video frame from the video stream generated by the vehicle camera into the trained feature extractor to extract the feature map of the current video frame.

[0051] Considering that this embodiment will subsequently obtain a heatmap through class activation mapping (CAM), and that pooling layers will lose the precise spatial mapping relationship between objects, thus affecting the localization effect of CAM, the backbone network should contain as few pooling layers as possible. Simultaneously, the deeper the network layers, the more abstract the extracted features, and the lower the sensitivity to location information; therefore, the number of network layers in the backbone network cannot be too deep. Based on these considerations, the feature extraction used in this embodiment is a lightweight convolutional neural network, meaning it has a small number of pooling layers (less than a preset first threshold) and a small number of network layers in its backbone network (less than a preset second threshold). It should be noted that in practical applications, the number of pooling layers in the lightweight convolutional neural network and the number of network layers in the backbone network should be set according to the actual video analysis effect. Optionally, in this embodiment, ResNet18 is specifically selected as the feature extractor. Using ResNet18 as the feature extractor can achieve good accuracy with minimal resource overhead. It is easy to understand that this feature extractor has been trained on an image dataset labeled with feature tags. It should be noted that in other embodiments of the present invention, when the real-time performance and accuracy of feature extraction meet the application requirements, other networks such as ResNet50, Googlenet, Mobilenet, and Darknet53 can also be selected as feature extractors.

[0052] Based on the extracted feature map of the current video frame, this embodiment further includes:

[0053] (S2) Divide the feature map into four non-overlapping feature map blocks along the horizontal and vertical directions, and cut out a feature map block in the center region of the original feature map to obtain five feature map blocks of the feature map.

[0054] CAM has the function of object localization and has very little time overhead, but its localization effect is poor for discretely distributed objects, especially small objects. In this embodiment, the feature map is cropped into multiple feature map blocks through step (S2), which helps to improve the localization accuracy.

[0055] The video frames generated by the vehicle-mounted camera have specific image structure information. Specifically, the upper left and upper right areas of the image are often background areas and do not contain target objects, while the lower right and lower right areas often contain target objects. Furthermore, the central area of ​​the image has the highest density of objects, but their distribution is relatively small, resulting in poor positioning performance when using CAM (Autonomous Computing) for subsequent localization. Based on the image structure information of the video frames generated by the vehicle-mounted camera, this embodiment adopts a '4+1' fixed structure cropping method, cropping the feature map into five feature map blocks, such as... Figure 2As shown, P1 to P5 represent the upper left, upper right, lower left, lower right, and center regions of the feature map, respectively. This feature map patch cropping method matches the image structure information, which is beneficial for the accurate extraction of RoI boxes.

[0056] In practical applications, the size ratio of each feature map patch is not fixed, but is determined according to the focal length of the vehicle camera. Specific methods include: the longer the focal length of the camera, the narrower the field of view, and the larger the image of a single object. Correspondingly, the proportion of feature map patches P1 and P2 is reduced. This ensures that feature map patches P1 and P2 are only background and do not contain target objects, thus eliminating the need for RoI extraction.

[0057] Accordingly, after completing the cropping of the feature map blocks, this embodiment also includes: removing the feature map blocks located in the upper left and upper right regions, that is, removing feature map blocks P1 and P2; since these two feature map blocks do not contain the target object, they are directly removed, which can effectively reduce the number of invalid RoI boxes, reduce the amount of computation, and reduce the amount of data transmission.

[0058] After completing the feature map clipping through the above steps (S2), this embodiment further includes:

[0059] (S3) Obtain the heatmap M of the feature patch with respect to each object category k through class activation mapping. k And in each heatmap, the values ​​greater than the preset threshold σ will be... m The pixel is set to 1, and the other pixels are set to 0, to obtain the mask map of each feature map block with respect to each object category k.

[0060] For any object category k, its heatmap M k In the diagram, each pixel value represents the probability that the pixel at the corresponding position in the feature map belongs to the k-th class of target objects;

[0061] With F c (x, y) represents the pixel value of the c-th channel feature map output by the feature extractor in the spatial coordinates (x, y), which is input into the global average pooling layer to obtain the mean scalar f. c =∑ x,y F c (x,y) is used to obtain the heatmap of object category k. in It is the weight of category k in feature channel c.

[0062] This embodiment significantly improves the target object localization effect by performing CAM-based localization on each feature map patch obtained through cropping. Furthermore, since convolutional operations do not alter the mapping relationships between objects, cropping the feature map first and then performing CAM-based object localization on each patch yields the same heatmap as cropping the original video frames first and then performing feature extraction and CAM on each video patch separately. However, the former only requires one feature extraction operation, while the latter requires five, consuming five times the computational resources. The time overhead of CAM object localization is negligible.

[0063] Through the above steps (S3), this embodiment can obtain a mask image for each feature map block with respect to each object category. In this mask image, the white area, i.e., the area with a pixel value of 1, represents the area where the object is located. Based on step (S3), this embodiment further includes:

[0064] (S4) Extract the white area in the mask image to obtain a series of bounding boxes, and extract the corresponding part of each bounding box in the feature map block to obtain a series of RoI boxes.

[0065] Based on the above steps (S1) to (S4), this embodiment achieves CAM-based RoI extraction, thereby accurately extracting RoI bounding boxes in video frames that are relevant to the video analysis task. Based on the RoI bounding boxes extracted in step (S4), this embodiment further includes:

[0066] (S5) Unload each RoI box to the edge node, and the DNN model in the edge node completes the target visual task analysis of the current video frame.

[0067] This embodiment only unloads the RoI boxes related to the video analysis task. While ensuring the accuracy of video analysis, it can effectively reduce the amount of data transmission and ensure the real-time updating of video analysis results. Thus, even under poor network conditions, it can achieve high-accuracy video analysis.

[0068] Figure 3 The diagram illustrates the analysis of a video frame using the method described in this embodiment. (a) is the original video frame, (b) is the heatmap obtained after CAM positioning, (c) is the corresponding mask image, (d) is the extracted RoI bounding box, (e) is the effective RoI bounding box obtained after removing feature patches P1 and P2, and (f) is the video analysis result obtained after unloading the effective RoI bounding box shown in (e) to the edge node. Figure 3 As can be seen, based on the method provided in this embodiment, the amount of data finally unloaded to the edge node is relatively small, only a few RoI boxes. From the final result, this embodiment can also accurately detect the corresponding target objects.

[0069] In a preferred embodiment, step (S5) of this embodiment adapts to the target object distribution characteristics of the current video stream and adjusts the transmission frequency of the RoI boxes when unloading them to the edge nodes; specifically, step (S5) of this embodiment further includes the following steps before unloading each RoI box to the edge nodes:

[0070] For any feature map block B within the current video frame, if the interval between the video frame to which the previously unloaded feature map block B belonged and the current video frame is less than the current transmission frame interval T... B If the condition is met, the RoI bounding boxes extracted from feature map B are discarded directly; otherwise, the RoI bounding boxes extracted from feature map B are retained.

[0071] Furthermore, in step (S5), after unloading each RoI box to the edge node, the following steps are also included:

[0072] For any feature map block B in the current video frame, if no target object is detected after the RoI box extracted from it is unloaded to the edge node, and the current transmission frequency of feature map block B is greater than the lower bound of the transmission frequency, then increase the transmission frame interval of feature map block B.

[0073] Furthermore, this embodiment also includes: for any feature map patch B within the current video frame, if the extracted RoI bounding box is unloaded to an edge node, and a target is detected within at least one RoI bounding box, then the transmission frame interval T of feature map patch B is set. B The initial transmission frame interval is used to restore the transmission frequency to the frame rate of the video stream.

[0074] Optionally, in this embodiment, the transmission frequency of each block is initialized to 30fps (the frame rate of the video stream is 30fps). The specific increase in the transmission frame interval can be set according to the actual detection requirements. In this embodiment, if no target object appears in the block after 5 frames, the transmission interval is increased to 10 frames, and the transmission frequency decreases sequentially according to this pattern. Simultaneously, the minimum transmission frequency for each block is set to 1fps. Once an object is detected in the block, its transmission frequency returns to 30fps.

[0075] Based on the continuity of the video stream, if no target object is detected in any of the RoI boxes extracted from a certain feature map block, then no target object will be detected in any of the RoI boxes extracted from the same feature map block in subsequent consecutive video frames. This embodiment adjusts the transmission frequency based on the detection status of the target object after the RoI boxes in each feature map block are unloaded, which can effectively reduce the unloading of RoI boxes and further reduce the amount of data transmission.

[0076] In a preferred embodiment, step (S5) of this embodiment also controls the resolution of each RoI box to be offloaded to the edge for visual task analysis by adaptively adjusting the downsampling rate, so as to transmit as little data as possible while ensuring high analysis accuracy.

[0077] At time t, N valid RoI boxes need to be unloaded using a function. Calculate the utility value for each frame, which is used to measure the effectiveness of assigning different downsampling rates r to the N valid RoI boxes in that frame. i Then, the detection results are input into the DNN model. Among them, This indicates that the i-th RoI box is assigned a downsampling rate r. i The accuracy of the analysis is then input into the DNN model. This indicates the computing resources required for edge analysis of this RoI frame.

[0078] By maximizing the utility function under the constraints of available bandwidth and edge resources, the optimal downsampling rate allocated to different RoI boxes can be obtained, thereby maximizing bandwidth and edge resource utilization while ensuring high accuracy of the DNN model's analysis results. Accordingly, in step (S5) of this embodiment, the following optimization problem is solved. Determine the downsampling rate r of each RoI box unloaded to the edge node. i :

[0079]

[0080] stC1:

[0081] C2:

[0082] C3: i ≥r min

[0083] Where t represents the current time, r i This represents the downsampling rate assigned to the i-th RoI box. This indicates that the i-th RoI box is assigned a downsampling rate r. i The size of the image after that, B t This indicates the currently available bandwidth resources of the edge node. This represents the GPU resources required for the DNN model to analyze the i-th RoI box in the edge node. It is the maximum GPU utilization currently available at the edge node, r min Indicates the lower bound of the downsampling rate; u t This represents the utility value of the current video frame. N represents the total number of RoI boxes that were uninstalled; This indicates that the i-th RoI box is assigned a downsampling rate r. i The accuracy of the analysis is then input into the DNN model. This represents the computing resources required for edge analysis of the i-th RoI box.

[0084] In summary, this embodiment uses the image structure information of the high-definition video stream generated by the vehicle-mounted camera to achieve CAM-based RoI extraction, and further filters out the effective RoI boxes. Finally, only these effective RoI boxes are unloaded to the edge nodes to perform video analysis tasks, which effectively reduces the amount of data transmission while ensuring the accuracy of video analysis.

[0085] Figure 4 The image shows a comparison of the final results obtained based on different video analysis methods. (a) is the result obtained by directly downsampling the video frames and inputting them into the DNN detector, while (b) is the final result obtained based on the video analysis method provided in this embodiment. (Comparison) Figure 4 As shown in (a) and (b), the video analysis method provided by this invention can detect all target objects present in the image. However, existing methods cannot detect small targets at a distance from the field of view. Therefore, based on the video analysis method provided by this invention, the original vehicle-mounted high-definition video frames can be extracted by RoI and unloaded to edge nodes for processing, which can greatly improve the accuracy of visual tasks.

[0086] Example 2:

[0087] A video analysis device based on edge-assisted and RoI extraction includes: a computer-readable storage medium and a processor;

[0088] A computer-readable storage medium contains a computer program;

[0089] The processor is used to read the computer program in the computer-readable storage medium and execute the video analysis method based on edge-assisted and RoI extraction provided in Embodiment 1 above.

[0090] Example 3:

[0091] An autonomous driving system includes: an onboard camera, and the video analysis device based on edge assistance and RoI extraction provided in Embodiment 2 above.

[0092] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video analysis method based on edge-assisted extraction and RoI extraction, characterized in that, Includes the following steps: (S1) Input the current video frame from the video stream generated by the vehicle camera into the trained feature extractor to extract the feature map of the current video frame; (S2) Divide the feature map into four non-overlapping feature map blocks along the horizontal and vertical directions, and cut out a feature map block in the central region of the original feature map to obtain five feature map blocks of the feature map; (S3) Obtain feature maps for each object category through class activation mapping. k Heat map M k And in each heatmap, values ​​exceeding a preset threshold will be... The pixel is set to 1, and the rest are set to 0, to obtain the feature map patch for each object category. k The mask image; (S4) Extract the white area in the mask image to obtain a series of bounding boxes, and extract the corresponding part of each bounding box in the feature map to obtain a series of RoI boxes; (S5) Unload each RoI box to the edge node, and the DNN model in the edge node completes the target visual task analysis of the current video frame; In step (S5), before unloading each RoI box to the edge node, the following is also included: For any feature map block B within the current video frame, if the interval between the video frame to which the previously unloaded feature map block B belonged and the current video frame is less than the current transmission frame interval T... B If the condition is met, the RoI bounding boxes extracted from feature map B are discarded directly; otherwise, the RoI bounding boxes extracted from feature map B are retained. Furthermore, in step (S5), after unloading each RoI box to the edge node, the process further includes: For any feature map patch B within the current video frame, if no target object is detected after the extracted RoI boxes are unloaded to edge nodes, and the current transmission frequency of feature map patch B is greater than the lower bound of the transmission frequency, then add a new feature map patch. B The transmission frame interval.

2. The video analysis method based on edge-assisted and RoI extraction as described in claim 1, characterized in that, Between step (S2) and step (S3), the method further includes: removing feature blocks located in the upper left and upper right regions.

3. The video analysis method based on edge-assisted extraction and RoI extraction as described in claim 2, characterized in that, The proportion of feature patches located in the upper left and upper right regions is inversely correlated with the focal length of the vehicle-mounted camera.

4. The video analysis method based on edge-assisted and RoI extraction as described in claim 1, characterized in that, Also includes: For any feature map block within the current video frame B If the unloading condition is met, and a target is detected within at least one RoI bounding box extracted from feature patch B in the current video frame, then the transmission frame interval T of feature patch B is set. B The initial transmission frame interval is set so that its transmission frequency is equal to the frame rate of the video stream.

5. The video analysis method based on edge-assisted extraction and RoI extraction as described in any one of claims 1 to 3, characterized in that, In step (S5), the following optimization problem is solved. Determine the downsampling rate of each RoI box unloaded to the edge node. r i : in, t Indicates the current moment. r i Indicates assignment to the first i Downsampling rate of each RoI bounding box Indicates the first Each RoI box is assigned a downsampling rate. The size of the image after processing This indicates the currently available bandwidth resources of the edge node. The analysis of the DNN model in the edge nodes is shown in the first... GPU resources required for each RoI frame This is the maximum GPU utilization currently available at the edge node. This indicates the lower bound of the downsampling rate; This represents the utility value of the current video frame. ; N Indicates the total number of RoI boxes that were uninstalled; Indicates the first Each RoI box is assigned a downsampling rate. The accuracy of the analysis is then input into the DNN model. Indicates the edge analysis of the first i The computing resources required for each RoI frame.

6. The video analysis method based on edge-assisted and RoI extraction as described in any one of claims 1 to 3, characterized in that, The feature extractor is a convolutional neural network, wherein the number of pooling layers is less than a preset first threshold, and the number of network layers in its backbone network is less than a preset second threshold.

7. The video analysis method based on edge-assisted and RoI extraction as described in claim 6, characterized in that, The feature extractor is ResNet18.

8. A video analysis device based on edge-assisted and RoI extraction, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium stores a computer program; The processor is used to read the computer program in the computer-readable storage medium and execute the video analysis method based on edge-assisted and RoI extraction as described in any one of claims 1 to 7.

9. An autonomous driving system, characterized in that, include: The vehicle-mounted camera, and the video analysis device based on edge-assisted and RoI extraction as described in claim 8.

Images