Method and system for narrow bandwidth video transmission based on time-space importance analysis

Abstract

The application relates to the field of video coding technology and discloses a narrow-bandwidth video transmission method and system based on time domain-space domain importance analysis, which comprises the following steps: acquiring a video frame, a bandwidth level and a single message payload upper limit; determining frame importance according to an interframe variation and a target coverage, and screening key frames; dividing image blocks into a core area, a transition area and a background area according to texture entropy and a target confidence; arranging area data according to area types and frame importance, and combining bandwidth level differences for compression; and performing fragmentation and encapsulation according to the single message payload upper limit, recombining key frames at a receiving end, complementing missing transition areas and background areas, and performing directional retransmission on missing core areas. Through key frame screening, area hierarchical compression and differential recovery, the application reduces the link capacity occupied by similar pictures and background data, ensures the preferential transmission of core video content, and improves the recovery efficiency of the core area.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and more specifically, to a narrow-bandwidth video transmission method and system based on temporal-spatial importance analysis. Background Technology

[0002] With the development of satellite communication, edge computing, and video surveillance technologies, field inspections, maritime monitoring, disaster early warning, and unmanned equipment are increasingly adopting satellite links to transmit video or image data. The current transmission process typically involves a data acquisition terminal acquiring video frames or still images, performing target detection, image cropping, and compression encoding on the acquired data, and then splitting the encoded data into multiple data packets according to the single data capacity allowed by the satellite communication protocol. These packets are then sent to a backend server via the satellite link. The backend server reassembles the data according to the image identifier and fragment sequence number carried in the data packets and performs retransmission processing for data packets that fail verification or are not received.

[0003] In the prior art, Chinese invention patent application CN120897038A discloses an image transmission method and system suitable for geostationary orbit satellites. This method involves cropping the acquired image by determining the target edge coordinates, encoding and dynamically fragmenting the cropped image, and reassembling the fragments at the receiving end. The sending end then retransmits the corresponding fragments according to a list of lost fragments. According to this processing method, in the case of continuous video acquisition, each moment's image is processed primarily as an independent image. When the monitored scene changes only slightly over a period of time, images with similar content may still be repeatedly cropped, encoded, and transmitted, easily consuming the limited capacity of the satellite link and preventing timely transmission of images corresponding to sudden target changes or significant scene changes. Furthermore, when some fragments are not received, retransmission is uniformly performed according to the lost fragment list without considering the impact of different missing contents on target identification and scene judgment. This may cause data with low relevance to the main observation content to repeatedly occupy retransmission resources, increasing the recovery time of effective images.

[0004] Therefore, it is necessary to design a narrow-bandwidth video transmission method and system based on temporal-spatial importance analysis to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes a narrow-bandwidth video transmission method and system based on temporal-spatial importance analysis, aiming to solve the problems of similar frames repeatedly occupying limited communication capacity during narrow-band video transmission, and the delay in the recovery of key frames caused by indiscriminate retransmission after fragment loss.

[0006] This invention proposes a narrow-bandwidth video transmission method based on temporal-spatial importance analysis, comprising: Get the continuous video frames, bandwidth level, and maximum payload per message; Compare adjacent frames to obtain inter-frame variation, perform target detection on video frames to obtain target coverage and target confidence; determine frame importance based on the inter-frame variation and target coverage, and identify video frames that reach the filtering threshold corresponding to the bandwidth level as key frames; The keyframes are divided into image blocks, and the image blocks are classified into core regions, transition regions or background regions according to the texture entropy of the image blocks and the target confidence of the covered targets. The keyframe region data is used as a transmission unit and arranged in the order of core region, transition region and background region. Within the same region, the keyframes are arranged in descending order of their importance. The compression degree of each type of region data is determined according to the bandwidth level to form a data sequence to be sent. The data sequence to be sent is fragmented and encapsulated based on the single message payload limit; the receiving end reassembles key frames based on the information carried by each fragment, fills in the missing transition area or background area according to the adjacent area of ​​the same frame and the adjacent key frames, generates a directed retransmission request for the missing core area and retransmits the corresponding fragment.

[0007] Furthermore, obtaining the inter-frame variation, target coverage, and target confidence includes: Adjacent video frames are converted into grayscale images, the average value of the absolute value of the grayscale difference of the corresponding pixels is calculated and normalized to obtain the inter-frame change; target regions in the video frames are identified, the proportion of the union area of ​​each target region to the area of ​​the video frame is determined as the target coverage, and the confidence level carried by the target detection result is determined as the target confidence level.

[0008] Further, determining the keyframe includes: The frame importance is obtained by weighting and summing the normalized inter-frame change and the target coverage according to the change weight and the target weight; a filtering threshold is set for each bandwidth level, and the lower the available bandwidth corresponding to the bandwidth level, the higher the filtering threshold.

[0009] Furthermore, when obtaining the texture entropy of the image patch and the target confidence of the covered target, the following steps are included: The keyframes are divided into sections with a base size of 16×16 pixels. The texture entropy is calculated based on the grayscale distribution of pixels within the image block. The target confidence of the overlapping target is associated with the image block based on the overlap relationship between the target region and the image block. When the image block overlaps with multiple targets, the highest target confidence is selected.

[0010] Furthermore, when classifying the image patch into a core region, transition region, or background region, the following steps are included: Image blocks that cover the target and whose target confidence reaches the first confidence threshold are assigned to the core region; image blocks whose target confidence reaches the second confidence threshold but does not reach the first confidence threshold, as well as image blocks adjacent to the core region and whose texture entropy reaches the texture threshold, are assigned to the transition region; the remaining image blocks are assigned to the background region; the first confidence threshold is greater than the second confidence threshold.

[0011] Furthermore, forming the data sequence to be sent includes: The region data of the core region, transition region, and background region are arranged in descending order. The data of the same type of region are arranged in descending order of the frame importance of the keyframe to which they belong. The core region within the same keyframe is arranged in descending order of target confidence, and the transition region and background region are arranged in descending order of texture entropy. The compression degree of the three types of region data is set for each bandwidth level. The compression degree of the core region is lower than that of the transition region, and the compression degree of the transition region is lower than that of the background region.

[0012] Furthermore, the process of fragmenting and encapsulating the data sequence to be transmitted includes: The fragment data length is determined based on the difference between the single message payload limit and the message header length. The data of each region is divided into multiple fragments. A frame number, region identifier, region type, fragment sequence number, total number of fragments and check code are added to each fragment. The fragments are then sent in the order of the data sequence to be sent.

[0013] Furthermore, when filling in missing transition or background areas, this includes: The receiving end collects each fragment according to the frame number and region identifier, and identifies the missing fragment based on the fragment sequence number, the total number of fragments, and the check code. When the missing fragment belongs to the background region, repair data is generated based on the texture features of the adjacent background region in the same frame. When the missing fragment belongs to the transition region, repair data is generated based on the image information of the adjacent region in the same frame and the corresponding region in the reconstructed adjacent keyframe.

[0014] Furthermore, when performing targeted retransmission of the missing core area, it includes: The receiving end generates the directed retransmission request based on the frame number, region identifier, and fragment sequence number of the missing fragment in the core region; According to the directed retransmission request, the sending end extracts the target fragment identified by the frame number, the region identifier and the fragment sequence number from the sending buffer, and retransmits the target fragment; The receiving end writes the target fragment into the core region determined by the frame number and the region identifier, and performs verification according to the check code.

[0015] Compared with existing technologies, the advantages of this invention are as follows: By combining the inter-frame variation of adjacent frames with the target coverage to determine frame importance, and setting a filtering threshold according to bandwidth level, video frames with significant image changes or containing target information are given priority in the transmission process, reducing the repeated occupation of limited communication capacity by static and similar images. After dividing keyframes into image blocks, core regions, transition regions, and background regions are distinguished based on texture entropy and target confidence. The transmission order and compression degree are determined according to region type, frame importance, and bandwidth level, so that limited bandwidth prioritizes carrying data from core regions and high-importance keyframes. The receiving end completes missing transition and background regions using adjacent regions of the same frame and adjacent keyframes, and only initiates targeted retransmission for missing core regions, avoiding the use of indiscriminate retransmission for all types of missing segments, reducing the amount of duplicated data transmission, shortening the recovery waiting time of core images, and improving the transmission and reconstruction efficiency of key video content under narrow bandwidth conditions.

[0016] On the other hand, this application also provides a narrow-bandwidth video transmission system based on temporal-spatial importance analysis, for applying the above-mentioned narrow-bandwidth video transmission method based on temporal-spatial importance analysis, including: The data acquisition unit is used to acquire continuous video frames, bandwidth levels, and the maximum payload of a single message. The key screening unit is used to compare adjacent frames to obtain the inter-frame change, perform target detection on video frames to obtain target coverage and target confidence; determine the frame importance based on the inter-frame change and the target coverage, and identify video frames that reach the screening threshold corresponding to the bandwidth level as key frames. The region segmentation unit is used to divide the keyframe into image blocks and classify the image blocks into core regions, transition regions or background regions according to the texture entropy of the image blocks and the target confidence of the covered target. The sequence generation unit is used to take the regional data of key frames as transmission units, arrange them in the order of core region, transition region and background region, and arrange them in descending order of the frame importance of key frames within the same region; determine the compression degree of each type of regional data according to the bandwidth level to form a data sequence to be sent; The fragmentation processing unit is used to fragment and encapsulate the data sequence to be sent based on the single message payload limit; the receiving end reassembles key frames based on the information carried by each fragment, fills in the missing transition area or background area according to the adjacent area of ​​the same frame and the adjacent key frames, generates a directed retransmission request for the missing core area and retransmits the corresponding fragment.

[0017] It is understandable that the narrow-bandwidth video transmission method and system based on the time-space importance analysis described above have the same beneficial effects, and will not be elaborated further here. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a narrow-bandwidth video transmission method based on temporal-spatial importance analysis provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the regional data sorting, fragmentation encapsulation, and missing fragment processing provided in an embodiment of the present invention; Figure 3 This is a functional block diagram of a narrow-bandwidth video transmission system based on temporal-spatial importance analysis, provided in an embodiment of the present invention. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] In some embodiments of this application, see Figures 1-2 As shown, this application proposes a narrow-bandwidth video transmission method based on temporal-spatial importance analysis, including: S100: Obtain continuous video frames, bandwidth level, and maximum payload per message; S200: Compare adjacent frames to obtain inter-frame changes, perform target detection on video frames to obtain target coverage and target confidence; determine frame importance based on inter-frame changes and target coverage, and identify video frames that reach the filtering threshold corresponding to the bandwidth level as key frames. S300: Divide keyframes into image blocks and classify them into core regions, transition regions, or background regions based on the texture entropy of the image blocks and the target confidence of the covered targets. S400: The regional data of key frames are used as transmission units and arranged in the order of core region, transition region and background region. Within the same region, they are arranged in descending order of the frame importance of the key frames. The compression degree of each type of regional data is determined according to the bandwidth level to form a data sequence to be sent. S500: The data sequence to be transmitted is fragmented and encapsulated based on the upper limit of the single message payload; the receiving end reassembles the key frame based on the information carried by each fragment, fills in the missing transition area or background area according to the adjacent area of ​​the same frame and the adjacent key frame, generates a directional retransmission request for the missing core area and retransmits the corresponding fragment.

[0021] In one embodiment, the present invention is executed by a video acquisition end and a receiving end working together. The video acquisition end includes a camera device, an edge processor, a memory, and a narrowband communication module, while the receiving end is a server. The video acquisition end is used to acquire video frames, filter keyframes, divide image blocks, compress and arrange region data, and complete segmentation and encapsulation; the receiving end is used to collect segments, reassemble keyframes, fill in missing regions, and process targeted retransmission of core regions.

[0022] The camera device generates consecutive video frames in chronological order of capture time, assigning each frame a frame number and capture time. Consecutive video frames use the same image size, color space, and pixel arrangement. Adjacent frames are two video frames that are consecutive in the capture sequence. The first video frame after the start of video capture has no preceding frame; this first video frame is designated as the initial keyframe and used as the preceding frame for inter-frame comparison of the next video frame.

[0023] Bandwidth level indicates the effective transmission capacity that the current narrowband link can provide. The video acquisition end uses a 10-second bandwidth statistics period to record the amount of data acknowledged as received by the receiving end within that period, and obtains the current effective bandwidth based on the data volume and the bandwidth statistics period. When the narrowband communication module can directly output the effective bandwidth, the output result of the narrowband communication module can also be read.

[0024] When the effective bandwidth is 0–2 Kbps, it is determined to be the first bandwidth level; when the effective bandwidth is greater than 2 Kbps but not higher than 5 Kbps, it is determined to be the second bandwidth level; when the effective bandwidth is greater than 5 Kbps but not higher than 8 Kbps, it is determined to be the third bandwidth level; and when the effective bandwidth is greater than 8 Kbps but not higher than 10 Kbps, it is determined to be the fourth bandwidth level. When two consecutive bandwidth statistics periods fall within the new bandwidth range, the current bandwidth level is updated to avoid repeatedly switching bandwidth levels when the effective bandwidth changes near the boundary of adjacent bandwidth ranges.

[0025] After the bandwidth level is updated, the compression level of uncompressed area data is determined according to the updated bandwidth level, while area data that has already been compressed or fragmented retains its original processing parameters. The bandwidth statistics period and bandwidth range can be adjusted according to the narrowband communication link used.

[0026] The single message payload limit is the maximum data length allowed to be carried by a message in the current communication protocol. The video acquisition terminal reads the single message payload limit from the protocol configuration of the narrowband communication module. When using different communication protocols, the single message payload limit can be [1] 124 Bytes, 128 Bytes, or 256 Bytes. After the communication protocol is switched, the single message payload limit is read again, and the data length of subsequent segments is determined according to the updated single message payload limit.

[0027] After obtaining the current and previous video frames, both frames are converted into grayscale images of the same size. Grayscale conversion can be performed using weighted grayscale conversion, extraction of the luminance channel, or calculation of the average of the three color channels. This implementation uses the BT.601 weighted grayscale conversion method, processing the red, green, and blue channels with weights of 0.299, 0.587, and 0.114 respectively to obtain the grayscale value of each pixel. When the video data is in YUV format, the Y luminance channel can also be directly extracted as a grayscale image.

[0028] Pixels at the same position in two grayscale images are read one by one. The absolute value of the grayscale difference at each pixel position is calculated. The average of all absolute grayscale differences is then calculated, and the average value is divided by 255 to normalize the result to the range of 0 to 1, thus obtaining the inter-frame variation. When the inter-frame variation is 0, it means that the two video frames have the same grayscale value at all pixel positions; when the inter-frame variation is close to 1, it means that there is a significant change in the image between adjacent frames.

[0029] For example, after comparing the current video frame with the previous video frame pixel by pixel, the average value of the absolute values ​​of all grayscale differences is 102. Dividing 102 by 255 gives an inter-frame change of 0.40.

[0030] The video acquisition end uses a lightweight object detection network capable of outputting target region and target confidence score to process the current video frame. Object detection can employ YOLO-Nano, Tiny-YOLO, MobileNet-SSD, or other object detection networks with similar output capabilities. This implementation uses a YOLO-Nano object detection network deployed in an edge processor.

[0031] Before object detection, the current video frame is scaled to the input size required by the object detection network. During scaling, the aspect ratio of the video frame is maintained, and empty spaces are filled with fixed grayscale values. The object detection network outputs candidate bounding boxes, object categories, and object confidence scores. For overlapping candidate bounding boxes belonging to the same category, a non-maximum suppression method is used for filtering, retaining the candidate bounding box with the highest confidence score and deleting other candidate bounding boxes whose overlap with this bounding box meets the suppression criteria. The remaining detection boxes after filtering are considered the target regions.

[0032] Target categories are pre-configured based on the application scenario. In a field inspection scenario, target categories can include personnel, vehicles, ships, smoke, and flames. Target confidence is in the range of 0 to 1, representing the degree of confidence with which the target detection network classifies a bounding box as the corresponding target category.

[0033] When only one target is detected in the current video frame, the target coverage is obtained by dividing the number of pixels covered by that target region by the total number of pixels in the video frame. When multiple targets are detected, each target region is mapped to video frame coordinates, and a binary mask with the same size as the video frame is created; covered positions are marked as 1, and uncovered positions are marked as 0. The number of pixels with a value of 1 is counted and divided by the total number of pixels in the video frame to obtain the target coverage. When multiple target regions overlap, the overlapping positions are still marked as 1 in the binary mask and are not counted repeatedly.

[0034] For example, a video frame contains 307,200 pixels. After removing overlapping portions, multiple target regions cover a total of 92,160 pixels, resulting in a target coverage of 0.30. The target confidence score carried by each target detection result is saved along with the corresponding target region for use in image block region segmentation.

[0035] Frame importance is determined based on inter-frame variation and target coverage. A variation weight is assigned to the inter-frame variation, and a target weight is assigned to the target coverage; the sum of the variation weight and the target weight is 1. The inter-frame variation is multiplied by the variation weight to obtain the result for the variation component; the target coverage is multiplied by the target weight to obtain the result for the target component; the two results are then added together to obtain the frame importance.

[0036] The change weights and target weights are obtained in advance using sample videos. These sample videos include static images, target entry, target exit, target movement, and sudden scene changes, and the video frames to be transmitted are labeled by annotators. Multiple sets of change weights and target weights are set at adjustment intervals of 0.1. The frame importance of each set of sample video frames is calculated, and they are sorted in descending order of frame importance.

[0037] Given the same number of keyframes, the number of manually labeled video frames selected is counted. The weighted combination with the highest number of manually labeled video frames selected is selected as the candidate combination; if multiple candidate combinations exist, the number of unlabeled video frames selected is counted, and the candidate combination with the lowest number of selected frames is used. This implementation uses a variable weight of 0.4 and a target weight of 0.6.

[0038] For example, if the inter-frame variation of a video frame is 0.80, multiplying 0.80 by the variation weight of 0.4 gives 0.32; if the target coverage is 0.50, multiplying 0.50 by the target weight of 0.6 gives 0.30; adding 0.32 and 0.30 gives a frame importance of 0.62.

[0039] Each bandwidth level has its own filtering threshold. The first bandwidth level has the lowest available bandwidth, so the highest filtering threshold is used; the fourth bandwidth level has the highest available bandwidth, so the lowest filtering threshold is used.

[0040] The screening threshold is pre-calibrated using sample videos. Values ​​in the range of 0.10 to 0.90 are used to form candidate screening thresholds at intervals of 0.05. For each bandwidth level, sample video frames are screened using each candidate screening threshold. Video frames whose frame importance reaches the candidate screening threshold are designated as candidate keyframes. These candidate keyframes then undergo further image block segmentation, region classification, and compression processing. The amount of data to be transmitted per unit time is then calculated.

[0041] If the amount of data to be sent exceeds the amount of data that can be sent within the same bandwidth level in the same time period, the candidate screening thresholds used are excluded. Among the remaining candidate screening thresholds, the retention rate of manually labeled video frames is calculated, and the candidate screening threshold with the highest retention rate is taken as the labeling result of that sample video; if the retention rates are the same, the candidate screening threshold with the highest value is taken.

[0042] Multiple sample videos are calibrated separately, and the calibration results obtained under the same bandwidth level are arranged in ascending order of value. When the number of calibration results is odd, the value in the middle position of the sorted list is taken; when the number of calibration results is even, the average of the two middle values ​​is taken to obtain the screening threshold for that bandwidth level.

[0043] In this embodiment, the filtering thresholds for the first to fourth bandwidth levels are 0.75, 0.60, 0.45, and 0.30, respectively. When the frame importance of a video frame reaches the filtering threshold corresponding to the current bandwidth level, the video frame is identified as a key frame; when the frame importance does not reach the filtering threshold, the complete pixel data of the video frame is not added to the data sequence to be sent.

[0044] After determining the keyframes, divide the image into blocks starting from the top left corner of the keyframe, using a base size of 16×16 pixels. If the width or height of a keyframe is not divisible by 16, copy the outermost row or column of pixels from the keyframe to fill it, ensuring that the width and height after filling are both divisible by 16.

[0045] Each image block is assigned a region identifier. The region identifier consists of the row and column numbers of the image block. For example, the region identifier for the image block in the first row and first column is "1-1", and the region identifier for the image block in the tenth row and twentieth column is "10-20". The pixel data of an image block constitutes a region data unit, and there is a one-to-one correspondence between the image block, the region data unit, and the region identifier. The receiving end determines the writing position of the region data in the keyframe based on the region identifier.

[0046] The texture entropy of an image patch can be obtained using gray-level histogram entropy, local binary pattern histogram entropy, gray-level co-occurrence matrix entropy, or gradient orientation histogram entropy. Gray-level histogram entropy reflects the dispersion of gray values ​​in the image patch; local binary pattern histogram entropy reflects the dispersion of local texture patterns; gray-level co-occurrence matrix entropy reflects the complexity of gray-level combinations between adjacent pixels; and gradient orientation histogram entropy reflects the dispersion along the image edge direction.

[0047] This implementation uses grayscale histogram information entropy. The image block is converted to an 8-bit grayscale image, with 0-255 set as 256 grayscale levels. The number of pixels for each grayscale level in the image block is counted. The number of pixels for each grayscale level is divided by the total number of pixels in the image block to obtain the occurrence ratio of the corresponding grayscale level in the image block. For grayscale levels with a non-zero occurrence ratio, the product of this occurrence ratio and the absolute value of its base-2 logarithm is calculated. All products are summed to obtain the texture entropy of the image block.

[0048] When processing 8-bit grayscale images using this method, the texture entropy is in the range of 0 to 8. When all pixels in an image block have the same grayscale value, the texture entropy is 0; when the number of grayscale levels in the image block increases and the proportion of each grayscale level is close, the texture entropy increases.

[0049] Texture entropy calculation can be implemented using OpenCV, the MATLAB image processing toolbox, or a processing program written using array statistics. When using OpenCV, a grayscale image can be obtained through grayscale conversion functions, and the number of pixels across 256 grayscale levels can be counted using histogram statistics functions. Then, the texture entropy can be calculated following the aforementioned processing procedure. When using other image processing programs, the same grayscale level division, pixel statistics, and entropy calculation process can be followed.

[0050] When a target region overlaps with an image patch at the pixel level, the target confidence score carried by the target detection result is assigned to the image patch. Specifically, the top-left and bottom-right coordinates of the target detection box are mapped to keyframe coordinates, and the pixel range of the image patch is compared with the pixel range of the target detection box. If the two overlap in both the horizontal and vertical directions, it is determined that the image patch covers the target.

[0051] When an image patch overlaps with multiple target regions, the target confidence scores of each target region are compared, and the image patch with the highest target confidence score is assigned. When an image patch does not overlap with any target region, its target confidence score is recorded as 0.

[0052] Both the first and second confidence thresholds are pre-obtained based on the validation images from the object detection network, with the first confidence threshold being greater than the second confidence threshold. The validation images include the true object category and the true object location annotations. The validation images are input into the object detection network, and candidate confidence scores are generated at intervals of 0.05 within the range of 0.30 to 0.90.

[0053] For each candidate confidence level, remove target detection results with a confidence level lower than that candidate confidence level. Count the number of remaining target detection results that match the true target, the number that do not match the true target, and the number of undetected true targets. A target detection result is considered a match for a true target if the overlap ratio between the detection box and the true target box reaches 0.50 and the target categories are the same.

[0054] Calculate the proportion of detection results that do not match the true target out of all retained detection results. Sort the candidate confidence scores with this proportion not exceeding 10% in ascending order of value, and take the minimum value among them as the first confidence threshold.

[0055] Among the candidate confidence levels below the first confidence threshold, the proportion of detected real targets to all real targets is calculated. Candidate confidence levels with a proportion not less than 90% are sorted in ascending order, and the highest value among them is taken as the second confidence threshold. In this embodiment, the first confidence threshold is set to 0.70, and the second confidence threshold is set to 0.40.

[0056] The texture threshold is obtained based on the texture entropy distribution of image patches in the current keyframe. Image patches whose target confidence reaches the first confidence threshold are excluded, and the remaining image patches are arranged in ascending order of texture entropy. The texture entropy at the 60th, 65th, 70th, 75th and 80th percentile positions are obtained respectively, forming five candidate texture thresholds.

[0057] Transition regions are divided using candidate texture thresholds, and compression and reconstruction are performed on the divided keyframes. Candidate texture thresholds whose data volume exceeds the allowable data volume of the current bandwidth level are excluded, and contour continuity is checked on the reconstructed keyframes formed by the remaining candidate texture thresholds.

[0058] Contour continuity can be determined through manual annotation or edge map comparison. When using edge map comparison, the Canny edge detection method is used to extract edges in both the original and reconstructed keyframes. Within an inspection area formed by extending an image block outside the core region, the proportion of edge pixels in the original keyframe that can find adjacent edge pixels in the reconstructed keyframe is counted. When this proportion reaches a set contour preservation ratio, the target contour is considered continuous. In this implementation, the contour preservation ratio is set to 90%.

[0059] Candidate texture thresholds that simultaneously meet the data volume and contour continuity conditions are sorted in ascending order of their values, and the maximum value is taken as the texture threshold of the current keyframe, so that the number of image patches that are classified into the transition region among the candidate results that meet the contour continuity condition is minimized.

[0060] For example, the five candidate texture thresholds are 3.6, 3.9, 4.2, 4.5, and 4.8. When using 4.5 and 4.8, the target contour retention rate is less than 90%; when using 3.6, 3.9, and 4.2, the target contour retention rate reaches 90%, and the amount of data to be sent does not exceed the amount of data allowed by the current bandwidth level. The maximum value of 4.2 is determined as the texture threshold.

[0061] Region types are determined in the order of core region, transition region, and background region. If an image patch covers the target and the target confidence level reaches the first confidence threshold, the image patch is classified as a core region. Image patches already classified as core regions are no longer considered for transition or background region classification.

[0062] For image patches not classified into the core region, if the target confidence reaches the second confidence threshold but not the first confidence threshold, the image patch is classified into the transition region. For image patches whose target confidence does not reach the second confidence threshold, it is also necessary to determine whether they are adjacent to the core region and whether their texture entropy reaches the texture threshold.

[0063] The adjacency relationship of image blocks can be determined using four-neighbor or eight-neighbor methods. This implementation uses an eight-neighbor relationship. When an image block is in direct contact with any core region image block above, below, to the left, to the right, or in one of the four diagonal directions, the image block is determined to be adjacent to the core region. When an image block is adjacent to the core region and its texture entropy reaches the texture threshold, it is classified into the transition region. Image blocks that do not meet the conditions for core region and transition region are classified into the background region.

[0064] For example, one image patch overlaps with the vehicle target region, with a target confidence score of 0.82, reaching the first confidence threshold of 0.70, and is therefore classified as a core region. Another image patch overlaps with the edge of the vehicle target, with a target confidence score of 0.55, reaching the second confidence threshold of 0.40 but not the first confidence threshold of 0.70, and is classified as a transition region. A third image patch does not overlap with the vehicle target region but is adjacent to the core region, with a texture entropy of 4.8, reaching the texture threshold of 4.2, and is also classified as a transition region. The remaining image patches are classified as background regions.

[0065] When forming the data sequence to be sent, the regional data are arranged in the order of core region, transition region, and background region. All core region data precedes all transition region data, and all transition region data precedes all background region data.

[0066] When data from the same region comes from different keyframes, it is sorted in descending order of the frame importance of its respective keyframe. When the frame importance is the same, it is sorted in ascending order of the keyframe acquisition time. When data from the same region belongs to the same keyframe, the core regions are sorted in descending order of target confidence; when the target confidence is the same, they are sorted in ascending order of the row and column numbers in the region identifier. Transition regions and background regions are sorted in descending order of texture entropy; when the texture entropy is the same, they are sorted in ascending order of the region identifier.

[0067] For example, the frame importance of the first keyframe is 0.68, and the frame importance of the second keyframe is 0.62. When both frames include core region data, the core region data of the first keyframe is placed before the core region data of the second keyframe. When the second keyframe includes two core region data points with target confidence levels of 0.82 and 0.76, the core region data with a target confidence level of 0.82 is placed before the core region data with a target confidence level of 0.76.

[0068] The region data can be compressed using JPEG, JPEG2000, WebP intraframe compression, or a custom encoding method based on discrete cosine transform. This implementation uses the baseline JPEG encoding method. A 16×16 pixel image block is converted to the YCbCr color space and divided into four 8×8 pixel transform blocks. A discrete cosine transform is performed on each transform block, and the transform coefficients are quantized using a quantization table associated with the region type and bandwidth level. Then, zigzag scanning, run-length encoding, and Huffman encoding are performed to form the compressed region data.

[0069] The core region uses a quantization table with the smallest quantization step size, the transition region uses a quantization table with a quantization step size between that of the core region and the background region, and the background region uses a quantization table with the largest quantization step size. The higher the compression ratio, the smaller the amount of data after compression.

[0070] The video capture terminal pre-stores the mapping relationship between bandwidth levels and compression ratios for three types of regions. Under the first bandwidth level, the compression ratios for the core region, transition region, and background region are 8:1, 32:1, and 128:1, respectively; under the second bandwidth level, they are 6:1, 24:1, and 96:1, respectively; under the third bandwidth level, they are 4:1, 16:1, and 64:1, respectively; and under the fourth bandwidth level, they are 2:1, 8:1, and 32:1, respectively.

[0071] Compression ratio can be calibrated using sample keyframes. Multiple sets of JPEG quality parameters are set for the sample keyframes that have already undergone region classification. JPEG quality parameters can be set in intervals of 5 to 95. The core region, transition region, and background region are compressed using each quality parameter respectively, and the data length before and after compression is recorded.

[0072] After decompressing the core region, the decompressed image is re-input into the object detection network. If the original object category is still detected, and the overlap ratio between the original object detection box and the target detection box reaches 0.70, the core region is considered to meet the retention criteria. Candidate quality parameters that meet the retention criteria are sorted in ascending order of compressed data size, and the set with the smallest compressed data size is selected as the core region compression parameters.

[0073] After decompression of the transition region, the Canny edge detection method is used to extract the edges of the core region and the transition region, and the proportion of edge pixels that can connect at the region boundary is calculated. When the edge pixel connection ratio reaches 90%, the transition region is determined to meet the contour continuity condition. The candidate quality parameters that meet the contour continuity condition are sorted in ascending order of compressed data volume, and the set with the smallest compressed data volume is selected as the compression parameters for the transition region.

[0074] After decompression of the background region, the structural similarity before and after decompression is extracted. When the structural similarity reaches a set background retention value, the background region is determined to meet the scene retention condition. In this embodiment, the background retention value is set to 0.50, and the set of candidate quality parameters that results in the smallest compressed data volume among those meeting the scene retention condition is used as the background region compression parameters.

[0075] Candidate compression parameters for the three regions are applied together to the sample keyframes, and the total amount of data generated per unit time is calculated. When the total amount of data exceeds the allowable amount of data for the corresponding bandwidth level, the compression ratio of the background region is increased; when the background region still exceeds the allowable amount of data after reaching the set maximum compression ratio, the compression ratio of the transition region is increased; the compression ratio of the core region is only increased when it still exceeds the allowable amount of data after the first two adjustments.

[0076] After the regional data is compressed, it is associated with the frame number, region identifier, and region type. A data sequence to be transmitted is then formed according to the region type, the frame importance of the keyframe to which it belongs, and the sorting result within the same keyframe. The sending end transmits the regional data sequentially, starting from the first position of the data sequence.

[0077] If the current transmission cycle cannot transmit the entire data sequence, the untransmitted area data is carried over to the next transmission cycle. At the start of the next transmission cycle, the newly generated area data and the untransmitted area data are rearranged together according to the above rules.

[0078] Before fragmentation and encapsulation, the header length is determined based on the length of the fields contained in the header. The header includes the frame number, region identifier, region type, fragment sequence number, total number of fragments, and checksum. The fragment data length is obtained by subtracting the header length from the maximum payload of a single message.

[0079] If the length of the region data does not exceed the fragment data length, the region data is encapsulated into a fragment. If the length of the region data exceeds the fragment data length, starting from the beginning of the region data, fragments are extracted sequentially according to their lengths until all region data is fragmented. The actual data length of the last fragment can be less than the fragment data length. Set the total number of fragments according to the actual number of fragments formed, and assign fragment numbers sequentially starting from 1.

[0080] The frame number is used to determine the keyframe to which the fragment belongs; the region identifier is used to determine the image block to which the fragment belongs and its position in the keyframe; the region type is used to indicate whether the fragment belongs to the core region, transition region, or background region; the fragment sequence number is used to determine the current fragment's position in the region data; the total number of fragments is used to determine whether the region data is complete; and the checksum is used to determine whether an error occurred during the fragment's transmission.

[0081] The checksum can be CRC-8, CRC-16, or CRC-32. This implementation uses CRC-16-CCITT checksum, with an initial value set to hexadecimal FFFF and a polynomial of hexadecimal 1021. The sending end calculates the checksum together with the data in the packet header (excluding the checksum field) and the fragmented payload, and writes the calculation result into the checksum field.

[0082] The sending end transmits fragments of data from each region according to the data sequence to be transmitted. When a region of data includes multiple fragments, all fragments of that region of data are transmitted in ascending order of fragment number, and then the next region of data in the data sequence to be transmitted is transmitted.

[0083] The receiving end reads the frame number and region identifier from the fragments, and groups fragments with the same frame number and region identifier into the same fragment set. The fragments in the set are arranged according to their fragment sequence number, and the actual number of received fragments is compared with the total number of fragments. If the actual number of received fragments is less than the total number of fragments, or if there are discontinuous positions in the fragment sequence number, the missing fragment sequence number is identified as a missing fragment.

[0084] The receiver recalculates the checksum using the same CRC-16-CCITT parameters as the transmitter and compares it with the checksum carried in the fragment. If the two checksums match, the fragment is considered valid; if they differ, the fragment is considered invalid and treated as a missing fragment with the same fragment sequence number.

[0085] When all fragments of a region's data have been received and verified, the receiving end concatenates and compresses the region's data according to the fragment sequence number, decompresses the region's data using the JPEG decoding method corresponding to that of the sending end, and writes the decompressed image block into the specified position in the keyframe according to the region identifier.

[0086] When the missing segment belongs to the background region, the receiving end generates repair data based on the texture features of adjacent background regions in the same frame. Background region repair can employ nearest neighbor image block copying, texture feature-based block matching, bilinear interpolation, fast progress inpainting, or texture synthesis methods. This implementation uses the texture feature-based block matching method.

[0087] Centered on the missing background image patch, read the image patches within its eight neighboring areas that have been reconstructed and belong to the background region. For each adjacent background image patch, calculate the average gray level, texture entropy, and principal gradient direction. The average gray level is obtained by averaging the gray values ​​of all pixels within the image patch. The principal gradient direction is calculated using the Sobel operator for both the horizontal and vertical gradients, classifying the gradient direction of each pixel into four categories: horizontal, vertical, left-slanted, and right-slanted. The category with the most pixels is selected as the principal gradient direction.

[0088] Compare the average grayscale difference between adjacent background image patches and the boundary pixels surrounding the missing location. If the average grayscale difference is the same, compare the texture entropy difference between adjacent background image patches and the image patches surrounding the missing location; if the texture entropy difference is still the same, compare whether the principal gradient directions are consistent. Determine the texture source image patch in the order of minimum average grayscale difference, minimum texture entropy difference, and consistent principal gradient direction.

[0089] The texture source image patch is copied to the missing location. After copying, a linear gradient is applied to the pixels within a two-pixel width surrounding the repaired image patch. Pixels located at the edge of the repaired image patch use the boundary pixels of the surrounding complete image patch, and the proportion of texture source pixels is gradually increased as the patch moves inward to obtain background region repair data.

[0090] If no complete background image patch exists within the eight neighboring regions, the search range is expanded to include the two image patches surrounding the missing location, and the texture source is determined according to the same rules. If no complete background image patch is found after expanding the search range, the Telea fast-moving inpainting method can be used to propagate grayscale and color information from the complete pixels surrounding the missing area to the missing location, generating background area inpainting data.

[0091] When the missing segment belongs to a transition region, the receiver generates repair data based on image information from adjacent regions in the same frame and adjacent keyframes. Temporal location matching of the transition region can employ block matching, template matching, or optical flow estimation. This implementation uses a block matching method based on the sum of absolute differences.

[0092] Determine the row and column position of the missing transition image block in the current keyframe based on the region identifier, read the core region, transition region, and background region image blocks that have been reconstructed within its eight neighboring regions, and obtain the boundary pixels around the missing position.

[0093] Find the most recent keyframes that have been reassembled before and after the current keyframe. Read image blocks with the same region identifier in the previous and next keyframes, and use the image block as the center, the range of one image block above, below, left and right as the search range, and each image block within the search range is a candidate image block.

[0094] The top, bottom, left, and right edges of each candidate image patch are compared with the existing boundary pixels around the missing location in the current keyframe. The absolute value of the relative pixel grayscale difference is calculated, and all absolute grayscale difference values ​​are summed to obtain the boundary difference value of the candidate image patch. The candidate image patch with the smallest boundary difference value is determined as the source of temporal inpainting.

[0095] When multiple candidate image blocks with the same boundary difference exist, select the candidate image block whose time distance between its keyframe and the current keyframe is the shortest; if the time distance is still the same, select the candidate image block whose region identifier is the same as the missing region.

[0096] The temporal inpainting source is written to the missing location, and boundary correction is performed on the area two pixels wide surrounding the written image block. The boundary correction uses a linear interpolation method to gradually transition the edge pixels of the inpainted image block from the boundary pixels of adjacent regions in the same frame to the inner pixels of the temporal inpainting source, thus obtaining transition region inpainting data.

[0097] If there is only one reconstructed keyframe before or after the current keyframe, block matching is performed only in that keyframe. If there are no available candidate image blocks in either of the preceding or following keyframes, a bilinear interpolation method is used to generate transition region repair data based on the boundary pixels in the upper, lower, left, and right directions of the missing location, and this transition region is marked as the region to be repaired.

[0098] When a missing fragment belongs to a core region, the receiving end generates a directed retransmission request. When the sending end sends a fragment for the first time, it writes the fragment to the sending buffer and establishes a buffer index using the frame number, region identifier, and fragment sequence number. The sending buffer also records the region type, sending time, and retransmission count.

[0099] The receiving end writes the frame number, region identifier, and fragment sequence number of the missing fragment in the core area into the directed retransmission request. The sending end reads the directed retransmission request, retrieves the transmission buffer according to the frame number, region identifier, and fragment sequence number, selects the fragment jointly identified by the three identifiers as the target fragment, and retransmits the target fragment.

[0100] After receiving the target fragment, the receiving end adds it to the fragment set with the same frame number and region identifier, and writes it to the original missing location according to the fragment sequence number. If the target fragment passes CRC-16 verification and all fragments of the core region have been received, the core region is marked as complete. If the target fragment verification fails or other missing fragments still exist in the core region, a retransmission request is generated again.

[0101] This implementation sets the maximum number of retransmissions for the same target fragment to 3. If the verification fails after reaching the maximum number of retransmissions, the core area is marked as a region to be retransmitted, and the repaired data from the background or transition areas is not used to replace the core area. After the link is restored, the receiving end continues to request the target fragment of the region to be retransmitted. The transmission buffer is retained at least until the receiving end confirms that the core area is complete, or until the buffer retention time is reached.

[0102] In one complete embodiment, the invention is applied to narrowband transmission of video from field vehicle inspections. The video acquisition end captures color video at a resolution of 640×480 at 10 frames per second. The narrowband communication module calculates the effective bandwidth within the last 10 seconds, obtaining an effective bandwidth of 4Kbps, and determines the current bandwidth level as the second bandwidth level.

[0103] The filtering threshold for the second bandwidth level is 0.60, and the compression ratios for the core region, transition region, and background region are 6:1, 24:1, and 96:1, respectively. The current communication protocol has a single message payload limit of 124-128 bytes. The message header uses 2 bytes to record the frame number, 2 bytes to record the region identifier, 1 byte to record the region type, 2 bytes to record the fragment sequence number, 2 bytes to record the total number of fragments, and 2 bytes to record the checksum. It also uses 1 byte to record the effective data length in the fragment. The message header length is 12 bytes, and each fragment can carry 112 bytes of region data.

[0104] The video capture terminal obtains two adjacent video frames with frame numbers 1055 and 1056. The two video frames are processed using the BT.601 weighted grayscale conversion method. The absolute value of the grayscale difference is calculated pixel by pixel and averaged. After normalization, the inter-frame variation of video frame 1056 is found to be 0.80.

[0105] The YOLO-Nano object detection network detected a vehicle and a person in video frame 1056. After removing the overlap, the two target regions occupy 0.50% of the video frame area, resulting in a target coverage of 0.50%. With a change weight of 0.4, the calculated inter-frame change is 0.32; with a target weight of 0.6, the calculated target coverage is also 0.30. Adding these two results, the frame importance of video frame 1056 is 0.62. This frame importance reaches the second bandwidth level screening threshold of 0.60, thus identifying video frame 1056 as a keyframe.

[0106] The keyframe with frame number 1056 was divided into 40 columns and 30 rows of 16×16 pixels, resulting in 1200 image blocks. The texture entropy of each image block was obtained using grayscale histogram information entropy. For each image block, the number of pixels at 256 grayscale levels was counted, and the number of pixels at each grayscale level was converted into the occurrence ratio. The texture entropy was obtained by multiplying the occurrence ratio of each non-zero grayscale level by the absolute value of its base-2 logarithm.

[0107] The target detection network outputs a vehicle target confidence score of 0.82 and a person target confidence score of 0.76. When an image patch overlaps with a vehicle target region, a confidence score of 0.82 is assigned to that image patch; when an image patch overlaps with a person target region, a confidence score of 0.76 is assigned to that image patch; when an image patch overlaps with two target regions simultaneously, the highest target confidence score of 0.82 is assigned to that image patch.

[0108] In this embodiment, 0.70 is used as the first confidence threshold and 0.40 is used as the second confidence threshold. After excluding image blocks with a target confidence of 0.70, the texture entropy of the remaining image blocks is arranged in ascending order of value. The transition region is divided using the texture entropy at the 60th, 65th, 70th, 75th, and 80th percentile positions, respectively.

[0109] When using the texture entropy at the 75th and 80th percentile positions, the contour preservation rate of the reconstructed image did not reach 90%; when using the texture entropy at the 60th, 65th, and 70th percentile positions, the contour preservation rate reached 90%, and the amount of data to be sent did not exceed the allowable data amount of the second bandwidth level. The texture entropy at the 70th percentile position is 4.2, which is the maximum value among the candidate texture thresholds that meet the conditions. Therefore, 4.2 is determined as the texture threshold for the current keyframe.

[0110] Image patch labeled "14-17" overlaps with the vehicle target area, with a target confidence score of 0.82, and is therefore classified as a core region. Image patch labeled "14-18" overlaps with the edge of the vehicle target, with a target confidence score of 0.55, reaching the second confidence threshold but not the first confidence threshold, and is therefore classified as a transition region.

[0111] Image patch labeled "13-17" does not overlap with the vehicle target area, but is adjacent to the core area labeled "14-17". Its texture entropy is 4.8, reaching the texture threshold of 4.2, and it is also classified as a transition area. Image patch labeled "12-28" does not cover the target and is not adjacent to the core area, so it is classified as a background area.

[0112] The current data to be transmitted also includes another keyframe with a frame importance of 0.68. When forming the data sequence to be transmitted, all core region data is arranged before all transition region data, and all transition region data is arranged before all background region data. Within the same type of region, the keyframe data with a frame importance of 0.68 is arranged before the keyframe data with frame number 1056, which has a frame importance of 0.62.

[0113] Within frame number 1056, the core region data with a target confidence of 0.82 is arranged before the core region data with a target confidence of 0.76, and the transition region and background region are arranged in descending order of texture entropy.

[0114] The video acquisition unit uses baseline JPEG encoding to process three types of region data. Each 16×16 pixel image block is divided into four 8×8 pixel transform blocks, and color space conversion, discrete cosine transform, quantization, zigzag scanning, run-length encoding, and Huffman encoding are performed sequentially. Using compression parameters at the second bandwidth level, compression ratios of 6:1, 24:1, and 96:1 are achieved for the core region, transition region, and background region, respectively.

[0115] The original color pixel data of the core region, identified as "14-17", is 768 bytes long, which is compressed to 128 bytes. Each slice can carry 112 bytes of region data, and this core region data is divided into two slices. Both slices carry frame number 1056, region identifier "14-17", core region type, their respective slice sequence number, total number of slices 2, and CRC-16 checksum.

[0116] The receiver assembles and segments the background image according to frame number and region identifier. The background region with region identifier "12-28" has one missing segment. The receiver reads the reconstructed background image blocks within its eight neighboring regions and calculates the average grayscale difference between each background image block and the boundary pixels of the missing location.

[0117] If there is only one background image patch with the smallest average gray level difference, that patch is identified as the texture source. If there are background image patches with the same average gray level difference, the image patch with the smallest texture entropy difference and the same principal gradient direction is selected. The receiving end copies the texture source to the missing location and performs linear gradient processing on the outer two pixel width range to form background region restoration data.

[0118] The transition region, identified as "14-18," contains a missing segment. The receiving end reads the image blocks around this region that have already been reconstructed, obtaining the boundary pixels where the vehicle outline enters and leaves the missing position.

[0119] The receiving end reads candidate image blocks within the region marked "14-18" and its surrounding one image block from the previous and next keyframes. It then calculates the sum of the absolute values ​​of the grayscale differences between the outer pixels of each candidate image block and the boundary pixels of the current keyframe. The candidate image block in the next keyframe with the smallest boundary difference is selected as the source for temporal inpainting. The receiving end writes the temporal inpainting source to the missing location and uses linear interpolation to correct the width range of the two outer pixels, forming the transition region inpainting data.

[0120] The core area with the region identifier "14-17" should include two fragments. The receiving end only receives fragment sequence number 1 and not fragment sequence number 2. The receiving end generates a directed retransmission request based on frame number 1056, region identifier "14-17", and fragment sequence number 2. The sending end searches its transmission buffer according to the three identifiers, obtains the target fragment, and retransmits it.

[0121] After receiving the target fragment, the receiving end writes it into the fragment set with the region identifier "14-17" and concatenates it with fragment sequence number 1. The target fragment passes CRC-16 verification, both fragments have been received completely, and the core region with the region identifier "14-17" is marked as complete.

[0122] The receiving end writes the fully received region data, background region repair data, transition region repair data, and re-received core region data into the positions indicated by each region identifier, completes the key frame reassembly of frame number 1056, and outputs the reassembled key frame according to the frame number.

[0123] In summary, by combining the inter-frame variation of adjacent frames with the target coverage to determine frame importance, and setting a filtering threshold based on bandwidth level, video frames with significant image changes or containing target information are prioritized for transmission, reducing the redundant use of limited communication capacity by static and similar images. After dividing keyframes into image blocks, core regions, transition regions, and background regions are distinguished based on texture entropy and target confidence. The transmission order and compression level are determined according to region type, frame importance, and bandwidth level, ensuring that limited bandwidth prioritizes data from core regions and high-importance keyframes. The receiving end completes missing transition and background regions using adjacent regions within the same frame and adjacent keyframes, initiating targeted retransmission only for missing core regions. This avoids indiscriminate retransmission of all missing segments, reducing the amount of duplicated data transmission and shortening the recovery waiting time for core images, thereby improving the transmission and reconstruction efficiency of critical video content under narrow bandwidth conditions.

[0124] Based on another preferred embodiment described above, see [link to preferred embodiment]. Figure 3 As shown, this embodiment provides a narrow-bandwidth video transmission system based on temporal-spatial importance analysis, used to apply the aforementioned narrow-bandwidth video transmission method based on temporal-spatial importance analysis, including: The data acquisition unit is used to acquire continuous video frames, bandwidth levels, and the maximum payload of a single message. The key screening unit is used to compare adjacent frames to obtain the inter-frame change, perform target detection on video frames to obtain target coverage and target confidence; determine the frame importance based on the inter-frame change and target coverage, and identify video frames that reach the screening threshold corresponding to the bandwidth level as key frames. The region segmentation unit is used to divide the keyframe into image blocks. Based on the texture entropy of the image block and the target confidence of the covered target, the image block is assigned to the core region, transition region or background region. The sequence generation unit is used to take the regional data of key frames as transmission units, arrange them in the order of core region, transition region and background region, and arrange them in descending order of the frame importance of key frames within the same region; determine the compression degree of each type of regional data according to the bandwidth level to form a data sequence to be sent; The fragmentation processing unit is used to fragment and encapsulate the data sequence to be transmitted based on the upper limit of the single message payload; the receiving end reassembles the key frame based on the information carried by each fragment, fills in the missing transition area or background area according to the adjacent area of ​​the same frame and the adjacent key frame, generates a directional retransmission request for the missing core area and retransmits the corresponding fragment.

[0125] It is understandable that the narrow-bandwidth video transmission method and system based on the time-space importance analysis described above have the same beneficial effects, and will not be elaborated further here.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A narrow-bandwidth video transmission method based on temporal-spatial importance analysis, characterized in that, include: Get the continuous video frames, bandwidth level, and maximum payload per message; Compare adjacent frames to obtain inter-frame variation, perform target detection on video frames to obtain target coverage and target confidence; determine frame importance based on the inter-frame variation and target coverage, and identify video frames that reach the filtering threshold corresponding to the bandwidth level as key frames; The keyframes are divided into image blocks, and the image blocks are classified into core regions, transition regions or background regions according to the texture entropy of the image blocks and the target confidence of the covered targets. The keyframe region data is used as a transmission unit and arranged in the order of core region, transition region and background region. Within the same region, the keyframes are arranged in descending order of their importance. The compression degree of each type of region data is determined according to the bandwidth level to form a data sequence to be sent. The data sequence to be sent is fragmented and encapsulated based on the single message payload limit; the receiving end reassembles key frames based on the information carried by each fragment, fills in the missing transition area or background area according to the adjacent area of ​​the same frame and the adjacent key frames, generates a directed retransmission request for the missing core area and retransmits the corresponding fragment.

2. The narrow-bandwidth video transmission method based on temporal-spatial importance analysis according to claim 1, characterized in that, When obtaining the inter-frame variation, target coverage, and target confidence, the following steps are included: Adjacent video frames are converted into grayscale images, the average value of the absolute value of the grayscale difference of the corresponding pixels is calculated and normalized to obtain the inter-frame change; target regions in the video frames are identified, the proportion of the union area of ​​each target region to the area of ​​the video frame is determined as the target coverage, and the confidence level carried by the target detection result is determined as the target confidence level.

3. The narrow-bandwidth video transmission method based on temporal-spatial importance analysis according to claim 2, characterized in that, Determining the keyframe includes: The frame importance is obtained by weighting and summing the normalized inter-frame change and the target coverage according to the change weight and the target weight; a filtering threshold is set for each bandwidth level, and the lower the available bandwidth corresponding to the bandwidth level, the higher the filtering threshold.

4. The narrow-bandwidth video transmission method based on temporal-spatial importance analysis according to claim 1, characterized in that, When obtaining the texture entropy of an image patch and the target confidence of the covered target, the following are included: The keyframes are divided into sections with a base size of 16×16 pixels. The texture entropy is calculated based on the grayscale distribution of pixels within the image block. The target confidence of the overlapping target is associated with the image block based on the overlap relationship between the target region and the image block. When the image block overlaps with multiple targets, the highest target confidence is selected.

5. The narrow-bandwidth video transmission method based on temporal-spatial importance analysis according to claim 4, characterized in that, When classifying the image patch into a core region, transition region, or background region, the following is included: Image blocks that cover the target and whose target confidence reaches the first confidence threshold are assigned to the core region; image blocks whose target confidence reaches the second confidence threshold but does not reach the first confidence threshold, as well as image blocks adjacent to the core region and whose texture entropy reaches the texture threshold, are assigned to the transition region; the remaining image blocks are assigned to the background region; the first confidence threshold is greater than the second confidence threshold.

6. The narrow-bandwidth video transmission method based on temporal-spatial importance analysis according to claim 5, characterized in that, Forming the data sequence to be sent includes: The region data of the core region, transition region, and background region are arranged in descending order. The data of the same type of region are arranged in descending order of the frame importance of the keyframe to which they belong. The core region within the same keyframe is arranged in descending order of target confidence, and the transition region and background region are arranged in descending order of texture entropy. The compression degree of the three types of region data is set for each bandwidth level. The compression degree of the core region is lower than that of the transition region, and the compression degree of the transition region is lower than that of the background region.

7. The narrow-bandwidth video transmission method based on temporal-spatial importance analysis according to claim 6, characterized in that, When fragmenting and encapsulating the data sequence to be sent, the following steps are included: The fragment data length is determined based on the difference between the single message payload limit and the message header length. The data of each region is divided into multiple fragments. A frame number, region identifier, region type, fragment sequence number, total number of fragments and check code are added to each fragment. The fragments are then sent in the order of the data sequence to be sent.

8. The narrow-bandwidth video transmission method based on temporal-spatial importance analysis according to claim 7, characterized in that, When filling in missing transition or background areas, the following should be included: The receiving end collects each fragment according to the frame number and region identifier, and identifies the missing fragment based on the fragment sequence number, the total number of fragments, and the check code. When the missing fragment belongs to the background region, repair data is generated based on the texture features of the adjacent background region in the same frame. When the missing fragment belongs to the transition region, repair data is generated based on the image information of the adjacent region in the same frame and the corresponding region in the reconstructed adjacent keyframe.

9. The narrow-bandwidth video transmission method based on temporal-spatial importance analysis according to claim 8, characterized in that, When performing targeted retransmission of missing core regions, the following is included: The receiving end generates the directed retransmission request based on the frame number, region identifier, and fragment sequence number of the missing fragment in the core region; According to the directed retransmission request, the sending end extracts the target fragment identified by the frame number, the region identifier and the fragment sequence number from the sending buffer, and retransmits the target fragment; The receiving end writes the target fragment into the core region determined by the frame number and the region identifier, and performs verification according to the check code.

10. A narrow-bandwidth video transmission system based on temporal-spatial importance analysis, used to apply the narrow-bandwidth video transmission method based on temporal-spatial importance analysis as described in any one of claims 1-9, characterized in that, include: The data acquisition unit is used to acquire continuous video frames, bandwidth levels, and the maximum payload of a single message. The key screening unit is used to compare adjacent frames to obtain the inter-frame change, perform target detection on video frames to obtain target coverage and target confidence; determine the frame importance based on the inter-frame change and the target coverage, and identify video frames that reach the screening threshold corresponding to the bandwidth level as key frames. The region segmentation unit is used to divide the keyframe into image blocks and classify the image blocks into core regions, transition regions or background regions according to the texture entropy of the image blocks and the target confidence of the covered target. The sequence generation unit is used to take the regional data of key frames as transmission units, arrange them in the order of core region, transition region and background region, and arrange them in descending order of the frame importance of key frames within the same region; determine the compression degree of each type of regional data according to the bandwidth level to form a data sequence to be sent; The fragmentation processing unit is used to fragment and encapsulate the data sequence to be sent based on the single message payload limit; the receiving end reassembles key frames based on the information carried by each fragment, fills in the missing transition area or background area according to the adjacent area of ​​the same frame and the adjacent key frames, generates a directed retransmission request for the missing core area and retransmits the corresponding fragment.

Images