Autonomous controllable architecture software and hardware collaborative audio and video signal transmission method and system

Abstract

The invention relates to the technical field of audio and video coding, decoding and processing, provides an audio and video signal transmission method and system based on software and hardware cooperation of an autonomous controllable architecture, and solves the problem of low bandwidth utilization rate. The method comprises the following steps: synchronously acquiring bandwidth fluctuation, packet loss rate and transmission delay parameters of a mobile terminal link, and generating a network state level through association analysis; performing resolution scaling on a brightness component of a video frame, performing dynamic downsampling on a chrominance component, generating a resolution reduction frame adaptive to a network state, and generating a fault-tolerant coding stream by combining a packet loss rate regulation and control quantization step size, a frame group length and an inter-frame prediction mode; performing entropy decoding and motion compensation to align brightness and chrominance components, forming a decoding reconstruction frame, performing multi-scale feature extraction and detail compensation, generating an enhanced brightness graph, and performing independent reconstruction on chrominance decoding components; and outputting an original resolution video frame through color reconstruction fusion. According to the invention, high-definition video low-delay transmission under network fluctuation is realized, and both bandwidth adaptability and image quality restoration precision are considered.

Description

Technical Field

[0001] The present application relates to the field of audio and video encoding and decoding and processing technology, and in particular to an audio and video signal transmission method and system with autonomous and controllable architecture and coordinated software and hardware. Background Art

[0002] In mobile audio and video communication scenarios, such as remote medical consultations, emergency command systems, and mobile vehicle surveillance, network environments often face severe bandwidth fluctuations, high packet loss rates, and significant transmission delay jitter. These scenarios require transmission systems with adaptability, high fault tolerance, and low processing latency to meet interactive needs.

[0003] Currently, the mainstream solution uses SFU architecture transmission technology with full link state awareness. This solution collects network status through the server collaborating with multiple clients. The sender periodically reports the uplink network status, and the receiver feedbacks the downlink status. The server then aggregates the data and selects the worst downlink status to feed back to the sender.

[0004] However, while this solution improves adaptability to weak networks, it still suffers from several core flaws. First, the state perception granularity is insufficient, relying solely on macro-link status and failing to distinguish between the temporal and spatial characteristics of audio and video data. Second, the encoding strategy is disconnected from the content, using a unified redundancy mechanism to process all video frames without considering the complexity of the frame content, resulting in low bandwidth utilization. Summary of the Invention

[0005] The present application provides an audio and video signal transmission method and system with an autonomous and controllable architecture and software and hardware collaboration, to solve the problem of low bandwidth utilization in the prior art.

[0006] In a first aspect, the present application provides an audio and video signal transmission method with autonomous and controllable architecture and software and hardware collaboration, comprising: During audio and video transmission, the bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile link are synchronously collected, and a network status level is generated based on the correlation analysis of the bandwidth fluctuation parameters and the transmission delay parameters; According to the network status level, the frame adjustment device performs resolution scaling on the original luminance component of the video frame and selectively samples the original chrominance component to generate a reduced-resolution frame that matches the current network status level; Based on the total number of pixels and the spatial complexity parameter of the reduced-resolution frame, combined with the packet loss rate parameter, the quantization step range, the frame group length, and the inter-frame prediction mode are synchronously controlled to generate an error-tolerant coding stream with an integrated anti-packet loss structure; Performing an entropy decoding operation on the error-tolerant coded stream at a receiving end, and performing motion compensation alignment on the decoded luminance component and the decoded chrominance component of the reduced-resolution frame recovered by decoding to obtain a decoded reconstructed frame; Based on a neural network reconstruction algorithm, multi-scale feature extraction is performed on the luminance component of the decoded reconstructed frame and local detail features are superimposed for detail compensation to generate an enhanced luminance feature map. Chroma resolution reconstruction is performed on the decoded chroma component to generate a chroma reconstruction component. The enhanced luminance feature map and the chroma reconstruction component are synchronously reconstructed in color space to output a video frame restored to the original resolution.

[0007] Optionally, based on a neural network reconstruction algorithm, multi-scale feature extraction is performed on the luminance component of the decoded reconstructed frame and local detail features are superimposed to perform detail compensation to generate an enhanced luminance feature map, chrominance resolution reconstruction is performed on the decoded chrominance component to generate a chrominance reconstructed component, and color space synchronization reconstruction is performed on the enhanced luminance feature map and the chrominance reconstructed component to output a video frame restored to the original resolution, including: Performing a multi-level convolution operation on the brightness component of the decoded and reconstructed frame based on a neural network reconstruction algorithm to extract brightness features of different scales; From the brightness features of different scales, the brightness features of the finest scale are selected and superimposed and fused with the local detail features, and detail compensation is performed on the brightness component of the decoded and reconstructed frame to generate an enhanced brightness feature map; Performing bilinear interpolation reconstruction on the chrominance component of the decoded reconstructed frame to generate a chrominance reconstructed component matching the original resolution; The enhanced luminance feature map and the chrominance reconstruction component are synchronously superimposed according to a preset color space conversion rule, the original resolution is restored while ensuring color consistency, and the restored video frame is output.

[0008] Optionally, selecting the finest-scale brightness feature from the brightness features of different scales and superimposing and fusing it with the local detail feature, performing detail compensation on the brightness component of the decoded and reconstructed frame, and generating an enhanced brightness feature map, including: Selecting a finest-scale brightness feature from the brightness features of different scales, obtaining spatial position coordinates corresponding to the finest-scale brightness feature, and defining a pixel neighborhood of a fixed size with each of the spatial position coordinates as a center point; Performing weighted calculation on the feature values ​​of all positions within the pixel neighborhood to generate local detail features including all positions, wherein the weight coefficient of the spatial position coordinates is higher than that of the edge position; The finest-scale brightness feature and the local detail feature are added point by point according to the same spatial position coordinates to obtain a superposition fusion result after detail compensation, and a nonlinear activation operation is performed on the superposition fusion result to generate an enhanced brightness feature map.

[0009] Optionally, based on the total number of pixels and the spatial complexity parameter of the reduced-resolution frame, combined with the packet loss rate parameter, synchronously regulating the quantization step range, the frame group length, and the inter-frame prediction mode to generate an error-tolerant coding stream integrating an anti-packet loss structure, including: Calculating the data volume of a unit frame according to the total number of pixels of the reduced-resolution frame, and combining the data volume of the unit frame with a spatial complexity parameter to generate a frame complexity weight; Increasing an upper limit of a quantization step range based on a product of the frame complexity weight and the packet loss rate parameter, reversely mapping a maximum value of a frame group length according to the packet loss rate parameter, and using the maximum value as an upper limit threshold for limiting the number of reference frames for an inter-frame prediction mode; During the encoding process, the adjusted quantization step range, frame group length and inter-frame prediction mode parameters are injected into the encoder to generate an error-tolerant encoding stream.

[0010] Optionally, according to the network status level, performing resolution scaling on the original luminance component of the video frame and performing selective sampling on the original chrominance component by an image adjustment device to generate a reduced-resolution frame matching the current network status level includes: Querying a preset resolution scaling ratio table according to the network status level, determining a target scaling ratio of the brightness component, performing uniform sampling on the original brightness component according to the target scaling ratio, and generating a scaled brightness component; Perform non-uniform region division on the original chrominance component, select the area with spatial gradient below the threshold for sparse sampling based on the spatial gradient distribution of the original chrominance component, and obtain the sampled chrominance component; The scaled luminance component is combined with the sampled chrominance component to generate a reduced-resolution frame that matches the current network status level.

[0011] Optionally, performing an entropy decoding operation on the error-tolerant coded stream at a receiving end, and performing motion compensation alignment on the decoded luminance component and the decoded chrominance component of the reduced-resolution frame recovered by decoding to obtain a decoded reconstructed frame, comprising: Performing syntax element type judgment on the error-tolerant coding stream, determining the category of the element to be decoded and the corresponding context model index, and calculating the corresponding probability interval through an arithmetic decoding engine based on the current state value of the context model index; Comparing the value of the current parsing position in the error-tolerant coded stream with the boundary value of the probability interval, parsing out a binary character sequence, performing a debinarization operation on the binary character sequence, and restoring it to the decoded luminance component and the decoded chrominance component of the reduced-resolution frame; Extracting a motion vector of the decoded luminance component, calculating a spatial coordinate offset of the decoded chrominance component based on the motion vector, and adjusting a spatial position of the decoded chrominance component using the spatial coordinate offset to obtain an offset-compensated decoded chrominance component; The decoded chrominance component after offset compensation is matched with the decoded luminance component in the time domain, and the matched decoded luminance component and decoded chrominance component are merged to obtain a recombined decoded reconstructed frame.

[0012] Optionally, during audio and video transmission, bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile link are synchronously collected, and a network status level is generated based on correlation analysis between the bandwidth fluctuation parameters and the transmission delay parameters, including: Periodically obtaining bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile link within a fixed time window, and calculating variations of the bandwidth fluctuation parameters within a continuous time window; Dividing the transmission delay parameter into multiple level intervals, and associating each level interval with a change in the bandwidth fluctuation parameter, and determining a bandwidth stability score corresponding to each level interval based on a distribution state of the change in the corresponding level interval; Based on a preset scoring threshold mapping table, the bandwidth stability score corresponding to each level interval is converted into a discrete network status level.

[0013] In a second aspect, the present application provides an audio and video signal transmission system with an autonomous and controllable architecture and coordinated software and hardware, including: An analysis module is used to synchronously collect bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile link during audio and video transmission, and generate a network status level based on correlation analysis of the bandwidth fluctuation parameters and transmission delay parameters; a sampling module, configured to perform resolution scaling on the original luminance component of the video frame and selectively sample the original chrominance component according to the network status level through an image adjustment device, thereby generating a reduced-resolution frame that matches the current network status level; A generation module is configured to synchronously control the quantization step size range, the frame group length, and the inter-frame prediction mode based on the total number of pixels and the spatial complexity parameter of the reduced-resolution frame in combination with the packet loss rate parameter to generate an error-tolerant coding stream integrating an anti-packet loss structure; a reconstruction module, configured to perform an entropy decoding operation on the error-tolerant coded stream at a receiving end, and perform motion compensation alignment on the decoded luminance component and the decoded chrominance component of the reduced-resolution frame recovered by decoding to obtain a decoded reconstructed frame; A restoration module is used to perform multi-scale feature extraction on the luminance component of the decoded reconstructed frame based on a neural network reconstruction algorithm and superimpose local detail features to perform detail compensation, thereby generating an enhanced luminance feature map; perform chroma resolution reconstruction on the decoded chroma component to generate a chroma reconstruction component; and perform color space synchronous reconstruction on the enhanced luminance feature map and the chroma reconstruction component to output a video frame restored to the original resolution.

[0014] In a third aspect, the present application provides a computing device comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement an audio and video signal transmission method with an autonomous and controllable architecture and software and hardware collaboration as described in the first aspect above.

[0015] In a fourth aspect, the present application provides a computer storage medium storing a computer program. When the computer program is executed by a computer, it implements an audio and video signal transmission method with an autonomous and controllable architecture and software and hardware collaboration as described in the first aspect.

[0016] Beneficial effects of this application: This application collects bandwidth fluctuations, packet loss rate and transmission delay parameters and generates network status levels through correlation analysis, realizing dynamic perception and quantitative grading of network status, and providing decision-making basis for adaptive transmission; according to the network status level, the brightness component of the video frame is scaled and the chrominance component is selectively sampled to generate reduced-resolution frames, differentially reducing data load, maintaining the clarity of the subject outline under weak network conditions and improving bandwidth utilization; based on the total number of pixels, spatial complexity and packet loss rate of the reduced-resolution frame, the quantization step size, frame group length and inter-frame prediction mode are synchronously adjusted, and the anti-packet loss structure is dynamically injected to improve Fault tolerance and reduce the collapse of image quality under high packet loss rate; the receiving end performs entropy decoding on the fault-tolerant coded stream and aligns the motion compensation of the decoded luminance / chrominance components to eliminate displacement distortion during transmission and ensure the spatiotemporal synchronization of audio and video; the multi-scale feature extraction of the luminance component of the decoded reconstructed frame is carried out through a neural network, and local detail compensation is superimposed to generate an enhanced luminance feature map, restore high-frequency edge and texture details, and solve the blurring problem caused by resolution reduction; resolution reconstruction is performed on the chrominance component and color space synchronization is reconstructed with the luminance feature map to achieve high-fidelity restoration of the chrominance component and output video frames consistent with the original resolution.

[0017] Furthermore, multi-level convolution operations are performed on the luminance components of the decoded and reconstructed frames to extract multi-scale features, capturing luminance information from global structures to local micro-textures, providing a multi-granularity feature basis for detail compensation; the finest-scale luminance features are fused with local detail features for superposition compensation, enhancing high-frequency details such as edge sharpness and suppressing compression artifacts, thereby improving subjective picture clarity; bilinear interpolation reconstruction is performed on the decoded chrominance components to generate high-resolution chrominance data, restoring chrominance resolution with low computational complexity, and avoiding end-side delays caused by complex models; the enhanced luminance feature map and chrominance reconstruction components are synchronously superimposed according to the preset color space conversion rules to ensure the color consistency and alignment accuracy of the bright color channel and eliminate color distortion.

[0018] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A flowchart of an audio and video signal transmission method with autonomous and controllable architecture software and hardware collaboration provided by the present application is shown; Figure 2 A scenario diagram showing an audio and video signal transmission method with autonomous and controllable architecture and hardware collaboration provided by the present application is shown; Figure 3 The present invention provides a schematic diagram of the structure of an audio and video signal transmission system with an autonomous and controllable architecture and coordinated software and hardware; Figure 4 A schematic structural diagram of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0022] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0023] Research has found that existing audio and video transmission technologies have significant shortcomings in mobile network environments. Traditional methods typically use fixed encoding rules and uniform bandwidth allocation methods, making them difficult to cope with dynamic changes in network conditions, such as bandwidth fluctuations, packet loss, and transmission delays. This rigid approach leads to three core flaws: First, resource allocation lacks flexibility. High-resolution images are still transmitted when network conditions deteriorate, causing lag or wasted bandwidth, while image quality is not improved when the network is good. Second, the characteristics of different image elements in the video are not distinguished, and the same compression strategy is applied to critical image brightness and color information, resulting in distortion of dynamic images. Third, the fault-tolerance mechanism is inefficient, relying on static redundant data rather than dynamic adjustment strategies. This not only increases unnecessary data volume but also makes it difficult to protect the integrity of critical images. This problem is particularly prominent in weak network scenarios such as mobile live streaming and video conferencing. There is an urgent need for a new transmission method that can intelligently sense network conditions and dynamically adapt to content characteristics.

[0024] To address these issues, the present invention proposes a content-aware, dynamic collaborative optimization method. Its core approach is to collaboratively regulate the three factors of network status, content complexity, and fault-tolerant coding. Specifically, by synchronously monitoring the correlation between bandwidth fluctuations, packet loss rate, and transmission delay, multi-level network status labels are generated to achieve refined perception. Subsequently, video frames are adaptively processed based on the network status level, dynamically scaling the resolution of luminance data. A key area-prioritized sampling strategy is implemented for color data to preserve core color information while reducing data volume. Furthermore, based on the processed image spatial complexity and real-time packet loss rate, three encoding parameters are dynamically bound: compression fineness is adjusted to preserve details, keyframe intervals are shortened to reduce error propagation, and prediction modes are selected based on image motion characteristics. This generates a coded stream that balances low redundancy with high fault tolerance. Finally, at the receiving end, a neural network is used to perform multi-layer feature extraction and detail compensation on luminance, reconstruct resolution on chrominance, and finally integrate luminance information to output a complete image. This method forms a complete perception, decision-making, and reconstruction closed loop, significantly improving transmission efficiency over weak networks and resolving the conflict between idle resources and image distortion. It also operates plug-and-play on general-purpose hardware platforms, eliminating the need for specialized codecs. Its innovative breakthrough lies in upgrading traditional link adaptation to a content adaptation mechanism, which completely solves the key problems of extensive state perception and the disconnection between encoding strategies and content.

[0025] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0026] Figure 1 The present invention provides a flowchart of an audio and video signal transmission method with autonomous and controllable architecture software and hardware collaboration, such as Figure 1 As shown, the method includes: 101. During audio and video transmission, synchronously collect bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile terminal link, and generate a network status level based on correlation analysis between the bandwidth fluctuation parameters and the transmission delay parameters; Optionally, step 101 may specifically include the following steps: 1011. Periodically obtain a bandwidth fluctuation parameter, a packet loss rate parameter, and a transmission delay parameter of the mobile link within a fixed time window, and calculate a change in the bandwidth fluctuation parameter within a continuous time window; 1012. Divide the transmission delay parameter into multiple level intervals, associate each level interval with a variation of the bandwidth fluctuation parameter, and determine a bandwidth stability score corresponding to each level interval based on a distribution of the variation in the corresponding level interval; 1013. Based on a preset scoring threshold mapping table, convert the bandwidth stability score corresponding to each level interval into a discrete network status level.

[0027] In the above steps, the bandwidth fluctuation parameter refers to the degree of bandwidth change of the mobile link during audio and video transmission, measured in the number of data bits transmitted per second; the packet loss rate parameter indicates the proportion of data packets lost during data transmission, given as a percentage; the transmission delay parameter describes the time delay from the sender to the receiver, measured in milliseconds; the network status level represents the network quality as a discrete level; the change is calculated by continuously comparing the values ​​of the bandwidth fluctuation parameter in different time periods to determine the magnitude of the change; the level interval divides the transmission delay parameter into multiple ranges to evaluate network performance at different levels; the distribution state is used to observe the dispersion of the change within each level interval; the bandwidth stability score is a numerical score calculated based on the distribution state to represent network stability; the preset score threshold mapping table is a pre-defined table used to map the score to a specific discrete network status level.

[0028] In the embodiment of the present application, first, the network parameter collection and change calculation are realized through step 1011, so that the system periodically obtains the bandwidth fluctuation parameter, packet loss rate parameter and transmission delay parameter of the mobile link within a fixed time window, wherein the bandwidth fluctuation parameter adopts the adjacent time window difference calculation method: First, record the bandwidth value at time T , after a fixed time interval Then record the new bandwidth value , and then calculate the bandwidth change formula Get the bandwidth fluctuation change, where Indicates the change in megabits per second (Mbps). At the same time, the system synchronously records the transmission delay parameters and packet loss rate parameters at that moment, forming three data sets including bandwidth change, transmission delay and packet loss rate. For example, in the flight simulator intercom system monitoring scenario, the system collects the audio transmission link bandwidth every 1 second and records it for the first time. =500Mbps, recorded after 1 second =520Mbps, calculate the bandwidth fluctuation =1520−500=20Mbps. At this time, the transmission delay parameter is 150 milliseconds and the packet loss rate parameter is 0.3%.

[0029] Secondly, the delay grading and stability score calculation is completed through step 1012. The transmission delay parameter is first divided into four grade intervals: 0 to 100 milliseconds is excellent, 101 to 200 milliseconds is good, 201 to 300 milliseconds is medium, and more than 300 milliseconds is poor. The grading standard is set according to the flight simulation aviation communication standard. Then, the current transmission delay parameter value is classified into the corresponding interval. For example, a delay of 150 milliseconds is classified into the good interval. Then, the last N bandwidth fluctuation change samples are obtained within the corresponding delay interval, and the arithmetic mean of these samples is calculated. ,in is the sample value, N is the number of samples; and calculate the standard deviation ,in, Indicates the degree of data dispersion. Finally, the stability scoring formula Calculates the bandwidth stability score.

[0030] For example, when monitoring the flight intercom system, a 150 millisecond delay is classified as a good range, and 10 bandwidth change samples are taken. , calculate the mean , standard deviation , substituting into the scoring formula we get (Keep two decimal places).

[0031] Finally, step 1013 implements network status level mapping. A predefined scoring threshold mapping table specifies that 9.0 to 10.0 points correspond to an excellent network status, 7.0 to 8.9 points correspond to a good network status, 5.0 to 6.9 points correspond to an intermediate network status, and less than 5.0 points correspond to a poor network status. After the system obtains the bandwidth stability score calculated in step 1012, it searches the mapping table for the corresponding discrete network status level based on the numerical range of the score. In particular, when monitoring critical flight warning systems, if the score falls into an abnormal range, a logging mechanism is triggered. For example, in the engine warning sound transmission scenario, if the input bandwidth stability score is 9.74 points, the mapping table is searched to determine that it is in the 9.0 to 10.0 segment, and the output network status level is excellent. If the calculated score is 8.5 points, the mapping table is searched to determine that it is a good network status. At this time, the system automatically generates an alarm log to record the network status degradation event.

[0032] In a practical application, during network performance monitoring of a flight simulator's sound system, the system periodically collects the delay parameters of the sound transmission channel every two seconds: the time from audio signal generation to playback, as well as bandwidth fluctuation parameters and the amount of data transmitted per second. For example, if the delay value is 25 milliseconds in the first round of collection and 40 milliseconds in the second round, the calculated bandwidth fluctuation is 40-25 = 15 milliseconds. The system then categorizes the transmission delay parameters into three levels: 0-20 milliseconds is Excellent (Level 1), 21-50 milliseconds is Good (Level 2), and 51 milliseconds and above is Needs Optimization (Level 3). The current delay value of 40 milliseconds falls within the Good range, and the system analyzes the distribution of historical bandwidth fluctuations within this range. If historical data shows that most fluctuations are concentrated in the 5-15 millisecond range (e.g., 7 out of 10 similar data sets fall within this range), the bandwidth stability score is assigned a 7, with higher scores out of 10 indicating greater stability. Conversely, if fluctuations frequently exceed 20 milliseconds, the score is assigned a 5. Finally, based on the preset scoring threshold mapping table, for example, scores of 8 to 10 correspond to a stable network status level, 6 to 7 correspond to slight fluctuations, and 5 or below correspond to needs for optimization, the current instance's score of 7 is converted to a slight fluctuation level, thereby triggering the system to automatically increase audio buffer resource allocation to ensure real-time synchronization of cockpit ambient sound.

[0033] In the overall solution for step 101 above, discrete network status levels are automatically generated by synchronously collecting bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters from the mobile link, and analyzing the correlation between the bandwidth fluctuation and transmission delay parameters. This technical implementation involves calculating the change in the bandwidth fluctuation parameter within a periodic time window, dividing the transmission delay parameter into level intervals, and correlating them with the bandwidth variation distribution to evaluate the bandwidth stability score. This score is then converted into a network status level based on a preset mapping table, enabling efficient identification of network health and dynamic optimization of transmission strategies to improve the quality and stability of audio and video transmission.

[0034] 102. Based on the network status level, perform resolution scaling on the original luminance component of the video frame and perform selective sampling on the original chrominance component by an image adjustment device to generate a reduced-resolution frame that matches the current network status level; Optionally, step 102 may specifically include the following steps: 1021. Query a preset resolution scaling ratio table according to the network status level to determine a target scaling ratio of the brightness component, perform uniform sampling on the original brightness component according to the target scaling ratio, and generate a scaled brightness component; 1022. Perform non-uniform region division on the original chrominance component, and based on the spatial gradient distribution of the original chrominance component, select regions where the spatial gradient is lower than a threshold value for sparse sampling to obtain sampled chrominance components. 1023. Combine the scaled luminance component with the sampled chrominance component to generate a reduced-resolution frame that matches the current network status level.

[0035] In the above steps, the frame adjustment device is a hardware or software component used to dynamically adjust the size of the video frame. The video frame refers to the complete representation of a single image in a video sequence. The original brightness component is the original data part of the video frame that describes the pixel brightness information. The original chrominance component is the original data part of the video frame that describes the pixel color information. The resolution scaling table is a preset mapping table used to convert the network status level into a specific scaling value. The target scaling is the specific scaling value of the brightness component to be scaled according to the resolution scaling table. Uniform sampling is a method of extracting pixel values ​​at fixed intervals to process the brightness component. The scaled brightness component is the brightness data result after uniform sampling and scaling. Non-uniform area division It is a method of dividing an image into blocks of varying sizes to identify features in different regions. Spatial gradient distribution refers to the pattern of change in the rate of change of pixel values ​​in the image in the spatial direction. Spatial gradient is the value of the intensity of pixel change in the horizontal or vertical direction. The threshold is a preset numerical limit used to determine whether to perform a sampling operation. Sparse sampling is a method of processing chromaticity components by selecting pixels at a lower density. The sampled chromaticity components are the chromaticity data results obtained through sparse sampling. Combination is the process of merging the scaled luminance component and the sampled chromaticity component into a complete frame. The reduced-resolution frame is the final generated video frame output with reduced resolution. The current network status level is a discrete network quality assessment level obtained from the previous step, such as excellent or good.

[0036] In the embodiment of the present application, first, adaptive scaling of the brightness component is implemented through step 1021. A preset resolution scaling ratio table is queried based on the network status level to determine the target scaling ratio of the brightness component. The scaling ratio table is set according to the following principles: an excellent network status scaling ratio of 100% means maintaining the original resolution; a good network status scaling ratio of 80%; a medium network status scaling ratio of 60%; and a poor network status scaling ratio of 40%. After obtaining the target ratio, a uniform sampling operation is performed on the original brightness component, i.e., pixel values ​​are extracted at fixed intervals to generate a scaled brightness component. For example, in the processing of a flight simulator instrument panel video stream, when the network status level is good, a scaling ratio of 80% is determined by querying the table. The original 1080P brightness component (1920×1080 pixels) is uniformly sampled by 4 pixels for every 5 pixels in the row and column directions to generate a scaled brightness component of 864×576 pixels.

[0037] Secondly, the chroma component intelligent sampling is realized through step 1022. First, the original chroma component is non-uniformly divided into regions, and the image is divided into 16×16 pixel blocks; then, the spatial gradient value of each block is calculated. : ; The spatial gradient value G represents the gradient mean used to measure the intensity of color change within an image block; N represents the total number of pixels in the block; is the chrominance component The gradient in the horizontal direction reflects the amount of color level change; is the chrominance component The gradient in the vertical direction reflects the vertical change of color; is the chrominance component gradient in the horizontal direction; is the chrominance component Gradient in the vertical direction. and is the chrominance component, and Represents the horizontal and vertical gradients; finally, select the smooth area with gradient values ​​lower than the threshold T, such as T=15, for sparse sampling, and set the sampling density to increase by 10% for every 5 decrease in the gradient value to generate the sampled chrominance component. For example, when processing the flight terrain display video, the gradient value of the sky area G =8 is lower than the threshold of 15, sampling is performed at a density of 30%; the gradient value of the cockpit instrument area G =35 is higher than the threshold, 100% sampling is maintained, and the final amount of chroma data is reduced.

[0038] Finally, step 1023 completes the synthesis of the reduced-resolution frame, and the scaled luminance component generated in step 1021 and the sampled chrominance component generated in step 1022 are pixel-aligned and recombined, where the luminance component maintains complete spatial information, and the chrominance component is bilinearly interpolated based on the sampling position index to fill the unsampled area. Finally, a reduced-resolution frame that matches the network status level is generated. For example, in a good network state, the luminance component of 864×576 pixels is combined with the sparsely sampled chrominance component, and the interpolation formula is: ; in, Represents the chromaticity value reconstructed at the target interpolation point, that is, the color data of the unsampled position that needs to be calculated in the end; is the chromaticity value of the adjacent sampling point, is the distance weight, The reciprocal of the Euclidean distance is used to calculate the weight. A closer distance results in a larger reciprocal and a higher weight. This reconstructs complete chroma information and outputs downscaled frames at a resolution of 960×540, reducing the data volume to 45% of the original video, meeting aviation QTG standards.

[0039] In a practical application, in a video transmission optimization scenario for an aviation training simulator, when the network status level is assessed as "moderately volatile," for example, corresponding to level 2 in step 1013, the system initiates image resizing. First, based on the current level, the system queries the preset resolution scaling table to determine whether the luminance component needs to be scaled down to one-third of its original size. For example, if the original resolution is 1920×1080, the target size is 640×360. The system uniformly samples the original luminance component—for example, sampling every three pixels horizontally and every three pixels vertically—to generate the scaled luminance component. Next, the system processes the chrominance component: the image is divided into 5×5 non-uniform blocks and the spatial gradient of color variation within each block is calculated. If the variation value in a region is detected to be below a set threshold, such as a gradient value ≤8 in the sky background area, sparse sampling is performed, sampling only every four pixels in that region. For areas with more volatile instrument panel regions with gradient values ​​>8, the original sampling density is maintained. Ultimately, the resulting sampled chrominance components preserve color details in key areas. Finally, the 640×360 luminance component is combined with the selectively sampled chrominance components to generate a reduced-resolution frame. This frame maintains the natural gradient of the sky area while keeping the edges of the instrument panel text sharp. The overall data volume is reduced, adapting to the transmission requirements of current network fluctuations.

[0040] In the overall solution of step 102 above, the resolution of the video content is adaptively adjusted through the frame adjustment device based on the dynamically assessed network status level. Specifically, uniform resolution scaling based on a preset ratio table is performed on the original luminance component of the video frame, while spatial gradient-driven intelligent sampling is implemented on the original chrominance component. Selective sparse sampling is performed by identifying low-gradient texture areas, and ultimately the scaled luminance component is recombined with the optimized sampled chrominance component to generate a reduced-resolution frame that accurately matches the real-time network status. This solution significantly reduces the amount of transmitted data in high-latency or high-packet-loss network environments while ensuring that core visual information is not lost, achieving a dynamic balance between transmission quality and bandwidth usage.

[0041] 103. Based on the total number of pixels and the spatial complexity parameter of the reduced-resolution frame and the packet loss rate parameter, synchronously adjust the quantization step range, the frame group length, and the inter-frame prediction mode to generate an error-tolerant coding stream integrated with an anti-packet loss structure; Optionally, step 103 may specifically include the following steps: 1031. Calculate the data volume of a unit frame according to the total number of pixels of the reduced-resolution frame, and combine the data volume of the unit frame with a spatial complexity parameter to generate a frame complexity weight; 1032. Increase an upper limit of a quantization step range based on a product of the frame complexity weight and the packet loss rate parameter, reversely map a maximum value of a frame group length according to the packet loss rate parameter, and use the maximum value as an upper limit threshold for limiting the number of reference frames for an inter-frame prediction mode. 1033. During the encoding process, the adjusted quantization step range, frame group length, and inter-frame prediction mode parameters are injected into the encoder to generate an error-tolerant encoding stream.

[0042] In the above steps, the reduced-resolution frame refers to the video frame with reduced resolution after the processing in step 102; the total number of pixels refers to the total number of all pixels in the frame; the spatial complexity parameter is a numerical indicator describing the complexity of image texture details, which is calculated by edge density or gradient change; the packet loss rate parameter refers to the proportion of data packets lost during data transmission; the quantization step range refers to the range of quantization accuracy changes allowed during encoding compression, and the larger the quantization step, the higher the compression rate; the frame group length is the number of frames that are packaged into groups for encoding; the inter-frame prediction mode refers to the encoding method that uses the similarity of previous and next frames for compression; the frame complexity weight is the compression difficulty calculated by the total number of pixels and the spatial complexity parameter. Degree coefficient; the amount of data per unit frame refers to the original byte size of a single frame when it is not compressed; the upper limit value indicates the maximum numerical boundary allowed by the parameter; the upper limit threshold of the number of reference frames limits the maximum number of historical frames that can be referenced during inter-frame prediction; the fault-tolerant coding stream is a compressed video data stream with the ability to resist data loss; the fused anti-packet loss structure refers to the anti-packet loss mechanism formed by synchronously controlling the coding parameters; synchronous control means the simultaneous adjustment of the three parameters of quantization step range, frame group length, and inter-frame prediction mode; product increase refers to the operation of expanding the parameter range through multiplication operation; reverse mapping refers to the method of deriving the parameter value in inverse proportion according to the packet loss rate; injection encoder refers to inputting the adjusted parameters into the video compression system.

[0043] In the embodiment of the present application, first, the total number of pixels of the reduced resolution frame is obtained through step 1031 , which is calculated by multiplying the resolution width by the height. For example, in a flight simulator terrain display video, the reduced resolution frame resolution is 960 pixels wide by 540 pixels high, so the total number of pixels P = 960 × 540 = 518,400 pixels. Then, assuming the pixel depth is 8 bits, the formula Calculate the unit frame data volume, where Indicates the byte size of the original data of a single frame. The pixel depth unit is bit; Substituting the parameters into =8518,400×8=518,400 bytes. Then extract the space complexity parameter S , this parameter is calculated by image processing algorithms such as Sobel edge detector to calculate the image edge density. For example, the edge ratio of the terrain image is measured to be 35%, that is, S=0.35. Finally, generate the frame complexity weight coefficient , substituting into W =518400×0.35=181440. This weight is used to quantify the compression difficulty coefficient of the video frame. The entire process, from pixel statistics to weight output, is connected in sequence, providing basic data for subsequent parameter adjustment.

[0044] Secondly, step 1032 is first based on the frame complexity weight W and packet loss rate parameters L , calculate the upper limit of the quantization step range, which is calculated using the formula ,in, Indicates the default value of the basic quantization step size is 30, k The adjustment coefficient is fixed at 0.02 (used to standardize the frame complexity weight coefficient). W is the frame complexity weight, L is the packet loss rate parameter; for example, in flight simulator instrument panel video processing, the input W =181440 and L =0.005 (i.e. packet loss rate is 0.5%), substitute it into the formula to get =30+0.02×181440×0.005=48, which means the upper limit of quantization step is extended to 48 to improve compression efficiency.

[0045] Then according to the packet loss rate parameter L Reverse mapping the maximum frame group length, using the formula , where the symbol ⌊⋅⌋ represents the floor operation, L is the packet loss rate. For example, when the packet loss rate is 0.005, =⌊10×(1−0.005)⌋=⌊9.95⌋=9, which means that the maximum allowed frame group length is 9 frames to reduce the amount of data in a single packet. Then the maximum value is directly used as the upper limit threshold of the number of reference frames in the inter-frame prediction mode, that is, , limiting prediction to only the most recent nine frames to reduce the risk of error propagation. The entire process sequentially performs quantization step adjustment, frame group length mapping, and reference frame threshold setting, outputting parameters such as a quantization step limit of 48, a frame group length limit of 9 frames, and a reference frame limit of 9 frames, forming a coordinated control mechanism.

[0046] Finally, step 1033 generates an error-resilient coded stream. Specifically, the parameters adjusted in step 1032, including the quantization step size range (e.g., set to a minimum of 4 to a maximum of 48), the maximum frame group length (e.g., 9 frames), and the maximum inter-frame prediction reference frame length (e.g., 9 frames), are first input into a standard video encoder such as H.265. During compression, the encoder increases the quantization step size to 48 to improve compression efficiency, limits the frame group length to 9 frames to reduce the size of a single packet, and uses only the most recent 9 frames for inter-frame prediction, reducing the risk of error propagation. This incorporates a robust packet loss architecture. The final output is an error-resilient coded stream. For example, in a flight terrain video encoding scenario, decoding of the original video failed at a packet loss rate of 0.8%. However, after adjusting these parameters, the compression ratio increased by 35%, and the system could tolerate a packet loss rate of 1.2% while maintaining smooth decoding. The entire process, from parameter injection to stream output, is seamless, ensuring the authenticity of aviation training even in harsh network environments.

[0047] In actual applications, in a video transmission optimization scenario for a flight simulator, after the system receives a reduced-resolution frame, such as 640 pixels wide by 360 pixels high, which equals 230,400 pixels, it immediately initiates the fault-tolerance processing process: First, the data volume of a single frame is calculated. Each pixel contains three color values ​​of red, green, and blue, which occupy a total of 3 bytes of storage space. Therefore, the total data volume is 230,400 pixels multiplied by 3 bytes, which equals 691,200 bytes. Then, the picture features are analyzed and it is detected that 60% of the areas, such as the sky background, have a smooth color change, while the color of the instrument panel area changes significantly. The proportion of the smooth area, 0.6, is multiplied by the total data volume, that is, 691,200 times 0.6 to obtain 414,720, which is used as the picture complexity index. Based on the current network packet loss situation, for example, if 10 packets are lost for every 100 transmitted packets, dynamic parameter adjustments are made. The base quantization parameter is capped at 20 points. The image complexity of 414,720 is multiplied by the 10% packet loss rate, resulting in 41,472. This is then multiplied by a fine-tuning factor of 0.00005, resulting in an adjustment of approximately 2 points, raising the new upper limit to 22 points. Furthermore, the original maximum frame group length of 60 frames is divided by 1.1, which corresponds to the adjustment factor for 100% + 10% packet loss. This factor, 60 divided by 1.1, is approximately 54.5. Taking the integer part, the maximum frame group length of 54 frames is set. This 54-frame maximum reference frame length is then set. Finally, the system applies these three adjustments, adjusting the quantization parameter range to 5-22 points, the frame group length to a maximum of 54 frames, and the reference frame limit to 54 frames. These adjustments are then synchronized with the encoder, generating a video stream that is resilient to packet loss. This entire process ensures the clarity of key images, such as the instrument panel, while making the video stream adaptable to unstable network conditions.

[0048] In the overall solution of step 103 above, the total number of pixels of the reduced-resolution frame and the spatial complexity parameter are coordinated with the packet loss rate parameter to dynamically adjust the key encoding parameters to generate a fault-tolerant video stream with high resistance to packet loss. Specifically, a frame complexity weight is generated based on the frame data volume and spatial complexity, and the upper limit of the quantization step range is expanded by multiplying the weight by the packet loss rate. At the same time, the maximum threshold of the frame group length is reversely mapped according to the packet loss rate and used to constrain the upper limit of the number of reference frames in the inter-frame prediction mode. The regulated quantization step range, frame group length and reference frame number parameters are injected into the encoding process to form a three-fold composite protection mechanism, thereby significantly enhancing the anti-packet loss capability and decoding stability of the video stream in harsh network environments while maintaining basic image quality.

[0049] 104. Perform an entropy decoding operation on the error-tolerant coded stream at a receiving end, and perform motion compensation alignment on the decoded luminance component and the decoded chrominance component of the reduced-resolution frame recovered by decoding to obtain a decoded reconstructed frame; Optionally, step 104 may specifically include the following steps: 1041. Perform syntax element type judgment on the error-tolerant coding stream, determine the category of the element to be decoded and the corresponding context model index, and calculate the corresponding probability interval through an arithmetic decoding engine based on the current state value of the context model index; 1042. Compare the value of the current parsing position in the error-tolerant coded stream with the boundary value of the probability interval, parse out a binary character sequence, perform a debinarization operation on the binary character sequence, and restore it to the decoded luminance component and the decoded chrominance component of the reduced-resolution frame; 1043. Extract the motion vector of the decoded luminance component, calculate the spatial coordinate offset of the decoded chrominance component based on the motion vector, and adjust the spatial position of the decoded chrominance component using the spatial coordinate offset to obtain a decoded chrominance component after offset compensation. 1044. Perform motion trajectory matching in the time domain on the decoded chrominance component and the decoded luminance component after offset compensation, merge the matched decoded luminance component and the decoded chrominance component, and obtain a recombined decoded reconstructed frame.

[0050] In the above steps, the error-resilient coded stream refers to the generated packet loss-resistant compressed video data; the entropy decoding operation is the inverse process of encoding, which is used to recover the original information from the compressed data; the reduced-resolution frame refers to the processed frame with reduced resolution; the decoded luminance component and the decoded chrominance component are the luminance and color data respectively recovered after entropy decoding; the motion-compensated alignment refers to adjusting the chrominance spatial position to match the luminance based on the motion data; the decoded reconstructed frame is the final restored video frame; the syntax element type determination is to identify the different coding element categories in the data stream; the context model index identifies the probability model to which the current decoding element belongs; the current state value is the dynamic probability state of the model; the arithmetic decoding engine is an algorithm unit that uses probability intervals to recover data; the probability interval represents the numerical range of the probability of symbol occurrence; the boundary value comparison refers to the operation of comparing data values ​​with probability intervals; the binary character sequence is intermediate data composed of 0 / 1; the debinarization operation refers to restoring the binary sequence to its original value; the motion vector is vector data that describes the motion offset between frames; the spatial coordinate offset is the position correction value of the chrominance relative to the luminance; the offset compensation is the process of adjusting the chrominance position; and the temporal motion trajectory matching refers to the process of tracking the motion trajectory across frames.

[0051] In the embodiment of the present application, after receiving the error-tolerant coding stream in step 1041, the header data is parsed, the syntax element type is identified, the category of the current element to be decoded is determined, such as the motion vector element, and the predefined context model index number is queried according to the category to point to the model M1, and the current state value of the model is obtained. (like =0.6), the probability interval is calculated by the arithmetic decoding engine, where the interval calculation formula for the symbol "0" is , parameter description: Indicates that the default value of the cumulative probability is 0. Assuming 100 for the state range, the calculated interval is [0, 60) (0.6 × 100 = 60). For example, when decoding flight navigation system video, the motion vector element type is detected, the index model M1 state value is 0.6, and the symbol "0" is generated in the interval [0, 60), providing the basis for the next decoding step.

[0052] Next, the video component data is restored through step 1042. The specific process is to read the current value of the error-tolerant coding stream. V like V =45, compare it with the probability interval boundary of step 1041: If V belongs to [0, ) interval, outputs the symbol "0"; otherwise, outputs "1". This loop generates a binary sequence such as "0010". The original values ​​are then restored using a debinarization mapping table. This mapping follows a predefined rule, for example, converting the binary value "0010" to the value 12. Separating the data stream yields the decoded luminance component of the downscaled frame, such as a 512×288 matrix, and the decoded chrominance component, such as a 256×144 matrix. For example, when processing flight instrument panel video, an input value of 45 outputs the symbol "0" in the interval [0,60). The cumulative sequence "0010" is mapped to the value 12, ultimately restoring the luminance and chrominance component matrices.

[0053] Next, the chrominance components are aligned in step 1043. The specific process is to extract the motion vector from the decoded luminance component. , such as horizontal displacement Pixel, based on the chroma sampling format such as 4:2:0, the formula for calculating the spatial coordinate offset is ,in, Indicates that the subsampling ratio is 2, and the offset is calculated Pixel; perform bilinear interpolation compensation on the decoded chroma component, the interpolation formula is: ; in, Indicates the new chroma value after offset compensation, is the chromaticity value of the neighboring point, is the position weight, the actual operation is the chromaticity pixel coordinate from Adjust to For example, an aircraft altitude indicator chromatic pixel Offset to , take the neighboring points 、 、 、 Interpolate to calculate new chrominance values.

[0054] Finally, the reconstructed frame is generated through step 1044. The specific process is to perform time domain motion trajectory matching on the chrominance component and the luminance component after offset compensation, that is, to track the object trajectory by comparing the motion vectors of consecutive frames, such as the pointer position change. When the trajectory deviation When the pixel (threshold) is set, the chromaticity position is fine-tuned based on the brightness component; after matching, the components are merged: the brightness matrix is ​​directly used as the Y channel, and the interpolated chromaticity matrix is ​​used as the The channels are reassembled into YUV format and decoded to reconstruct the frame. For example, in the flight speedometer animation, the pointer brightness position of frame t is (100, 200). If the chromaticity trajectory deviation ΔD = 0.5 pixels does not exceed the threshold, it is directly merged and output as a 960×540 pixel reconstructed frame.

[0055] In practical applications, in a video receiver optimization scenario for an aviation simulator, the system immediately initiates the decoding and reconstruction process upon receiving a video stream resistant to network packet loss. The system first identifies the data stream type, for example, determining based on data features that the current segment belongs to image brightness information and calling model No. 3 from the pre-stored decoding rule library. The arithmetic decoding engine calculates the probability range, assuming the model indicates the current symbol probability range is 0.78 to 0.92. The system then reads the value 0.85 from the data stream. Since this value falls between 0.78 and 0.92, the system parses the binary sequence "1011" and converts it into a specific digital matrix, restoring the 640×360 resolution brightness component matrix and the sparsely sampled chrominance component matrix. Motion compensation then takes place: It extracts motion displacement information from the luminance component. For example, if a certain instrument panel area is detected to have shifted 32 pixels horizontally between two consecutive frames, the system automatically calculates that the chrominance component needs to be compensated by 32 times 0.5, which equals 16 pixels, since the chrominance resolution is half that of the luminance. The chrominance component coordinates are adjusted based on this offset, moving the chrominance data at the original coordinates (x, y) to the new location (x+16, y). Finally, timeline matching is performed, precisely combining the adjusted chrominance components, such as the red indicator light data on the cockpit panel, with the luminance components at the corresponding moment to generate a fully reconstructed image frame. This entire process fully restores the dynamic instrument panel image at the receiving end, eliminating artifacts such as smearing during transmission.

[0056] In the overall approach of step 104, the arithmetic decoding engine analyzes the probability interval boundaries based on the state value of the context model index. Numerical comparison and debinarization are then performed to restore the decoded luminance and chrominance components of the downscaled frame. The motion vector of the luminance component is then extracted, and the spatial coordinate offset of the chrominance component is calculated to implement component alignment compensation. Temporal motion trajectory matching is then used to achieve spatiotemporal synchronization between the chrominance and luminance components, ultimately generating a highly complete decoded reconstructed frame. This process significantly enhances the decoder's fault-tolerant and resilient capabilities in data loss scenarios, ensuring stable and smooth visual output even in fluctuating network environments.

[0057] 105. Based on a neural network reconstruction algorithm, multi-scale feature extraction is performed on the luminance component of the decoded reconstructed frame and local detail features are superimposed to perform detail compensation to generate an enhanced luminance feature map. Chroma resolution reconstruction is performed on the decoded chroma component to generate a chroma reconstruction component. The enhanced luminance feature map and the chroma reconstruction component are synchronously reconstructed in color space to output a video frame restored to the original resolution.

[0058] Optionally, step 105 may specifically include the following steps: 1051. Perform a multi-level convolution operation on the brightness component of the decoded and reconstructed frame based on a neural network reconstruction algorithm to extract brightness features of different scales; 1052. From the brightness features of different scales, select the brightness feature of the finest scale and superimpose and fuse it with the local detail feature, perform detail compensation on the brightness component of the decoded and reconstructed frame, and generate an enhanced brightness feature map; Among them, step 1052 may specifically include the following processes: selecting the brightness feature of the finest scale from the brightness features of different scales, and obtaining the spatial position coordinates corresponding to the brightness feature of the finest scale, and defining a pixel neighborhood of a fixed size with each of the spatial position coordinates as the center point; performing weighted calculation on the feature values ​​of all positions in the pixel neighborhood to generate a local detail feature containing all positions, wherein the weight coefficient of the spatial position coordinate is higher than that of the edge position; adding the brightness feature of the finest scale and the local detail feature point by point according to the same spatial position coordinates to obtain a superposition fusion result after detail compensation, performing a nonlinear activation operation on the superposition fusion result, and generating an enhanced brightness feature map.

[0059] 1053. Perform bilinear interpolation reconstruction on the chrominance component of the decoded reconstructed frame to generate a chrominance reconstructed component matching the original resolution; 1054. Synchronously superimpose the enhanced luminance feature map and the chrominance reconstruction component according to a preset color space conversion rule, restore the original resolution while ensuring color consistency, and output a restored video frame.

[0060] In the above steps, the neural network reconstruction algorithm refers to the image enhancement method based on deep learning; the decoded reconstructed frame refers to the restored low-resolution video frame; the luminance component and the chrominance component are the separate representations of the luminance and color data of the reconstructed frame; the multi-scale feature extraction is to capture the macroscopic to microscopic features of the image through convolution operations at different levels; the local detail feature describes the enhanced information of the image texture details; the detail compensation is the operation of adding details to the original luminance to enhance the clarity; the enhanced luminance feature map is the high-definition luminance data after compensation; the chrominance resolution reconstruction is the process of restoring the low-resolution chrominance data to the original resolution; the chrominance reconstruction component is the complete chrominance data after restoration; the color space synchronization Step reconstruction is the process of fusing brightness and chromaticity in a unified color space; the original resolution is the video size before resolution reduction; the restored video frame is the final output high-quality full-resolution frame; the multi-level convolution operation is the step of performing multiple feature extractions on the input data in the neural network; the finest scale brightness feature refers to the feature map with the highest resolution; the spatial position coordinates identify the coordinates of the feature point in the image; the pixel neighborhood refers to the square area centered on the coordinates; the weighted calculation is the weighted summation of the feature values ​​in the neighborhood; bilinear interpolation reconstruction is a method of amplifying the chromaticity component using linear interpolation of neighboring pixels; the preset color space conversion rule is the color model standard for merging brightness and chromaticity.

[0061] In this embodiment, multi-scale feature extraction is first performed on the luminance component of the decoded and reconstructed frame in step 1051. The specific process is as follows: the luminance component is input into a neural network for three-level convolution processing: the first layer uses a large 5×5 convolution kernel to extract macroscopic features, generating a feature map F1 of half the size; the second layer uses a medium 3×3 convolution kernel to process F1 to generate a feature map F2 of further half the size; and the third layer uses a small 1×1 convolution kernel to process F2 to generate a feature map F3 that retains detail. For example, the luminance component of a flight instrument panel (512×288 pixels) undergoes three-level processing: F1 of size 256×144 contains the overall instrument outline, F2 of size 128×72 contains the pointer outline, and F3 of size 128×72 contains the scale texture, thus fully implementing multi-scale feature extraction.

[0062] Next, in step 1052, the finest-scale feature map F3 is selected and the spatial coordinates (x, y) of its feature points are obtained. A 5×5 pixel neighborhood is defined with the coordinates as the center, and the local detail features are calculated using the weighted formula: ; Among them, the spatial coordinates Indicates the position of the currently processed feature point in the image; offset Is relative to the center coordinate The horizontal / vertical offset value of , ranging from -2 to 2, is used to traverse 25 positions in a 5×5 neighborhood; Relative position within the neighborhood The finest scale eigenvalue at ; weight coefficient is the weight assigned to the feature value at each offset position. The weight coefficient is w(0,0) = 0.4 at the center, decreasing linearly with distance to the edge, w(2,2) = 0.01. F3 is added to the local feature map point by point at the same coordinates, resulting in enhanced = F3 + localfeature. Finally, the enhanced brightness feature map is generated using the ReLU activation function f(x) = max(0,x). For example, at the pointer tip coordinate (120,85), the F3 value is 0.7, the neighborhood weighted localfeature is 0.25, and the superimposed value of 0.95 is output after ReLU, resulting in a significant improvement in sharpness.

[0063] Next, chroma resolution reconstruction is performed in step 1053. First, the chroma component of the decoded reconstructed frame is obtained. This component has a lower resolution than the original video size. For example, if the original resolution is 512×288, the chroma component is only 256×144. The system uses bilinear interpolation for reconstruction: for each pixel to be reconstructed in the chroma image, the positions of its four nearest neighboring pixels are found, and the horizontal and vertical distance ratios between the pixel and the four neighbors are calculated. The chroma values ​​of the four neighbors are then weighted averaged based on the distance ratios, with closer neighbors being given higher weights. For example, the closest pixel is given a weight of 40%, the next closest pixel is given a weight of 30%, the farther away pixel is given a weight of 20%, and the farther away pixel is given a weight of 10%. Finally, the reconstructed chroma value for the pixel is calculated using the weighted average. This process is repeated until all target pixels are reconstructed, restoring the chroma components to their original resolution. For example, in the chromaticity reconstruction of the flight map, the target point is located at the coordinates (200.5, 150.5), and the coordinates of the four adjacent points are (200, 150), (201, 150), (200, 151), and (201, 151). The new chromaticity value is calculated based on the position ratio weighting to eliminate the color mosaic effect and generate a full-size chromaticity reconstruction component.

[0064] Finally, step 1054 outputs the restored video frame. First, the enhanced luminance feature map generated in step 1052 is aligned with the chrominance reconstruction component generated in step 1053. A standard YUV-to-RGB color space conversion method is used: for each pixel position, the Y value of the luminance feature map and the Cb and Cr values ​​of the chrominance reconstruction component are calculated. The RGB primary color values ​​are then calculated according to aviation display standards: the red R value is calculated by converting the Y and Cr values, the green G value is calculated by converting the Y, Cb, and Cr values, and the blue B value is calculated by converting the Y and Cb values. The specific coefficients adhere to international standards to ensure color accuracy. During the synthesis process, a strict one-to-one correspondence between luminance and chrominance space positions is maintained, and the final output is an RGB video frame in its original resolution. For example, in the red area of ​​an aircraft altimeter, the luminance Y value is 180 and the chrominance Cr value is 120. Using international standard coefficients, the red R value is calculated to be 170, preserving the vibrant red and accurately recreating the color scheme of the aircraft instrument panel.

[0065] In a practical application, during the HD video reconstruction process for an aviation simulator, the system performs enhancement processing on the 640×360 pixel image recovered at the receiver. First, for the luminance component, a three-layer neural network extracts feature maps at different scales, with the smallest-scale feature map maintaining the original resolution. The system demarcates a 3×3 pixel region around each pixel, for example, the points at coordinates 100 and 200, as the analysis area. A weighted calculation is performed on all pixels within this region: the center point is assigned a higher weight, such as 70%, and the remaining 30% is evenly distributed among the eight surrounding points. The weighted calculation result is superimposed on the smallest-scale feature map and then subjected to nonlinear upscaling to generate a detail-enhanced luminance image. The chrominance component is also processed. The original sparsely sampled chrominance data is only 180×120 pixels. Reconstruction is performed using bilinear interpolation. When the color value at coordinates 150 and 300 needs to be filled in, the system searches for the data of the four nearest known sampling points and calculates the weighted average based on the ratio of their distance from the target location, directly obtaining a chrominance value of 129 at that point. Finally, simultaneous color space reconstruction is performed, combining the enhanced luminance image with the reconstructed chromaticity image according to standard conversion rules. For example, a luminance value of 210 at a certain location is combined with chromaticity values ​​of 129 and 125. The red, green, and blue color values ​​are calculated using a universal conversion formula, ultimately generating a restored image at the original 1920×1080 resolution. This process enhances the edge sharpness of the instrument panel digits and creates a natural and smooth transition between sky and background colors, fully restoring the visual quality prior to transmission.

[0066] In the overall solution of the above step 105, the decoded reconstructed frame is restored with refined image quality through a neural network reconstruction algorithm, multi-level convolution is implemented for the luminance component to extract multi-scale features, and the finest-scale luminance features are fused with the local detail features generated based on the weighting of the spatial position coordinates for superposition compensation, and an enhanced luminance feature map is generated through nonlinear activation; at the same time, bilinear interpolation is used to perform resolution reconstruction on the chrominance component, and finally the enhanced luminance feature map and the chrominance reconstruction component are synchronously reconstructed through a preset color space conversion rule, so as to achieve accurate restoration of the original resolution while strictly maintaining color consistency, and output a restored video frame with rich visual details and accurate colors.

[0067] The following is a complete embodiment of steps 101 to 105: like Figure 2As shown, during audio and video transmission in an aviation simulator training system, the system first monitors network status in real time: it collects transmission delay and bandwidth data every two seconds and categorizes delays into "Excellent" (0-20 milliseconds), "Good" (21-50 milliseconds), and "Needs Improvement" (51 milliseconds and above). The current delay of 40 milliseconds is considered "Excellent." Considering historical data showing bandwidth fluctuations concentrated between 5 and 15 milliseconds, the system automatically assigns a score of 7 and converts the network status to "Medium Fluctuation." Based on this rating, the system initiates image scaling: scaling the original 1920×1080 image to a 640×360 resolution by sampling every three pixels for the luminance component. The image is then divided into 5×5 blocks, sampling every four pixels in areas with smooth colors (such as the sky background) while maintaining the original sampling in areas with dense detail (such as the instrument panel), generating a reduced-resolution frame. The total number of pixels calculated is 640 times 360, which equals 230,400 pixels. This, combined with the image's 60% smooth area, yields a complexity value of 414,720.

[0068] Based on a 10% packet loss rate, the system dynamically adjusts encoding parameters: the quantization step size is adjusted from 20 plus 414,720 times 10% times 0.00005, which is approximately 2 minutes, to 22 minutes; the frame group length is increased from 60 frames to 60 divided by 1.1, which is approximately 54 frames; and the prediction reference frame limit is 54 frames. After these adjustments, the encoder generates a packet loss-resistant data stream. Upon receiving the data, the receiver first identifies the data type and uses a probabilistic model to convert the binary sequence "1011" into a luminance matrix. Based on the 32-pixel displacement of the instrument panel in the luminance matrix, the chrominance compensation required is calculated as 16 pixels, based on the ratio of halving the chrominance resolution, and the chrominance data coordinates are shifted to achieve alignment.

[0069] Finally, a three-layer neural network is used for processing: Multi-scale features of the luminance component are extracted. At the smallest scale layer, with each pixel as the center, such as the coordinates [100, 200], a weighted calculation is performed on the pixels in a 3×3 area (70% in the center and 30% in the periphery), and superimposed to generate a luminance map with sharpened instrument panel edges. Bilinear interpolation is performed on the 180×120 chroma sampling points to reconstruct the image to 640×360. The luminance value of 210 and the chroma values ​​of 129 / 125 are converted to red 206, green 212, and blue 212 according to the YUV-RGB conversion rules, reconstructing the original resolution image and fully restoring the details and natural colors of the cockpit instruments.

[0070] Figure 3 The present invention provides a structural diagram of an audio and video signal transmission system with an autonomous and controllable architecture and coordinated software and hardware, such as Figure 3 As shown, the system includes: An analysis module 31 is configured to synchronously collect bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile link during audio and video transmission, and generate a network status level based on correlation analysis between the bandwidth fluctuation parameters and the transmission delay parameters; a sampling module 32 configured to perform resolution scaling on the original luminance component of the video frame and selectively sample the original chrominance component according to the network status level, thereby generating a reduced-resolution frame that matches the current network status level; A generation module 33 is configured to synchronously control the quantization step size range, the frame group length, and the inter-frame prediction mode based on the total number of pixels and the spatial complexity parameter of the reduced-resolution frame in combination with the packet loss rate parameter to generate an error-tolerant coding stream integrating an anti-packet loss structure; A reconstruction module 34 is configured to perform an entropy decoding operation on the error-tolerant coded stream at a receiving end, and perform motion compensation alignment on the decoded luminance component and the decoded chrominance component of the reduced-resolution frame recovered by decoding to obtain a decoded reconstructed frame; The restoration module 35 is used to perform multi-scale feature extraction on the luminance component of the decoded reconstructed frame based on a neural network reconstruction algorithm and superimpose local detail features to perform detail compensation, generate an enhanced luminance feature map, perform chroma resolution reconstruction on the decoded chroma component, generate a chroma reconstruction component, and perform color space synchronous reconstruction on the enhanced luminance feature map and the chroma reconstruction component, and output a video frame restored to the original resolution.

[0071] Figure 3 The autonomous controllable architecture software and hardware coordinated audio and video signal transmission system can perform Figure 1 The implementation principles and technical effects of the autonomous and controllable architecture software and hardware coordinated audio and video signal transmission method described in the illustrated embodiment are not further elaborated. The specific manner in which each module and unit performs operations in the autonomous and controllable architecture software and hardware coordinated audio and video signal transmission system in the above embodiment has been described in detail in the relevant embodiments of the method and will not be elaborated on here.

[0072] In one possible design, Figure 3 The embodiment shown is an autonomous controllable architecture software and hardware coordinated audio and video signal transmission system that can be implemented as a computing device, such as Figure 4 As shown, the computing device may include a storage component 41 and a processing component 42; The storage component 41 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 42 .

[0073] The processing component 42 is used for the above Figure 1 The embodiment provides an audio and video signal transmission method with autonomous and controllable architecture and coordinated software and hardware.

[0074] The processing component 42 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0075] The storage component 41 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0076] Of course, a computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0077] The input / output interface provides an interface between the processing component and the peripheral interface module, which can be an output device, an input device, etc.

[0078] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0079] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0080] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The embodiment shown is a method for transmitting audio and video signals with an autonomous and controllable architecture and coordinated software and hardware.

[0081] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0082] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0083] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer or server) to execute the methods described in each embodiment or certain portions of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for transmitting audio and video signals with autonomous and controllable architecture and coordinated software and hardware, characterized in that: include: During audio and video transmission, the bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile link are synchronously collected, and a network status level is generated based on the correlation analysis of the bandwidth fluctuation parameters and the transmission delay parameters; According to the network status level, the frame adjustment device performs resolution scaling on the original luminance component of the video frame and selectively samples the original chrominance component to generate a reduced-resolution frame that matches the current network status level; Based on the total number of pixels and the spatial complexity parameter of the reduced-resolution frame, combined with the packet loss rate parameter, the quantization step range, the frame group length, and the inter-frame prediction mode are synchronously controlled to generate an error-tolerant coding stream with an integrated anti-packet loss structure; Performing an entropy decoding operation on the error-tolerant coded stream at a receiving end, and performing motion compensation alignment on the decoded luminance component and the decoded chrominance component of the reduced-resolution frame recovered by decoding to obtain a decoded reconstructed frame; Based on a neural network reconstruction algorithm, multi-scale feature extraction is performed on the luminance component of the decoded reconstructed frame and local detail features are superimposed for detail compensation to generate an enhanced luminance feature map. Chroma resolution reconstruction is performed on the decoded chroma component to generate a chroma reconstruction component. The enhanced luminance feature map and the chroma reconstruction component are synchronously reconstructed in color space to output a video frame restored to the original resolution.

2. The audio and video signal transmission method with autonomous and controllable architecture software and hardware collaboration according to claim 1 is characterized in that: The method includes: performing multi-scale feature extraction on the luminance component of the decoded reconstructed frame based on a neural network reconstruction algorithm and superimposing local detail features to perform detail compensation to generate an enhanced luminance feature map; performing chroma resolution reconstruction on the decoded chroma component to generate a chroma reconstruction component; performing color space synchronization reconstruction on the enhanced luminance feature map and the chroma reconstruction component, and outputting a video frame restored to the original resolution, including: Performing a multi-level convolution operation on the brightness component of the decoded and reconstructed frame based on a neural network reconstruction algorithm to extract brightness features of different scales; From the brightness features of different scales, the brightness features of the finest scale are selected and superimposed and fused with the local detail features, and detail compensation is performed on the brightness component of the decoded and reconstructed frame to generate an enhanced brightness feature map; Performing bilinear interpolation reconstruction on the chrominance component of the decoded reconstructed frame to generate a chrominance reconstructed component matching the original resolution; The enhanced luminance feature map and the chrominance reconstruction component are synchronously superimposed according to a preset color space conversion rule, the original resolution is restored while ensuring color consistency, and the restored video frame is output.

3. The audio and video signal transmission method with autonomous and controllable architecture software and hardware collaboration according to claim 2 is characterized in that: From the brightness features of different scales, the brightness features of the finest scale are selected and superimposed and fused with the local detail features, and detail compensation is performed on the brightness component of the decoded and reconstructed frame to generate an enhanced brightness feature map, including: Selecting a finest-scale brightness feature from the brightness features of different scales, obtaining spatial position coordinates corresponding to the finest-scale brightness feature, and defining a pixel neighborhood of a fixed size with each of the spatial position coordinates as a center point; Performing weighted calculation on the feature values ​​of all positions within the pixel neighborhood to generate local detail features including all positions, wherein the weight coefficient of the spatial position coordinates is higher than that of the edge position; The finest-scale brightness feature and the local detail feature are added point by point according to the same spatial position coordinates to obtain a superposition fusion result after detail compensation, and a nonlinear activation operation is performed on the superposition fusion result to generate an enhanced brightness feature map.

4. The audio and video signal transmission method with autonomous and controllable architecture software and hardware collaboration according to claim 1 is characterized in that: Based on the total number of pixels and spatial complexity parameters of the reduced-resolution frame and in combination with the packet loss rate parameter, the quantization step range, the frame group length, and the inter-frame prediction mode are synchronously controlled to generate an error-tolerant coding stream with an integrated anti-packet loss structure, including: Calculating the data volume of a unit frame according to the total number of pixels of the reduced-resolution frame, and combining the data volume of the unit frame with a spatial complexity parameter to generate a frame complexity weight; Increasing an upper limit of a quantization step range based on a product of the frame complexity weight and the packet loss rate parameter, reversely mapping a maximum value of a frame group length according to the packet loss rate parameter, and using the maximum value as an upper limit threshold for limiting the number of reference frames for an inter-frame prediction mode; During the encoding process, the adjusted quantization step range, frame group length and inter-frame prediction mode parameters are injected into the encoder to generate an error-tolerant encoding stream.

5. The audio and video signal transmission method with autonomous and controllable architecture software and hardware collaboration according to claim 1 is characterized in that: According to the network status level, the frame adjustment device performs resolution scaling on the original luminance component of the video frame and selectively samples the original chrominance component to generate a reduced-resolution frame that matches the current network status level, including: Querying a preset resolution scaling ratio table according to the network status level, determining a target scaling ratio of the brightness component, performing uniform sampling on the original brightness component according to the target scaling ratio, and generating a scaled brightness component; Perform non-uniform region division on the original chrominance component, select the area with spatial gradient below the threshold for sparse sampling based on the spatial gradient distribution of the original chrominance component, and obtain the sampled chrominance component; The scaled luminance component is combined with the sampled chrominance component to generate a reduced-resolution frame that matches the current network status level.

6. The audio and video signal transmission method with autonomous and controllable architecture software and hardware collaboration according to claim 1 is characterized in that: Performing an entropy decoding operation on the error-tolerant coded stream at a receiving end, and performing motion compensation alignment on the decoded luminance component and the decoded chrominance component of the reduced-resolution frame recovered by decoding to obtain a decoded reconstructed frame, including: Performing syntax element type judgment on the error-tolerant coding stream, determining the category of the element to be decoded and the corresponding context model index, and calculating the corresponding probability interval through an arithmetic decoding engine based on the current state value of the context model index; Comparing the value of the current parsing position in the error-tolerant coded stream with the boundary value of the probability interval, parsing out a binary character sequence, performing a debinarization operation on the binary character sequence, and restoring it to the decoded luminance component and the decoded chrominance component of the reduced-resolution frame; Extracting a motion vector of the decoded luminance component, calculating a spatial coordinate offset of the decoded chrominance component based on the motion vector, and adjusting a spatial position of the decoded chrominance component using the spatial coordinate offset to obtain an offset-compensated decoded chrominance component; The decoded chrominance component after offset compensation is matched with the decoded luminance component in the time domain, and the matched decoded luminance component and decoded chrominance component are merged to obtain a recombined decoded reconstructed frame.

7. The audio and video signal transmission method with autonomous and controllable architecture software and hardware collaboration according to claim 1 is characterized in that: During audio and video transmission, the bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile link are synchronously collected. Based on the correlation analysis of the bandwidth fluctuation parameters and transmission delay parameters, a network status level is generated, including: Periodically obtaining bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile link within a fixed time window, and calculating variations of the bandwidth fluctuation parameters within a continuous time window; Dividing the transmission delay parameter into multiple level intervals, and associating each level interval with a change in the bandwidth fluctuation parameter, and determining a bandwidth stability score corresponding to each level interval based on a distribution state of the change in the corresponding level interval; Based on a preset scoring threshold mapping table, the bandwidth stability score corresponding to each level interval is converted into a discrete network status level.

8. An audio and video signal transmission system with autonomous and controllable architecture and coordinated software and hardware, characterized by: include: An analysis module is used to synchronously collect bandwidth fluctuation parameters, packet loss rate parameters, and transmission delay parameters of the mobile link during audio and video transmission, and generate a network status level based on correlation analysis of the bandwidth fluctuation parameters and transmission delay parameters; a sampling module, configured to perform resolution scaling on the original luminance component of the video frame and selectively sample the original chrominance component according to the network status level through an image adjustment device, thereby generating a reduced-resolution frame that matches the current network status level; A generation module is configured to synchronously control the quantization step size range, the frame group length, and the inter-frame prediction mode based on the total number of pixels and the spatial complexity parameter of the reduced-resolution frame in combination with the packet loss rate parameter to generate an error-tolerant coding stream integrating an anti-packet loss structure; a reconstruction module, configured to perform an entropy decoding operation on the error-tolerant coded stream at a receiving end, and perform motion compensation alignment on the decoded luminance component and the decoded chrominance component of the reduced-resolution frame recovered by decoding to obtain a decoded reconstructed frame; A restoration module is used to perform multi-scale feature extraction on the luminance component of the decoded reconstructed frame based on a neural network reconstruction algorithm and superimpose local detail features to perform detail compensation, thereby generating an enhanced luminance feature map; perform chroma resolution reconstruction on the decoded chroma component to generate a chroma reconstruction component; and perform color space synchronous reconstruction on the enhanced luminance feature map and the chroma reconstruction component to output a video frame restored to the original resolution.

Images