Multi-task semantic compression method and device for traffic scene video stream, equipment and medium

Abstract

The invention relates to a multi-task semantic compression method and device for traffic scene video streaming, equipment and a medium, and the method comprises the steps: inputting the splicing features of a current frame and a reference frame into a second semantic compression model, extracting the motion information features between the current frame and the reference frame layer by layer in a motion information encoder through employing a multi-layer convolutional neural network, generating and decoding a compressed motion information representation to generate a prediction frame; extracting a residual feature between the predicted frame and the current frame in a residual encoder of the second semantic compression model, compressing the residual feature by adopting a multi-layer down-sampling convolution operation to determine a compressed residual feature, and fusing the compressed motion information representation and the compressed residual feature to obtain a motion information representation; generating a reconstructed image of the current frame; and calling a pre-trained target detection model for decoding to detect the traffic scene target in the reconstructed image. According to the method, the problem of resource waste is solved, the calculation efficiency is improved, and the calculation overhead is saved.

Description

Technical Field

[0001] This application relates to the field of multi-task semantic compression method for traffic scene video streams, as well as corresponding devices, electronic devices, and computer-readable storage media. Background Art

[0002] With the widespread adoption of applications such as video surveillance and autonomous driving, the amount of video data is increasing rapidly. Traditional video coding methods (such as H.264 / H.265) have limitations in terms of compression efficiency and intelligent analysis. These methods mainly focus on pixel-level compression and fail to fully utilize the semantic information in the video. At the same time, the surge in video data and the increasing demand for intelligent analysis are leading to a growing strain on computing resources.

[0003] Existing video compression technologies are generally divided into traditional compression algorithms such as H.264 / H.265, and deep video compression methods using neural networks. Multi-task frameworks typically use neural networks for video analysis. Existing video compression and multi-task frameworks have the following technical shortcomings:

[0004] (1) Although current deep learning video compression methods are gradually approaching traditional compression algorithms (such as H.264 and H.265) in terms of compression ratio, they have not yet shown a breakthrough advantage. These methods have failed to fully explore and utilize the semantic information of the video, and still focus on improving the compression ratio as the core goal. They lack intelligent processing of video content, and the process usually consumes a lot of computing power, but only completes the video compression task and directly discards the semantic information of the video. This not only leads to more computing power consumption compared with traditional algorithms, but also fails to achieve multi-task processing, resulting in a waste of computing resources.

[0005] (2) While existing multi-task frameworks have achieved significant results in the field of video analytics, their application processes have obvious limitations. Videos typically need to be compressed using H.264 or H.265 before transmission, and then decompressed to complete the analysis task. This sequential processing not only increases system complexity but also introduces additional computational and storage overhead, leading to a significant reduction in resource utilization efficiency. Furthermore, the decompressed video data often contains redundant information, while key semantic information directly related to the analysis task fails to be extracted efficiently, further exacerbating the waste of computing power. This model is particularly unsuitable for scenarios with limited computing resources and high real-time requirements, making it difficult to meet practical needs and necessitating a more efficient solution.

[0006] In summary, existing deep learning video compression methods typically require videos to be compressed using H.264 or H.265 before transmission, and then decompressed to complete the analysis task. This sequential processing not only increases system complexity but also introduces additional computational and storage overhead, leading to a significant reduction in resource utilization efficiency. The applicant has made corresponding explorations to address this problem. Summary of the Invention

[0007] The purpose of this application is to solve the above-mentioned problems by providing a multi-task semantic compression method, corresponding device, electronic device and computer-readable storage medium for traffic scene video streams.

[0008] To achieve the various objectives of this application, the following technical solution is adopted:

[0009] A multi-task semantic compression method for traffic scene video streams, proposed to meet one of the purposes of this application, includes:

[0010] A traffic scene video stream is acquired, and multiple traffic scene image frames in the traffic scene video stream are extracted and classified according to the frame skipping step size to determine the key frame of the traffic scene image frame and the reference frame of the current frame. The key frame represents the first traffic scene image in the traffic scene video stream, the current frame represents the traffic scene image frames including the second traffic scene image and subsequent traffic scene image frames, and the reference frame represents the traffic scene image frame preceding the current frame.

[0011] The keyframes are input into a pre-trained first semantic compression model. The spatial dimension of the keyframes is gradually compressed in the main encoder through downsampling operations to extract the deep high-dimensional features of the keyframes for semantic compression.

[0012] The concatenated features of the current frame and the reference frame are input into the pre-trained second semantic compression model. In the motion information encoder, a multi-layer convolutional neural network is used to extract the motion information features between the current frame and the reference frame layer by layer, generate the compressed motion information representation and decode it to generate the prediction frame of the current frame.

[0013] The residual features between the predicted frame and the current frame are extracted in the residual encoder of the second semantic compression model. The residual features are compressed by multi-layer downsampling convolution operation to determine the compressed residual features. The compressed motion information representation and the compressed residual features are fused to generate the reconstructed image of the current frame.

[0014] The pre-trained target detection model is invoked to decode the compressed motion information representation and the compressed residual features to detect traffic scene targets in the reconstructed image until the multi-task loss function reaches a convergence state, thereby completing the multi-task semantic compression of the traffic scene video stream.

[0015] Optionally, the first semantic compression model includes a master encoder, a master decoder, and a super-prior model, wherein the super-prior model includes a super-prior encoder and a super-prior decoder; the second semantic compression model includes a motion information encoder, a motion information decoder, a residual encoder, a residual decoder, and a feature fusion module.

[0016] Optionally, the step of inputting the keyframes into a pre-trained first semantic compression model, and progressively compressing the spatial dimension of the keyframes in the main encoder through downsampling operations to extract the deep, high-dimensional features of the keyframes for semantic compression includes:

[0017] The first frame of the traffic scene image is used as the key frame input, and the key frame is normalized.

[0018] The normalized keyframes are input into the main encoder for encoding. The main encoder adopts a multi-layer convolutional neural network structure, which contains four downsampling layers. Each downsampling layer uses a convolution operation with a stride of 2 and introduces non-linear characteristics through the ReLU activation function to obtain deep, high-dimensional features.

[0019] The deep, high-dimensional features are input into a quantizer for quantization, and the quantized features are then sent to a main decoder for decoding. The main decoder is symmetrical to the main encoder and uses deconvolution layers for upsampling. The image features are reconstructed through a step-by-step restoration process to obtain the restored traffic scene image.

[0020] Optionally, the step of inputting the concatenated features of the current frame and the reference frame into a pre-trained second semantic compression model, and using a multi-layer convolutional neural network in the motion information encoder to extract the motion information features between the current frame and the reference frame layer by layer, generating a compressed motion information representation and decoding it to generate a predicted frame of the current frame includes:

[0021] The concatenation features of the current frame and the reference frame are input into the motion information encoder, wherein the motion information encoder adopts a four-layer convolutional neural network structure, extracts the motion information features between the current frame and the reference frame layer by layer, and generates a compressed motion information representation.

[0022] The compressed motion information representation is input to the motion information decoder, which decodes the compressed motion information representation, restores the motion features through a deconvolutional network, and outputs motion prediction information to generate a prediction frame between the current frame and the reference frame.

[0023] Optionally, the residual features between the predicted frame and the current frame are extracted in the residual encoder of the second semantic compression model, and the residual features are compressed by multi-layer downsampling convolution operation to determine the compressed residual features. The compressed motion information representation and the compressed residual features are then fused to generate the reconstructed image of the current frame.

[0024] The predicted frame and the current frame are input into the residual encoder to obtain compressed residual features, wherein the residual encoder includes multiple downsampling convolution operations;

[0025] The compressed residual features are input into the residual decoder to decode the compressed residual features, and then fused with the motion information features. The residual details are then upsampled through a deconvolutional network to restore the residual details.

[0026] The decoded motion information representation and residual features are input into the feature fusion module, and the features are combined by channel stitching. The features are then integrated through a convolutional layer to generate the reconstructed image of the current frame.

[0027] Optionally, the step of determining the frame skipping step size includes:

[0028] Obtain the frame rate corresponding to the traffic scene video stream and the frame rate corresponding to the traffic camera;

[0029] The frame skipping step size is determined by the ratio between the frame rate corresponding to the traffic scene video stream and the frame rate corresponding to the traffic camera.

[0030] Optionally, the basic network architecture of the object detection model is the YOLO model; the traffic scene targets include one or more of the following: cars, trucks, vans, bicycles, pedestrians, and license plates; the multi-task loss function is constructed from the semantic compression loss function and the object detection loss function.

[0031] A multi-task semantic compression apparatus for traffic scene video streams, provided to suit another purpose of this application, includes:

[0032] The image frame extraction and classification module is configured to acquire a traffic scene video stream, extract and classify multiple traffic scene image frames in the traffic scene video stream according to the frame skipping step size, so as to determine the key frame and the reference frame of the traffic scene image frame. The key frame represents the first traffic scene image in the traffic scene video stream, the current frame represents the traffic scene image frame after the second traffic scene image, and the reference frame represents the traffic scene image frame before the current frame.

[0033] The keyframe semantic compression module is configured to input the keyframe into a pre-trained first semantic compression model, and gradually compress the spatial dimension of the keyframe through downsampling operations in the main encoder to extract the deep high-dimensional features of the keyframe for semantic compression.

[0034] The prediction frame generation module is configured to input the concatenated features of the current frame and the reference frame into a pre-trained second semantic compression model, and use a multi-layer convolutional neural network in the motion information encoder to extract the motion information features between the current frame and the reference frame layer by layer, generate a compressed motion information representation and decode it to generate the prediction frame of the current frame.

[0035] The reconstructed image generation module is configured to extract the residual features between the predicted frame and the current frame in the residual encoder of the second semantic compression model, compress the residual features using a multi-layer downsampling convolution operation to determine the compressed residual features, and fuse the compressed motion information representation and the compressed residual features to generate the reconstructed image of the current frame.

[0036] The multi-task semantic compression module is configured to call a pre-trained target detection model to decode the compressed motion information representation and the compressed residual features to detect traffic scene targets in the reconstructed image, until the multi-task loss function reaches a convergence state, thereby completing the multi-task semantic compression of the traffic scene video stream.

[0037] An electronic device provided for another purpose of this application includes a central processing unit and a memory, the central processing unit being configured to invoke a computer program stored in the memory to perform the steps of the multi-task semantic compression method for traffic scene video streams described in this application.

[0038] A computer-readable storage medium is provided for another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the multi-task semantic compression method for the traffic scene video stream, which, when invoked by a computer, executes the steps included in the corresponding method.

[0039] Compared to existing technologies, this application addresses the problems of conventional deep learning video compression methods, which typically require video to be compressed using H.264 or H.265 before transmission and then decompressed for analysis. This sequential processing not only increases system complexity but also introduces additional computational and storage overhead, leading to a significant reduction in resource utilization efficiency. This application addresses these issues by providing the following beneficial effects, including but not limited to:

[0040] Firstly, traditional video compression methods (such as H.264 / H.265) primarily improve compression ratios by reducing video quality. While this works well in bandwidth-constrained scenarios, it often neglects the deep semantic information of the video during compression, focusing only on spatial compression and thus wasting computational resources. In contrast, the multi-task semantic compression method based on convolutional neural networks (CNNs) proposed in this application can extract high-dimensional features of the video using deep learning techniques while maintaining video quality, enabling more efficient and intelligent compression. By combining semantic information with compression, this method not only improves the video compression ratio but also outperforms traditional compression algorithms such as H.264 to some extent, thus providing a more efficient solution for video stream transmission and storage.

[0041] Secondly, traditional multi-task frameworks often follow a sequential processing pattern of "compression-transmission-decompression-analysis." In this process, video needs to be compressed before transmission, then decompressed before analysis, which introduces additional computational and storage overhead. This application, however, extracts deep semantic information directly during the compression process, enabling deep integration of video compression and analysis tasks within the same framework, avoiding redundant decompression steps. This not only reduces the waste of computational resources but also improves the overall system's computational efficiency, making it particularly suitable for applications with limited computational resources or high real-time requirements.

[0042] Third, this application employs a shared semantic feature extraction mechanism, enabling the extracted video semantic information to simultaneously serve multiple downstream tasks (such as object detection and video reconstruction). This mechanism improves feature reuse efficiency, thereby reducing computational overhead. Traditional multi-task frameworks typically require compression before object detection or video analysis, which is not only time-consuming but also inefficient. The multi-task semantic compression method in this application directly provides necessary semantic information for subsequent tasks during the compression process, significantly improving the efficiency of multi-task processing.

[0043] Fourth, the system architecture of this application provides an integrated processing solution starting from the camera end. This architecture achieves seamless integration of data acquisition, compression, and transmission, avoiding the latency issues caused by multiple data conversions and processing in traditional systems. Especially in application scenarios requiring real-time video stream processing, such as traffic monitoring and autonomous driving, this method can effectively meet the requirements of high real-time performance and high resource efficiency, while reducing system complexity.

[0044] Furthermore, this application extracts semantics during the compression process, transforming video compression from a mere reduction of data volume into an optimized data foundation for subsequent tasks such as object detection and video analysis. By introducing a pre-trained object detection model, traffic scene targets can be efficiently detected in the decoded and reconstructed images, thus meeting the needs of efficient video analysis in practical applications.

[0045] In summary, this application not only solves the resource waste problem of existing video compression and multi-task frameworks, but also improves computational efficiency, reduces computational overhead, and enhances the real-time performance and flexibility of the system by deeply integrating compression and multi-task processing. It has broad application prospects, especially in scenarios that require real-time video processing and efficient resource utilization, such as intelligent transportation and video surveillance. Attached Figure Description

[0046] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 This is a flowchart illustrating the multi-task semantic compression method for traffic scene video streams in this embodiment of the application.

[0048] Figure 2 This is a schematic diagram of the multi-task framework in the embodiments of this application;

[0049] Figure 3 This is a flowchart illustrating the multi-task semantic compression of traffic scene video streams in this application embodiment;

[0050] Figure 4 This is a flowchart of I-frame inference in an embodiment of this application;

[0051] Figure 5 This is an exemplary architecture of the encoder in the I-frame inference model of this application embodiment;

[0052] Figure 6 This is an exemplary architecture of the decoder in the I-frame inference model of this application embodiment;

[0053] Figure 7 This is an exemplary architecture of the P-frame inference model in the embodiments of this application;

[0054] Figure 8 This is an exemplary architecture of the target detection network in the embodiments of this application;

[0055] Figure 9 This is an exemplary architecture of the SPP module in the embodiments of this application;

[0056] Figure 10 This is an exemplary architecture of module C3 in the embodiments of this application;

[0057] Figure 11 This is a schematic diagram of the multi-task semantic compression device for traffic scene video streams in the embodiments of this application;

[0058] Figure 12 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0059] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0060] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application’s specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0062] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0063] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0064] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0065] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0066] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0067] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0068] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0069] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0070] Please see Figure 1 In one embodiment of the multi-task semantic compression method for traffic scene video streams of this application, the method includes:

[0071] Step S10: Obtain the traffic scene video stream, extract multiple traffic scene image frames from the traffic scene video stream according to the frame skipping step size and classify them to determine the key frame of the traffic scene image frame and the reference frame of the current frame. The key frame represents the first traffic scene image in the traffic scene video stream, the current frame represents the traffic scene image frames including the second traffic scene image and the subsequent traffic scene image frames, and the reference frame represents the traffic scene image frame preceding the current frame.

[0072] The multi-task semantic compression system for traffic scene video streams in terminal devices can acquire traffic scene video streams, extract and classify multiple traffic scene image frames in the traffic scene video streams according to the frame skipping step size, and determine the key frame and the reference frame of the current frame of the traffic scene image frame. The key frame represents the first frame of traffic scene image in the traffic scene video stream, the current frame represents the second frame of traffic scene image and subsequent traffic scene image frames, and the reference frame represents the previous traffic scene image frame of the current frame.

[0073] Specifically, the multi-task model in this application aims to effectively reduce data transmission volume through semantic compression and decompression techniques, thereby improving data processing efficiency. In the target detection task, it provides accurate target recognition capabilities for various traffic scenarios through intelligent identification and extraction of key information. Furthermore, the privacy protection function ensures the security of user data, making it particularly suitable for scenarios requiring user privacy protection. Overall, this model comprehensively achieves efficient data processing, information security, and intelligent target recognition, improving the applicability and security of various tasks in terms of data transmission, storage, and processing. Figure 1 This application introduces a multi-task framework from the perspective of the implemented tasks. The framework primarily implements semantic compression of video into a bitstream, followed by different processing of the bitstream according to different needs. These processes are parallel and independent. The main processing includes video restoration, object detection, and video storage. Video restoration involves using a neural network to restore the semantically compressed bitstream back to the original video while maintaining high resolution. Object detection involves detecting traffic scene objects in the original video, primarily pedestrians, bicycles, and various types of vehicles. Video storage involves storing the semantically compressed bitstream according to the required storage specifications.

[0074] like Figure 2 As shown, Figure 1 In addition to the tasks mentioned above, there are two Level 2 tasks: data entry and privacy protection. Data entry refers to storing various data detected by the data targets, such as the number of pedestrians and traffic flow, into the database. Privacy protection refers to blurring the face and vehicle areas detected in the object detection in the restored video, in order to protect public data.

[0075] In some embodiments, the multi-task framework implementation includes semantic compression and video restoration. In this application, deep learning methods are used for video compression. During compression, this application continuously extracts feature information from the video to obtain deep-level features, which are then converted into a bitstream using an entropy coding algorithm, thus constituting semantic compression. Meanwhile, in this application, video restoration is a subset of semantic compression; therefore, the two parts are described together.

[0076] First, let's introduce the semantic compression part, whose overall reasoning is as follows: Figure 3 As shown, in Figure 3 The document indicates that in semantic compression, this application employs two distinct inference methods: I-frame (keyframe) inference and P-frame (predictive frame) inference. I-frame inference refers to the first frame of the input video stream or video file. Since there is no previous frame as a reference, the I-frame model (image compression model) is used to infer the deep features of the image before entropy coding is applied to obtain the corresponding bitstream. P-frame inference, starting from the second frame, uses the previous frame as a reference frame and calculates the relationship between the reference frame and the current frame to obtain deep features, which are typically referred to as video features. Subsequently, the data is compressed into a bitstream using entropy coding (asymmetric digital system).

[0077] In some embodiments, the step of determining the frame skipping step size includes:

[0078] Step S101: Obtain the frame rate corresponding to the traffic scene video stream and the frame rate corresponding to the traffic camera;

[0079] Step S102: Determine the frame skipping step size based on the ratio between the frame rate corresponding to the traffic scene video stream and the frame rate corresponding to the traffic camera.

[0080] Step S20: Input the keyframe into the pre-trained first semantic compression model, and gradually compress the spatial dimension of the keyframe through downsampling operation in the main encoder to extract the deep high-dimensional features of the keyframe for semantic compression of the keyframe.

[0081] A traffic scene video stream is acquired, and multiple traffic scene image frames are extracted and classified according to the frame skipping step size. After determining the key frames and reference frames of the current frame, the key frames are input into a pre-trained first semantic compression model. The spatial dimension of the key frames is progressively compressed in the main encoder through downsampling operations to extract the deep high-dimensional features of the key frames for semantic compression. The first semantic compression model includes a main encoder, a main decoder, and a super-prior model, wherein the super-prior model includes a super-prior encoder and a super-prior decoder.

[0082] like Figure 4 As shown, from Figure 4 As can be seen, the I-frame inference model (the first semantic compression model) includes a main encoder and a main decoder. The entire process includes:

[0083] Given the first frame of the traffic scene image from the traffic scene video stream as input, the traffic scene image is then normalized, as shown in formula (1), where img i,j This represents the pixel value at row number i and column number j in the original image. The value of pixel j in row i of the normalized image;

[0084]

[0085] The normalized data is then fed into the main encoder for encoding. The specific network structure is as follows: Figure 5 As shown, a deep, high-dimensional feature is obtained, and the feature is sent to the quantizer for quantization, as shown in formula (2), where floor(x) represents flooring down.

[0086]

[0087] The quantized deep features need to be further processed according to the actual traffic scenario, which will be described in detail later; then the features are sent to the main decoder for decoding, as follows: Figure 6 As shown, the restored traffic scene image is then obtained.

[0088] In a specific embodiment, the keyframe is input into a pre-trained first semantic compression model, and the spatial dimension of the keyframe is progressively compressed in the main encoder through downsampling operations to extract the deep, high-dimensional features of the keyframe for semantic compression. This step includes:

[0089] Step S201: Take the first frame of the traffic scene image as the key frame input and perform a normalization operation on the key frame;

[0090] Step S202: Input the normalized keyframes into the main encoder for encoding. The main encoder adopts a multi-layer convolutional neural network structure, which includes 4 downsampling layers. Each downsampling layer uses a convolution operation with a stride of 2 and introduces non-linear characteristics through the ReLU activation function to obtain deep high-dimensional features.

[0091] Step S203: Input the deep, high-dimensional features into the quantizer for quantization, and send the quantized features into the main decoder for decoding. The main decoder is symmetrical to the main encoder and uses a deconvolution layer for upsampling. The image features are reconstructed by gradually restoring them to obtain the restored traffic scene image.

[0092] Step S30: Input the splicing features of the current frame and the reference frame into the pre-trained second semantic compression model. In the motion information encoder, a multi-layer convolutional neural network is used to extract the motion information features between the current frame and the reference frame layer by layer, generate the compressed motion information representation and decode it to generate the prediction frame of the current frame.

[0093] The keyframes are input into a pre-trained first semantic compression model. In the main encoder, the spatial dimension of the keyframes is progressively compressed through downsampling operations to extract deep, high-dimensional features of the keyframes. After semantic compression of the keyframes, the concatenated features of the current frame and the reference frame are input into a pre-trained second semantic compression model. In the motion information encoder, a multi-layer convolutional neural network is used to extract motion information features between the current frame and the reference frame layer by layer, generating a compressed motion information representation and decoding it to generate a predicted frame for the current frame. The second semantic compression model includes a motion information encoder, a motion information decoder, a residual encoder, a residual decoder, and a feature fusion module.

[0094] P-frame inference shares some similarities with I-frame inference, the main difference being the inclusion of a reference frame. The network extracts motion information features (motion) between the reference frame and the current frame, as well as residual features (res) between the predicted frame inferred by the neural network and the current frame. Typically, the previous frame of the video is used as the reference frame for the current frame. The specific model structure and inference logic are as follows: Figure 7 As shown. Among them, in Figure 7 Xmotion and Xres are deep, high-dimensional features, and like I-frame inference, they also need to be quantized by a quantizer.

[0095] In a specific embodiment, the steps of inputting the concatenated features of the current frame and the reference frame into a pre-trained second semantic compression model, and using a multi-layer convolutional neural network in the motion information encoder to extract the motion information features between the current frame and the reference frame layer by layer, generating a compressed motion information representation and decoding it to generate a predicted frame of the current frame, include:

[0096] Step S301: Input the splicing features of the current frame and the reference frame into the motion information encoder. The motion information encoder adopts a four-layer convolutional neural network structure to extract the motion information features between the current frame and the reference frame layer by layer to generate a compressed motion information representation.

[0097] Step S302: Input the compressed motion information representation into the motion information decoder, decode the compressed motion information representation, restore the motion features through a deconvolution network, and output motion prediction information to generate a prediction frame between the current frame and the reference frame.

[0098] Step S40: Extract the residual features between the predicted frame and the current frame in the residual encoder of the second semantic compression model, compress the residual features using a multi-layer downsampling convolution operation to determine the compressed residual features, and fuse the compressed motion information representation and the compressed residual features to generate the reconstructed image of the current frame.

[0099] The concatenated features of the current frame and the reference frame are input into a pre-trained second semantic compression model. In the motion information encoder, a multi-layer convolutional neural network is used to extract the motion information features between the current frame and the reference frame layer by layer, generate a compressed motion information representation, and decode it to generate a prediction frame for the current frame. Then, the residual features between the prediction frame and the current frame are extracted in the residual encoder of the second semantic compression model. The residual features are compressed using a multi-layer downsampling convolution operation to determine the compressed residual features. Finally, the compressed motion information representation and the compressed residual features are fused to generate the reconstructed image of the current frame.

[0100] In a further embodiment, the residual features between the predicted frame and the current frame are extracted in the residual encoder of the second semantic compression model, and the residual features are compressed by multi-layer downsampling convolution operation to determine the compressed residual features. The compressed motion information representation and the compressed residual features are then fused to generate the reconstructed image of the current frame.

[0101] Step S401: Input the predicted frame and the current frame into the residual encoder to obtain the compressed residual features, wherein the residual encoder includes multi-layer downsampling convolution operations;

[0102] Step S402: Input the compressed residual features into the residual decoder to decode the compressed residual features, and fuse them with the motion information features. Then, perform upsampling through a deconvolution network to restore the residual detail information.

[0103] Step S403: Input the decoded motion information representation and residual features into the feature fusion module, use channel stitching to achieve feature combination, and integrate the features through a convolutional layer to generate the reconstructed image of the current frame.

[0104] Step S50: Call the pre-trained target detection model to decode the compressed motion information representation and the compressed residual features to detect traffic scene targets in the reconstructed image until the multi-task loss function reaches convergence, thereby completing the multi-task semantic compression of the traffic scene video stream.

[0105] The residual features between the predicted frame and the current frame are extracted in the residual encoder of the second semantic compression model. Multi-layer downsampling convolution operations are used to compress the residual features to determine the compressed residual features. The compressed motion information representation and the compressed residual features are then fused to generate the reconstructed image of the current frame. A pre-trained object detection model is then called to decode the compressed motion information representation and the compressed residual features to detect traffic scene targets in the reconstructed image. This process continues until the multi-task loss function converges, thus completing the multi-task semantic compression of the traffic scene video stream. The basic network architecture of the object detection model is the YOLO model. The traffic scene targets include one or more of the following: cars, trucks, vans, bicycles, pedestrians, and license plates. The multi-task loss function is constructed from the semantic compression loss function and the object detection loss function.

[0106] In this application, video restoration is a subset of semantic compression operations. Specifically, video restoration refers to the decoder portion of both the first semantic compression model (I-frame inference model) and the second semantic compression model (P-frame inference model). Figure 4 as well as Figure 7 The portion within the dashed box. This application uses different decoders to achieve different tasks. The keyframe image reconstruction in the above embodiment uses one decoder, while the object detection model implemented in this embodiment uses another decoder. The specific structure of the object detection model is as follows... Figure 8As shown, it includes: Backbone, Neck, and Head. Specifically, the Backbone is responsible for extracting the basic features of the input image. Through multi-layer convolution and pooling operations, it gradually abstracts feature representations at different scales. The Backbone includes convolutional layers (CONV), a C3 module, and an SPPF module. The convolutional layer (CONV) can be CONV(256,256,3,2), which represents CONV(number of input channels, number of output channels, kernel size, stride). It extracts local features through convolution operations, and the stride of 2 can reduce the feature map size (downsampling). The C3 module can be C3(256,256,j=3), which is a commonly used efficient feature extraction module in the YOLO series. It is designed based on the residual concept, fusing multi-branch features to maintain feature expressive power while reducing computational cost. j represents the number of residual blocks or branch parameters. The SPPF module can be SPPF(512), which performs multi-scale max pooling on the input feature map and then concatenates them to quickly increase the receptive field and enhance the feature extraction capability for targets at different scales. It is more efficient than the traditional SPP. The input image passes through CONV→C3→CONV→C3→SPPF in sequence, gradually extracting and enhancing features, and outputting multi-scale basic features to Neck.

[0107] Furthermore, in the Neck network, the multi-scale features output by the Backbone are fused and enhanced. Through operations such as upsampling, concatenation, and convolution, a richer feature hierarchy is constructed, allowing the model to better adapt to the detection of targets of different sizes. Upsampling enlarges the low-resolution feature map and fuses it with the high-resolution feature map, compensating for details lost during downsampling and restoring spatial information. The concatenation operation concatenates feature maps of different scales and channels along the channel dimension, fusing multi-scale features and allowing the model to utilize both fine-grained (high-resolution) and semantic (low-resolution) information simultaneously. The C3 module further refines the fused features, enhancing feature representation; for example, C3(12,256) optimizes the concatenated features. The Backbone output features are processed through Upsampling → Concatenation → C3 → Convolution →

[0108] Operations such as UpSample→Concat→... are performed to fuse multiple cross-scale features and output multiple sets of fused features to the Head (head network).

[0109] Furthermore, in the Head (head network), based on the fused features output from the Neck, the final target detection prediction is performed, and the prediction results (such as target category, position offset, etc.) are directly output through convolutional layers. Convolutional layers (Conv2d): such as Conv2d(128,nax(nc=5,1,1)), complete the mapping from features to prediction results through 1×1 convolution, where nc represents the number of categories; multiple sets of features output from the Neck are passed through different convolutional layers in the Head to predict detection results at different scales (fitting large, medium, and small targets), and finally output information such as the category and location of the target in the traffic scene.

[0110] In some embodiments, such as Figure 9 As shown, the SPP module independently demonstrates the internal logic of SPP. It pools the input features using MaxPool with different kernel sizes (3×3, 9×9, 15×15), then concatenates them and integrates them using convolution, which quickly expands the receptive field and allows the model to "see" more global information.

[0111] In some embodiments, such as Figure 10 As shown, the C3 module contains convolution (CONV) and residual branches (BottleneckCSP, with shortcut connections). Through multi-branch feature fusion and residual learning, it ensures feature extraction capabilities while maintaining lightweight design, thereby improving the model's adaptability to complex scenarios.

[0112] In some embodiments, this application deploys semantic compression models at edge devices, including highway monitoring, home monitoring, and enterprise monitoring. Specifically, cameras are installed at the edge, and the video data collected by the cameras is compressed into deep features using the semantic compression methods mentioned in previous chapters. These features are then further compressed into a bitstream using a range asymmetric digital system (ARDS). This bitstream is then transmitted to a data center via a network. The data center receives the bitstream, stores it in a storage device, and then uses an ADS to reconstruct the deep features. Depending on the operator's needs, different decoders can be called to perform tasks such as real-time display and object detection. To facilitate data display, the system incorporates privacy protection features in real-time display. Specifically, object detection is performed simultaneously during real-time display to obtain the coordinates of faces and license plates in the video, and then the affected areas are blurred.

[0113] In some embodiments, this application provides a multi-task semantic compression method based on convolutional neural networks, the method comprising the following steps:

[0114] Step S100: Data collection. Collect video data of resolutions such as 360P, 720P, and 1080P in the monitored scene.

[0115] Step S200: Data segmentation. Extract video segments at equal time intervals. For example, extract at 1-minute intervals. In each video, extract three consecutive traffic scene image frames every minute. Then, store the traffic scene image frames in a certain file organization format on the hard disk. Note that frame skipping is required here according to different traffic scenes. For example, in this case, the monitoring camera at the intersection has a frame rate of 20fps, while the video collected in step 1 has a frame rate of 60fps. Therefore, it is necessary to extract the video every 3 frames instead of extracting three frames consecutively.

[0116] The frame skipping step size can be determined based on formula (3), where T step Indicates the frame skipping step, FPS orin This indicates the frame rate (FPS) corresponding to the traffic scene video stream. target This indicates the frame rate corresponding to the traffic camera.

[0117]

[0118] Step S300: Data Labeling. The image data obtained in Step S200 is labeled to obtain training labels for the object detection network. The required label categories for the traffic scene are defined, including but not limited to cars, trucks, vans, bicycles, pedestrians, license plates, and faces. The ImgLabel tool is used to label all traffic scene image frames obtained in Step S200 with the defined categories. All traffic scene image frames and their corresponding labels are traversed, and the paths of the labeled traffic scene image frames are recorded. These recorded paths are then shuffled and divided into training and validation sets in a 7:3 ratio.

[0119] In some embodiments, a multi-task model is constructed to obtain deep features (semantic information) of the video by performing different feature extraction methods on different frames of the video, to obtain high-quality restored video by decoding the semantic information, and to obtain target detection information by decoding the semantic information. The semantic compression model is constructed according to the video compression requirements, based on key frames (I-frames) and prediction frames (P-frames). Since the I-frame is the first frame of the traffic scene image and has no reference, image compression is applied to directly extract and compress features to obtain the traffic scene image frame features.

[0120] The P-frame utilizes its relationship with the previous frame to extract motion information features and residual features between the current frame and the reference frame. A fusion network is then used to reconstruct the traffic scene image frame, resulting in a high-quality reconstructed traffic scene image frame. The target detection model decodes the motion information features and residual features obtained from the P-frame to identify traffic scene targets, including but not limited to cars, trucks, vans, bicycles, pedestrians, license plates, and faces.

[0121] In a further embodiment, model training utilizes a weighted average loss function to train the multi-task model for both compression and detection tasks. Initialization involves placing the semantic compression network parameters and the object detection network parameters into different optimizers, setting the batch size to B, defining the loss weights λ and β, and defining the Gaussian interference variance σ.

[0122] In some embodiments, a multi-task loss function is defined as shown in Equation (4) to balance the gradient descent speeds of the semantic compression task and the object detection task, where loss detect The loss value represents the loss value for the object detection task. compress This represents the loss value for the semantic compression task.

[0123] loss = loss detect +βloss compress (4)

[0124] In a further embodiment, a loss function for semantic compression is defined as shown in Equation (5), which is used to balance the relationship between image quality and compression ratio in semantic compression, where loss... bpp This represents the compression ratio loss. psnr This indicates a loss of accuracy.

[0125] loss compress =loss bpp +λloss psnr (5)

[0126] In a further embodiment, the Gaussian noise formula is defined as shown in formula (6), where random(-4σ, 4σ) represents taking a random number between -4σ and 4σ. The obtained noise is then added to the output features motion and residual in the P-frame network using formulas (6) and (7), where motion... i,j This represents the momentum information extracted from the P-frame network, where P(motion) i,j ) represents motion i,j Corresponding to random Gaussian noise, Indicates motion after adding random noise. i,j ,residual i,j This represents the momentum information extracted from the P-frame network, P(residual) i,j ) indicates residual i,j Corresponding to random Gaussian noise, Residual after adding random noise i,j Then, the noise-added features are fed into subsequent networks for further training, thereby improving the network's ability to adapt to quantization loss.

[0127]

[0128] In a further embodiment, a traffic scene video stream is recorded according to the target frame rate and resolution, and then labeled. Then, the trained model is used to perform semantic compression video restoration and target detection. The features obtained by semantic compression are encoded into a bit stream by rANS and stored on the hard disk. The compression ratio is obtained by formula (9), where bpp represents the number of bits used per pixel. The quality ratio of the original video and the restored video is calculated by formula (10). The accuracy of target detection is obtained by calculating the mAP of the video target detection, where PSNR represents peak signal-to-noise ratio and MSE represents mean square error, which is used to measure the difference between the predicted value and the actual observed value.

[0129]

[0130] PSNR = 20·log 10 (255)-10·log 10 (MSE) (10)

[0131] As can be seen from the above embodiments, compared with the prior art, the deep learning video compression methods in the present application typically require the video to be compressed using H.264 or H.265 before transmission, and then decompressed to complete the analysis task. This serial processing method not only increases the complexity of the system, but also introduces additional computational and storage overhead, leading to a significant reduction in resource utilization efficiency. The present application has the following beneficial effects, including but not limited to:

[0132] Firstly, traditional video compression methods (such as H.264 / H.265) primarily improve compression ratios by reducing video quality. While this works well in bandwidth-constrained scenarios, it often neglects the deep semantic information of the video during compression, focusing only on spatial compression and thus wasting computational resources. In contrast, the multi-task semantic compression method based on convolutional neural networks (CNNs) proposed in this application can extract high-dimensional features of the video using deep learning techniques while maintaining video quality, enabling more efficient and intelligent compression. By combining semantic information with compression, this method not only improves the video compression ratio but also outperforms traditional compression algorithms such as H.264 to some extent, thus providing a more efficient solution for video stream transmission and storage.

[0133] Secondly, traditional multi-task frameworks often follow a sequential processing pattern of "compression-transmission-decompression-analysis." In this process, video needs to be compressed before transmission, then decompressed before analysis, which introduces additional computational and storage overhead. This application, however, extracts deep semantic information directly during the compression process, enabling deep integration of video compression and analysis tasks within the same framework, avoiding redundant decompression steps. This not only reduces the waste of computational resources but also improves the overall system's computational efficiency, making it particularly suitable for applications with limited computational resources or high real-time requirements.

[0134] Third, this application employs a shared semantic feature extraction mechanism, enabling the extracted video semantic information to simultaneously serve multiple downstream tasks (such as object detection and video reconstruction). This mechanism improves feature reuse efficiency, thereby reducing computational overhead. Traditional multi-task frameworks typically require compression before object detection or video analysis, which is not only time-consuming but also inefficient. The multi-task semantic compression method in this application directly provides necessary semantic information for subsequent tasks during the compression process, significantly improving the efficiency of multi-task processing.

[0135] Fourth, the system architecture of this application provides an integrated processing solution starting from the camera end. This architecture achieves seamless integration of data acquisition, compression, and transmission, avoiding the latency issues caused by multiple data conversions and processing in traditional systems. Especially in application scenarios requiring real-time video stream processing, such as traffic monitoring and autonomous driving, this method can effectively meet the requirements of high real-time performance and high resource efficiency, while reducing system complexity.

[0136] Furthermore, this application extracts semantics during the compression process, transforming video compression from a mere reduction of data volume into an optimized data foundation for subsequent tasks such as object detection and video analysis. By introducing a pre-trained object detection model, traffic scene targets can be efficiently detected in the decoded and reconstructed images, thus meeting the needs of efficient video analysis in practical applications.

[0137] In summary, this application not only solves the resource waste problem of existing video compression and multi-task frameworks, but also improves computational efficiency, reduces computational overhead, and enhances the real-time performance and flexibility of the system by deeply integrating compression and multi-task processing. It has broad application prospects, especially in scenarios that require real-time video processing and efficient resource utilization, such as intelligent transportation and video surveillance.

[0138] Please see Figure 11A multi-task semantic compression device for traffic scene video streams, provided to meet one of the purposes of this application, includes an image frame extraction and classification module 1100, a key frame semantic compression module 1200, a prediction frame generation module 1300, a reconstructed image generation module 1400, and a multi-task semantic compression module 1500. The image frame extraction and classification module 1100 is configured to acquire a traffic scene video stream, extract and classify multiple traffic scene image frames from the video stream according to the frame skipping step size, and determine the keyframes and reference frames of the traffic scene image frames. The keyframe represents the first traffic scene image in the video stream, the current frame represents the traffic scene image frames following the second traffic scene image, and the reference frame represents the preceding traffic scene image frame. The keyframe semantic compression module 1200 is configured to input the keyframes into a pre-trained first semantic compression model, and progressively compress the spatial dimension of the keyframes through downsampling operations in the main encoder to extract deep, high-dimensional features for semantic compression. The predicted frame generation module 1300 is configured to input the concatenated features of the current frame and the reference frame into a pre-trained second semantic compression model, and generate a predicted frame based on motion information. The encoder employs a multi-layer convolutional neural network to extract motion information features between the current frame and the reference frame layer by layer, generating a compressed motion information representation and decoding it to generate a predicted frame for the current frame. The reconstructed image generation module 1400 is configured to extract residual features between the predicted frame and the current frame in the residual encoder of the second semantic compression model, compress the residual features using multi-layer downsampling convolution operations to determine the compressed residual features, and fuse the compressed motion information representation and the compressed residual features to generate a reconstructed image of the current frame. The multi-task semantic compression module 1500 is configured to call a pre-trained target detection model to decode the compressed motion information representation and the compressed residual features to detect traffic scene targets in the reconstructed image until the multi-task loss function reaches convergence, thereby completing the multi-task semantic compression of the traffic scene video stream.

[0139] Based on any embodiment of this application, please refer to Figure 12 Another embodiment of this application also provides an electronic device, which can be implemented by a computer device, such as... Figure 12The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store control information sequences. When executed by the processor, the computer-readable instructions enable the processor to implement a multi-task semantic compression method for traffic scene video streams. The processor provides computational and control capabilities, supporting the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to execute the multi-task semantic compression method for traffic scene video streams of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0140] In this embodiment, the processor is used to execute... Figure 11 The memory stores the specific functions of each module, and stores the program code and various data required to execute the above modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules in the multi-task semantic compression device for traffic scene video streams of this application, and the server can call the server's program code and data to execute the functions of all modules.

[0141] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the multi-task semantic compression method for traffic scene video streams described in any embodiment of this application.

[0142] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the multi-task semantic compression method for traffic scene video streams described in any embodiment of this application.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0144] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A multi-task semantic compression method for traffic scene video streams, characterized in that, include: A traffic scene video stream is acquired, and multiple traffic scene image frames in the traffic scene video stream are extracted and classified according to the frame skipping step size to determine the key frame of the traffic scene image frame and the reference frame of the current frame. The key frame represents the first traffic scene image in the traffic scene video stream, the current frame represents the traffic scene image frames including the second traffic scene image and subsequent traffic scene image frames, and the reference frame represents the traffic scene image frame preceding the current frame. The keyframes are input into a pre-trained first semantic compression model. The spatial dimension of the keyframes is gradually compressed in the main encoder through downsampling operations to extract the deep high-dimensional features of the keyframes for semantic compression. The concatenated features of the current frame and the reference frame are input into the pre-trained second semantic compression model. In the motion information encoder, a multi-layer convolutional neural network is used to extract the motion information features between the current frame and the reference frame layer by layer, generate the compressed motion information representation and decode it to generate the prediction frame of the current frame. The residual features between the predicted frame and the current frame are extracted in the residual encoder of the second semantic compression model. The residual features are compressed by multi-layer downsampling convolution operation to determine the compressed residual features. The compressed motion information representation and the compressed residual features are fused to generate the reconstructed image of the current frame. The pre-trained target detection model is invoked to decode the compressed motion information representation and the compressed residual features to detect traffic scene targets in the reconstructed image until the multi-task loss function reaches a convergence state, thereby completing the multi-task semantic compression of the traffic scene video stream.

2. The multi-task semantic compression method for traffic scene video streams according to claim 1, characterized in that, The first semantic compression model includes a master encoder, a master decoder, and a super-prior model, wherein the super-prior model includes a super-prior encoder and a super-prior decoder; The second semantic compression model includes a motion information encoder, a motion information decoder, a residual encoder, a residual decoder, and a feature fusion module.

3. The multi-task semantic compression method for traffic scene video streams according to claim 2, characterized in that, The steps of inputting the keyframes into a pre-trained first semantic compression model, and progressively compressing the spatial dimension of the keyframes in the main encoder through downsampling operations to extract the deep, high-dimensional features of the keyframes for semantic compression include: The first frame of the traffic scene image is used as the key frame input, and the key frame is normalized. The normalized keyframes are input into the main encoder for encoding. The main encoder adopts a multi-layer convolutional neural network structure, which contains four downsampling layers. Each downsampling layer uses a convolution operation with a stride of 2 and introduces non-linear characteristics through the ReLU activation function to obtain deep, high-dimensional features. The deep, high-dimensional features are input into a quantizer for quantization, and the quantized features are then sent to a main decoder for decoding. The main decoder is symmetrical to the main encoder and uses deconvolution layers for upsampling. The image features are reconstructed through a step-by-step restoration process to obtain the restored traffic scene image.

4. The multi-task semantic compression method for traffic scene video streams according to claim 2, characterized in that, The steps of inputting the concatenated features of the current frame and the reference frame into a pre-trained second semantic compression model, extracting motion information features between the current frame and the reference frame layer by layer in the motion information encoder, generating a compressed motion information representation and decoding it to generate a predicted frame for the current frame include: The concatenation features of the current frame and the reference frame are input into the motion information encoder, wherein the motion information encoder adopts a four-layer convolutional neural network structure, extracts the motion information features between the current frame and the reference frame layer by layer, and generates a compressed motion information representation. The compressed motion information representation is input to the motion information decoder, which decodes the compressed motion information representation, restores the motion features through a deconvolutional network, and outputs motion prediction information to generate a prediction frame between the current frame and the reference frame.

5. The multi-task semantic compression method for traffic scene video streams according to claim 4, characterized in that, The residual features between the predicted frame and the current frame are extracted in the residual encoder of the second semantic compression model. The residual features are compressed by multi-layer downsampling convolution operation to determine the compressed residual features. The compressed motion information representation and the compressed residual features are fused to generate the reconstructed image of the current frame. The predicted frame and the current frame are input into the residual encoder to obtain compressed residual features, wherein the residual encoder includes multiple downsampling convolution operations; The compressed residual features are input into the residual decoder to decode the compressed residual features, and then fused with the motion information features. The residual details are then upsampled through a deconvolutional network to restore the residual details. The decoded motion information representation and residual features are input into the feature fusion module, and the features are combined by channel stitching. The features are then integrated through a convolutional layer to generate the reconstructed image of the current frame.

6. The multi-task semantic compression method for traffic scene video streams according to claim 1, characterized in that, The steps to determine the frame skipping step size include: Obtain the frame rate corresponding to the traffic scene video stream and the frame rate corresponding to the traffic camera; The frame skipping step size is determined by the ratio between the frame rate corresponding to the traffic scene video stream and the frame rate corresponding to the traffic camera.

7. The multi-task semantic compression method for traffic scene video streams according to any one of claims 1 to 6, characterized in that, The basic network architecture of the target detection model is the YOLO model; the traffic scene targets include one or more of the following: cars, trucks, vans, bicycles, pedestrians, and license plates; the multi-task loss function is constructed from the semantic compression loss function and the target detection loss function.

8. A multi-task semantic compression device for traffic scene video streams, characterized in that, include: The image frame extraction and classification module is configured to acquire a traffic scene video stream, extract and classify multiple traffic scene image frames in the traffic scene video stream according to the frame skipping step size, so as to determine the key frame and the reference frame of the traffic scene image frame. The key frame represents the first traffic scene image in the traffic scene video stream, the current frame represents the traffic scene image frame after the second traffic scene image, and the reference frame represents the traffic scene image frame before the current frame. The keyframe semantic compression module is configured to input the keyframe into a pre-trained first semantic compression model, and gradually compress the spatial dimension of the keyframe through downsampling operations in the main encoder to extract the deep high-dimensional features of the keyframe for semantic compression. The prediction frame generation module is configured to input the concatenated features of the current frame and the reference frame into a pre-trained second semantic compression model, and use a multi-layer convolutional neural network in the motion information encoder to extract the motion information features between the current frame and the reference frame layer by layer, generate a compressed motion information representation and decode it to generate the prediction frame of the current frame. The reconstructed image generation module is configured to extract the residual features between the predicted frame and the current frame in the residual encoder of the second semantic compression model, compress the residual features using a multi-layer downsampling convolution operation to determine the compressed residual features, and fuse the compressed motion information representation and the compressed residual features to generate the reconstructed image of the current frame. The multi-task semantic compression module is configured to call a pre-trained target detection model to decode the compressed motion information representation and the compressed residual features to detect traffic scene targets in the reconstructed image, until the multi-task loss function reaches a convergence state, thereby completing the multi-task semantic compression of the traffic scene video stream.

9. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images