Coding method, compression network training method and related equipment

Abstract

The invention discloses a coding method, a compression network training method and related equipment, and belongs to the field of image processing, and the coding method comprises the steps: carrying out the saliency feature extraction of a to-be-coded feature map of a to-be-coded image, and obtaining a target saliency map, the target saliency map is used for representing the importance of each region of the to-be-coded image; performing coding processing on the to-be-coded feature map based on the target saliency map to obtain a coded code stream; wherein the coding processing comprises the following steps: determining a code rate weight of the to-be-coded feature map according to the target saliency map; and performing compression coding on the to-be-coded feature map according to the code rate weight.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an encoding method, a compression network training method and related equipment. BACKGROUND

[0002] In the field of computer vision, object detection and image segmentation are important tasks, which have wide applications in the fields of autonomous driving, video surveillance, medical image analysis, etc. At present, with the development of deep learning technology, methods based on deep neural networks have achieved great success in object detection and image segmentation tasks. However, with the growth of image and video data scale, traditional deep neural networks face challenges in model size and computational complexity.

[0003] In computer vision tasks, feature pyramid networks have greatly improved the performance of multi-scale problems in object detection. Different scales of features are extracted from the original image, and as the size decreases, the location information contained is reduced, and the semantic information becomes rich. However, for compression networks, the current compression is usually carried out according to a fixed compression code rate. However, when the compression code rate is set to be small, the image will be distorted, and when the compression code rate is set to be large, the compression effect will be poor. Therefore, in the prior art, there is a problem of how to balance the distortion of the image and the compression code rate. SUMMARY

[0004] The embodiments of the present application provide an encoding method, a compression network training method and related equipment, which can solve the problem of how to balance the distortion of the image and the compression code rate.

[0005] In a first aspect, an encoding method is provided, comprising:

[0006] performing saliency feature extraction on a to-be-encoded feature map of a to-be-encoded image to obtain a target saliency map, the target saliency map being used to represent the importance of each region of the to-be-encoded image;

[0007] performing encoding processing on the to-be-encoded feature map based on the target saliency map to obtain an encoding code stream;

[0008] wherein the encoding processing comprises: determining a code rate weight of the to-be-encoded feature map according to the target saliency map; and performing compression encoding on the to-be-encoded feature map according to the code rate weight.

[0009] In a second aspect, a compression network training method is provided, comprising:

[0010] The sample image is used for iterative training of the encoding network model, and a target compression network is obtained. The target compression network is used for encoding processing of a to-be-encoded feature map of a to-be-encoded image based on a target saliency map, and the target saliency map is used for representing importance of each region of the to-be-encoded image.

[0011] The encoding network model comprises a feature pyramid network and a compression network. The feature pyramid network is used for feature extraction of the sample image in different scales to obtain a second feature map. The compression network is used for compression encoding of the second feature map.

[0012] In one iteration process, a loss value between an image output by the encoding network model and the sample image is calculated based on a target loss function. The target compression network is the compression network obtained through training when the loss value reaches convergence. The target loss function is a minimum mean square error loss function determined based on a target weight. The target weight is used for adjusting code rate weights allocated by the compression network to saliency regions and non-saliency regions.

[0013] In a third aspect, an encoding apparatus is provided, comprising:

[0014] A first feature extraction module is configured to perform saliency feature extraction on a to-be-encoded feature map of a to-be-encoded image to obtain a target saliency map. The target saliency map is used for representing importance of each region of the to-be-encoded image.

[0015] An encoding module is configured to perform encoding processing on the to-be-encoded feature map based on the target saliency map to obtain an encoded bitstream.

[0016] The encoding processing comprises determining a code rate weight of the to-be-encoded feature map according to the target saliency map and performing compression encoding on the to-be-encoded feature map according to the code rate weight.

[0017] In a fourth aspect, a compression network training apparatus is provided, comprising:

[0018] A training module is configured to perform iterative training of an encoding network model by using a sample image to obtain a target compression network. The target compression network is used for encoding processing of a to-be-encoded feature map of a to-be-encoded image based on a target saliency map. The target saliency map is used for representing importance of each region of the to-be-encoded image.

[0019] The encoding network model comprises a feature pyramid network and a compression network. The feature pyramid network is used for feature extraction of the sample image in different scales to obtain a second feature map. The compression network is used for compression encoding of the second feature map.

[0020] In one iteration process, a loss value between an image output by the encoding network model and the sample image is calculated based on a target loss function, and the target compression network is trained to obtain the compression network under the condition that the loss value reaches convergence.

[0021] In a fifth aspect, an electronic device is provided, which includes a processor and a memory, the memory storing programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method according to the first aspect.

[0022] In a sixth aspect, an electronic device is provided, which includes a processor and a communication interface, wherein,

[0023] The processor is configured to perform salient feature extraction on a to-be-encoded feature map of a to-be-encoded image to obtain a target saliency map, the target saliency map being used to represent the importance of each region of the to-be-encoded image; and perform encoding processing on the to-be-encoded feature map based on the target saliency map to obtain an encoded bitstream; wherein the encoding processing includes: determining a code rate weight of the to-be-encoded feature map according to the target saliency map; and performing compression encoding on the to-be-encoded feature map according to the code rate weight.

[0024] Alternatively, the processor is configured to perform iterative training on an encoding network model by using a sample image to obtain a target compression network, the target compression network being used to perform encoding processing on a to-be-encoded feature map of a to-be-encoded image based on a target saliency map, the target saliency map being used to represent the importance of each region of the to-be-encoded image.

[0025] The encoding network model includes a feature pyramid network and a compression network, the feature pyramid network being used to perform feature extraction of different scales on the sample image to obtain a second feature map, and the compression network being used to perform compression encoding on the second feature map.

[0026] In one iteration process, a loss value between an image output by the encoding network model and the sample image is calculated based on a target loss function, and the target compression network is trained to obtain the compression network under the condition that the loss value reaches convergence.

[0027] In a seventh aspect, a readable storage medium is provided, and the readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the method according to the first aspect or implement the steps of the method according to the second aspect.

[0028] In an eighth aspect, a chip is provided, and the chip includes a processor and a communication interface, the communication interface is coupled with the processor, and the processor is configured to execute a program or instructions to implement the method according to the first aspect or implement the method according to the second aspect.

[0029] In a ninth aspect, a computer program / program product is provided, and the computer program / program product includes computer instructions, which, when executed by at least one processor, implement the method according to the first aspect or implement the method according to the second aspect.

[0030] In the embodiments of the present application, a target saliency map is obtained by performing saliency feature extraction on a to-be-encoded feature map of a to-be-encoded image, and the target saliency map is used to represent the importance of each region of the to-be-encoded image; the to-be-encoded feature map is encoded based on the target saliency map to obtain an encoded code stream; wherein the encoding process includes: determining a code rate weight of the to-be-encoded feature map according to the target saliency map; and performing compression encoding on the to-be-encoded feature map according to the code rate weight. In this way, the compression code rate can be allocated in the saliency region, and thus the distortion rate of the saliency region can be reduced under the condition that the overall compression code rate is unchanged. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is a structural schematic diagram of a coding and decoding system to which the embodiments of the present application can be applied;

[0032] Figure 2 is a flowchart of an encoding method provided by the embodiments of the present application;

[0033] Figure 3 is a schematic diagram of a saliency feature compression framework for an image to which the encoding method provided by the embodiments of the present application can be applied;

[0034] Figure 4 is a schematic diagram of a saliency temporal feature fusion framework for a video to which the encoding method provided by the embodiments of the present application can be applied;

[0035] Figure 5 is a flowchart of a compression network training method provided by the embodiments of the present application;

[0036] Figure 6 is a structural schematic diagram of an encoding device provided by the embodiments of the present application;

[0037] Figure 7 is a structural schematic diagram of a compression network training apparatus provided by an embodiment of the present application;

[0038] Figure 8 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0039] Figure 9 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0040] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a particular order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second" are generally of a kind, not limited to the number of objects, for example, the first object can be one or more. In addition, "or" in the present application means at least one of the connected objects. For example, "A or B" covers three scenarios, namely, scenario one: including A and not including B; scenario two: including B and not including A; scenario three: including A and B. The character " / " generally represents that the objects before and after are in an "or" relationship.

[0041] The term "indicate" in the present application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). Among them, the direct indication can be understood as that the sender explicitly informs the receiver of specific information, operations to be performed or requested results, etc. in the sent indication; the indirect indication can be understood as that the receiver determines the corresponding information according to the indication sent by the sender, or judges and determines the operation to be performed or the requested result according to the judgment result.

[0042] Figure 1 is a schematic diagram of a coding system 10 provided by an embodiment of the present application. The technical solutions of the embodiments of the present application relate to coding (CODEC) of video data (including encoding or decoding). Among them, the video data includes original uncoded video, coded video, decoded (for example, reconstructed) video or syntax elements, etc.

[0043] As Figure 1As shown, the codec system 10 includes a source device 100, which provides encoded video data to be decoded and displayed by a destination device 110. Specifically, the source device 100 provides the video data to the destination device 110 via a communication medium 120. The source device 100 and the destination device 110 may include any one or more of a desktop computer, a notebook (i.e., laptop) computer, a tablet computer, a set-top box, a mobile phone, a wearable device (e.g., a smart watch or a wearable camera), a television, a camera, a display device, an in-vehicle device, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, a digital media player, a video game console, a video conferencing device, a video streaming device, a broadcast receiver device, a broadcast transmitter device, a spacecraft, an aircraft, a robot, a satellite, and the like.

[0044] exist Figure 1 In the example of FIG, the source device 100 includes a data source 101, a memory 102, an encoder 200, and an output interface 104. The destination device 110 includes an input interface 111, a decoder 300, a memory 113, and a display device 114. The source device 100 represents an example of a video encoding device, and the destination device 110 represents an example of a video decoding device. In other examples, the source device 100 and the destination device 110 may not include Figure 1 Some components in, or can also include Figure 1 For example, source device 100 may receive video data from an external data source (such as an external camera). Similarly, destination device 110 may interface with an external display device rather than including an integrated display device. For another example, memory 102 and memory 113 may be external memories.

[0045] Although Figure 1 Source device 100 and destination device 110 are shown as separate devices, but in some examples, they may be integrated into one device. In such embodiments, the functions corresponding to source device 100 and the functions corresponding to destination device 110 may be implemented using the same hardware or software, or using separate hardware or software, or any combination thereof.

[0046] In some examples, source device 100 and destination device 110 can perform one-way video transmission or two-way video transmission. If it is two-way video transmission, source device 100 and destination device 110 can operate in a substantially symmetrical manner, that is, each of source device 100 and destination device 110 includes an encoder and a decoder.

[0047] Data source 101 represents a source of video data (i.e., raw, uncoded video data) and provides encoder 200 with successive pictures containing video data that encoder 200 encodes. Data source 101 of source device 100 can include a video capture device, such as a video camera, a video archive containing previously captured raw video, or a video feed interface to receive video from a video content provider. As another alternative, data source 101 can generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In these cases, encoder 200 encodes the captured, pre-captured, or computer-generated video data. Encoder 200 can rearrange the pictures from the received order (sometimes referred to as "display order") into the encoding order. Encoder 200 can generate a bitstream including encoded video data. Source device 100 can then output the encoded video data via output interface 104 onto a

[0048] Memory 102 of source device 100 and memory 113 of destination device 110 represent general purpose memories. In some examples, memory 102 can store raw video data from data source 101, and memory 113 can store decoded video data from decoder 300. Additionally or alternatively, memory 102, 113 can store software instructions executable by, e.g., encoder 200 and decoder 300, respectively. Although memory 102 and memory 113 are shown separately from, and as distinct from, encoder 200 and decoder 300 in this example, it will be understood that encoder 200 and decoder 300 can also each include an internal memory for a functionally similar or equivalent purpose. If encoder 200 and decoder 300 are deployed on the same hardware device, memory 102 and memory 113 can be one one memory. Furthermore, memory 102, 113 can store encoded video data that is output from encoder 200 and input to decoder 300, for example. In some examples, portions of memory 102, 113 can be allocated to one or more video buffers, e.g., to store raw, decoded, or encoded video data.

[0049] In some examples, source device 100 can output encoded data from output interface 104 to storage 113. Similarly, destination device 110 can access encoded data from storage 113 via input interface 111. Storage 113 or storage 102 can include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, Digital Versatile Discs (DVDs), Compact Disc Read-Only Memory (CD-ROMs), flash drive, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0050] Output interface 104 can include any type of medium or device capable of sending encoded video data from source device 100 to destination device 110. For example, output interface 104 can include a transmitter or a transceiver, e.g., an antenna, configured to transmit encoded video data from source device 100 directly to destination device 110 in real-time. The encoded video data can be modulated according to a communication standard of a wireless communication protocol and transmitted to destination device 110.

[0051] Communication medium 120 can include transient media, such as wireless broadcasts or wired networks transmitted over physical lines. For example, communication medium 120 can include radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cable). Communication medium 120 can form a portion of a packet-based network, such as a local area network, a wide area network, or a global network, such as the Internet. Communication medium 120 can also be in a form of storage media, such as a hard drive, flash drive, compact disc, digital video disc, Blu-ray disc, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0052] In some embodiments, communication medium 120 can include routers, switches, base stations, or any other equipment that can be used to facilitate communication from source device 100 to destination device 110. For example, a server (not shown) can receive the encoded video from source device 100 and provide the encoded video data to destination device 110, e.g., via network transmission. The server can include a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery Over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Services (MBMS) or evolved MBMS (eMBMS) server, or a Network-attached storage (NAS) device, etc. The server can implement one or more HTTP streaming protocols, such as the MPEG Media Transport (MMT) protocol, Dynamic Adaptive Streaming over HTTP (DASH) protocol, HTTP Live Streaming (HLS) protocol, or Real Time Streaming Protocol (RTSP), etc.

[0053] Destination device 110 can access the encoded video data from a server, e.g., through a wireless channel (e.g., Wi-Fi connection) or wired connection (e.g., Digital subscriber line (DSL), cable modem, etc.) for accessing encoded video data stored on the server.

[0054] Output interface 104 and input interface 111 can represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to the IEEE 802.11 standard or the IEEE 802.15 standard (e.g., ZigBee™), the Bluetooth standard, etc., or other physical components. In examples where output interface 104 and input interface 111 comprise wireless components, output interface 104 and input interface 111 can be configured to transfer data, such as encoded video data, according to a WIFI, Ethernet, cellular (such as 4G, LTE (Long-Term Evolution), LTE-A (Long-Term Evolution Advanced), 5G, 6G, etc.), standards.

[0055] The techniques provided by this disclosure can be employed in coding of video for use in a variety of multimedia applications, such as one or more of the following: video conferencing, digital TV, wireless TV distribution, streaming video, encoding video on digital versatile discs (DVDs), decoding video stored on DVDs, or other applications.

[0056] Input interface 111 of destination device 110 receives an encoded video bitstream from communication medium 120. The encoded video bitstream can include syntax elements and coded data units (e.g., slices, pictures, groups of pictures, frames, blocks, etc.) that, when decoded, produce decoded video data. Display device 114 displays the decoded video data to a user. Display device 114 can comprise a cathode ray tube (CRT), a liquid-crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0057] Encoder 200 and decoder 300 can be implemented as one or more of various processing circuitry, which can include one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuitry, hardware, or any combinations thereof. When the techniques are implemented partially in software, a device can store instructions for the software in a suitable, non- transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques provided by this disclosure.

[0058] The encoder 200 and the decoder 300 can be based on the following video coding standards: H.263, H.264, H.265 (also known as High Efficiency Video Coding (HEVC)), H.266 (also known as Versatile Video Coding (VVC)), Moving Picture Experts Group 2 (MPEG-2), MPEG-4, VP8, VP9, Alliance for Open Media Video 1 (AV1), Audio Video coding Standard 1 (AVS1), AVS2, AVS3, or a next generation video standard protocol, without limitation.

[0059] Generally, the encoder 200 and the decoder 300 can perform block-based coding of pictures. The term “block” generally refers to a structure comprising data to be processed (e.g., encoded, decoded, or otherwise used in the encoding or decoding process). For example, a block can comprise a two-dimensional matrix of samples of luma or chroma data. For example, the encoder 200 and the decoder 300 can code video data represented in YUV format.

[0060] For the convenience of understanding, the following describes some contents related to the embodiments of the present application:

[0061] I. Conventional image coding schemes.

[0062] Traditional coding schemes mostly follow three steps: transform, quantization and entropy coding. JPEG, as the most widely used compression standard, uses a series of transform, quantization and entropy coding to reduce spatial and other coding redundancies as much as possible. In order to achieve the reduction of spatial redundancy, the image is first divided into small blocks, and the image is transformed from time domain to frequency domain by Discrete Cosine Transform (DCT) to achieve a more compact representation, and then the transformed image information is quantized and input to the entropy coding process. Among them, quantization is the only lossy process in the whole compression method. Due to the universality of wavelet transform for non-stationary processes, in order to reduce the information loss in the quantization process, JPEG2000 uses Discrete Walsh Transform (DWT) instead of DCT to achieve better compression quality. In the High Efficiency Video Coding (HEVC) standard, the intra coding unit can independently compress a single frame, and the BPG codec is proposed for image compression based on this similarity. At present, the compression standard is designed for more extensive image compression tasks, so it has no advantage in front of some special highly correlated image sets.

[0063] II. Image coding based on deep learning

[0064] With the development of deep learning, neural network based methods have been widely applied in image processing, one of which is image compression. In the deep learning based scheme, the compression task is regarded as an encoding process, and the end-to-end learning method is used for training. The specific process can be decomposed into an encoding process, and the corresponding reconstruction task can be regarded as a decoding process. The encoder-decoder structure is widely used in the learning based compression method. Many scholars have conducted research in this field. For example, the related technology proposes a compression model based on recursive neural network (RNN), which uses an auto-encoder to extract the features of the image as a transformation process. The residual structure is used for feature extraction in the encoder and decoder. The related technology also proposes a new optimization method using compression rate and recovery distortion. The smooth adjustable compression ratio makes it very popular in subsequent methods. In order to obtain better performance, a compression method based on hyper-prior is proposed to reduce spatial redundancy on this basis. The entropy coding module is improved in the related technology and is attached to the hyper-prior structure to further reduce the coding redundancy. The related technology uses the advantages of residual structure and attention mechanism to propose a better structure of auto-encoder, and proposes a Gaussian mixture likelihood entropy model to improve its flexibility and accuracy. The above methods can certainly change the compression ratio through parameter adjustment, however, when the compression is at a low bit rate, because these methods do not pay special attention to extreme cases, the compression quality usually drops very quickly.

[0065] The dynamic image expert group (MPEG) established a machine-oriented video coding (VCM) group to establish a preliminary anchor point for compressing internal features from the Mask R-CNN architecture for multi-task detection and segmentation. In Mask R-CNN, the feature pyramid network (FPN) has greatly improved the performance for multi-scale problems in object detection, extracts different scale features from the original image, and as the size decreases, the location information contained is reduced, and the semantic information becomes rich.

[0066] In computer vision tasks, feature pyramid networks have greatly improved performance on the multi-scale problem in object detection. Different scales of features are extracted from the original image, and as the size decreases, the location information contained is reduced, and the semantic information becomes rich. However, for compression networks, there is still a lot of redundancy in the image in space. Reducing the code rate of these non-important regions will not affect the accuracy of machine vision tasks. Important regions are generally considered to be salient regions, which refer to regions with saliency in an image and usually contain the main information of objects. Using saliency information can effectively guide the model to focus attention on important regions, thereby improving the efficiency and performance of the model. For this purpose, the encoding method of the present application is proposed.

[0067] The encoding method provided by the embodiments of the present application will be described in detail below in conjunction with the drawings and some embodiments and their application scenarios.

[0068] Referring to Figure 2 The encoding method provided by the embodiments of the present application is shown in Figure 2 The encoding method comprises the following steps.

[0069] Step 201, performing saliency feature extraction on a to-be-encoded feature map of a to-be-encoded image to obtain a target saliency map, wherein the target saliency map is used to represent the importance of each region of the to-be-encoded image.

[0070] Step 202, performing encoding processing on the to-be-encoded feature map based on the target saliency map to obtain an encoded bitstream.

[0071] The encoding processing comprises: determining a code rate weight of the to-be-encoded feature map according to the target saliency map; and performing compression encoding on the to-be-encoded feature map according to the code rate weight.

[0072] In the embodiments of the present application, the encoding method can be applied to an encoding end, and the encoding end is provided with an encoding network model, which comprises a feature pyramid network and a target compression network. The feature pyramid network can perform feature extraction on a to-be-encoded image to obtain a to-be-encoded feature map, and output the to-be-encoded feature map to the target compression network. Then, the target compression network can perform saliency feature extraction based on the to-be-encoded feature map to obtain a target saliency map. Finally, the target compression network performs encoding processing on the to-be-encoded feature map based on the target saliency map to obtain an encoded bitstream.

[0073] Optionally, the region where the object subject is located in an image often has the characteristics of more complex details and more prominent brightness contrast. The region selected by the detection frame of the commonly used data set for object detection and image segmentation tasks, i.e. the region where the object subject is located, can be regarded as a salient region.

[0074] Optionally, in the training stage, the training set (i.e. sample images) can be used to test the bounding box information of the target detection result to generate a saliency map for different regions of the image. When calculating the loss function, the proportion of the saliency region is increased, so that the compression network focuses on compressing the saliency region where the main body of the object is located, while reducing other non-important regions, without reducing the accuracy of tasks such as image segmentation and target detection. For example, the rectangular box information of the training sample image is sent into the network, and the area within the rectangular box contributes more loss when calculating the distortion of the reconstructed image in the loss function stage. In this way, the compression network learns to allocate more compression code rate to the saliency region where the main body of the object is located, thereby improving the task accuracy of the target task network under the condition of constant compression code rate.

[0075] After completing the iterative training, in the target compression network, the code rate weight of the to-be-encoded feature map is determined according to the target saliency map, so that the compression code rate can be allocated to the saliency region, thereby reducing the distortion rate of the saliency region under the condition of constant overall compression code rate.

[0076] Optionally, the above feature pyramid network can first extract features that are helpful for task processing from the image through a convolutional neural network. Through a series of convolution, pooling and other operations, the input image can be converted into high-dimensional feature representation layer by layer. These feature representations have rich semantic information and can be used for image classification, target detection, image segmentation and other tasks. Compared with transmitting the image itself, transmitting the features consumes less bandwidth. Then, additional feature pyramid layers are added to the convolutional neural network to realize multi-scale feature representation. The feature pyramid network can obtain rich semantic information at different scales through bottom-up feature extraction and top-down feature fusion, and can effectively process targets of different sizes. The processing flow of a general feature pyramid network is as follows: the original image is sent into the feature pyramid network to extract features, then multi-scale feature extraction is performed from bottom to top, and feature fusion is performed from top to bottom to improve the performance of image tasks such as target detection.

[0077] Optionally, in the computer vision framework, after the edge device collects images or videos, multi-scale features with smaller code rate occupation are generally selected for transmission, and the image tasks with large amount of calculation are performed on the remote server.

[0078] Optionally, the main encoding and reconstruction object of the target compression network is multi-scale feature, i.e. the above to-be-encoded feature map. The target compression network learns to predict the saliency region during the training stage, and can generate a saliency map corresponding to different scale features frame by frame during actual testing, and guide the compression code rate allocation in the compression process.

[0079] In the embodiments of the present application, a salient feature is extracted from a to-be-encoded feature map of a to-be-encoded image to obtain a target saliency map, which is used to represent the importance of each region of the to-be-encoded image; the to-be-encoded feature map is encoded based on the target saliency map to obtain an encoded code stream; wherein the encoding process includes: determining the code rate weight of the to-be-encoded feature map according to the target saliency map; and compressively encoding the to-be-encoded feature map according to the code rate weight. In this way, the compression code rate can be allocated in the salient region, and thus the distortion rate of the salient region can be reduced under the condition that the overall compression code rate remains unchanged.

[0080] Optionally, in some embodiments, the target saliency map includes a salient region and a non-salient region, wherein the importance corresponding to the salient region is greater than the importance corresponding to the non-salient region, and the code rate weight of the salient region is greater than the code rate weight of the non-salient region.

[0081] In the embodiments of the present application, the above-mentioned code rate weight can be understood or replaced as an allocated compression code rate, or can be understood or replaced as an importance weight value, wherein the greater the code rate weight, the greater the corresponding allocated compression code rate, that is, the greater the importance weight value.

[0082] Optionally, in some embodiments, the importance weight value corresponding to the salient region can be a first value, and the importance weight value corresponding to the non-salient region can be a second value, wherein the first value is greater than the second value. As shown in Figure 3 and Figure 4 As shown, the salient region in the target saliency map A can be a white region, and the non-salient region can be a black region.

[0083] Optionally, in some embodiments, the to-be-encoded feature map includes at least two first feature maps of different scales, and the saliency map includes a saliency map corresponding to each scale of the first feature map.

[0084] In the embodiments of the present application, the above-mentioned first feature map is a multi-channel feature map, specifically, the target compression network can perform salient feature extraction on each scale of the first feature map to obtain a saliency map corresponding to each scale of the first feature map, or perform salient feature extraction on a first feature map of a certain scale (such as the largest scale) to obtain a saliency map corresponding to the first feature map of the scale, and then obtain a saliency map corresponding to the first feature map of other scales through scale scaling.

[0085] Optionally, in some embodiments, before the salient feature is extracted from the to-be-encoded feature map of the to-be-encoded image to obtain the target saliency map, the method further includes:

[0086] The input image sequence is subjected to feature extraction of different scales to obtain the to-be-encoded feature map of the to-be-encoded image.

[0087] In the embodiments of the present application, the target compression network can be understood as a feature compression network, which is used for compression processing of the image sequence. That is, the encoding network model is applied in a saliency feature compression framework for images, as shown in Figure 3 The saliency feature compression framework for images includes an encoding network model, a decoding network model, and an image task network. The image task network is used for target detection or image segmentation.

[0088] Optionally, in some embodiments, before the saliency feature extraction of the to-be-encoded feature map of the to-be-encoded image is performed to obtain the target saliency map, the method further includes:

[0089] The input video sequence is subjected to feature extraction of different scales to obtain the to-be-encoded feature map.

[0090] The to-be-encoded image is N frames of video images contained in one encoding process of the video sequence, and N is an integer greater than 1.

[0091] In the embodiments of the present application, the target compression network can be understood as a time sequence feature compression network, which is used for compression processing of the video sequence. That is, the encoding network model is applied in a saliency time sequence feature fusion framework for videos, as shown in Figure 4 The saliency time sequence feature fusion framework for videos includes an encoding network model, a decoding network model, and an image task network. The image task network is used for target tracking.

[0092] Optionally, in some embodiments, the target saliency map corresponding to the first frame of video image in the N frames of video images is the saliency map of the first frame of video image, and the target saliency map corresponding to the nth frame of video image in the N frames of video images is determined based on saliency region superposition of the saliency map of the nth frame of video image and the saliency map of the (n-1)th frame of video image, n is an integer greater than 1 and less than or equal to N.

[0093] In the embodiments of the present application, the saliency region in the target saliency map corresponding to the nth frame of video image in the N frames of video images is the union of the saliency region of the saliency map of the nth frame of video image and the saliency region of the saliency map of the (n-1)th frame of video image. For example, the saliency region of the saliency map of the nth frame of video image is region A, and the saliency region of the saliency map of the (n-1)th frame of video image is region B, and the saliency region in the target saliency map corresponding to the nth frame of video image in the N frames of video images is region A and region B. Region A and region B can not overlap or partially overlap.

[0094] It should be understood that in the embodiments of the present application, the idea of the saliency feature compression framework for images is extended to videos, and the redundant information of a video sequence is removed from the spatial dimension of a single frame of video and the temporal dimension between multiple frames. On the basis of focusing on encoding the salient region when compressing a single frame of image, when encoding the inter-frame information, the salient regions of adjacent frames are superimposed, so that when encoding the residual information of adjacent frames, the code rate of the region where the main object is located can be emphasized by using the superimposed salient information of adjacent frames. On the basis of greatly reducing the redundant information, the accuracy of video image tasks such as target tracking is preserved.

[0095] It should be noted that in the image sequence task test and the video sequence task test respectively, the experimental results show that: under the condition of saving a large amount of code rate, the accuracy of the detection result does not decrease, but increases to a certain extent, which shows that the method has achieved good results in image tasks and video tasks.

[0096] Optionally, the coding method of the present application uses the redundancy of the salient region, which is not limited to a single machine vision task or network structure, but can be used in any machine vision task and network structure, such as commonly used detection and segmentation networks such as Fast R-CNN, Mask R-CNN, YOLO5, etc. Here, no longer tedious.

[0097] In order to better understand the present application, the design process of the loss function of the target compression network in the training process is described in detail below.

[0098] In the training process of the image feature compression framework based on information redundancy, the code rate loss function is kept unchanged, and a weight is added to the minimum mean square error loss function used to measure the reconstruction loss.

[0099] Image compression is a constrained optimization problem. The goal of image compression is to minimize the distortion D of the image as much as possible under a given code rate R, which means that the compression algorithm should keep the image file as small as possible while minimizing the impact on image quality. Therefore, the compression algorithm needs to find a balance point between code rate and distortion to ensure the best image quality under a given code rate. For example, rate-distortion optimization is used to solve this problem, which can be expressed in mathematical formula as follows:

[0100] min J, J = D + λ × R;

[0101] Where J is the final measurement data, and λ is the Lagrange multiplier.

[0102] By adjusting lambda, a compression model with different compression rates can be obtained. The scheme of the present application mainly modifies the distortion D for calculating the image reconstruction quality. The loss function of the target compression network generally uses the minimum mean square error to calculate the reconstruction distortion. The minimum mean square error is a commonly used distortion measurement index in image compression, which measures the pixel-level difference between the compressed image and the original image. The minimum mean square error calculates the square of the difference between the pixel value of the compressed image and the pixel value of the original image, and then takes the average. By minimizing the minimum mean square error distortion loss, the image compression algorithm can minimize the distortion of the image, thereby improving the quality of the compressed image, where N is the total number of pixels of the image, x i represents the information of the current pixel of the original image (i.e., the pixel information of the sample image), represents the information of the current pixel of the reconstructed image (i.e., the pixel information of the image output by the target compression network). The loss function D MSE of the target compression network (i.e., the distortion D) can be represented as:

[0103]

[0104] During the training of the network, a saliency map can be generated according to the bounding box, and the image can be divided into a salient region (the number of pixels in the region is S) and a non-salient region (the number of pixels in the region is N-S) according to the saliency map. The minimum mean square error is changed to a minimum mean square error based on a target weight. That is, a target loss function D weighted-MSE is used to calculate the loss value during the training process of the target network model, which is:

[0105]

[0106] where weight represents the target weight, S represents the number of pixels in the salient region of a sample image, N represents the total number of pixels in a sample image, x i represents the pixel information of the sample image, represents the pixel information of the image output by the encoding network model.

[0107] By adjusting weight, the code rate allocated by the target compression network when compressing different regions (i.e., salient regions and non-salient regions) is adjusted, so that more efficient image redundancy compression is realized without affecting the image and video tasks, and the code rate is saved.

[0108] Optionally, the image output by the encoding network model can be understood as a reconstructed image.

[0109] It should be understood that in some embodiments, the target weight can be two weight values, for example, including W1 and W2, and the above target loss function D weighted-MSE may be:

[0110]

[0111] It should be noted that the traditional image, video compression network (i.e. target compression network) selects the minimum mean square error operation between the input image before compression and the image after encoding reconstruction, and the result of the operation is used to represent the distortion of the image after the target compression network, as a loss function. However, in the embodiments of the present application, in the training stage, the detection frame region of the training set is regarded as the saliency region of the current frame to be encoded, and according to the saliency region and the non-saliency region, when calculating the loss function, the saliency region loss function is multiplied by a larger weight, and the non-saliency region is multiplied by a small weight, that is, in this way, the weight of the loss function of the saliency region is strengthened under the premise that the overall compression code rate of the current frame is unchanged, and then more code rate is allocated to the saliency region. Through this loss function, after training, the target compression network has the prediction ability of the saliency region, and when calling the target compression network to encode the input picture and video, the target compression network can still allocate more code rate to the predicted saliency region according to the saliency prediction ability.

[0112] Reference Figure 5 The embodiments of the present application also provide a compression network training method, as shown in Figure 5 The compression network training method comprises the following steps:

[0113] Step 501, iteratively training an encoding network model by using a sample image to obtain a target compression network, the target compression network is used for encoding processing of a to-be-encoded feature map of a to-be-encoded image based on a target saliency map, and the target saliency map is used to represent the importance of each region of the to-be-encoded image;

[0114] Among them, the encoding network model comprises a feature pyramid network and a compression network, the feature pyramid network is used for feature extraction of the sample image in different scales to obtain a second feature map; and the compression network is used for compression encoding of the second feature map.

[0115] Among them, in one iteration process, a loss value between an image output by the encoding network model and the sample image is calculated based on a target loss function, and the target compression network is obtained by training the compression network under the condition that the loss value reaches convergence; the target loss function is a minimum mean square error loss function determined based on a target weight, and the target weight is used to adjust the code rate weight allocated by the compression network to the saliency region and the non-saliency region.

[0116] In the embodiments of the present application, the loss value reaching convergence can be understood as the loss value approaching the preset value and being relatively stable (such as the change of the loss value in continuous multiple iterations being small).

[0117] Optionally, the target loss function D weighted-MSE is:

[0118]

[0119] wherein, weight represents a target weight, S represents a number of significant region pixels of a sample image, N represents a total number of pixels of a sample image, x i represents pixel information of a sample image, represents pixel information of an image output by the encoding network model.

[0120] Optionally, the target saliency map includes a saliency region and a non-saliency region, wherein the saliency region corresponds to a greater importance than the non-saliency region, and a code rate weight of the saliency region is greater than a code rate weight of the non-saliency region.

[0121] Optionally, the to-be-encoded feature map includes at least two first feature maps of different scales, and the saliency map includes a saliency map corresponding to each first feature map of a scale.

[0122] The encoding method provided in the embodiments of the present application can be executed by an encoding device. In the embodiments of the present application, the encoding method is executed by an encoding device as an example to illustrate the encoding device provided in the embodiments of the present application.

[0123] With reference to Figure 6 , the embodiments of the present application further provide an encoding device, as shown in Figure 6 , the encoding device 600 includes:

[0124] A first feature extraction module 601 is configured to perform saliency feature extraction on a to-be-encoded feature map of a to-be-encoded image to obtain a target saliency map, wherein the target saliency map is used to represent the importance of each region of the to-be-encoded image.

[0125] An encoding module 602 is configured to perform encoding processing on the to-be-encoded feature map based on the target saliency map to obtain an encoding bitstream.

[0126] The encoding processing includes determining a code rate weight of the to-be-encoded feature map according to the target saliency map, and performing compression encoding on the to-be-encoded feature map according to the code rate weight.

[0127] Optionally, the target saliency map includes a saliency region and a non-saliency region, wherein the saliency region corresponds to a greater importance than the non-saliency region, and a code rate weight of the saliency region is greater than a code rate weight of the non-saliency region.

[0128] Optionally, the to-be-encoded feature map comprises at least two first feature maps of different scales, and the saliency map comprises a saliency map corresponding to each first feature map of a scale.

[0129] Optionally, the encoding device 600 further comprises:

[0130] a second feature extraction module configured to perform feature extraction of different scales on a to-be-encoded image in an input image sequence to obtain the to-be-encoded feature map.

[0131] Optionally, the encoding device 600 further comprises:

[0132] a second feature extraction module configured to perform feature extraction of different scales on an input video sequence to obtain the to-be-encoded feature map.

[0133] Optionally, the to-be-encoded image is N video images contained in one encoding process in the video sequence, and N is an integer greater than 1.

[0134] Optionally, a target saliency map corresponding to a first video image in the N video images is a saliency map of the first video image, and a target saliency map corresponding to an nth video image in the N video images is determined based on saliency region superposition of a saliency map of the nth video image and a saliency map of an (n-1)th video image, n is an integer greater than 1 and less than or equal to N.

[0135] The compression network training method provided in the embodiments of the present application can be executed by a compression network training device. In the embodiments of the present application, the compression network training method is executed by a compression network training device as an example to illustrate the compression network training device provided in the embodiments of the present application.

[0136] Referring to Figure 7 , the embodiments of the present application also provide a compression network training device, as shown in Figure 7 , the compression network training device 700 comprises:

[0137] a training module 701 configured to perform iterative training on an encoding network model by using sample images to obtain a target compression network, the target compression network being configured to perform encoding processing on a to-be-encoded feature map of a to-be-encoded image based on a target saliency map, and the target saliency map being configured to represent importance of each region of the to-be-encoded image.

[0138] Optionally, the encoding network model comprises a feature pyramid network and a compression network, the feature pyramid network being configured to perform feature extraction of different scales on the sample images to obtain a second feature map, and the compression network being configured to perform compression encoding on the second feature map.

[0139] In one iteration process, a loss value between an image output by the encoding network model and the sample image is calculated based on a target loss function, and the target compression network is trained based on a case where the loss value reaches convergence.

[0140] Optionally, the target loss function D weighted-MSE is:

[0141]

[0142] wherein weight represents a target weight, S represents a number of significant region pixels of a sample image, N represents a total number of pixels of a sample image, x i represents pixel information of the sample image, represents pixel information of the image output by the encoding network model.

[0143] Optionally, the target saliency map includes a significant region and a non-significant region, wherein the significant region corresponds to a greater importance than the non-significant region, and a code rate weight of the significant region is greater than a code rate weight of the non-significant region.

[0144] Optionally, the to-be-encoded feature map includes at least two first feature maps of different scales, and the saliency map includes a saliency map corresponding to each first feature map of a scale.

[0145] The encoding device and the compression network training device in the embodiments of the present application can be an electronic device, for example, an electronic device with an operating system, or a component in an electronic device, for example, an integrated circuit or a chip. The electronic device can be a terminal or other equipment other than a terminal. Illustratively, the terminal can include, but is not limited to, the types of the terminal 11 listed above, and the other equipment can be a server, a network attached storage (NAS), etc., which are not limited in the embodiments of the present application.

[0146] The encoding device and the compression network training device provided in the embodiments of the present application can implement the processes of the method embodiments Figures 2 to 5 and achieve the same technical effects. To avoid repetition, the details are not described here.

[0147] As Figure 8As shown, the embodiments of the present application also provide an electronic device 800, comprising a processor 801 and a memory 802, the memory 802 stores programs or instructions which can run on the processor 801, the programs or instructions are executed by the processor 801 to realize the steps of the above-mentioned encoding method embodiments or, realize the above-mentioned compression network training method, and can achieve the same technical effects, to avoid repetition, which will not be described here. Optionally, the memory 802 can be Figure 1 The memory 102 in the embodiment shown, the processor 801 can realize Figure 1 The function of the encoder 200 in the embodiment shown.

[0148] The embodiments of the present application also provide an electronic device, comprising a processor and a communication interface, the communication interface and the processor are coupled, the processor is used to run programs or instructions, realize the steps in the method embodiments as Figure 2 Or Figure 5 The embodiment shown. The electronic device embodiment corresponds to the above-mentioned encoding method and compression network training method embodiments, each implementation process and implementation manner of the above-mentioned method embodiments can be applied to the electronic device embodiment, and can achieve the same technical effects. Specifically, Figure 9 To realize the hardware structure of an electronic device of the embodiment of the present application.

[0149] The electronic device 900 includes but is not limited to at least part of the components such as radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909 and processor 910.

[0150] Those skilled in the art can understand that the electronic device 900 can also include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected with the processor 910 through a power management system, so as to realize the functions of power management system such as management of charging, discharging and power consumption management. Figure 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, the electronic device can include more or less components than the figure, or combine certain components, or different component arrangement, which will not be described here.

[0151] It should be understood that in the embodiments of the present application, the input unit 904 can include a graphics processing unit (GPU) 9041 and a microphone 9042. The graphics processor 9041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 906 can include a display panel 9061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 can include two parts of a touch detection device and a touch controller. The other input devices 9072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc., which will not be described here.

[0152] In the embodiments of the present application, after the radio frequency unit 901 receives the downlink data from the network side device, it can be transmitted to the processor 910 for processing. In addition, the radio frequency unit 901 can send uplink data to the network side device. Generally, the radio frequency unit 901 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

[0153] The memory 909 can be used to store software programs or instructions and various data. The memory 909 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), etc. In addition, the memory 909 can include a volatile memory or a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 909 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0154] The processor 910 can include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 910.

[0155] The processor 910 is configured to perform the following steps: performing salient feature extraction on a to-be-encoded feature map of a to-be-encoded image to obtain a target saliency map, the target saliency map being used to represent the importance of each region of the to-be-encoded image; performing encoding processing on the to-be-encoded feature map based on the target saliency map to obtain an encoded code stream; wherein the encoding processing includes: determining a code rate weight of the to-be-encoded feature map according to the target saliency map; and performing compression encoding on the to-be-encoded feature map according to the code rate weight.

[0156] Alternatively, the processor 910 is configured to perform iterative training on the coding network model by using the sample image, to obtain a target compression network, the target compression network being configured to perform coding processing on a to-be-coded feature map of a to-be-coded image based on a target saliency map, the target saliency map being configured to represent importance of each region of the to-be-coded image.

[0157] The coding network model comprises a feature pyramid network and a compression network, the feature pyramid network being configured to perform feature extraction of different scales on the sample image to obtain a second feature map, and the compression network being configured to perform compression coding on the second feature map.

[0158] In one iteration process, a loss value between an image output by the coding network model and the sample image is calculated based on a target loss function, and the target compression network is obtained by training the compression network when the loss value reaches convergence; the target loss function is a minimum mean square error loss function determined based on a target weight, and the target weight is used to adjust a code rate weight allocated by the compression network to a saliency region and a non-saliency region.

[0159] It can be understood that the implementation process of each implementation manner mentioned in the embodiment can refer to the related description of the above method embodiment, and achieve the same or corresponding technical effect. To avoid repetition, it will not be repeated here.

[0160] The embodiment of the present application also provides a readable storage medium, which stores a program or instructions, and the program or instructions are executed by a processor to realize each process of the above-mentioned coding method or compression network training method embodiment, and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0161] The processor is the processor in the terminal in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. In some examples, the readable storage medium can be a non-transitory readable storage medium.

[0162] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface and the processor are coupled, and the processor is configured to run a program or instructions to realize each process of the above-mentioned coding method or compression network training method embodiment, and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0163] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system on chip (SOC), which can also include an independent display chip, etc.

[0164] The embodiment of the present application further provides a computer program / program product, which comprises computer instructions, and the computer program / program product is executed by at least one processor to implement each process of the above-mentioned encoding method or compression network training method embodiment, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0165] It should be noted that, in this document, the terms "comprising", "including", or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements does not only include those elements, but also includes other elements that are not expressly listed or other elements inherent in such process, method, article, or apparatus. Without more limitations, an element defined by the sentence "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element. In addition, it should be pointed out that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing functions in the order shown or discussed, but can also include performing functions in a substantially simultaneous manner or in reverse order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, features described with reference to certain examples can be combined in other examples.

[0166] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of computer software product and general hardware platform, of course, it can also be realized by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disc, optical disc, etc.), and includes a plurality of instructions for making the terminal or network side device execute the method described in each embodiment of the present application.

[0167] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms of embodiments under the inspiration of the present application without departing from the scope of the present application and the protection scope of the claims, and these embodiments all belong to the protection scope of the present application.

Claims

1. An encoding method characterized by, The method comprises the following steps: performing saliency feature extraction on a to-be-encoded feature map of a to-be-encoded image to obtain a target saliency map, the target saliency map being used to represent the importance of each region of the to-be-encoded image; performing encoding processing on the to-be-encoded feature map based on the target saliency map to obtain an encoded code stream; wherein the encoding processing comprises: determining a code rate weight of the to-be-encoded feature map according to the target saliency map; and performing compression encoding on the to-be-encoded feature map according to the code rate weight.

2. The method of claim 1, wherein, The target saliency map comprises a saliency region and a non-saliency region, wherein the importance corresponding to the saliency region is greater than the importance corresponding to the non-saliency region, and the code rate weight of the saliency region is greater than the code rate weight of the non-saliency region.

3. The method according to claim 1 or 2, characterized in that, The to-be-encoded feature map comprises at least two first feature maps of different scales, and the saliency map comprises a saliency map corresponding to each first feature map of a scale.

4. The method according to any one of claims 1 to 3, characterized in that, Before the step of performing saliency feature extraction on the to-be-encoded feature map of the to-be-encoded image to obtain the target saliency map, the method further comprises: performing feature extraction of different scales on a to-be-encoded image in an input image sequence to obtain the to-be-encoded feature map.

5. The method according to any one of claims 1 to 3, characterized in that, Before the step of performing saliency feature extraction on the to-be-encoded feature map of the to-be-encoded image to obtain the target saliency map, the method further comprises: performing feature extraction of different scales on an input video sequence to obtain the to-be-encoded feature map; wherein the to-be-encoded image is N video images contained in one encoding process in the video sequence, and N is an integer greater than 1.

6. The method of claim 5, wherein, The target saliency map corresponding to a first video image in the N video images is a saliency map of the first video image, and the target saliency map corresponding to an nth video image in the N video images is determined based on saliency region superposition of a saliency map of the nth video image and a saliency map of an (n-1)th video image, n is an integer greater than 1 and less than or equal to N.

7. A method of compressing network training, the method comprising: The method comprises the following steps: iteratively training an encoding network model using sample images to obtain a target compression network, the target compression network being used to perform encoding processing on a to-be-encoded feature map of a to-be-encoded image based on a target saliency map, the target saliency map being used to represent the importance of each region of the to-be-encoded image; wherein the encoding network model comprises a feature pyramid network and a compression network, the feature pyramid network being used to perform feature extraction of different scales on the sample images to obtain a second feature map, and the compression network being used to perform compression encoding on the second feature map; wherein in one iteration process, a loss value between an image output by the encoding network model and the sample image is calculated based on a target loss function, the target compression network being the compression network trained under the condition that the loss value reaches convergence; and the target loss function is a minimum mean square error loss function determined based on a target weight, the target weight being used to adjust the code rate weight allocated by the compression network to the saliency region and the non-saliency region.

8. The method of claim 7, wherein, The target loss function D weighted-MSE is: wherein weight represents a target weight, S represents the number of pixels of a salient region of a sample image, N represents the total number of pixels of a sample image, x i represents pixel information of a sample image, represents pixel information of an image output by the encoding network model.

9. The method according to claim 7 or 8, characterized in that, The target saliency map includes a saliency region and a non-saliency region, wherein the importance corresponding to the saliency region is greater than the importance corresponding to the non-saliency region, and the code rate weight of the saliency region is greater than the code rate weight of the non-saliency region.

10. The method according to any one of claims 7 to 9, characterized in that, The to-be-encoded feature map includes at least two first feature maps of different scales, and the saliency map includes a saliency map corresponding to each first feature map of a scale.

11. An encoding apparatus, comprising: Comprise: A first feature extraction module is configured to perform saliency feature extraction on a to-be-encoded feature map of a to-be-encoded image to obtain a target saliency map, wherein the target saliency map is used to represent the importance of each region of the to-be-encoded image. An encoding module is configured to perform encoding processing on the to-be-encoded feature map based on the target saliency map to obtain an encoded bitstream. The encoding processing includes determining a code rate weight of the to-be-encoded feature map according to the target saliency map, and performing compression encoding on the to-be-encoded feature map according to the code rate weight.

12. The apparatus of claim 11, wherein, The target saliency map includes a saliency region and a non-saliency region, wherein the importance corresponding to the saliency region is greater than the importance corresponding to the non-saliency region, and the code rate weight of the saliency region is greater than the code rate weight of the non-saliency region.

13. The apparatus of claim 11 or 12, wherein, Further comprise: A second feature extraction module is configured to perform feature extraction of different scales on an input video sequence to obtain the to-be-encoded feature map. The to-be-encoded image is N frames of video images contained in one encoding process of the video sequence, and N is an integer greater than 1.

14. The apparatus of claim 13, wherein, The target saliency map corresponding to a first frame of video image in the N frames of video images is a saliency map of the first frame of video image, and the target saliency map corresponding to an nth frame of video image in the N frames of video images is determined based on saliency region superposition of a saliency map of the nth frame of video image and a saliency map of an (n-1)th frame of video image, n is an integer greater than 1 and less than or equal to N.

15. A compressed network training apparatus, comprising: Comprise: A training module is configured to iteratively train an encoding network model using sample images to obtain a target compression network, wherein the target compression network is used to perform encoding processing on a to-be-encoded feature map of a to-be-encoded image based on a target saliency map, and the target saliency map is used to represent the importance of each region of the to-be-encoded image. The encoding network model includes a feature pyramid network and a compression network, the feature pyramid network is used to perform feature extraction of different scales on the sample images to obtain a second feature map, and the compression network is used to perform compression encoding on the second feature map. In one iteration process, a loss value between an image output by the encoding network model and the sample image is calculated based on a target loss function, the target compression network is obtained by training the compression network when the loss value reaches convergence, the target loss function is a minimum mean square error loss function determined based on a target weight, and the target weight is used to adjust the code rate weight allocated by the compression network to the saliency region and the non-saliency region.

16. The apparatus of claim 15, wherein, The target loss function D weighted-MSE is: wherein weight represents a target weight, S represents the number of pixels of a salient region of a sample image, N represents the total number of pixels of a sample image, x i represents pixel information of a sample image, represents pixel information of an image output by the encoding network model.

17. An electronic device, comprising: A computer program product, comprising a computer readable storage medium having stored thereon instructions that, when executed, by a processor, perform the steps of the encoding method of any one of claims 1 to 6, or the steps of the compressed network training method of any one of claims 7 to 10.

18. A readable storage medium, characterized by, A computer program product, comprising a computer readable storage medium having stored thereon instructions that, when executed, by a processor, perform the steps of the encoding method of any one of claims 1 to 6, or the steps of the compressed network training method of any one of claims 7 to 10.

19. A computer program product, characterised in that, A computer program product, comprising a computer readable storage medium having stored thereon instructions that, when executed, by a processor, perform the steps of the encoding method of any one of claims 1 to 6, or the steps of the compressed network training method of any one of claims 7 to 10.

Images