Image compression method and device and electronic equipment

Abstract

The invention discloses an image compression method and device and electronic equipment, and belongs to the technical field of coding and decoding, and the method comprises the steps that a coding end obtains a video sequence, and the video sequence comprises multiple frames of video images; the coding end performs feature extraction on each frame of video image in the multiple frames of video images to obtain K feature maps corresponding to each frame of video image, the K feature maps correspond to different scales, and K is greater than or equal to 2; the coding end performs feature fusion on the K feature maps corresponding to each frame of video image to obtain a fused feature map corresponding to each frame of video image; the coding end carries out feature fusion on the fusion feature maps corresponding to every M frames of video images in the multiple frames of video images according to a time sequence to obtain multiple frames of feature maps corresponding to every M frames of video images, and M is greater than or equal to 2; and obtaining a code stream corresponding to the video sequence based on the multi-frame feature map corresponding to every M frames of video images.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of communication, and particularly relates to an image compression method and device and electronic equipment. BACKGROUND

[0002] In a computer vision task, a coding end can extract features of different scales of original video images to improve the accuracy of target detection, but this increases the code rate of video compression. Therefore, how to balance the accuracy of target detection and the code rate of video compression is an urgent problem to be solved. SUMMARY

[0003] The embodiments of the present application provide an image compression method, device and electronic equipment, which can balance the accuracy of target detection and the code rate of video compression.

[0004] In a first aspect, an image compression method is provided, which is executed by a coding end, and the method comprises the following steps:

[0005] The coding end acquires a video sequence, wherein the video sequence comprises multiple video images;

[0006] The coding end extracts features of each video image in the multiple video images to acquire K feature maps corresponding to each video image, wherein the K feature maps correspond to different scales, and K is greater than or equal to 2;

[0007] The coding end fuses the K feature maps corresponding to each video image to acquire a fused feature map corresponding to each video image;

[0008] The coding end fuses the fused feature maps corresponding to each M video image in the multiple video images in a time sequence to acquire multiple feature maps corresponding to each M video image, wherein M is greater than or equal to 2;

[0009] Based on the multiple feature maps corresponding to each M video image, a code stream corresponding to the video sequence is acquired.

[0010] In a second aspect, an image compression device is provided, which comprises the following modules:

[0011] A first acquisition module is configured to acquire a video sequence, wherein the video sequence comprises multiple video images;

[0012] A feature extraction module is configured to extract features of each video image in the multiple video images to acquire K feature maps corresponding to each video image, wherein the K feature maps correspond to different scales, and K is greater than or equal to 2;

[0013] The feature fusion module is configured to perform feature fusion on the K feature maps corresponding to each frame of video image to obtain a fused feature map corresponding to each frame of video image.

[0014] The time sequence feature fusion module is configured to perform feature fusion on the fused feature maps corresponding to each M frame of video image in the plurality of frames of video image in a time sequence to obtain a plurality of frames of feature maps corresponding to each M frame of video image, where M is greater than or equal to 2.

[0015] The second obtaining module is configured to obtain a code stream corresponding to the video sequence based on the plurality of frames of feature maps corresponding to each M frame of video image.

[0016] In a third aspect, an electronic device is provided, which includes a processor and a memory, the memory storing programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method according to the first aspect.

[0017] In a fourth aspect, an electronic device is provided, which includes a processor and a communication interface, the communication interface coupled to the processor, and the processor configured to run programs or instructions to implement the steps of the method according to the first aspect.

[0018] In a fifth aspect, an electronic device is provided, which includes a memory configured to store video data, and a processing circuitry configured to implement the steps of the method according to the first aspect.

[0019] In a sixth aspect, a readable storage medium is provided, which stores programs or instructions, and the programs or instructions, when executed by a processor, implement the steps of the method according to the first aspect.

[0020] In a seventh aspect, a chip is provided, which includes a processor and a communication interface, the communication interface coupled to the processor, and the processor configured to run programs or instructions to implement the steps of the method according to the first aspect.

[0021] In an eighth aspect, a computer program / program product is provided, which is stored in a storage medium, and the program / program product is executed by at least one processor to implement the steps of the method according to the first aspect.

[0022] In an embodiment of the present application, the encoding end can obtain K feature maps of different scales corresponding to each frame of video image in the video sequence, and then perform feature fusion on the K feature maps to obtain a fused feature map corresponding to the video image, and further perform feature fusion on the fused feature maps corresponding to the M frames of video image to obtain a multi-frame feature map, and generate a code stream of the video sequence based on the multi-frame feature map. Since the multi-frame feature map eliminates the inter-frame redundant features of the M frames of video image, it can reduce the bit rate compared to directly generating a code stream based on the fused feature map corresponding to the M frames of video image. At the same time, since the multi-frame feature map fuses the features of the M frames of video image, target detection based on the multi-frame feature map can ensure detection accuracy. Therefore, the image compression method provided by the embodiment of the present application can take into account both code stream and detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the encoding and decoding system provided in an embodiment of the present application.

[0024] Figure 2 This is a schematic diagram of an image compression method provided in an embodiment of the present application.

[0025] Figure 3 This is a schematic diagram of a coding framework provided in an embodiment of the present application.

[0026] Figure 4 This is a schematic diagram of obtaining a fusion feature map provided in an embodiment of the present application.

[0027] Figure 5 This is a schematic diagram of a timing fusion network provided in an embodiment of the present application.

[0028] Figure 6 This is a schematic block diagram of an image compression device provided according to an embodiment of the present application.

[0029] Figure 7 This is a schematic block diagram of an electronic device provided according to an embodiment of the present application.

[0030] Figure 8 This is a schematic diagram of the hardware structure of a terminal provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0032] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0033] Figure 1 is a schematic diagram of a codec system 10 provided in an embodiment of the present application. The technical solution of the embodiment of the present application relates to encoding and decoding (CODEC) (including encoding or decoding) of video data. The video data includes original unencoded video, encoded video, decoded (e.g., reconstructed) video, or syntax elements.

[0034] like Figure 1 As shown, the codec system 10 includes a source device 100, which provides encoded video data to be decoded and displayed by a destination device 110. Specifically, the source device 100 provides the video data to the destination device 110 via a communication medium 120. The source device 100 and the destination device 110 may include any one or more of a desktop computer, a notebook (i.e., laptop) computer, a tablet computer, a set-top box, a mobile phone, a wearable device (e.g., a smart watch or a wearable camera), a television, a camera, a display device, an in-vehicle device, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, a digital media player, a video game console, a video conferencing device, a video streaming device, a broadcast receiver device, a broadcast transmitter device, a spacecraft, an aircraft, a robot, a satellite, and the like.

[0035] exist Figure 1 In the example of FIG, the source device 100 includes a data source 101, a memory 102, an encoder 200, and an output interface 104. The destination device 110 includes an input interface 111, a decoder 300, a memory 113, and a display device 114. The source device 100 represents an example of a video encoding device, and the destination device 110 represents an example of a video decoding device. In other examples, the source device 100 and the destination device 110 may not include Figure 1 Some components in, or can also include Figure 1Source device 100 and destination device 110 can be examples of a device as described in more detail below. Source device 100 and destination device 110 can comprise, for example, a wireless communication device, a mobile communication device, a personal computer, a laptop computer, a tablet computer, a desktop computer, a personal digital assistant, a digital camera, a digital recording device, a mobile audio device, a video gaming console, a wireless earphone, a wireless speaker, a wearable device, or any other suitable device. Source device 100 and destination device 110 can also be referred to as a video encoding device and a video decoding device, respectively. In some examples, source device 100 and destination device 110 can be integrated in the same device. In some examples, source device 100 and destination device 110 can be separate devices.

[0036] Although Figure 1 Although source device 100 and destination device 110 are illustrated as separate devices, they can alternately be integrated in the same device in some examples. In such embodiments, the functions corresponding to source device 100 and the functions corresponding to destination device 110 can be implemented using the same hardware or software, or using separate hardware or software, or any combination thereof.

[0037] In some examples, source device 100 and destination device 110 can perform unidirectional video transmission or bidirectional video transmission. If bidirectional video transmission, then source device 100 and destination device 110 can operate in a substantially symmetrical manner. That is, each of source device 100 and destination device 110 can include appropriate encoders and decoders.

[0038] Data source 101 represents a source of video data (i.e., raw, unencoded video data) and provides successive blocks of video data to encoder 200, which encodes the data of the pictures. Data source 101 of source device 100 can include a video capture device, such as a video camera, a video archive containing previously captured raw video, or a video feed interface to receive video from a video content provider. As another alternative, data source 101 can generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In these cases, encoder 200 encodes the captured, pre-captured, or computer-generated video data. Encoder 200 can rearrange the pictures from the received order (sometimes referred to as "display order") into the encoding order. Encoder 200 can generate a bitstream including encoded video data. Source device 100 can then output the encoded video data via output interface 104 onto a communication medium 120 for reception and retrieval by, for example, input interface 111 of destination device 110.

[0039] The memory 102 of the source device 100 and the memory 113 of the destination device 110 represent general storage memory. In some examples, the memory 102 can store raw video data from the data source 101, and the memory 113 can store decoded video data from the decoder 300. Additionally or alternatively, the memories 102, 113 can store software instructions executable by, for example, the encoder 200 and the decoder 300, respectively. Although the memory 102 and the memory 113 are shown separately from the encoder 200 and the decoder 300 in this example, it should be understood that the encoder 200 and the decoder 300 can also include internal memories for similar or equivalent purposes. If the encoder 200 and the decoder 300 are deployed on the same hardware device, the memory 102 and the memory 113 can be one memory. Moreover, the memories 102, 113 can store encoded video data that is output from the encoder 200 and input to the decoder 300, for example. In some examples, portions of the memories 102, 113 can be allocated as one or more video buffers, for example, for storing raw, decoded, or encoded video data.

[0040] In some examples, the source device 100 can output encoded data from the output interface 104 to the memory 113. Similarly, the destination device 110 can access the encoded data from the memory 113 via the input interface 111. The memory 113, or the memory 102, can comprise any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, Digital Versatile Discs (DVDs), Compact Disc Read-Only Memory (CD-ROMs), flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0041] The output interface 104 can include any type of medium or device capable of transmitting the encoded video data from the source device 100 to the destination device 110. For example, the output interface 104 can include a transmitter or a transceiver, e.g., an antenna, configured to transmit encoded video data from the source device 100 to the destination device 110 directly or in a wireless manner. The encoded video data can be modulated according to a communication standard of a wireless communication protocol and transmitted to the destination device 110.

[0042] Communication medium 120 can include transient media, such as a wireless broadcast or wired network transmission. For example, communication medium 120 can include radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cable). Communication medium 120 can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. Communication medium 120 can also be in a form of a storage medium, such as a hard disk, flash drive, compact disk, digital video disk, Blu-ray disk, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0043] In some embodiments, communication medium 120 can include routers, switches, base stations, or any other equipment that can be used to facilitate communication from source device 100 to destination device 110. For example, a server (not shown) can receive encoded video from source device 100 and provide the encoded video data to destination device 110, e.g., via network transmission. The server can include a web server (e.g., for a website), a server configured to provide file transfer protocol services (such as File Transfer Protocol (FTP) or File Delivery Over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Services (MBMS) or evolved Multimedia Broadcast Multicast Service (eMBMS) server, or a Network-attached storage (NAS) device, etc. The server can implement one or more HTTP streaming protocols, such as the MPEG Media Transport (MMT) protocol, Dynamic Adaptive Streaming over HTTP (DASH) protocol, HTTP Live Streaming (HLS) protocol, or Real Time Streaming Protocol (RTSP), etc.

[0044] Destination device 110 can access the encoded video data from the server, e.g., through a wireless channel for accessing encoded video data stored on a server (e.g., a Wi-Fi connection) or a wired connection (e.g., a Digital subscriber line (DSL), a cable modem, etc.).

[0045] Output interface 104 and input interface 111 can represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to the IEEE 802.11 specification or the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth specification, or other physical components utilized to communicate data. In examples where output interface 104 and input interface 111 comprise wireless components, output interface 104 and input interface 111 can be configured to communicate data, such as encoded video data, according to a WIFI, Ethernet, a cellular network specification such as 4G, LTE (Long-Term Evolution), LTE-Advanced, 5G, 6G, or other telecommunication standards.

[0046] The techniques of this disclosure can be applied to video coding in support of one or more multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions, digital video that is encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0047] Input interface 111 of destination device 110 receives an encoded video bitstream from communication medium 120. The encoded video bitstream can include syntax elements and coded data units (e.g., slices, pictures, groups of pictures, or other units) that, when decoded, form a decoded video data. Display device 114 displays the decoded video data to a user. Display device 114 can comprise a Cathode ray tube (CRT), a liquid-crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0048] The encoder 200 and the decoder 300 may be implemented as one or more of a variety of processing circuits, which may include a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), discrete logic, hardware, or any combination thereof. When the technology is implemented in whole or in part in software, the device may store instructions for the software in an appropriate non-transitory computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology provided in the embodiments of the present application.

[0049] Deep learning-based image processing, including image compression, has been introduced in some scenarios. In deep learning-based solutions, the compression task is considered an encoding process and trained using end-to-end learning methods. The specific process can be decomposed into an encoding process, and the corresponding reconstruction task can be considered a decoding process.

[0050] To facilitate understanding of the embodiments of the present application, the related technologies of the present application are explained.

[0051] 1. Computer vision technology, also known as artificial intelligence (AI) vision technology, is a key area of ​​computer science. It aims to use cameras and computers to replace the human eye in machine vision tasks such as identifying, tracking, and measuring objects. Further image processing is performed to make the computer-processed images more suitable for human observation or transmission to instruments for detection. Its basic principle is that an image acquisition device converts captured objects into image signals, which are then transmitted to a specialized image processing system. These signals are converted into digital signals based on information such as pixel distribution, brightness, and color. The image system then performs operations on these signals to extract the target's features. Finally, the resulting judgments control the operation of field equipment. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0052] In recent years, computer vision technology has been widely applied in various fields. In the field of autonomous driving, it helps cars to plan driving routes, detect and avoid obstacles, recognize traffic signals, and perform other functions. In the field of industrial manufacturing, it is used in robot vision systems, defect detection, quality control, part recognition, and assembly. In the field of medical diagnosis, computer vision technology is applied to image analysis, disease diagnosis, and treatment monitoring. In addition, it also plays an important role in security monitoring, augmented reality, garbage classification, agriculture, and gaming and entertainment.

[0053] 2、Feature Pyramid Network (FPN)

[0054] Feature Pyramid Network is a deep neural network used for object detection and semantic segmentation. Its core idea is to construct a feature pyramid at different scales to extract object features at different scales, thereby improving the accuracy of detection and segmentation.

[0055] Feature Pyramid Network can be applied to any machine vision task and network structure, such as Faster Region-based Convolutional Neural Networks (Faster R-CNN), Mask R-CNN, and YOLO5, which are used for detection and segmentation tasks.

[0056] In computer vision tasks, objects may appear in different sizes and shapes, so the detection or segmentation system needs to be able to handle information at different scales. Feature Pyramid Network obtains feature maps at different scales by downsampling and upsampling images at different levels, and fuses or cascades these feature maps. This fusion allows the network to obtain both detailed information from low-level features and semantic information from high-level features, which helps to effectively analyze and extract the overall and local features of objects.

[0057] Therefore, in object detection, Feature Pyramid Network enables the detector to better handle objects of different scales, improving the accuracy and robustness of detection. In image segmentation tasks, Feature Pyramid Network can obtain multi-scale context information, helping to achieve more fine-grained segmentation results. Therefore, Feature Pyramid Network can better adapt to objects of different scales, improving the performance of object detection and image segmentation.

[0058] In computer vision tasks, using Feature Pyramid Network to extract multi-scale features of objects can improve the accuracy of object detection, but it also increases the code rate of video compression. Therefore, how to balance the accuracy of object detection and the code rate of video compression is a problem that needs to be solved.

[0059] The following describes the encoding method and encoding method provided by the embodiment of the present application through some embodiments and their application scenarios in conjunction with the accompanying drawings. Among them, the image compression method can be executed by the encoding end, for example, Figure 1 The original table 100 or the encoder 200 in the embodiment. The encoding end may be implemented by software, hardware, or a combination thereof. When implemented by hardware, the encoding end may be referred to as an encoding end device.

[0060] Figure 2 This is a schematic diagram of an image compression method provided by an embodiment of the present application. Figure 2 As shown, the method 400 includes at least part of the following:

[0061] S401: The encoding end obtains a video sequence, where the video sequence includes multiple frames of video images.

[0062] The specific implementation reference of the encoding end obtaining the video sequence is Figure 1 For the sake of brevity, the relevant implementation of the source device 100 obtaining video data will not be repeated here.

[0063] It should be understood that the embodiments of the present application do not limit the application scenarios of the image compression method 400. For example, the image compression method can be applied to various computer vision tasks, such as target detection or target tracking, etc., and the present application is not limited to this.

[0064] S402: The encoding end extracts features from each frame of the multiple video images to obtain K feature maps corresponding to each frame of the video images, where the K feature maps correspond to different scales, K≥2, for example, K=3.

[0065] For example, the encoder can extract features from each frame of the multi-frame video image based on a feature extraction network to obtain K feature maps of different scales corresponding to each frame of the video image. The feature extraction network can include a feature pyramid network.

[0066] In some specific embodiments, the feature extraction network includes multiple feature extraction layers for performing feature extraction on each frame of video image to obtain K feature maps of different scales corresponding to each frame of video image. Optionally, the multiple feature extraction layers include three feature extraction layers, and the K feature maps may include three feature maps.

[0067] For example, each frame of video image is input into the feature extraction network, and features are extracted from each frame of video image through the multiple feature extraction layers to obtain a first feature map, a second feature map, and a third feature map corresponding to each frame of video image, wherein the first feature map, the second feature map, and the third feature map correspond to different scales.

[0068] Optionally, each feature extraction layer can include a convolution layer, different feature extraction layers correspond to different sizes of convolution kernels, and each frame of video image is convolved by different sizes of convolution kernels to obtain K feature maps of different sizes.

[0069] In some embodiments, the encoding end extracts features from each frame of video image in the plurality of frames of video images to obtain K feature maps corresponding to the each frame of video image, including:

[0070] The encoding end extracts features from the video image to obtain a first feature map;

[0071] The encoding end performs up-sampling processing on the first feature map to obtain a second feature map, and performs down-sampling processing on the first feature map to obtain a third feature map, wherein the K feature maps corresponding to the video image include the first feature map, the second feature map and the third feature map.

[0072] Therefore, the second feature map is a high-resolution feature map in the K feature maps, the third feature map is a low-resolution feature map in the K feature maps, and the first feature map is a medium-resolution feature map in the K feature maps. The larger the size of the feature map, the richer the detailed information (such as position information) it carries; the smaller the size, the richer the semantic information it carries. By obtaining feature maps of different scales and resolutions, rich semantic information and detailed information can be obtained, which can improve the accuracy of target detection.

[0073] Optionally, the encoding end can also perform normalization processing on the pixel values in the K feature maps obtained. Alternatively, the pixel values in the first feature map are normalized, and then the first feature map after normalization is up-sampled and down-sampled to obtain the second feature map and the third feature map.

[0074] S403, the encoding end extracts features from K feature maps corresponding to each frame of video image to obtain a fusion feature map corresponding to each frame of video image. Wherein, the fusion feature map fuses the features of the K feature maps.

[0075] For example, the encoding end uses a feature fusion network to fuse K feature maps corresponding to each frame of video image to obtain a fusion feature map corresponding to each frame of video image, and the specific fusion algorithm is not limited in the present application.

[0076] Optionally, the feature fusion network can include a plurality of convolutional layers and a fusion module, the plurality of convolutional layers can be used to convert K feature maps of different sizes into feature maps of the same size, and the fusion module is further used to fuse the feature maps of the same size into one feature map, for example, the feature maps of the same size are spliced into one feature map, and the spliced feature map is taken as a fusion feature map. Optionally, the spliced feature map can also be subjected to convolutional processing to obtain the fusion feature map.

[0077] S404, the encoding end performs feature fusion on the fusion feature map corresponding to each M frame of video image in the plurality of frames of video image in time sequence, and obtains a plurality of frames of feature map corresponding to each M frame of video image, wherein M≥2.

[0078] Optionally, the M frames of video image can be M frames of video image that are continuous in time sequence.

[0079] Optionally, the M frames of video image can be considered as a group of video images with strong inter-frame similarity in time sequence. Optionally, M can be 2, 4, 8, 16, 32, etc.

[0080] For example, the encoding end can take M frames of video image that are continuous in time sequence in a video sequence as a group of pictures (GOP), and further perform feature fusion on the fusion feature map corresponding to each frame of video image in the group of pictures to obtain a plurality of frames of feature map corresponding to the group of pictures. It can be understood that the plurality of frames of feature map fuses the inter-frame similar features and inter-frame difference features of the M frames of video image, and compresses the inter-frame redundant features of the M frames of video image. That is, the plurality of frames of feature map has a smaller bit rate and includes more feature information than the fusion feature map corresponding to the M frames of video image. Therefore, generating a code stream of the video sequence based on the plurality of frames of feature map can reduce the code rate, and performing target detection based on the plurality of frames of feature map can ensure the accuracy of the detection result because the plurality of frames of feature map retains the inter-frame similar features and inter-frame difference features of the M frames of video image.

[0081] It should be noted that, since the video image and the fusion feature map have a one-to-one correspondence, the M fusion feature maps corresponding to the M frames of video image can also be considered as a group of pictures, and a plurality of frames of feature map can be obtained by performing feature fusion based on the M fusion feature maps. The plurality of frames of feature map fuses the features in the M fusion feature maps. The plurality of frames of feature map, also referred to as a plurality of fusion feature maps, for example, when M=16, the plurality of frames of feature map, also referred to as 16 frames of feature map or 16 frames of fusion feature map.

[0082] In some embodiments, the encoding end performs feature fusion on the fusion feature map corresponding to each M frame of video image in the plurality of frames of video image in time sequence to obtain a target feature map corresponding to each M frame of video image, including:

[0083] The encoding end performs feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first multi-frame feature map, where L≥2;

[0084] The encoding end performs feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first multi-frame feature map, where L≥2;

[0085] The encoding end performs feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first multi-frame feature map, where L≥2;

[0086] The encoding end performs feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first multi-frame feature map, where L≥2;

[0087] Optionally, L can be 2.

[0088] Optionally, J can be equal to the integer value of (M-L) / (L-1), for example, the integer value is rounded up.

[0089] Taking L=2 as an example, the encoding end performs feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first multi-frame feature map, comprising:

[0090] The encoding end performs feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first multi-frame feature map, where L≥2;

[0091] Taking L=2 as an example, the encoding end performs feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first multi-frame feature map, comprising:

[0092] The encoding end performs feature fusion on the jth multi-frame feature map and the j+2 fusion feature map to obtain a j+1 multi-frame feature map, where the j+2 fusion feature map is a fusion feature map corresponding to a j+2 frame of video image in the M frames of video images, and j=1, 2, …, M-2.

[0093] For example, the encoding end can perform feature fusion on the first multi-frame feature map and the third fusion feature map to obtain a second multi-frame feature map, where the third fusion feature map is a fusion feature map corresponding to a third frame of video image in the M frames of video images.

[0094] For another example, the encoding end can perform feature fusion on the M-2 multi-frame feature map and the M fusion feature map to obtain an M-1 multi-frame feature map, where the M fusion feature map is a fusion feature map corresponding to an M frame of video image in the M frames of video images.

[0095] In some embodiments of the present application, the encoding end performs feature fusion on the fusion feature map corresponding to each M frame of video image in the M frames of video images in time sequence to obtain a multi-frame feature map corresponding to each M frame of video image, including:

[0096] The encoding end performs feature fusion on the fusion feature map corresponding to each M frame of video image in the M frames of video images based on a time sequence fusion network to obtain a multi-frame feature map corresponding to each M frame of video image.

[0097] It should be understood that the present application does not limit the specific implementation of generating a multi-frame feature map based on the fusion feature map corresponding to the M frames of video images, for example, the M frames of video images can be fused by a one-level feature fusion network (i.e., the time sequence fusion network can include a one-level feature fusion network), or the M frames of video images can also be fused by a multi-level feature fusion network (i.e., the time sequence fusion network can include a multi-level feature fusion network), for example, the output of each level of feature fusion network is taken as the input of the next level of feature fusion network, and the input of the next level of feature fusion network is introduced into the next one or more fusion feature maps in time sequence, so that the output of the last level of feature fusion network is relative to the fusion of the features in the fusion feature map corresponding to the M frames of video images.

[0098] In one specific embodiment, the time sequence fusion network includes J+1 levels of feature fusion network, and J=(M-L) / (L-1). For example, L=2, and the time sequence fusion network can include 15 levels of feature fusion network.

[0099] In some embodiments, the encoding end performs feature fusion on the fusion feature map corresponding to each M frame of video image in the M frames of video images based on a time sequence fusion network to obtain a multi-frame feature map corresponding to each M frame of video image, including:

[0100] perform feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first plurality of feature maps through a first feature fusion network in the J+1-level feature fusion network;

[0101] perform feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first plurality of feature maps through a first feature fusion network in the J+1-level feature fusion network;

[0102] perform feature fusion on the fusion feature maps corresponding to the first L video images in each M video images to obtain a first plurality of feature maps through a first feature fusion network in the J+1-level feature fusion network;

[0103] For example, when L=2, the input of the first feature fusion network can include a first fusion feature map (denoted as y1) and a second fusion feature map (denoted as y2), and the output is a first plurality of feature maps (denoted as p1); the input of the second feature fusion network can include the first plurality of feature maps (i.e., p1) and a third fusion feature map (denoted as y3), which is the fusion feature map corresponding to the third video image in the M video images, and the output is a second plurality of feature maps (denoted as p2); and so on. The input of the M-1-level feature fusion network can include an M-2 plurality of feature maps (denoted as P m-2 ) and an M fusion feature map (denoted as y M ), which is the fusion feature map corresponding to the M video image in the M video images, and the output is an M-1 plurality of feature maps (denoted as P m-1 ).

[0104] Optionally, each feature fusion network in the J+1-level feature fusion network can be implemented through a deep learning model (such as a convolutional neural network (CNN)), for example, a deep learning model is trained to learn the correlation between video images, and then the features of the video images are extracted through the deep learning model to realize feature-level fusion. Learning the inter-frame relationship through a deep learning model can effectively eliminate inter-frame redundancy.

[0105] Optionally, each feature fusion network includes a first convolutional layer and a second convolutional layer, the output of the first convolutional layer is used as the input of the second convolutional layer, the first convolutional layer is used to perform convolution processing on two input feature maps to obtain an output feature map, and the second convolutional layer is used to perform convolution processing on the difference image between the output feature map and the two feature maps, and the second convolutional layer is used to output the plurality of feature maps.

[0106] For example, for the first-level feature fusion network, the two feature maps include a first fusion feature map and a second fusion feature map, wherein the first fusion feature map is a fusion feature map corresponding to a first frame of video image in the M frames of video images, and the second fusion feature map is a fusion feature map corresponding to a second frame of video image in the M frames of video images.

[0107] For example, for the j+1-level feature fusion network, the two feature maps include a j multi-frame feature map and a j+2 fusion feature map, wherein the j+2 fusion feature map is a fusion feature map corresponding to a j+2 frame of video image in the M frames of video images, and j=1, 2, …, M-2.

[0108] Taking the first-level feature fusion network as an example, the first convolutional layer can perform convolutional processing on y1 and y2 to output a feature map q, and the second convolutional layer can perform convolutional processing on q and a difference feature map (i.e., y2-y1) between y2 and y1 to obtain p1. The feature map q includes inter-frame similar features of y1 and y2.

[0109] Therefore, the inter-frame similar features of the M frames of video images can be obtained based on the time sequence fusion network, so that the inter-frame redundant features can be eliminated, and the code rate can be reduced compared with directly generating a code stream based on the fusion feature maps corresponding to the M frames of video images. At the same time, since the multi-frame feature maps fuse the features of the M frames of video images, target detection based on the multi-frame feature maps can ensure detection accuracy. Therefore, the image compression method provided in the embodiments of the present application can balance the code stream and detection accuracy.

[0110] S405, the encoding end obtains a code stream corresponding to the video sequence based on the multi-frame feature maps corresponding to each M frames of video images.

[0111] In some embodiments, the encoding end obtains a code stream corresponding to the video sequence based on the multi-frame feature maps corresponding to each M frames of video images, including:

[0112] The encoding end performs feature compression based on the multi-frame feature maps corresponding to each M frames of video images to obtain a code stream corresponding to the video sequence.

[0113] It should be understood that the present application does not limit the compression algorithm adopted by the encoding end for the multi-frame feature maps, for example, a lossy compression algorithm, specifically, for example, a compression algorithm based on transformation, a compression algorithm based on prediction encoding or quantization, etc. Alternatively, a lossless compression algorithm, for example, an entropy encoding algorithm, can also be used.

[0114] In some embodiments of the present application, the method 400 further includes:

[0115] The encoding end sends a code stream corresponding to the video sequence to a decoding end.

[0116] For example, after the decoding end receives the code stream corresponding to the video sequence, the decoding end can perform decoding processing on the code stream to obtain a reconstructed video sequence. For example, the decoding end can obtain the reconstructed video sequence by using a corresponding behavior of the encoding end for obtaining the code stream by encoding the video sequence.

[0117] In some embodiments, the image compression method can be implemented by a neural network model, and the present application does not limit the specific network type. For example, the neural network model can be a deep learning model, such as a CNN model, for example, Fast R-CNN, Mask R-CNN, YOLO5, and the like.

[0118] In some embodiments, the neural network model can include a feature extraction network, a feature fusion network, a time sequence fusion network, and a feature compression network.

[0119] The feature extraction network is configured to perform feature extraction on each of the plurality of video images to obtain K feature maps corresponding to each of the plurality of video images.

[0120] The feature fusion network is configured to perform feature fusion on the K feature maps corresponding to each of the plurality of video images to obtain a fusion feature map corresponding to each of the plurality of video images.

[0121] The time sequence fusion network is configured to perform feature fusion on the fusion feature maps corresponding to each M video image of the plurality of video images to obtain a plurality of feature maps corresponding to each M video image.

[0122] The feature compression network is configured to perform feature compression on the plurality of feature maps corresponding to each M video image to obtain a code stream corresponding to the video sequence.

[0123] In some embodiments, the neural network model can be trained using a target loss function. The target loss function is determined based on at least one of a rate loss function and a distortion loss function. The rate loss function is used to ensure that the bit rate of the code stream generated by the encoding end is as small as possible, and the distortion loss function is used to constrain the reconstruction quality of the reconstructed video sequence. Therefore, determining the target loss function based on the rate loss function and the distortion loss function is beneficial to ensure that the trained neural network model takes into account both the code rate and the reconstruction quality of the video sequence.

[0124] In some specific implementations, the rate loss function is represented by the following formula:

[0125]

[0126] wherein, is a latent variable of the fused feature map output by the feature fusion network, is introduced edge information, represents the probability distribution of the pixel value of under the condition that the latent variable is the probability distribution of the pixel value of is the probability distribution of the edge information , and N represents the number of training samples, for example, the number of video images included in a video sequence used for training.

[0127] In some specific implementations, the distortion loss function is used to measure the difference between the K feature maps corresponding to the video image output by the feature extraction network and the K feature maps corresponding to the video image reconstructed at the decoding end.

[0128] For example, the distortion loss function can be represented as:

[0129]

[0130] wherein F1, F2, …, F K represent the K feature maps of a video image, F'1, F'2, …, F' K represent the K feature maps reconstructed at the decoding end, and l2 represents the difference between the original feature maps and the reconstructed feature maps.

[0131] In some specific implementations, the target loss function is determined according to the following formula:

[0132] L total = λ·(D1+D2…+D K )+R

[0133] wherein L total represents the target loss function, D1 represents the distortion loss function of the first feature map of the video image, D2 represents the distortion loss function of the second feature map of the video image, and D K represents the distortion loss function of the Kth feature map of the video image, R represents the rate loss function, and λ is related to the compression rate of the video sequence. In this way, by adjusting λ, a neural network model with different compression rates can be obtained, so as to adapt to different application scenarios.

[0134] Hereinafter, in conjunction with Figure 3 , a coding framework provided by the embodiments of the present application is described, which can execute the coding process in Figure 3 . The coding framework can include a feature extraction process, a feature fusion process, and a time-based feature fusion process. Optionally, it also includes a feature compression process.

[0135] In the feature extraction process, the encoding end can perform feature extraction on each frame of video images in the video sequence to obtain K feature maps of different scales, for example, three feature maps of different scales (denoted as F1, F2 and F3 respectively) are extracted based on a feature extraction network (such as a feature pyramid network), and optionally, the pixel values in the three scale feature maps can also be normalized.

[0136] In the feature fusion process, the K feature maps of different scales extracted in the feature extraction process can be fused to obtain a fusion feature map (denoted as y) corresponding to the video image, for example, as shown in Figure 4 , the encoding end can fuse the three feature maps F1, F2 and F3 of different scales to obtain the fusion feature map y, and then M video images corresponding to the fusion feature maps in time sequence can form a group of images, for example, the fusion feature maps (denoted as y1, y2, …, y 15 ,y 16 ) corresponding to 16 consecutive video images can form a group of images.

[0137] In the time sequence-based feature fusion process, the fusion feature maps can be arranged in time sequence, and every M fusion feature maps can form a group of images, and a time sequence fusion network can be used to determine the multi-frame feature maps corresponding to each group of images. Optionally, the time sequence fusion network can include a multi-level feature fusion network for fusing M fusion feature maps in time sequence. Taking M = 16 and L = 2 as an example, the network structure of the time sequence fusion network can be as shown in Figure 5 , the time sequence fusion network includes 15 levels of feature fusion networks, wherein the first level of feature fusion network is used to fuse the first fusion feature map (denoted as y1) and the second fusion feature map (denoted as y2) in time sequence in the M fusion feature maps, and output the first multi-frame feature map (denoted as p1); the second level of feature fusion network is used to fuse the first multi-frame feature map output by the first level of feature fusion network and the third fusion feature map (denoted as y3) in time sequence in the M fusion feature maps, and output the second multi-frame feature map (denoted as p2); and so on, the 15th level of feature fusion network is used to fuse the 14th multi-frame feature map (denoted as p 14 ) output by the 14th level of feature fusion network and the 16th fusion feature map (denoted as y16) in time sequence in the M fusion feature maps, and output the 15th multi-frame feature map (denoted as p15), and the output of each level of feature fusion network in the 15 levels of feature fusion networks can be represented as:

[0138] p1 = concat (y1, y2)

[0139] p2 = concat (y3, p1)

[0140] …

[0141] p 15 =concat(y 16 ,p 14 )

[0142] wherein concat denotes a feature fusion function.

[0143] In the feature compression process, the multi-frame feature maps output by the previous process can be compressed to obtain a code stream corresponding to the video sequence. For example, a High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), or entropy coding algorithm can be used.

[0144] In summary, the encoding end can obtain K feature maps of different scales corresponding to each video image in the video sequence, then perform feature fusion on the K feature maps to obtain a fusion feature map corresponding to the video image, further perform feature fusion on the fusion feature maps corresponding to M video images to obtain multi-frame feature maps, and generate a code stream of the video sequence based on the multi-frame feature maps. Since the multi-frame feature maps eliminate the inter-frame redundant features of the M video images, the code rate can be reduced compared to directly generating a code stream based on the fusion feature maps corresponding to the M video images. Moreover, since the multi-frame feature maps fuse the features of the M video images, target detection based on the multi-frame feature maps can ensure detection accuracy. Therefore, the image compression method provided in the embodiments of the present application can balance the code stream and detection accuracy.

[0145] The image compression method provided in the embodiments of the present application can be executed by an image compression device. In the embodiments of the present application, the image compression method is executed by an image compression device as an example to illustrate the image compression device provided in the embodiments of the present application.

[0146] Figure 6 A schematic block diagram of an image compression device 600 according to an embodiment of the present application is shown. As shown in the figure, the image compression device 600 includes: Figure 6

[0147] A first acquisition module 610 is configured to acquire a video sequence, wherein the video sequence includes multiple video images.

[0148] A feature extraction module 620 is configured to perform feature extraction on each video image in the multiple video images to obtain K feature maps corresponding to the video image, wherein the K feature maps correspond to different scales, and K≥2.

[0149] ​The feature fusion module 630 is configured to perform feature fusion on the K feature maps corresponding to each frame of video image to obtain a fused feature map corresponding to each frame of video image.

[0150] The time sequence feature fusion module 640 is configured to perform feature fusion on the fused feature maps corresponding to every M frames of video image in a time sequence to obtain a plurality of feature maps corresponding to each M frames of video image, where M is greater than or equal to 2.

[0151] The second obtaining module 650 is configured to obtain a bitstream corresponding to the video sequence based on the plurality of feature maps corresponding to each M frames of video image.

[0152] In some embodiments, the time sequence feature fusion module 640 is further configured to:

[0153] perform feature fusion on the fused feature maps corresponding to the first L frames of video image in the every M frames of video image to obtain a first plurality of feature maps, where L is greater than or equal to 2;

[0154] perform feature fusion on the fused feature maps corresponding to the j-th plurality of feature maps and a j-th group of video images after the first L frames of video image in a time sequence to obtain a (j+1)-th plurality of feature maps, where each group of video images includes (L-1) video images, where j = 1, 2, …, J, where J = (M-L) / (L-1), and when j = 1, the j-th plurality of feature maps is the first plurality of feature maps;

[0155] determine the (J+1)-th plurality of feature maps as the plurality of feature maps corresponding to the M frames of video image.

[0156] In some embodiments, the time sequence feature fusion module 640 is further configured to:

[0157] perform feature fusion on the first fused feature map and the second fused feature map to obtain the first plurality of feature maps, where the first fused feature map is the fused feature map corresponding to the first frame of video image in the M frames of video image, and the second fused feature map is the fused feature map corresponding to the second frame of video image in the M frames of video image.

[0158] In some embodiments, the time sequence feature fusion module 640 is further configured to:

[0159] perform feature fusion on the j-th plurality of feature maps and a (j+2)-th fused feature map to obtain a (j+1)-th plurality of feature maps, where the (j+2)-th fused feature map is the fused feature map corresponding to a (j+2)-th frame of video image in the M frames of video image, and j = 1, 2, …, M-2.

[0160] In some embodiments, the time sequence feature fusion module 640 is further configured to:

[0161] perform feature fusion on the fusion feature maps corresponding to the first L video images in the every M video images to obtain a first plurality of feature maps;

[0162] perform feature fusion on the fusion feature maps corresponding to the first L video images in the every M video images to obtain a first plurality of feature maps;

[0163] perform feature fusion on the fusion feature maps corresponding to the first L video images in the every M video images to obtain a first plurality of feature maps;

[0164] perform feature fusion on the fusion feature maps corresponding to the first L video images in the every M video images to obtain a first plurality of feature maps.

[0165] In some embodiments, L = 2, each feature fusion network in the J+1 feature fusion networks comprises a first convolutional layer and a second convolutional layer, the output of the first convolutional layer is taken as the input of the second convolutional layer, the first convolutional layer is configured to perform convolutional processing on two input feature maps to obtain an output feature map, the second convolutional layer is configured to perform convolutional processing on a difference image between the output feature map and the two feature maps, and the second convolutional layer is configured to output the plurality of feature maps.

[0166] For the first feature fusion network, the two feature maps comprise a first fusion feature map and a second fusion feature map, wherein the first fusion feature map is a fusion feature map corresponding to a first video image in the M video images, and the second fusion feature map is a fusion feature map corresponding to a second video image in the M video images.

[0167] For the j+1 feature fusion network, the two feature maps comprise the jth plurality of feature maps and a j+2 fusion feature map, wherein the j+2 fusion feature map is a fusion feature map corresponding to a j+2 video image in the M video images, and j = 1, 2, …, M-2.

[0168] In some embodiments, the feature extraction module 610 is further configured to:

[0169] The feature extraction network is used for feature extraction on each of the video images to obtain K feature maps corresponding to each of the video images.

[0170] In some embodiments, the feature extraction module 610 is further configured to:

[0171] extract features from the video images to obtain a first feature map;

[0172] perform up-sampling processing on the first feature map to obtain a second feature map, and perform down-sampling processing on the first feature map to obtain a third feature map, wherein the K feature maps corresponding to the video images include the first feature map, the second feature map, and the third feature map.

[0173] In some embodiments, the feature fusion module 620 is further configured to:

[0174] The feature fusion network is used for feature fusion on the K feature maps corresponding to each of the video images to obtain a fusion feature map corresponding to each of the video images.

[0175] In some embodiments, the second obtaining module 650 is further configured to:

[0176] perform feature compression on the multiple feature maps corresponding to each of the M video images to obtain a code stream corresponding to the video sequence.

[0177] In some embodiments, the image compression apparatus 600 further includes

[0178] The communication module is configured to send the code stream corresponding to the video sequence to a decoding end.

[0179] In some embodiments, the image compression apparatus is implemented based on a neural network model, wherein the neural network model includes a feature extraction network, a feature fusion network, a temporal fusion network, and a feature compression network.

[0180] The feature extraction network is used for feature extraction on each of the video images in the multiple video images to obtain K feature maps corresponding to each of the video images.

[0181] The feature fusion network is used for feature fusion on the K feature maps corresponding to each of the video images to obtain a fusion feature map corresponding to each of the video images.

[0182] The temporal fusion network is used for feature fusion on the fusion feature maps corresponding to each of the M video images in the multiple video images to obtain multiple feature maps corresponding to each of the M video images.

[0183] The feature compression network is configured to perform feature compression on the multiple feature maps corresponding to each M frames of video images to obtain a code stream corresponding to the video sequence.

[0184] In some embodiments, the neural network model is trained based on a target loss function, wherein the target loss function is determined according to at least one of a rate loss function and a distortion loss function.

[0185] The rate loss function is represented by the following formula:

[0186]

[0187] wherein, is a latent variable of a fusion feature map output by the feature fusion network, is introduced edge information, represents that the pixel value probability distribution of the latent variable is measured under the condition that the pixel value probability distribution of the latent variable is measured under the condition that the pixel value probability distribution of the latent variable is measured under the condition that the pixel value probability distribution of the latent variable is measured under the condition that the pixel value probability distribution of the latent variable

[0188] The distortion loss function is used to measure the difference between the K feature maps corresponding to the video images output by the feature extraction network and the K feature maps corresponding to the video images reconstructed at the decoding end.

[0189] In some embodiments, the target loss function is determined according to the following formula:

[0190] L total = λ · (D1+D2…+D K )+R

[0191] wherein, L total represents the target loss function, D1 represents the distortion loss function of the first feature map of the video image, D2 represents the distortion loss function of the second feature map of the video image, D K represents the distortion loss function of the Kth feature map of the video image, R represents the rate loss function, and λ is related to the compression rate of the video sequence.

[0192] Therefore, in the embodiment of the present application, the image compression apparatus can obtain K feature maps of different scales corresponding to each video image in the video sequence, then perform feature fusion on the K feature maps to obtain a fused feature map corresponding to the video image, further perform feature fusion on the fused feature maps corresponding to M video images to obtain a plurality of feature maps, and generate a code stream of the video sequence based on the plurality of feature maps. Since the plurality of feature maps eliminate the inter-frame redundant features of the M video images, the code rate can be reduced compared with directly generating a code stream based on the fused feature maps corresponding to the M video images, and since the plurality of feature maps fuse the features of the M video images, target detection based on the plurality of feature maps can ensure detection accuracy.

[0193] The image compression apparatus provided by the embodiment of the present application can implement the method embodiment Figures 2 to 5 The method embodiment implements each process and achieves the same technical effect. To avoid repetition, the details are not described here.

[0194] As shown in Figure 7 The embodiment of the present application also provides an electronic device 800, which includes a processor 801 and a memory 802, and the memory 802 stores programs or instructions executable on the processor 801. For example, when the electronic device 800 is an encoding end device, the programs or instructions are executed by the processor 801 to implement each step of the above-mentioned image compression method embodiment and achieve the same technical effect. Optionally, the memory 802 can be the memory 102 or the memory 113 in the embodiment shown in Figure 1 The processor 801 can implement the functions of the encoder 200 or the decoder 300 in the embodiment shown in Figure 1

[0195] The embodiment of the present application also provides an electronic device, which includes a memory configured to store video data, and a processing circuit configured to implement each step of the above-mentioned image compression method embodiment. Optionally, the memory can be the memory 102 or the memory 113 in the embodiment shown in Figure 1 The processing circuit can implement the functions of the encoder 200 in the embodiment shown in Figure 1

[0196] The embodiment of the present application also provides an electronic device, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps in the method embodiment shown in Figures 2 to 5 The device embodiment corresponds to the above-mentioned method embodiment, and each implementation process and implementation manner of the above-mentioned method embodiment can be applied to the terminal embodiment and achieve the same technical effect.

[0197] ​​The electronic device may be a terminal, or may be other devices other than a terminal, such as a server, a network attached storage (NAS), etc.

[0198] Among them, the terminal can be a mobile phone, tablet personal computer, laptop computer, notebook computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile Internet device (MID), augmented reality (AR), virtual reality (VR) equipment, mixed reality (MR) equipment, robot, wearable device, flight vehicle, vehicle user equipment (VUE), shipborne equipment, pedestrian user equipment (PUE), smart home (home appliances with wireless communication function, such as refrigerator, TV, washing machine or furniture, etc.), game console, personal computer (PC), ATM or self-service machine and other terminal-side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, vehicle-mounted devices can also be called vehicle-mounted terminals, vehicle-mounted controllers, vehicle-mounted modules, vehicle-mounted components, vehicle-mounted chips, or vehicle-mounted units, etc. It should be noted that the specific type of terminal is not limited in the embodiments of this application.

[0199] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), or cloud computing services based on big data and artificial intelligence platforms.

[0200] For example, the electronic devices mentioned above may include but are not limited to Figure 1 The type of source device 100 or destination device 110 shown.

[0201] Taking an electronic device as an example, Figure 8 A hardware structure schematic diagram of a terminal is shown.

[0202] The terminal 900 includes, but is not limited to, at least part of components such as a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910.

[0203] Those skilled in the art can understand that the terminal 900 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 910 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. Figure 8 The terminal structure shown in the figure is not a limitation on the terminal, and the terminal can include more or fewer components than the figure, or combine certain components, or different component arrangements, which are not described here.

[0204] It should be understood that in the embodiments of the present application, the input unit 904 can include a graphics processing unit (GPU) 9041 and a microphone 9042. The graphics processor 9041 processes image data of a still picture or a video obtained by an image acquisition device (such as a camera) in a video acquisition mode or an image acquisition mode, or can process obtained point cloud data. The display unit 906 can include a display panel 9061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 can include a touch detection device and a touch controller. The other input devices 9072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc., which are not described here.

[0205] In the embodiments of the present application, the radio frequency unit 901 can transmit the downlink data received from the network side device to the processor 910 for processing. In addition, the radio frequency unit 901 can send uplink data to the network side device. Generally, the radio frequency unit 901 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

[0206] The memory 909 can be used to store software programs or instructions and various data. The memory 909 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 909 can include a volatile memory or a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 909 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0207] The processor 910 can include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 910.

[0208] In some implementations, the processor 910 is configured to:

[0209] obtain a video sequence, wherein the video sequence includes a plurality of video images;

[0210] perform feature extraction on each video image in the plurality of video images to obtain K feature maps corresponding to each video image, wherein the K feature maps correspond to different scales, and K≥2;

[0211] perform feature fusion on the K feature maps corresponding to each frame of video image to obtain a fused feature map corresponding to each frame of video image;

[0212] perform feature fusion on the fused feature maps corresponding to each M frame of video image in the plurality of frames of video image in sequence to obtain a plurality of frames of feature maps corresponding to each M frame of video image, wherein M≥2;

[0213] obtain a code stream corresponding to the video sequence based on the plurality of frames of feature maps corresponding to each M frame of video image.

[0214] Therefore, in the embodiments of the present application, the terminal can obtain K feature maps of different scales corresponding to each frame of video image in the video sequence, then perform feature fusion on the K feature maps to obtain a fused feature map corresponding to the video image, further perform feature fusion on the fused feature maps corresponding to M frames of video image to obtain a plurality of frames of feature maps, and generate a code stream of the video sequence based on the plurality of frames of feature maps. Since the plurality of frames of feature maps eliminates the inter-frame redundant features of the M frames of video image, the code rate can be reduced compared with directly generating a code stream based on the fused feature maps corresponding to the M frames of video image, and since the plurality of frames of feature maps fuse the features of the M frames of video image, target detection based on the plurality of frames of feature maps can ensure detection accuracy.

[0215] It can be understood that the implementation processes of the implementation manners mentioned in the embodiments can refer to the related descriptions of the method embodiments and achieve the same or corresponding technical effects. To avoid repetition, they will not be described here again.

[0216] The embodiments of the present application also provide a readable storage medium having a program or instructions stored thereon, which is executed by a processor to implement various processes of the above image compression method embodiments and achieve the same technical effects. To avoid repetition, they will not be described here again.

[0217] The processor is the processor in the terminal in the above embodiments. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk, etc. In some examples, the readable storage medium can be a non-transitory readable storage medium.

[0218] The embodiments of the present application also provide a chip including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to run a program or instructions to implement various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, they will not be described here again. Figures 2 to 5 The processor is the processor in the terminal in the above embodiments. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk, etc. In some examples, the readable storage medium can be a non-transitory readable storage medium.

[0219] It should be understood that the chip mentioned in the embodiments of the present application may include a system-level chip (also referred to as a system chip, a chip system or a system-on-chip chip), and may also include an independent display chip, etc.

[0220] The embodiment of the present application further provides a computer program / program product, which is stored in a storage medium and is executed by at least one processor to implement the above Figures 2 to 5 The various processes of the method embodiment shown can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0221] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0222] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general-purpose hardware platform, or of course, by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes a number of instructions for enabling a terminal or network-side device to execute the methods described in each embodiment of the present application.

[0223] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of this application and the scope of protection of the claims. These implementation methods are all within the protection of this application.

Claims

1. An image compression method characterized by, The method comprises the following steps: An encoding end acquires a video sequence, wherein the video sequence comprises multiple video images; The encoding end extracts features of each video image in the multiple video images to acquire K feature maps corresponding to each video image, wherein the K feature maps correspond to different scales, and K≥2; The encoding end fuses the K feature maps corresponding to each video image to acquire a fused feature map corresponding to each video image; The encoding end fuses the fused feature maps corresponding to every M video image in the multiple video images in a time sequence to acquire multiple feature maps corresponding to every M video image, wherein M≥2; The encoding end acquires a code stream corresponding to the video sequence based on the multiple feature maps corresponding to every M video image.

2. The method of claim 1, wherein, The encoding end fuses the fused feature maps corresponding to every M video image in the multiple video images in a time sequence to acquire multiple feature maps corresponding to every M video image, comprising: The encoding end fuses the fused feature maps corresponding to the first L video images in every M video image to acquire a first multiple feature maps, wherein L≥2; The encoding end fuses the fused feature maps corresponding to the jth multiple feature map and the (j+1)th group of video images after the first L video images in a time sequence to acquire a (j+2)th multiple feature map, wherein each group of video images comprises (L-1) video images, j=1, 2, …, J, J=(M-L) / (L-1), and when j=1, the jth multiple feature map is the first multiple feature map; The (J+1)th multiple feature map is determined as the multiple feature maps corresponding to the M video images.

3. The method of claim 2, wherein, When L=2, the encoding end fuses the fused feature maps corresponding to the first L video images in every M video image to acquire a first multiple feature maps, comprising: The encoding end fuses a first fused feature map and a second fused feature map to acquire the first multiple feature maps, wherein the first fused feature map is a fused feature map corresponding to a first video image in the M video images, and the second fused feature map is a fused feature map corresponding to a second video image in the M video images.

4. The method of claim 2, wherein, When L=2, the encoding end fuses the fused feature maps corresponding to the jth multiple feature map and the (j+1)th group of video images after the first L video images in a time sequence to acquire a (j+2)th multiple feature map, comprising: The encoding end fuses the jth multiple feature map and a (j+2)th fused feature map to acquire a (j+1)th multiple feature map, wherein the (j+2)th fused feature map is a fused feature map corresponding to a (j+2)th video image in the M video images, j=1, 2, …, M-2.

5. The method according to any one of claims 1-4, characterized in that, The encoding end fuses the fused feature maps corresponding to every M video image in the multiple video images in a time sequence to acquire multiple feature maps corresponding to every M video image, comprising: The encoding end performs feature fusion on the fusion feature maps corresponding to every M frame of video image in the plurality of frames of video image based on a time sequence fusion network, to obtain a plurality of frames of feature maps corresponding to the every M frame of video image, wherein the time sequence fusion network comprises J+1 level feature fusion networks, and J=(M-L) / (L-1); In the method, the encoding end performs feature fusion on the fusion feature maps corresponding to every M frame of video image in the plurality of frames of video image based on a time sequence fusion network, to obtain a plurality of frames of feature maps corresponding to the every M frame of video image, and the method comprises the following steps: performing feature fusion on the fusion feature maps corresponding to the first L frames of video image in the every M frame of video image through a first level feature fusion network in the J+1 level feature fusion networks, to obtain a first plurality of frames of feature maps; performing feature fusion on the fusion feature maps corresponding to the jth group of video images after the first L frames of video image and the jth plurality of frames of feature maps through a (j+1)th level feature fusion network in the J+1 level feature fusion networks, to obtain a (j+1)th plurality of frames of feature maps, wherein j=1, 2, …, J, and when j=1, the jth plurality of frames of feature maps is the first plurality of frames of feature maps; determining a (J+1)th plurality of frames of feature maps obtained through a (J+1)th level feature fusion network in the J+1 level feature fusion networks as the plurality of frames of feature maps corresponding to the M frame of video image.

6. The method of claim 5, wherein, L=2, each level feature fusion network in the J+1 level feature fusion networks comprises a first convolutional layer and a second convolutional layer, the output of the first convolutional layer is taken as the input of the second convolutional layer, the first convolutional layer is used for performing convolutional processing on two input feature maps to obtain an output feature map, and the second convolutional layer is used for performing convolutional processing on a difference image between the output feature map and the two feature maps, and the second convolutional layer is used for outputting the plurality of frames of feature maps; In the method, the encoding end performs feature fusion on the fusion feature maps corresponding to every M frame of video image in the plurality of frames of video image based on a time sequence fusion network, to obtain a plurality of frames of feature maps corresponding to the every M frame of video image, and the method comprises the following steps: In the first level feature fusion network, the two feature maps comprise a first fusion feature map and a second fusion feature map, wherein the first fusion feature map is a fusion feature map corresponding to a first frame of video image in the M frame of video image, and the second fusion feature map is a fusion feature map corresponding to a second frame of video image in the M frame of video image; 7. The method according to any one of claims 1 to 6, characterized in that, In the (j+1)th level feature fusion network, the two feature maps comprise a jth plurality of frames of feature maps and a (j+2)th fusion feature map, wherein the (j+2)th fusion feature map is a fusion feature map corresponding to a (j+2)th frame of video image in the M frame of video image, and j=1, 2, …, M-2. The method comprises the following steps:

8. The method according to any one of claims 1-7, characterized in that, performing feature compression on the plurality of frames of feature maps corresponding to the every M frame of video image to obtain a code stream corresponding to the video sequence. The image compression method is implemented based on a neural network model, wherein the neural network model comprises a feature extraction network, a feature fusion network, a time sequence fusion network, and a feature compression network; In the method, the encoding end performs feature fusion on the fusion feature maps corresponding to every M frame of video image in the plurality of frames of video image based on a time sequence fusion network, to obtain a plurality of frames of feature maps corresponding to the every M frame of video image, and the method comprises the following steps: the feature extraction network is used for performing feature extraction on every frame of video image in the plurality of frames of video image to obtain K feature maps corresponding to the every frame of video image; The feature fusion network is configured to perform feature fusion on the K feature maps corresponding to each frame of video image to obtain a fused feature map corresponding to each frame of video image. The time sequence feature fusion network is configured to perform feature fusion on the fused feature maps corresponding to each M frame of video image in the plurality of frames of video image based on time sequence to obtain a plurality of frames of feature maps corresponding to each M frame of video image. The feature compression network is configured to perform feature compression on the plurality of frames of feature maps corresponding to each M frame of video image to obtain a code stream corresponding to the video sequence.

9. The method of claim 8, wherein, The neural network model is trained based on a target loss function, wherein the target loss function is determined according to at least one of a rate loss function and a distortion loss function. The rate loss function is represented by the following formula: wherein, is a latent variable of a fusion feature map output by the feature fusion network, is introduced edge information, represents that under the condition that the pixel value probability distribution of the latent variable is the probability distribution of the edge information , and N represents the number of training samples;​ The distortion loss function is configured to measure the difference between the K feature maps output by the feature extraction network and the K feature maps of the video image reconstructed at the decoding end.

10. The method of claim 9, wherein, The target loss function is determined according to the following formula: L total = λ · (D1+D2…+D K )+R wherein L total represents the target loss function, D1 represents a distortion loss function of a first feature map of the video image, D2 represents a distortion loss function of a second feature map of the video image, D K represents a distortion loss function of a Kth feature map of the video image, R represents the rate loss function, and λ is related to a compression rate of the video sequence.

11. An image compression apparatus characterized by comprising: The method comprises the following steps: A first acquisition module is configured to acquire a video sequence, wherein the video sequence comprises a plurality of frames of video image. A feature extraction module is configured to perform feature extraction on each frame of video image in the plurality of frames of video image to obtain K feature maps corresponding to each frame of video image, wherein the K feature maps correspond to different scales, and K≥2. A feature fusion module is configured to perform feature fusion on the K feature maps corresponding to each frame of video image to obtain a fused feature map corresponding to each frame of video image. A time sequence feature fusion module is configured to perform feature fusion on the fused feature maps corresponding to each M frame of video image in the plurality of frames of video image based on time sequence to obtain a plurality of frames of feature maps corresponding to each M frame of video image, wherein M≥2. A second acquisition module is configured to acquire a code stream corresponding to the video sequence based on the plurality of frames of feature maps corresponding to each M frame of video image.

12. The apparatus of claim 11, wherein, The time sequence feature fusion module is further configured to: perform feature fusion on the fused feature maps corresponding to the first L frames of video image in the each M frame of video image to obtain a first plurality of frames of feature maps, wherein L≥2; perform feature fusion on the fused feature maps corresponding to each group of video image after the first L frames of video image based on time sequence to obtain a j+1 plurality of frames of feature maps, wherein each group of video image comprises (L-1) frames of video image, j=1, 2, …, J, J=(M-L) / (L-1), and when j=1, the j plurality of frames of feature maps is the first plurality of frames of feature maps; determine a J+1 plurality of frames of feature maps as the plurality of frames of feature maps corresponding to the M frame of video image.

13. The apparatus of claim 11 or 12, wherein, The time sequence feature fusion module is further configured to: perform feature fusion on the fused feature maps corresponding to each M frame of video image in the plurality of frames of video image based on a time sequence fusion network to obtain a plurality of frames of feature maps corresponding to each M frame of video image, wherein the time sequence fusion network comprises J+1 levels of feature fusion network, J=(M-L) / (L-1). The first multi-frame feature map is obtained by performing feature fusion on the fusion feature maps corresponding to the first L video images in each M video images through a first feature fusion network in the J+1-level feature fusion network. The first multi-frame feature map is obtained by performing feature fusion on the fusion feature maps corresponding to the first L video images in each M video images through a first feature fusion network in the J+1-level feature fusion network. The first multi-frame feature map is obtained by performing feature fusion on the fusion feature maps corresponding to the first L video images in each M video images through a first feature fusion network in the J+1-level feature fusion network.

14. The apparatus of claim 13, wherein, L=2, each feature fusion network in the J+1-level feature fusion network includes a first convolutional layer and a second convolutional layer, the output of the first convolutional layer is taken as the input of the second convolutional layer, the first convolutional layer is used for performing convolutional processing on two input feature maps to obtain an output feature map, and the second convolutional layer is used for performing convolutional processing on a difference image between the output feature map and the two feature maps, and the second convolutional layer is used for outputting the multi-frame feature map. For the first feature fusion network, the two feature maps include a first fusion feature map and a second fusion feature map, wherein the first fusion feature map is a fusion feature map corresponding to a first video image in the M video images, and the second fusion feature map is a fusion feature map corresponding to a second video image in the M video images. For the j+1-level feature fusion network, the two feature maps include the j multi-frame feature map and a j+2 fusion feature map, wherein the j+2 fusion feature map is a fusion feature map corresponding to a j+2 video image in the M video images, and j=1, 2, …, M-2.

15. An electronic device, comprising: The computer program is stored in the storage medium, and the computer program is executed by the at least one processor to implement the steps of the image compression method according to any one of claims 1-10. The computer program is stored in the storage medium, and the computer program is executed by the at least one processor to implement the steps of the image compression method according to any one of claims 1-10.

16. A readable storage medium, characterized by, The computer program is stored in the storage medium, and the computer program is executed by the at least one processor to implement the steps of the image compression method according to any one of claims 1-10.

17. A chip, characterized by The computer program is stored in the storage medium, and the computer program is executed by the at least one processor to implement the steps of the image compression method according to any one of claims 1-10.

18. A computer program, characterized in that, ​

Images