Image processing method and terminal

Abstract

The embodiment of the present application provides an image processing method and terminal, which belongs to the field of image compression and decompression technology. The above-mentioned image processing method includes: the decoding end obtains the target code stream; the target code stream includes a first code stream and a second code stream; the decoding end decodes the first code stream through a compression prediction network to obtain N reconstructed feature maps, and decodes the second code stream through a residual network to obtain N residual feature maps, the N first feature maps correspond one-to-one to the N residual feature maps, the residual feature maps are used to characterize the residual between the corresponding reconstructed feature maps and the original feature maps, the loss function of the residual network is determined based on the loss generated by the visual detection task, and N is a positive integer greater than or equal to 2; the decoding end determines N first target feature maps based on the N reconstructed feature maps and the N residual feature maps.

Description

Technical Field

[0001] The present application belongs to the technical field of image compression and decompression, and specifically relates to an image processing method and terminal. Background Art

[0002] Machine vision inspection tasks typically rely on multi-scale architectures, such as the Mask Region-based Convolutional Neural Network (Mask R-CNN) architecture, to extract multiple feature maps corresponding to the original image. These feature maps are then compressed to produce a bitstream, which is then decoded to reconstruct feature maps. Visual inspection of the image is then performed based on the reconstructed feature maps. However, the accuracy of the detection results for these visual inspection tasks is relatively low. Summary of the Invention

[0003] The embodiments of the present application provide an image processing method and terminal, which can solve the technical problem of low accuracy of detection results in visual detection tasks.

[0004] In a first aspect, an image processing method is provided, comprising:

[0005] The decoding end obtains a target code stream; the target code stream includes a first code stream and a second code stream;

[0006] The decoding end decodes the first code stream through the compression prediction network to obtain N reconstructed feature maps, and decodes the second code stream through the residual network to obtain N residual feature maps, the N first feature maps and the N residual feature maps Figure 1 One-to-one, the residual feature map is used to represent the residual between the corresponding reconstructed feature map and the original feature map. The loss function of the residual network is determined based on the loss generated by the visual detection task, and N is a positive integer greater than or equal to 2;

[0007] The decoding end determines N first target feature maps based on the N reconstructed feature maps and the N residual feature maps.

[0008] In a second aspect, an image processing method is provided, comprising:

[0009] The encoder obtains N original feature maps corresponding to the image, where N is a positive integer greater than or equal to 2;

[0010] The encoding end compresses the third target feature map through a compression prediction network to obtain a first bitstream; the third target feature map is a partial feature map of the N original feature maps;

[0011] The encoder determines a second bitstream based on the residual network, the N original feature maps, and the first bitstream;

[0012] The encoding end generates a target bitstream based on the first bitstream and the second bitstream.

[0013] According to a third aspect, an image processing apparatus is provided, comprising:

[0014] An acquisition module, configured to acquire a target code stream; the target code stream includes a first code stream and a second code stream;

[0015] A decoding module is configured to decode the first code stream through a compression prediction network to obtain N reconstructed feature maps, and decode the second code stream through a residual network to obtain N residual feature maps, wherein the N first feature maps and the N residual feature maps are Figure 1 One-to-one, the residual feature map is used to represent the residual between the corresponding reconstructed feature map and the original feature map. The loss function of the residual network is determined based on the loss generated by the visual detection task, and N is a positive integer greater than or equal to 2;

[0016] A determination module is used to determine N first target feature maps based on the N reconstructed feature maps and the N residual feature maps.

[0017] In a fourth aspect, an image processing apparatus is provided, comprising:

[0018] An acquisition module is used to obtain N original feature maps corresponding to the image, where N is a positive integer greater than or equal to 2;

[0019] A first encoding module is configured to compress a third target feature map through a compression prediction network to obtain a first bitstream; the third target feature map is a partial feature map of the N original feature maps;

[0020] A second encoding module, configured to determine a second bitstream based on the residual network, the N original feature maps, and the first bitstream;

[0021] A generating module is configured to generate a target code stream based on the first code stream and the second code stream.

[0022] In a fifth aspect, a terminal is provided, which includes a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0023] In a sixth aspect, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented.

[0024] In the seventh aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect, or to implement the method described in the second aspect.

[0025] In an eighth aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium, and the computer program / program product is executed by at least one processor to implement the steps of the method described in the first aspect, or to implement the steps of the method described in the second aspect.

[0026] In an embodiment of the present application, a target bitstream is obtained; N reconstructed feature maps are obtained by decoding the first bitstream using a compression prediction network, and N residual feature maps are obtained by decoding the second bitstream using a residual network; and N first target feature maps are determined based on the N reconstructed feature maps and the N residual feature maps. In an embodiment of the present application, N residual feature maps are obtained by decoding the second bitstream using a residual network. The loss function of the residual network is determined based on the loss generated by a subsequent visual inspection task. The N reconstructed feature maps are then padded with the N residual feature maps to determine N first target feature maps, thereby reducing distortion of the first target feature maps and improving the accuracy of the detection results of the subsequent visual inspection task. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is one of the flowcharts of the image processing method provided in the embodiment of the present application;

[0028] Figure 2 Schematic diagram of the structure of the compression prediction network provided by the embodiment of the present application;

[0029] Figure 3 This is one of the structural diagrams of the residual network provided in the embodiment of the present application;

[0030] Figure 4 This is the second structural diagram of the residual network provided in the embodiment of the present application;

[0031] Figure 5 This is the second flowchart of the image processing method provided in the embodiment of the present application;

[0032] Figure 6 This is a schematic diagram of the application process of the image processing method provided in the embodiment of the present application;

[0033] Figure 7 This is one of the structural diagrams of the image processing device provided in the embodiment of the present application;

[0034] Figure 8 This is the second structural diagram of the image processing device provided in the embodiment of the present application;

[0035] Figure 9 is a structural diagram of a communication device provided in an embodiment of the present application;

[0036] Figure 10 This is a schematic diagram of the hardware structure of the terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The following will be combined with the accompanying drawings in the embodiments of this application to clearly describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0038] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0039] The term "indication" in this application can be either a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, the operation to be performed, or the requested result, etc. in the instruction sent; an indirect indication can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the operation to be performed or the requested result, etc. based on the judgment result.

[0040] The image processing device corresponding to the image processing method in the embodiment of the present application may be a terminal, which may also be referred to as a terminal device or a user terminal (UE). The terminal may be a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a notebook computer, a personal digital assistant (PDA), a handheld computer, a netbook, an ultra-mobile personal computer (UMPC), a mobile Internet device (MID), an augmented reality (AR), a virtual reality (VR) device, a robot, a wearable device (Wearable Device), an aircraft (flight vehicle), a vehicle user equipment (VUE), a shipborne device, a pedestrian user equipment (PUE), a smart home (home appliance with wireless communication function, such as a refrigerator, a television, a washing machine or furniture, etc.), a game console, a personal computer (PC), an ATM or a self-service machine, and other terminal-side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, vehicle-mounted devices can also be called vehicle-mounted terminals, vehicle-mounted controllers, vehicle-mounted modules, vehicle-mounted components, vehicle-mounted chips, or vehicle-mounted units, etc. It should be noted that the specific type of terminal is not limited in the embodiments of this application.

[0041] The present application provides an image processing method, which can be applied to a decoding end. The image processing method provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings through some embodiments and their application scenarios.

[0042] See also Figure 1 , Figure 1 This is one of the flow charts of the image processing method provided in the embodiment of the present application. The image processing method provided in this embodiment includes the following steps:

[0043] S101: The decoding end obtains the target bit stream.

[0044] In this step, the decoding end obtains the target code stream transmitted by the encoding end, and the above-mentioned target code stream includes a first code stream and a second code stream, wherein the first code stream is the encoding result of the reconstructed feature map, and the second code stream is the encoding result of the residual feature map.

[0045] S102: The decoding end decodes the first bit stream through a compression prediction network to obtain N reconstructed feature maps, and decodes the second bit stream through a residual network to obtain N residual feature maps.

[0046] In this step, the first bit stream can be decoded by the compression prediction network to obtain N reconstructed feature maps; the second bit stream can be decoded by the residual network to obtain N residual feature maps, where the N first feature maps and the N residual feature maps are Figure 1 One-to-one, the residual feature map is used to represent the residual between the corresponding reconstructed feature map and the original feature map. The loss function of the residual network is determined based on the loss generated by the visual detection task, and N is a positive integer greater than or equal to 2.

[0047] S103: The decoding end determines N first target feature maps based on the N reconstructed feature maps and the N residual feature maps.

[0048] In this step, the N reconstructed feature maps can be added to the corresponding N residual feature maps to obtain N first target feature maps. Optionally, the above steps can be performed in a residual network. In subsequent steps, visual detection of the image is performed based on the N first target feature maps.

[0049] Optionally, the above-mentioned visual detection task may be performing target detection on an image synthesized from N first target feature maps using a Fast Region-Convolutional Neural Network (Fast R-CNN).

[0050] In an embodiment of the present application, N residual feature maps are obtained by decoding the second code stream through a residual network. The loss function of the above-mentioned residual network is determined based on the loss generated by the subsequent visual detection task, and then the N reconstructed feature maps are respectively complemented by the N residual feature maps to determine N first target feature maps, thereby reducing the distortion of the first target feature map and improving the accuracy of the detection results of the subsequent visual detection task.

[0051] As a possible implementation, decoding the first code stream through a compression prediction network to obtain N reconstructed feature maps includes:

[0052] The decoding end performs arithmetic decoding and inverse quantization processing on the first bit stream to obtain a reconstructed first latent variable;

[0053] The decoding end decodes the reconstructed first latent variable through the N first decoders included in the preset compression prediction network to obtain the N reconstructed feature maps; the N first decoders and the N reconstructed feature maps Figure 1 One to one correspondence, and the scale of the reconstructed feature map output by each first decoder is different.

[0054] See also Figure 2 ,like Figure 2 As shown, the compression prediction network includes 4 first decoders, that is, the value of N is 4. At the decoding end, the first bit stream is arithmetic decoded and dequantized to obtain the reconstructed first latent variable (i.e. Figure 2 After C1′ in ( ), the reconstructed first latent variables are decoded by four first decoders respectively to obtain four corresponding reconstructed feature maps. Wherein, the latent variables can be understood as the variables obtained after compressing the feature maps.

[0055] It should be noted that in Figure 2 In the application scenario shown, the scales of the reconstructed feature maps output by the four first decoders in the compression prediction network are different, that is, the number of upsampling times corresponding to each first decoder is different. Since the sizes of the reconstructed feature map P5', the reconstructed feature map P4', the reconstructed feature map P3' and the reconstructed feature map P2' increase successively, the number of upsampling times of the first decoders corresponding to these four reconstructed feature maps increases successively.

[0056] As a possible implementation, Figure 2 The compression prediction network shown is a Reversed Single-Input-Multi-Output (RSIMO) network.

[0057] As another possible implementation, decoding the first code stream through a compression prediction network to obtain N reconstructed feature maps includes:

[0058] The decoding end performs arithmetic decoding and inverse quantization processing on the first bit stream to obtain a reconstructed first latent variable;

[0059] The decoding end decodes the reconstructed first latent variable through a first decoder included in a preset compression prediction network to obtain M reconstructed feature maps;

[0060] The decoding end downsamples or upsamples one or more of the M reconstructed feature maps to obtain NM reconstructed feature maps, where M is an integer greater than or equal to 1 and less than N.

[0061] M reconstructed feature maps are obtained through a first decoder, and then upsampling or downsampling is performed according to one or more of the M reconstructed feature maps to obtain NM reconstructed feature maps, that is, N reconstructed feature maps are obtained.

[0062] Optionally, when M is 1, the M reconstructed feature maps are the feature maps with the smallest resolution among the N reconstructed feature maps. For example, when M is 1, a first decoder is used to decode the reconstructed first latent variable to obtain a reconstructed feature map P5', and then the reconstructed feature map P5' is upsampled to obtain a reconstructed feature map P2', a reconstructed feature map P3' and a reconstructed feature map P4'. The upsampling coefficients can be 8, 4 and 2 respectively, and the upsampling method can be bilinear interpolation (bilinear) or the like, which is not specifically limited in this application. The reconstructed feature map P5' is the feature map with the smallest resolution among the reconstructed feature map P2', the reconstructed feature map P3', the reconstructed feature map P4' and the reconstructed feature map P5'.

[0063] Optionally, when M is 2, the M reconstructed feature maps are the feature map with the smallest resolution and the feature map with the largest resolution among the N reconstructed feature maps. For example, when M is 2, a first decoder is used to decode the reconstructed first latent variable to obtain a reconstructed feature map P5' and a reconstructed feature map P2', and then the reconstructed feature map P5' is upsampled to obtain a reconstructed feature map P4', and the reconstructed feature map P2' is downsampled to obtain a reconstructed feature map P3'.

[0064] In the embodiment of the present application, higher video compression efficiency can be achieved through the above two possible implementation methods, especially the second method. The residual correlation of the reconstructed feature map obtained after upsampling or downsampling is high, so the compression efficiency of the subsequent residual network is also correspondingly high.

[0065] Optionally, the second code stream is an encoding result of the N residual feature maps, and the residual network includes one decoder;

[0066] The step of decoding the second code stream through a residual network to obtain N residual feature maps includes:

[0067] The decoding end performs arithmetic decoding on the second code stream to obtain a reconstructed second latent variable;

[0068] The decoding end decodes the reconstructed second latent variable through a decoder included in the residual network to obtain the N residual feature maps.

[0069] Optionally, the residual network includes a decoder, and the second code stream is the encoding result of N residual feature maps. Figure 3 ,like Figure 3As shown, the residual network includes a decoder, which performs arithmetic decoding on the second bit stream at the decoding end to obtain the reconstructed second latent variable (i.e. Figure 3 After that, the reconstructed second latent variable is decoded by the decoder to obtain four residual feature maps, and the four residual feature maps are added to the four reconstructed feature maps P2' to P5' to obtain the first target feature map.

[0070] The size of the feature map input to the encoder in the residual network is the same as the size of the feature map output by the decoder in the residual network. This residual network can generalize multi-layer features to reduce the number of parameters in the residual network.

[0071] It should be noted that, in the residual network, the step of adding N residual feature maps to the corresponding reconstructed feature maps to obtain N first target feature maps can also be performed.

[0072] Optionally, the second bitstream is an encoding result of a second target feature map, the second target feature map is a partial feature map of the N residual feature maps, the residual network includes N second decoders, and the scale of the residual feature map output by each second decoder is different;

[0073] The step of decoding the second code stream through a residual network to obtain N residual feature maps includes:

[0074] The decoding end performs arithmetic decoding on the second code stream to obtain a reconstructed second latent variable;

[0075] The decoding end decodes the reconstructed second latent variable through N second decoders included in the residual network to obtain the N residual feature maps.

[0076] Optionally, the residual network includes N second decoders, the second bitstream is the encoding result of the second target feature map, and the second target feature map is a partial feature map of the N residual feature maps. Optionally, the second target feature map is a feature map with the highest resolution among the N residual feature maps, or the second target feature map is a feature map with the second highest resolution among the N residual feature maps.

[0077] See also Figure 4 ,like Figure 4 As shown, the residual network includes four second decoders, which perform arithmetic decoding on the second bit stream at the decoding end to obtain the reconstructed second latent variable (i.e. Figure 4 After that, the reconstructed second latent variable is decoded by four second decoders to obtain four residual feature maps. The four residual feature maps are added to the reconstructed feature maps P2' to P5' to obtain four first target feature maps.

[0078] It should be noted that in Figure 4 The scales of the residual feature maps output by the four second decoders in the residual network shown are different, that is, the number of downsampling times corresponding to each second decoder is different.

[0079] In this embodiment, the second decoder in the residual network only needs to decode some of the N residual feature maps, thereby reducing the workload of the decoder and improving decoding efficiency.

[0080] As a possible implementation, Figure 4 The residual network shown is a Single-Input-Multi-Output (SIMO) network.

[0081] Optionally, the loss function of the compression prediction network is determined based on a weighted sum of a first rate loss function and N distortion loss functions.

[0082] When N is 4, that is, when the first bitstream is decoded by the compression prediction network to obtain the reconstructed feature map P2', the reconstructed feature map P3', the reconstructed feature map P4' and the reconstructed feature map P5', the loss function of the compression prediction network can be determined by the following formula (1):

[0083] L MSE =λ1·(D2+D3+D4+D5)+R1 (1)

[0084] Among them, D2 represents the distortion loss function associated with the reconstruction feature map P2', D3 represents the distortion loss function associated with the reconstruction feature map P3', D4 represents the distortion loss function associated with the reconstruction feature map P4', D5 represents the distortion loss function associated with the reconstruction feature map P5', R1 represents the first rate loss function, λ1 is the weight parameter used to balance the feature map distortion and the encoding and decoding rate, L MSE Represents the loss function of the compressed prediction network.

[0085] Optionally, the N distortion loss functions and the N reconstruction features Figure 1 One to one correspondence, and the distortion loss function is determined based on the loss between the corresponding reconstructed feature map and the original feature map; the first rate loss function is determined based on edge information and the reconstructed first latent variable.

[0086] When N is 4, that is, when the first code stream is decoded by the compression prediction network to obtain the reconstructed feature map P2', the reconstructed feature map P3', the reconstructed feature map P4' and the reconstructed feature map P5', the distortion loss function associated with each feature map can be determined by the following formula:

[0087]

[0088]

[0089]

[0090]

[0091] Among them, D2 represents the distortion loss function associated with the reconstructed feature map P2', D3 represents the distortion loss function associated with the reconstructed feature map P3', D4 represents the distortion loss function associated with the reconstructed feature map P4', D5 represents the distortion loss function associated with the reconstructed feature map P5', the l2 parameter is used to represent the loss between the original feature map and the reconstructed feature map, and N is the number of training samples.

[0092] Alternatively, the first rate loss function may be determined by the following formula (2):

[0093]

[0094] Where R1 represents the first rate loss function, N is the number of training samples, For edge information, is the first latent variable for reconstruction, Indicates The first latent variable reconstructed under the condition The probability distribution of pixel values, Representing edge information The image is divided into edges, corners, and regions. Edge information is the information on the edge of the image. Edge information is one of the important features in depth images and can be used for tasks such as target detection, scene segmentation, and object tracking.

[0095] Optionally, the loss function of the residual network is determined based on a weighted sum of a second rate loss function and a task loss function.

[0096] Alternatively, the loss function of the residual network can be determined by the following formula (3):

[0097] L Task =λ2·T+R2 (3)

[0098] Among them, L Task Represents the loss function of the residual network, λ2 is the weight parameter used to balance the loss of the visual detection task and the encoding and decoding rate, R2 represents the second rate loss function, and T represents the task loss function.

[0099] Optionally, the task loss function is determined based on the loss generated by visual detection of the N reconstructed feature maps, and the second rate loss function is determined based on edge information and a reconstructed second latent variable.

[0100] Alternatively, the task loss function can be expressed by the following formula (4):

[0101] T=L cls +L box +L mask (4)

[0102] Among them, T represents the task loss function, L cls Represents the classification value of the bounding box in the visual detection task, L box Represents the regression loss value of the bounding box in the visual detection task, L mask Represents the loss value of the mask part in the visual detection task.

[0103] Alternatively, the second rate loss function may be determined by the following formula (5):

[0104]

[0105] Where R2 represents the first rate loss function, N is the number of training samples, For edge information, is the second latent variable for reconstruction, Indicates The second latent variable reconstructed under the condition The probability distribution of pixel values, Representing edge information The probability distribution of .

[0106] Optionally, the network structure of the compression prediction network can be selected and designed according to actual needs. For example, the network structure of the compression prediction network is a Balle network structure or a Cheng network structure.

[0107] Optionally, the network structure of the residual network can be the same as the network structure of the compression prediction network, or can be selected and designed according to actual needs. For example, the network structure of the residual network is a Balle network structure or a Cheng network structure.

[0108] The training process of the above-mentioned compression prediction network can be as follows: inputting training samples into the compression prediction network to obtain the latent variables and reconstructed feature maps output by the compression prediction network. The rate loss function is adjusted according to the latent variables, the number of training samples, and the edge information. The distortion loss function is adjusted according to the number of samples and the loss between the original feature map and the reconstructed feature map. The compression prediction network is then adjusted based on the above-mentioned rate loss function and distortion loss function. When the loss function value represented by the loss function of the compression prediction network is less than or equal to a preset threshold, the compression prediction network training is determined to be completed.

[0109] The training process of the aforementioned network may include freezing the network parameters of the compressed prediction network after training of the compressed prediction network is completed, and then iteratively training the residual network. During the iterative training of the residual network, the output of the residual network, i.e., the first target feature map, may be input into the visual inspection network for evaluation of the visual inspection task, and the residual network may be iteratively optimized based on the task loss function.

[0110] It should be understood that the loss function of the residual network can randomly train single-layer features during each iterative training, thereby increasing the training speed of the residual network and improving the performance of subsequent visual inspection tasks.

[0111] It should be understood that after the compression prediction network training is completed, the network parameters in the compression prediction network are frozen, and then the residual network is iteratively trained to achieve segmented training of the compression prediction network and the residual network, avoiding the correlation between the two networks during the iterative training process, thereby improving the performance of the encoding and decoding ends.

[0112] See also Figure 5 , Figure 5 This is the second flow chart of the image processing method provided in the embodiment of the present application. The image processing method provided in this embodiment includes the following steps:

[0113] S501: The encoder obtains N original feature maps corresponding to the image.

[0114] In this step, the encoder obtains N original feature maps, where N is a positive integer greater than or equal to 2.

[0115] In an optional embodiment, the N original feature maps may be obtained by performing a feature map extraction operation on the image. Specifically, four original feature maps corresponding to the image may be extracted using Feature Pyramid Networks (FPN).

[0116] In other optional embodiments, multiple original feature maps corresponding to the image can also be extracted through a convolutional neural network. The specific implementation method of obtaining N original feature maps corresponding to the image is not limited here.

[0117] The resolution of each original feature map is different, or the image features contained in each original feature map are different.

[0118] S502: The encoding end compresses the third target feature map through a compression prediction network to obtain a first bit stream.

[0119] In this step, some of the N original feature maps can be determined as third target feature maps, and the third target feature maps are compressed by the compression prediction network to obtain the first bitstream. For specific implementation methods, please refer to the subsequent embodiments.

[0120] S503: The encoder determines a second bitstream based on the residual network, the N original feature maps, and the first bitstream.

[0121] In this step, the second bitstream can be determined based on the residual network, the N original feature maps, and the first bitstream. For specific implementation methods, please refer to the subsequent embodiments.

[0122] S504: The encoding end generates a target bitstream based on the first bitstream and the second bitstream.

[0123] In this step, after obtaining the first code stream and the second code stream, the encoder can mix the first code stream and the second code stream to generate a target code stream.

[0124] In this embodiment, compared with the method in the related art of compressing all the original feature maps corresponding to the image to obtain the target code stream, the embodiment of the present application only needs to compress part of the original feature maps to obtain the target code stream, thereby reducing the bit rate of the encoded code stream.

[0125] Optionally, compressing the third target feature map through a compression prediction network to obtain a first bitstream includes:

[0126] The encoder performs downsampling processing on the third target feature map through a compression prediction network;

[0127] The encoder end compresses the third target feature map after downsampling through the compression prediction network, obtaining a first latent variable corresponding to the third target feature map;

[0128] The encoding end quantizes the first latent variable and performs arithmetic coding on the quantized first latent variable to obtain the first code stream.

[0129] Optionally, the original feature map with the lowest resolution among the N original feature maps may be determined as the third target feature map.

[0130] Specifically, in the implementation scenario of extracting four original feature maps corresponding to an image through Feature Pyramid Networks (FPN), please refer to Table 1:

[0131] Table 1:

[0132] Original feature map Size ratio Mean Average Precision P2 64 74.22％ P3 16 76.29％ P4 4 18.38％ P5 1 30.50％

[0133] From Table 1, we can see that the average precision corresponding to the original feature maps P4 and P5 with smaller sizes (i.e., lower resolution) is lower, which means that the original feature maps P4 and P5 have a greater impact on the performance of subsequent visual detection tasks.

[0134] Optionally, the original feature map with the lowest resolution can be determined as the third target feature map, the compression prediction network compresses the third target feature map, and in subsequent steps, N reconstructed feature maps are predicted based on the compression results of the third target feature map. For the encoding end, only the third target feature map needs to be compressed to obtain the first code stream, thereby saving the code rate for compressing other original feature maps with larger resolutions.

[0135] In other implementations, one original feature map may be randomly selected from two original feature maps with lower resolutions and determined as the third target feature map.

[0136] Specifically, the above compression process is as follows: the encoding end downsamples the third target feature map through the compression prediction network to obtain the first latent variable, quantizes the above first latent variable, and arithmetically encodes the quantized first latent variable to obtain the first code stream.

[0137] For easier understanding, see Figure 2 ,in, Figure 2 P5 in represents the third target feature map, and C1 represents the first latent variable.

[0138] Optionally, determining the second bitstream according to the residual network, the N original feature maps, and the first bitstream includes:

[0139] The encoding end decodes the first code stream through a compression prediction network to obtain N reconstructed feature maps, and the N reconstructed feature maps are compared with the N original feature maps. Figure 1 One to one correspondence;

[0140] The encoding end obtains N residual feature maps according to the N original feature maps and the N reconstructed feature maps;

[0141] The encoding end encodes the N residual feature maps through a residual network to obtain the second code stream.

[0142] In this embodiment, after obtaining the first bit stream, the compression prediction network can decode the first bit stream to obtain N reconstructed feature maps. As described above, the N reconstructed feature maps are compared with the N original feature maps. Figure 1 Then, the N original feature maps are subtracted from the corresponding N reconstructed feature maps to obtain N residual feature maps. As mentioned above, the residual feature maps are used to represent the residual between the corresponding reconstructed feature maps and the original feature maps.

[0143] Specifically, if Figure 3 As shown, the residual feature map is encoded by the encoder of the residual network to obtain the second latent variable, that is, Figure 3 C2 in , performs arithmetic coding on the first latent variable to obtain a second code stream.

[0144] Optionally, determining the second bitstream according to the residual network, the N original feature maps, and the first bitstream includes:

[0145] The encoding end decodes the first code stream through a compression prediction network to obtain N reconstructed feature maps, and the N reconstructed feature maps are compared with the N original feature maps. Figure 1 One to one correspondence;

[0146] The encoding end obtains N residual feature maps according to the N original feature maps and the N reconstructed feature maps;

[0147] The encoding end encodes the second target feature map through a residual network to obtain the second code stream, where the second target feature map is a partial feature map of the N residual feature maps.

[0148] In this embodiment, after obtaining N residual feature maps, the feature map with the highest resolution among the N residual feature maps is determined as the second target feature map, and the above second target feature map is encoded by the encoder of the residual network to obtain a second code stream.

[0149] In this embodiment, the encoder in the residual network only needs to encode some of the N residual feature maps to generate a second bit stream, thereby reducing the bit rate of the second bit stream.

[0150] For example, in Figure 4 In the application scenario shown, the residual feature map obtained by subtracting the original feature map P2 from the reconstructed feature map P2' is determined as the second target feature map, and Figure 4 The encoder of the residual network shown encodes the second target feature map to obtain a second code stream.

[0151] To understand the overall technical solution, please refer to Figure 6 ,exist Figure 6 In the application scenario shown, the original feature map P5 is transmitted to the compression prediction network to obtain the reconstructed feature maps P2' to P5', the original feature maps P2 to P5 are subtracted from the reconstructed feature maps P2' to P5' to obtain the residual feature map, the above residual feature map is input into the prediction network, and the reconstructed feature maps P2' to P5' are added to the output of the residual network to obtain the first target feature map.

[0152] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0153] like Figure 7 As shown, the embodiment of the present application further provides an image processing device 700, including:

[0154] An acquisition module 701 is configured to acquire a target bitstream, wherein the target bitstream includes a first bitstream and a second bitstream.

[0155] The decoding module 702 is configured to decode the first code stream through the compression prediction network to obtain N reconstructed feature maps, and decode the second code stream through the residual network to obtain N residual feature maps, wherein the N first feature maps and the N residual feature maps are Figure 1 One-to-one, the residual feature map is used to represent the residual between the corresponding reconstructed feature map and the original feature map. The loss function of the residual network is determined based on the loss generated by the visual detection task, and N is a positive integer greater than or equal to 2;

[0156] The determination module 703 is configured to determine N first target feature maps based on the N reconstructed feature maps and the N residual feature maps.

[0157] Optionally, the decoding module 702 is configured to decode the first code stream through a compression prediction network to obtain N reconstructed feature maps, including:

[0158] The decoding module 702 is configured to perform arithmetic decoding and inverse quantization on the first bit stream to obtain a reconstructed first latent variable; decode the reconstructed first latent variable through the N first decoders included in the preset compression prediction network to obtain the N reconstructed feature maps; the N first decoders and the N reconstructed feature maps are connected. Figure 1 One to one correspondence, and the scale of the reconstructed feature map output by each first decoder is different.

[0159] Optionally, the decoding module 702 is configured to decode the first code stream through a compression prediction network to obtain N reconstructed feature maps, including:

[0160] The decoding module 702 is used to perform arithmetic decoding and inverse quantization on the first code stream to obtain a reconstructed first latent variable; decode the reconstructed first latent variable through a first decoder included in a preset compression prediction network to obtain M reconstructed feature maps; downsample or upsample one or more of the M reconstructed feature maps to obtain NM reconstructed feature maps, where M is an integer greater than or equal to 1 and less than N.

[0161] Optionally, the second code stream is an encoding result of the N residual feature maps, and the residual network includes one decoder;

[0162] The decoding module 702 is configured to decode the second bitstream using a residual network to obtain N residual feature maps, including:

[0163] The decoding module 702 is configured to perform arithmetic decoding on the second bitstream to obtain a reconstructed second latent variable;

[0164] The reconstructed second latent variable is decoded by a decoder included in the residual network to obtain the N residual feature maps.

[0165] Optionally, the second bitstream is an encoding result of a second target feature map, the second target feature map is a partial feature map of the N residual feature maps, the residual network includes N second decoders, and the scale of the residual feature map output by each second decoder is different;

[0166] The decoding module 702 is configured to decode the second bitstream through a residual network to obtain N residual feature maps, including:

[0167] The decoding module 702 performs arithmetic decoding on the second code stream to obtain a reconstructed second latent variable; and decodes the reconstructed second latent variable through the N second decoders included in the residual network to obtain the N residual feature maps.

[0168] Optionally, the loss function of the compression prediction network is determined based on a weighted sum of a first rate loss function and N distortion loss functions.

[0169] Optionally, the N distortion loss functions and the N reconstruction features Figure 1 One to one correspondence, and the distortion loss function is determined based on the loss between the corresponding reconstructed feature map and the original feature map; the first rate loss function is determined based on edge information and the reconstructed first latent variable.

[0170] Optionally, the loss function of the residual network is determined based on a weighted sum of a second rate loss function and a task loss function.

[0171] Optionally, the task loss function is determined based on the loss generated by visual detection of the N reconstructed feature maps, and the second rate loss function is determined based on edge information and a reconstructed second latent variable.

[0172] In an embodiment of the present application, N residual feature maps are obtained by decoding the second code stream through a residual network. The loss function of the above-mentioned residual network is determined based on the loss generated by the subsequent visual detection task, and then the N reconstructed feature maps are respectively complemented by the N residual feature maps to determine N first target feature maps, thereby reducing the distortion of the first target feature map and improving the accuracy of the detection results of the subsequent visual detection task.

[0173] This device embodiment is similar to the above Figure 1 Corresponding to the embodiment of the image processing method shown, each implementation process and implementation method of the decoding end in the above method embodiment can be applied to the device embodiment and can achieve the same technical effect.

[0174] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0175] like Figure 8 As shown, the embodiment of the present application further provides an image processing device 800, including:

[0176] An acquisition module 801 is configured to acquire N original feature maps corresponding to an image, where N is a positive integer greater than or equal to 2.

[0177] A first encoding module 802 is configured to compress a third target feature map through a compression prediction network to obtain a first bitstream; the third target feature map is a partial feature map of the N original feature maps;

[0178] A second encoding module 803 is configured to determine a second bitstream based on the residual network, the N original feature maps, and the first bitstream;

[0179] The generating module 804 is configured to generate a target bitstream based on the first bitstream and the second bitstream.

[0180] Optionally, the first encoding module 802 is specifically configured to:

[0181] Downsampling the third target feature map through a compression prediction network;

[0182] Obtaining a first latent variable corresponding to the third target feature map by compressing and downsampling the third target feature map through the compression prediction network;

[0183] The first latent variable is quantized, and arithmetic coding is performed on the quantized first latent variable to obtain the first code stream.

[0184] Optionally, the second encoding module 803 is specifically configured to:

[0185] The first code stream is decoded by a compression prediction network to obtain N reconstructed feature maps, and the N reconstructed feature maps are compared with the N original feature maps. Figure 1 One to one correspondence;

[0186] Obtaining N residual feature maps according to the N original feature maps and the N reconstructed feature maps;

[0187] The N residual feature maps are encoded through a residual network to obtain the second code stream.

[0188] Optionally, the second encoding module 803 is further specifically configured to:

[0189] The first code stream is decoded by a compression prediction network to obtain N reconstructed feature maps, and the N reconstructed feature maps are compared with the N original feature maps. Figure 1 One to one correspondence;

[0190] Obtaining N residual feature maps according to the N original feature maps and the N reconstructed feature maps;

[0191] The second target feature map is encoded through the residual network to obtain the second code stream, where the second target feature map is a partial feature map of the N residual feature maps.

[0192] In this embodiment, compared with the method in the related art of compressing all the original feature maps corresponding to the image to obtain the target code stream, the embodiment of the present application only needs to compress part of the original feature maps to obtain the target code stream, thereby reducing the bit rate of the encoded code stream.

[0193] The image processing device provided in the embodiment of the present application can achieve Figure 5 The various processes implemented by the method embodiment achieve the same technical effect and are not described here again to avoid repetition.

[0194] The image processing device in the embodiments of the present application can be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or chip. The electronic device can be a terminal, or it can be other devices other than a terminal. For example, the terminal can include but is not limited to the types of terminals listed above, and other devices can be servers, network attached storage (NAS), etc., which are not specifically limited in the embodiments of the present application.

[0195] The image processing device provided in the embodiment of the present application can achieve Figure 1 Each process implemented by the method embodiment, or capable of implementing Figure 5 The various processes implemented by the method embodiment achieve the same technical effect and are not described here again to avoid repetition.

[0196] Alternatively, as Figure 9 As shown, an embodiment of the present application also provides a communication device 900, including a processor 901 and a memory 902, wherein the memory 902 stores programs or instructions that can be run on the processor 901. For example, when the communication device 900 is a terminal, the program or instructions are executed by the processor 901 to implement the various steps of the above-mentioned image processing method embodiment and can achieve the same technical effect.

[0197] The embodiment of the present application further provides a terminal, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to execute a program or instruction to implement the following Figure 1 The steps in the method embodiment shown, or the implementation as shown Figure 5 The steps in the method embodiment shown. This terminal embodiment corresponds to the above-mentioned terminal side method embodiment, and each implementation process and implementation method of the above-mentioned method embodiment can be applied to this terminal embodiment and can achieve the same technical effect. Specifically, Figure 10 A schematic diagram of the hardware structure of a terminal for implementing an embodiment of the present application.

[0198] The terminal 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010.

[0199] Those skilled in the art will understand that the terminal 1000 may also include a power supply (such as a battery) to power each component, and the power supply may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 10 The terminal structure shown in the figure does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0200] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0201] In the embodiment of the present application, after receiving downlink data from a network-side device, the RF unit 1001 may transmit the data to the processor 1010 for processing; the RF unit 1001 may also send uplink data to the network-side device. Typically, the RF unit 1001 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low-noise amplifier, a duplexer, and the like.

[0202] The memory 1009 can be used to store software programs or instructions and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0203] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0204] The processor 1001 is configured to perform the following operations:

[0205] Get the target stream;

[0206] Decoding the first bitstream through a compression prediction network to obtain N reconstructed feature maps, and decoding the second bitstream through a residual network to obtain N residual feature maps;

[0207] Determine N first target feature maps based on the N reconstructed feature maps and the N residual feature maps.

[0208] Alternatively, the processor 1001 is further configured to perform the following operations:

[0209] Get N original feature maps corresponding to the image;

[0210] Compressing the third target feature map through a compression prediction network to obtain a first bitstream;

[0211] Determine a second bitstream according to the residual network, the N original feature maps, and the first bitstream;

[0212] A target code stream is generated based on the first code stream and the second code stream.

[0213] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0214] The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. In some examples, the readable storage medium may be a non-transitory readable storage medium.

[0215] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0216] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0217] An embodiment of the present application further provides a computer program / program product, which is stored in a storage medium. The computer program / program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0218] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0219] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general-purpose hardware platform, or of course, by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes a number of instructions for enabling a terminal or network-side device to execute the methods described in each embodiment of the present application.

[0220] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of this application and the scope of protection of the claims. These implementation methods are all within the protection of this application.

Claims

1. An image processing method, characterized in that: include: The decoding end obtains a target code stream; the target code stream includes a first code stream and a second code stream; The decoding end decodes the first bitstream through a compression prediction network to obtain N reconstructed feature maps, and decodes the second bitstream through a residual network to obtain N residual feature maps, wherein the N reconstructed feature maps correspond one-to-one to the N residual feature maps, and the residual feature maps are used to represent the residual between the corresponding reconstructed feature maps and the original feature maps. The loss function of the residual network is determined based on the loss generated by the visual detection task, and N is a positive integer greater than or equal to 2; The decoding end determines N first target feature maps based on the N reconstructed feature maps and the N residual feature maps; The loss function of the compression prediction network is determined based on a weighted summation of a first rate loss function and N distortion loss functions, the N distortion loss functions correspond one-to-one to the N reconstructed feature maps, and the distortion loss function is determined based on the loss between the corresponding reconstructed feature map and the original feature map; the first rate loss function is determined based on edge information and a reconstructed first latent variable; the reconstructed first latent variable is obtained by the decoding end performing arithmetic decoding and inverse quantization on the first bitstream; The loss function of the residual network is determined based on a weighted summation of a second rate loss function and a task loss function, the task loss function is determined according to the loss generated by visual detection of the N reconstructed feature maps, and the second rate loss function is determined based on edge information and a reconstructed second latent variable; the reconstructed second latent variable is obtained by the decoding end performing arithmetic decoding on the second code stream.

2. The method according to claim 1, characterized in that The step of decoding the first code stream through a compression prediction network to obtain N reconstructed feature maps includes: The decoding end performs arithmetic decoding and inverse quantization processing on the first bit stream to obtain a reconstructed first latent variable; The decoding end decodes the reconstructed first latent variable through N first decoders included in a preset compression prediction network to obtain the N reconstructed feature maps; the N first decoders correspond one-to-one to the N reconstructed feature maps, and the scale of the reconstructed feature map output by each first decoder is different.

3. The method according to claim 1, characterized in that The step of decoding the first code stream through a compression prediction network to obtain N reconstructed feature maps includes: The decoding end performs arithmetic decoding and inverse quantization processing on the first bit stream to obtain a reconstructed first latent variable; The decoding end decodes the reconstructed first latent variable through a first decoder included in a preset compression prediction network to obtain M reconstructed feature maps; The decoding end downsamples or upsamples one or more of the M reconstructed feature maps to obtain NM reconstructed feature maps, where M is an integer greater than or equal to 1 and less than N.

4. The method according to any one of claims 1 to 3, characterized in that The second code stream is an encoding result of the N residual feature maps, and the residual network includes a decoder; The step of decoding the second code stream through a residual network to obtain N residual feature maps includes: The decoding end performs arithmetic decoding on the second code stream to obtain a reconstructed second latent variable; The decoding end decodes the reconstructed second latent variable through a decoder included in the residual network to obtain the N residual feature maps.

5. The method according to any one of claims 1 to 3, characterized in that The second bitstream is an encoding result of a second target feature map, the second target feature map is a partial feature map of the N residual feature maps, the residual network includes N second decoders, and the scale of the residual feature map output by each second decoder is different; The step of decoding the second code stream through a residual network to obtain N residual feature maps includes: The decoding end performs arithmetic decoding on the second code stream to obtain a reconstructed second latent variable; The decoding end decodes the reconstructed second latent variable through N second decoders included in the residual network to obtain the N residual feature maps.

6. An image processing method, characterized in that: include: The encoder obtains N original feature maps corresponding to the image, where N is a positive integer greater than or equal to 2; The encoding end compresses the third target feature map through a compression prediction network to obtain a first bit stream; The third target feature map is a partial feature map of the N original feature maps; The encoder determines a second bitstream based on the residual network, the N original feature maps, and the first bitstream; The encoding end generates a target bitstream based on the first bitstream and the second bitstream; The determining the second bitstream according to the residual network, the N original feature maps, and the first bitstream includes: The encoder decodes the first bitstream through a compression prediction network to obtain N reconstructed feature maps, where the N reconstructed feature maps correspond one-to-one to the N original feature maps; The encoding end obtains N residual feature maps according to the N original feature maps and the N reconstructed feature maps; The encoding end encodes the N residual feature maps through a residual network to obtain the second bitstream; The residual feature map is used to represent the residual between the corresponding reconstructed feature map and the original feature map, and the loss function of the residual network is determined based on the loss generated by the visual detection task; The loss function of the compression prediction network is determined based on a weighted sum of a first rate loss function and N distortion loss functions, wherein the N distortion loss functions correspond one-to-one to the N reconstructed feature maps, and the distortion loss function is determined based on the loss between the corresponding reconstructed feature map and the original feature map; the first rate loss function is determined based on edge information and a reconstructed first latent variable; the reconstructed first latent variable is obtained by performing arithmetic decoding and dequantization processing on the first bitstream at the decoding end; The loss function of the residual network is determined based on the weighted summation of the second rate loss function and the task loss function, the task loss function is determined according to the loss generated by visual detection of the N reconstructed feature maps, and the second rate loss function is determined based on edge information and the reconstructed second latent variable; the reconstructed second latent variable is obtained by arithmetic decoding of the second code stream by the decoding end.

7. The method according to claim 6, characterized in that The compressing the third target feature map by the compression prediction network to obtain the first bitstream includes: The encoder performs downsampling processing on the third target feature map through a compression prediction network; The encoder end compresses the third target feature map after downsampling through the compression prediction network, obtaining a first latent variable corresponding to the third target feature map; The encoding end quantizes the first latent variable and performs arithmetic coding on the quantized first latent variable to obtain the first code stream.

8. The method according to claim 6, characterized in that The encoding end encodes the N residual feature maps through a residual network to obtain the second bitstream, including: The encoding end encodes the second target feature map through a residual network to obtain the second code stream, where the second target feature map is a partial feature map of the N residual feature maps.

9. An image processing device, characterized in that: include: An acquisition module, configured to acquire a target code stream; the target code stream includes a first code stream and a second code stream; A decoding module, configured to decode the first bitstream through a compression prediction network to obtain N reconstructed feature maps, and decode the second bitstream through a residual network to obtain N residual feature maps, wherein the N reconstructed feature maps correspond one-to-one to the N residual feature maps, and the residual feature maps are used to represent the residual between the corresponding reconstructed feature maps and the original feature maps. The loss function of the residual network is determined based on the loss generated by the visual detection task, and N is a positive integer greater than or equal to 2; A determination module, configured to determine N first target feature maps based on the N reconstructed feature maps and the N residual feature maps; The loss function of the compression prediction network is determined based on a weighted summation of a first rate loss function and N distortion loss functions, the N distortion loss functions correspond one-to-one to the N reconstructed feature maps, and the distortion loss function is determined based on the loss between the corresponding reconstructed feature map and the original feature map; the first rate loss function is determined based on edge information and a reconstructed first latent variable; the reconstructed first latent variable is obtained by performing arithmetic decoding and inverse quantization on the first bitstream at the decoding end; The loss function of the residual network is determined based on the weighted summation of the second rate loss function and the task loss function, the task loss function is determined according to the loss generated by visual detection of the N reconstructed feature maps, and the second rate loss function is determined based on edge information and the reconstructed second latent variable; the reconstructed second latent variable is obtained by arithmetic decoding of the second code stream by the decoding end.

10. The device according to claim 9, characterized in that The decoding module is configured to decode the first code stream through a compression prediction network to obtain N reconstructed feature maps, including: The decoding module is used to perform arithmetic decoding and inverse quantization on the first code stream to obtain a reconstructed first latent variable; the reconstructed first latent variable is decoded by N first decoders included in a preset compression prediction network to obtain the N reconstructed feature maps; the N first decoders correspond one-to-one to the N reconstructed feature maps, and the scale of the reconstructed feature map output by each first decoder is different.

11. The device according to claim 9, characterized in that The decoding module is configured to decode the first code stream through a compression prediction network to obtain N reconstructed feature maps, including: The decoding module is used to perform arithmetic decoding and inverse quantization on the first code stream to obtain a reconstructed first latent variable; decode the reconstructed first latent variable through a first decoder included in a preset compression prediction network to obtain M reconstructed feature maps; downsample or upsample one or more of the M reconstructed feature maps to obtain NM reconstructed feature maps, where M is an integer greater than or equal to 1 and less than N.

12. The device according to any one of claims 9 to 11, characterized in that The second code stream is an encoding result of the N residual feature maps, and the residual network includes a decoder; The decoding module is configured to decode the second code stream through a residual network to obtain N residual feature maps, including: The decoding module is used to perform arithmetic decoding on the second code stream to obtain a reconstructed second latent variable; and the reconstructed second latent variable is decoded by a decoder included in the residual network to obtain the N residual feature maps.

13. The device according to any one of claims 9 to 11, characterized in that The second bitstream is an encoding result of a second target feature map, the second target feature map is a partial feature map of the N residual feature maps, the residual network includes N second decoders, and the scale of the residual feature map output by each second decoder is different; The decoding module is configured to decode the second code stream through a residual network to obtain N residual feature maps, including: The decoding module is used to perform arithmetic decoding on the second code stream to obtain a reconstructed second latent variable; The reconstructed second latent variable is decoded by the N second decoders included in the residual network to obtain the N residual feature maps.

14. An image processing device, characterized in that: include: An acquisition module is used to obtain N original feature maps corresponding to the image, where N is a positive integer greater than or equal to 2; A first encoding module is used to compress the third target feature map through a compression prediction network to obtain a first code stream; The third target feature map is a partial feature map of the N original feature maps; A second encoding module, configured to determine a second bitstream based on the residual network, the N original feature maps, and the first bitstream; A generating module, configured to generate a target bitstream based on the first bitstream and the second bitstream; The second encoding module is specifically configured to: Decoding the first bitstream through a compression prediction network to obtain N reconstructed feature maps, where the N reconstructed feature maps correspond one-to-one to the N original feature maps; Obtaining N residual feature maps according to the N original feature maps and the N reconstructed feature maps; Encoding the N residual feature maps through a residual network to obtain the second bitstream; The residual feature map is used to represent the residual between the corresponding reconstructed feature map and the original feature map, and the loss function of the residual network is determined based on the loss generated by the visual detection task; The loss function of the compression prediction network is determined based on a weighted sum of a first rate loss function and N distortion loss functions, wherein the N distortion loss functions correspond one-to-one to the N reconstructed feature maps, and the distortion loss function is determined based on the loss between the corresponding reconstructed feature map and the original feature map; the first rate loss function is determined based on edge information and a reconstructed first latent variable; the reconstructed first latent variable is obtained by performing arithmetic decoding and dequantization processing on the first bitstream at the decoding end; The loss function of the residual network is determined based on the weighted summation of the second rate loss function and the task loss function, the task loss function is determined according to the loss generated by visual detection of the N reconstructed feature maps, and the second rate loss function is determined based on edge information and the reconstructed second latent variable; the reconstructed second latent variable is obtained by arithmetic decoding of the second code stream by the decoding end.

15. The device according to claim 14, characterized in that The first encoding module is specifically configured to: Downsampling the third target feature map through a compression prediction network; Obtaining a first latent variable corresponding to the third target feature map by compressing and downsampling the third target feature map through the compression prediction network; The first latent variable is quantized, and arithmetic coding is performed on the quantized first latent variable to obtain the first code stream.

16. The device according to claim 14, characterized in that The second encoding module is further specifically configured to: The second target feature map is encoded through the residual network to obtain the second code stream, where the second target feature map is a partial feature map of the N residual feature maps.

17. A terminal, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented, or the steps of the image processing method according to any one of claims 6 to 8 are implemented.

18. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 5 are implemented, or the steps of the image processing method according to any one of claims 6 to 8 are implemented.

Images