Method and apparatus for depth image coding oriented to saliency segmentation, electronic device, medium and product

Abstract

The application discloses a saliency segmentation-oriented depth image coding method and device, electronic equipment, medium and product, and relates to the technical field of image coding. The method comprises the following steps: acquiring the saliency values of each pixel point in a target image, determining the saliency probability distribution of the target image according to the saliency values of each pixel point; calculating the weight distribution value of each pixel point based on a preset activation function and the saliency probability distribution; performing channel clipping on the original latent features of the target image according to the weight distribution value to obtain coded latent features; and encoding the coded latent features based on a preset coding network to obtain an encoding result. For any one of the weight distribution values, the weight distribution value is related to the distribution proportion of the corresponding pixel point in a first storage bit stream, and the greater the weight distribution value, the higher the corresponding distribution proportion in the first storage bit stream. The application solves the technical problem of low performance of depth coding in downstream saliency detection tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image coding, in particular to a saliency segmentation-oriented deep image coding method and device, electronic equipment, storage medium and computer program product. BACKGROUND

[0002] In the existing deep learning-based image coding technology, since the saliency segmentation task only focuses on part of the region in the image, and the existing technology does not optimize this feature, a large number of bits are wasted in the non-saliency region, affecting the coding efficiency, and simply compressing the input image into a potential feature with the same scale without considering the difference in saliency information contained in different images and different regions in the image not only wastes bit resources but also may cause loss of saliency information, affecting the performance of subsequent tasks. Therefore, there is a problem of low performance of deep coding in the downstream saliency detection task in the current deep learning-based image coding technology. SUMMARY

[0003] The main purpose of the present application is to provide a saliency segmentation-oriented deep image coding method and device, electronic equipment, storage medium and computer program product, aiming to solve the technical problem of low performance of deep coding in the downstream saliency detection task.

[0004] To achieve the above-mentioned purpose, the present application provides a saliency segmentation-oriented deep image coding method, which comprises:

[0005] obtaining the saliency value of each pixel point in the target image, and determining the saliency probability distribution of the target image according to the saliency value of each pixel point;

[0006] calculating the weight distribution value of each pixel point based on a preset activation function and the saliency probability distribution;

[0007] channel pruning the original potential feature of the target image according to the weight distribution value to obtain an encoded potential feature, and encoding the encoded potential feature based on a preset encoding network to obtain an encoding result, wherein for any one of the weight distribution values, the weight distribution value is related to the allocation proportion of the corresponding pixel point in the first storage bit stream, and the greater the weight distribution value, the higher the corresponding allocation proportion in the first storage bit stream.

[0008] In an embodiment, the step of channel pruning the original potential feature of the target image according to the weight distribution value to obtain an encoded potential feature comprises:

[0009] obtaining the original potential feature of the target image through a preset encoder;

[0010] generating a bit allocation weight map based on the weight allocation values, and expanding the bit allocation weight map into a three-dimensional mask;

[0011] Performing element-wise multiplication of the three-dimensional mask and the original latent feature to clip the neural network channels whose weight distribution values ​​are lower than a preset weight distribution threshold to obtain the encoded latent feature.

[0012] In one embodiment, the depth image coding method for saliency segmentation further includes:

[0013] For any preset training sample image in the training phase, feature extraction is performed on the preset training sample image to obtain a training coding latent feature corresponding to the preset training sample image, wherein the training coding latent feature includes a first basic feature and a first enhanced feature;

[0014] Encoding the training coding potential features based on the initial coding network to obtain a training coding result;

[0015] Decoding and restoring the training encoding result based on the initial decoding network to obtain a training reconstructed image;

[0016] updating network parameters in the initial encoding network and the initial decoding network based on a mean square error between the training reconstructed image and the preset training sample image;

[0017] After the preset training conditions are met, the initial encoding network is used as the preset encoding network, and the initial decoding network is used as the preset decoding network for restoring the encoding result.

[0018] In one embodiment, the step of encoding the training coding potential features based on the initial coding network to obtain a training coding result includes:

[0019] Performing entropy coding on the first basic feature by using a super priori entropy model to generate a second storage bitstream;

[0020] performing entropy coding on the first enhanced feature by using a full decomposition entropy model to generate a third storage bitstream;

[0021] An all-zero feature is introduced, and entropy coding is performed on the first enhanced feature in combination with a full decomposition entropy model to generate a fourth storage bitstream, wherein the dimension of the all-zero feature is the same as that of the first enhanced feature.

[0022] In one embodiment, the step of decoding and restoring the training encoding result based on the initial decoding network to obtain a training reconstructed image includes:

[0023] decoding the second storage bit stream, the third storage bit stream and the fourth storage bit stream, respectively corresponding to generate a basic reconstructed feature, an enhanced reconstructed feature and a full zero reconstructed feature;

[0024] concatenating the basic reconstructed feature and the enhanced reconstructed feature at a channel level to generate a first decoded feature, and concatenating the basic reconstructed feature and the full zero reconstructed feature at the channel level to generate a second decoded feature, the first decoded feature representing an optimal reconstructed scale, and the second decoded feature representing a worst reconstructed scale;

[0025] if the target image is restored according to the first decoded feature, a first reconstructed image of the optimal reconstructed scale is generated;

[0026] if the target image is restored according to the second decoded feature, a second reconstructed image of the worst reconstructed scale is generated.

[0027] In an embodiment, the pixel points are represented in a preset feature space, for any one of the pixel points, a relative distance between the pixel point and a saliency segmentation network in the preset feature space represents an absolute value of a saliency value corresponding to the pixel point, if the relative distance is smaller, it represents that the pixel point contains more saliency information, and a saliency value corresponding weight distribution value of the pixel point is larger, if the relative distance is larger, it represents that the pixel point contains less saliency information, and a saliency value corresponding weight distribution value of the pixel point is smaller.

[0028] In addition, to achieve the above object, the present application further provides a depth image coding device for saliency segmentation, the depth image coding device for saliency segmentation comprises:

[0029] a probability obtaining module, configured to obtain saliency values of pixel points in a target image, and determine a saliency probability distribution of the target image according to the saliency values of the pixel points;

[0030] a weight calculating module, configured to calculate weight distribution values of the pixel points based on a preset activation function and the saliency probability distribution;

[0031] a coding module, configured to perform channel clipping on original latent features of the target image according to the weight distribution values to obtain coded latent features, and code the coded latent features based on a preset coding network to obtain a coding result, wherein for any one of the weight distribution values, the weight distribution value is related to an allocation proportion of a corresponding pixel point in a first storage bit stream, and the larger the weight distribution value is, the higher the corresponding allocation proportion in the first storage bit stream is.

[0032] In addition, to achieve the above object, the present application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the depth image coding method based on saliency segmentation as described above.

[0033] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the depth image coding method based on saliency segmentation as described above.

[0034] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the depth image coding method based on saliency segmentation as described above.

[0035] The present application provides a depth image coding method based on saliency segmentation, which comprises: obtaining the saliency value of each pixel point in a target image, determining the saliency probability distribution of the target image according to the saliency value of each pixel point; calculating the weight distribution value of each pixel point based on a preset activation function and the saliency probability distribution; performing channel clipping on the original latent feature of the target image according to the weight distribution value to obtain a coded latent feature, and encoding the coded latent feature based on a preset encoding network to obtain an encoding result, wherein for any one of the weight distribution values, the weight distribution value is related to the distribution proportion of the corresponding pixel point in a first storage bit stream, and the greater the weight distribution value, the higher the corresponding distribution proportion in the first storage bit stream.

[0036] The present application accurately identifies the key information in the image by obtaining the saliency value of each pixel point in the target image, provides an important basis for the subsequent encoding process, converts the saliency value into the saliency probability, makes the encoding process more stable and reliable, reduces the encoding error caused by the saliency value fluctuation, reasonably allocates the encoding resources through the weight distribution based on the saliency probability, makes the pixel points containing more saliency information occupy a larger proportion in the bit stream, so that more information is retained, improves the encoding efficiency and quality, and through the coded latent feature extraction and encoding based on the weight distribution value, the system can more accurately capture the important information in the image and generate high-quality encoding bit stream. Compared with the related scheme which lacks a bit allocation method for the saliency segmentation task, resulting in bit waste in the invalid area, the present application adaptively allocates bits in the region, reasonably allocates the weight according to the importance of different regions for the saliency segmentation task, and only retains the useful information for the downstream saliency segmentation task to improve the encoding performance of the image. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, the other drawings can also be obtained based on these drawings without any creative effort.

[0039] Figure 1 A flowchart is provided for the depth image coding method for saliency segmentation according to an embodiment of the present application.

[0040] Figure 2 An elevation rolloff function diagram is provided for the depth image coding method for saliency segmentation according to an embodiment of the present application.

[0041] Figure 3 A flowchart is provided for the depth image coding method for saliency segmentation according to an embodiment of the present application.

[0042] Figure 4 A saliency segmentation geometry diagram is provided for the depth image coding method for saliency segmentation according to an embodiment of the present application.

[0043] Figure 5 A flowchart is provided for the depth image coding method for saliency segmentation according to an embodiment of the present application.

[0044] Figure 6 A flowchart is provided for the depth image coding method for saliency segmentation according to an embodiment of the present application.

[0045] Figure 7 A module structure diagram is provided for the depth image coding device for saliency segmentation according to an embodiment of the present application.

[0046] Figure 8 A device structure diagram is provided for the hardware running environment involved in the depth image coding method for saliency segmentation according to an embodiment of the present application.

[0047] The object implementation, functional features and advantages of the present application will be further explained with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein merely serve to explain the technical solutions of the present application, and are not used to limit the present application.

[0049] For better understanding of the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0050] The main solution of the embodiment of the present application is: obtaining the saliency values of each pixel point in the target image, determining the saliency probability distribution of the target image according to the saliency values of each pixel point; based on the preset activation function and the saliency probability distribution, calculating the weight distribution value of each pixel point; according to the weight distribution value, the original latent feature of the target image is channel cropped to obtain the encoded latent feature, and the encoded latent feature is encoded based on the preset encoding network to obtain the encoding result, wherein for any one of the weight distribution values, the weight distribution value is related to the distribution proportion of the corresponding pixel point in the first storage bit stream, and the greater the weight distribution value, the higher the corresponding distribution proportion in the first storage bit stream.

[0051] In the present embodiment, for the sake of description, the following describes an image encoding system as the execution subject.

[0052] Since the prior art only focuses on the overall fidelity of the reconstructed image or the human eye perception, it is not optimized for the downstream saliency segmentation task, lacks a task-specific bit allocation method, compresses the input image into a latent feature with the same scale, ignores the differences in saliency information in different images and regions, thereby wasting bits in invalid regions, while losing key saliency information, and reducing the efficiency in the saliency segmentation task.

[0053] The present application provides a solution, obtains the saliency values of each pixel point in the target image and determines the saliency probability, calculates the weight distribution value of the saliency probability using a preset activation function, determines and encodes the latent feature of the target image according to the weight distribution value, wherein the weight distribution value determines the distribution proportion of the corresponding pixel point in the first storage bit stream, and through saliency analysis and weight distribution, the allocation of coding resources is optimized, so that important information obtains a higher proportion of bit stream, thereby improving the coding efficiency and image reconstruction quality, and solving the problem of low performance of deep coding in the downstream saliency detection task.

[0054] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, an image encoding system, etc. capable of realizing the above functions. The following will take the image encoding system as an example to describe the present embodiment and each of the following embodiments.

[0055] Based on this, the embodiment of the present application provides a deep image coding method for saliency segmentation, which is described in detail with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the deep image coding method for saliency segmentation of the present application is shown.

[0056] In this embodiment, the depth image coding method based on saliency segmentation includes steps S01-S03:

[0057] In step S01, the saliency values of each pixel point in the target image are obtained, and the saliency probability distribution of the target image is determined according to the saliency values of each pixel point.

[0058] It should be noted that the target image refers to the image that needs to be encoded and processed, and the pixel point is the basic unit of the image, each pixel point contains color, brightness and other information, the saliency value is a value reflecting the prominence of each pixel point in the image relative to the background or surrounding pixels, the saliency probability is a probability value obtained by normalizing the saliency value, indicating the importance of the pixel point in the image, and the saliency probability distribution refers to the probability distribution of the saliency value on each pixel point in the target image.

[0059] In addition, it should be noted that the target image is received as input, and a saliency detection algorithm (such as contrast-based saliency detection, learning-based saliency detection, etc.) is used to calculate the saliency value of each pixel point in the image, and the calculation formula of the saliency value is:

[0060]

[0061] where w represents the saliency value of the current pixel point, represents the saliency detection algorithm, I represents the input target image, and Φ represents the saliency segmentation network. The saliency segmentation network is a trained neural network used to distinguish salient regions and background regions in the image. The saliency value reflects the importance or visual attraction of the pixel point in the image. By normalizing the saliency values, the sum of the saliency probabilities of all pixel points is 1, and the formula for normalizing is:

[0062]

[0063] where w norm represents the saliency probability, w represents the saliency value of the current pixel point, w max represents the maximum saliency value, and w min represents the minimum saliency value.

[0064] It can be understood that since the existing depth coding technology does not consider the saliency information of the image, there is a lack of bit allocation method for saliency segmentation task, resulting in waste of bits in invalid areas, so step S01 is performed to obtain the saliency value of each pixel point through saliency detection, and the saliency probability is determined, providing a basis for subsequent weight allocation.

[0065] Step S02, based on the preset activation function and the saliency probability distribution, the weight distribution value of each pixel point is calculated;

[0066] It should be noted that the preset activation function is a function for calculating the saliency probability weight distribution value, which is the ascending cosine roll-off activation function (ACRD) here, and the weight distribution value is a value calculated according to the saliency probability of each pixel point, which is used to represent the importance of each pixel point in the encoding process.

[0067] In addition, it should be noted that the ascending cosine roll-off activation function is applied to the saliency probability to calculate the weight distribution value of each pixel point. The application designs a novel ascending cosine roll-off activation function, as shown in Figure 2 , Figure 2 The ascending cosine roll-off function diagram is provided, and the function can calculate the weight distribution value of any pixel point in the target image according to the size of the saliency probability, and the calculation formula is:

[0068]

[0069] Wherein, x represents the saliency probability of any pixel point in the saliency probability distribution, that is, w norm , α represents the bandwidth of the ascending cosine roll-off activation function, which is a parameter between 0 and 1, which affects the transition bandwidth and height of the function.

[0070] It can be understood that since the existing depth encoding technology simply and uniformly compresses the input image into potential features with the same scale, it does not consider that different images and different regions of the same image contain different saliency information, resulting in waste of bits in invalid regions and loss of rich saliency information, so step S02 is performed, and a novel ascending cosine roll-off activation function is designed to reasonably allocate weights according to the size of the saliency probability, providing an optimized bit resource allocation scheme for the subsequent encoding process and improving the encoding efficiency.

[0071] Step S03, according to the weight distribution value, the original potential feature of the target image is channel cropped to obtain the encoding potential feature, and the encoding result is obtained based on the preset encoding network, wherein for any one of the weight distribution values, the weight distribution value is related to the allocation proportion of the corresponding pixel point in the first storage bit stream, and the greater the weight distribution value, the higher the corresponding allocation proportion in the first storage bit stream.

[0072] It should be noted that the encoding potential feature refers to the feature after the original potential feature is cropped, which contains the main information of the target image, and the original potential feature refers to the feature data extracted from the input image without processing, and the calculation formula of the original potential feature is:

[0073]

[0074] wherein y represents the extracted original latent feature, g a represents a feature extraction network for extracting an original latent feature of a target image, I represents an input target image, represents network parameters in the feature extraction network, the network parameters refer to parameters such as weights and biases in the encoding network, which determine how the feature extraction network extracts features of the target image and generates the original latent feature, the preset encoder is a complex neural network structure for encoding the encoded latent feature, which contains multiple encoding layers, and each encoding layer contains a large number of encoding parameters, the first storage bit stream refers to a bit stream for storing encoded image data, used for storing or transmitting image information, and the allocation ratio refers to a corresponding allocation ratio of the pixel points in the first storage bit stream, and the greater the weight allocation value, the higher the allocation ratio.

[0075] In addition, it should be noted that the bit allocation weight map is generated according to the weight allocation value, the bit allocation weight map represents the weight allocation value of each pixel point in the encoding process, and reflects the number of bits that should be allocated to each pixel point in the encoding process. The expanded bit allocation weight map is multiplied by the obtained original latent feature to obtain the cropped feature, i.e., the encoded latent feature. The encoded latent feature is respectively subjected to entropy encoding using two entropy models of hyper-prior and full decomposition. Entropy encoding is a lossless compression method that can compress data according to statistical characteristics while ensuring data integrity. In addition, the encoding network and the saliency segmentation network are decoupled. For the encoding process, various encoding networks such as autoencoders and high-efficiency video coding (HEVC) image encoders can be adapted, so that the encoding network can be compatible with multiple saliency segmentation networks.

[0076] It can be understood that, since the existing deep encoding technology adopts an autoregressive entropy model, which is time-consuming, step S03 is performed to improve the flexibility and efficiency of encoding by using two lightweight entropy models (hyper-prior entropy model and full decomposition entropy model) for entropy encoding, so that the encoded image data is more compact and efficient.

[0077] In a feasible implementation, in step S03, the step of channel cropping the original latent feature of the target image according to the weight allocation value to obtain the encoded latent feature includes steps A01-A03:

[0078] Step A01, obtaining the original latent feature of the target image by using a preset encoder;

[0079] It should be noted that a pre-trained encoder (usually a deep neural network) is used to process the target image. The preset encoder refers to a trained neural network model used to extract latent features from the input image. These latent features usually contain the main information and structure of the image and are the basis for the subsequent encoding process. After the target image is processed, the output corresponds to the original latent features. The original latent features refer to the unprocessed feature data extracted from the input image by the preset encoder.

[0080] Step A02: generating a bit allocation weight map based on the weight allocation value, and expanding the bit allocation weight map into a three-dimensional mask;

[0081] It should be noted that the weight allocation value is represented in the form of a two-dimensional image, that is, a bit allocation weight map is formed. While keeping the content of the bit allocation weight map unchanged, a dimension (usually the channel dimension) is added so that each channel corresponds to the same weight map, resulting in the weight map that originally contained only two dimensions (width and height) becoming a three-dimensional mask containing three dimensions (width, height and channel). The three-dimensional mask refers to the expansion of the bit allocation weight map into a three-dimensional image that matches the original potential features, which is used to adjust the weights of the original potential features in subsequent steps.

[0082] In step A03, the three-dimensional mask is element-wise multiplied with the original latent feature to trim the neural network channels whose weight distribution values ​​are lower than a preset weight distribution threshold to obtain the encoded latent feature.

[0083] It should be noted that element-wise multiplication is a mathematical operation that multiplies corresponding elements of two arrays (or matrices) of the same shape to obtain a new array (or matrix) used to adjust the weights of the three-dimensional mask and the original latent features. The expanded three-dimensional mask is element-wise multiplied with the original latent features. For each corresponding element in the three-dimensional mask and the original latent features, a multiplication operation is performed. Since each channel in the mask corresponds to the same weight map, this operation actually performs a weighted processing on each channel of the original latent features, pruning neural network channels whose weight assignment values ​​are lower than a preset weight assignment threshold. The preset weight assignment threshold is used to determine whether the weight of the neural network channel is important. If a weight assignment value is lower than the preset weight assignment threshold, then the weight is considered unimportant and can be removed or set to zero during the pruning process. Regions with higher weights (i.e., regions with higher significance) will receive higher weights during the encoding process, thereby retaining more information.

[0084] In this embodiment, the preset encoder can more effectively extract the potential features of the image, improve the accuracy and effectiveness of feature extraction in the encoding process, provide high-quality feature data for the subsequent steps, realize bit allocation based on saliency information by generating a bit allocation weight map and expanding it into a three-dimensional mask, so that the salient region obtains more bit resources in the encoding process, improves the quality of the encoded image, and realizes feature clipping and optimization based on the weight distribution value through element point multiplication operation, removes unimportant feature information, and improves the quality of the encoded image and the encoding efficiency.

[0085] Based on the first embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and the following will not be described in detail. On this basis, please refer to Figure 3 , the depth image encoding method based on saliency segmentation further includes steps S11-S15:

[0086] Step S11, for any one of the preset training sample images in the training stage, the feature extraction is performed on the preset training sample image to obtain the training encoding potential feature corresponding to the preset training sample image, wherein the training encoding potential feature includes the first basic feature and the first enhanced feature.

[0087] It should be noted that the preset training sample image refers to the image stored in the training sample in advance and needing to be trained, the saliency value of each pixel point in the preset training sample image is obtained, the saliency probability distribution of the target image is determined according to the saliency value of each pixel point, the weight distribution value of each pixel point is calculated based on the preset activation function and the saliency probability distribution, the original potential feature of the preset training sample image is channel clipped according to the weight distribution value, and the corresponding training encoding potential feature is obtained. The training encoding potential feature includes the first basic feature and the first enhanced feature. The first basic feature refers to the most basic feature extracted from the target image, which contains the basic information and structure of the image. The first enhanced feature refers to the feature obtained by further processing the target image or the first basic feature, which contains more detailed and textured information.

[0088] Step S12, encoding the training encoding potential feature based on the initial encoding network to obtain the training encoding result;

[0089] It should be noted that the initial encoding network is a neural network designed for realizing potential feature encoding according to historical experience at the initial training. The training encoding potential feature is entropy encoded through the initial encoding network, and the generated bit stream is the training encoding result.

[0090] Step S13, decoding and restoring the training encoding result based on the initial decoding network to obtain the training reconstruction image;

[0091] It should be noted that the initial decoding network is a neural network designed based on historical experience during the initial training to realize potential feature decoding. The training encoding results are decoded through the initial decoding network to restore the corresponding reconstructed features, thereby obtaining the training reconstructed image.

[0092] Step S14, updating the network parameters in the initial encoding network and the initial decoding network based on the mean square error between the training reconstructed image and the preset training sample image;

[0093] It should be noted that by comparing the training reconstructed image and the preset training sample image pixel by pixel and calculating the mean square error between them, the gap between the training reconstructed image and the target image at the optimal reconstruction scale and the worst reconstruction scale can be determined. The mean square error (MSE) is an indicator to measure the degree of difference between two images. The calculation formula is the average value of the sum of the squares of all corresponding pixel value differences. After obtaining the mean square error, the system inputs it into the initial encoding network and the initial decoding network as a feedback signal. The system uses the back propagation algorithm to adjust these encoding parameters according to the size of the mean square error to reduce the difference between the initial encoding network and the initial decoding network.

[0094] Step S15: After the preset training conditions are met, the initial encoding network is used as the preset encoding network, and the initial decoding network is used as the preset decoding network for restoring the encoding result.

[0095] It should be noted that after the preset training conditions are met (such as the number of training rounds, the mean square error reaches a certain threshold, etc.), the initial encoding network will be used as the preset encoding network, and the initial decoding network will be used as the preset decoding network to restore the encoding result. The preset encoding network and the preset decoding network will be used in actual deep image encoding and decoding applications.

[0096] In this embodiment, through feature extraction, the information in the image can be more effectively utilized, and the accuracy and efficiency of the subsequent encoding and decoding processes can be improved. Through encoding, high-dimensional image features can be converted into low-dimensional encoding results while retaining information useful for downstream tasks. Through decoding and restoration, it can be evaluated whether the encoding process retains sufficient information to reconstruct the original image, thereby verifying the performance of the encoding network and the decoding network. By minimizing the mean square error, the parameters of the encoding network and the decoding network can be gradually optimized, thereby improving their performance in downstream tasks (such as saliency detection). By setting reasonable training conditions, it can be ensured that the model fully learns during the training process and avoids overfitting or underfitting, thereby obtaining a final model with good performance.

[0097] In a feasible implementation, the initial encoding network includes a super priori entropy model and a full decomposition entropy model. In step S12, the step of encoding the encoding potential features includes steps B01 to B03:

[0098] Step B01, entropy encoding the first basic feature through a super prior entropy model to generate a second storage bit stream;

[0099] It should be noted that the system processes the input first basic feature through the initial encoding network, and entropy encodes the first basic feature using a super prior entropy model. The super prior entropy model is an advanced probability model that can reduce the entropy estimation error caused by the incomplete consistency of the cropping method during training and inference. Entropy encoding is a probability-based encoding method that compresses data by reducing redundancy, ultimately generating a second storage bit stream. The second storage bit stream refers to the bit stream obtained after entropy encoding by the super prior entropy model, which contains basic information of the image and has low redundancy.

[0100] Step B02, entropy encoding the first enhanced feature through a full decomposition entropy model to generate a third storage bit stream;

[0101] It should be noted that the system processes the first enhanced feature through the initial encoding network, and entropy encodes the first enhanced feature using a full decomposition entropy model. The full decomposition entropy model is an entropy encoding method that decomposes features into multiple independent components and encodes them separately, which is used to reduce code rate and reduce encoding and decoding time. Ultimately, a third storage bit stream is generated. The third storage bit stream is the bit stream obtained after entropy encoding by the full decomposition entropy model, which contains image details and texture information and has high encoding efficiency.

[0102] Step B03, introducing a full zero feature, combining the full decomposition entropy model to entropy encode the first enhanced feature to generate a fourth storage bit stream, wherein the dimension of the full zero feature is the same as that of the first enhanced feature.

[0103] It should be noted that the system introduces a full zero feature and combines it with the first enhanced feature. The full zero feature is a feature vector with all elements equal to zero, and its dimension is the same as that of the first enhanced feature. The combined features are entropy encoded using the full decomposition entropy model, ultimately generating a fourth storage bit stream. The fourth storage bit stream is the bit stream obtained after entropy encoding by introducing the full zero feature and combining the full decomposition entropy model, which contains information of the first enhanced feature and the influence of the full zero feature.

[0104] In this embodiment, through entropy coding of the super-prior entropy model, the first basic feature is effectively compressed into a second storage bit stream. This bit stream not only contains the basic information of the image, but also has low redundancy, thereby saving storage space. Through entropy coding of the full decomposition entropy model, the first enhanced feature is effectively compressed into a third storage bit stream. The bit stream not only contains the details and texture information of the image, but also has higher coding efficiency and lower bit rate. By introducing all-zero features and combining them with the full decomposition entropy model for encoding, the system can evaluate the impact of different feature combinations on coding performance, which helps to discover more effective feature combinations and coding strategies, thereby improving overall coding performance.

[0105] In a feasible implementation, in step S13, the step of decoding and restoring the training encoding result based on the initial decoding network to obtain the training reconstructed image includes steps B11 to B14:

[0106] Step B11: decoding the second stored bit stream, the third stored bit stream, and the fourth stored bit stream in the initial decoding network to generate a basic reconstruction feature, an enhanced reconstruction feature, and an all-zero reconstruction feature, respectively;

[0107] It should be noted that the second stored bit stream, the third stored bit stream and the fourth stored bit stream are decoded to generate basic reconstruction features, enhanced reconstruction features and all-zero reconstruction features. The basic reconstruction features are the basic reconstruction information of the image obtained by decoding the second stored bit stream, which includes the basic information and structure of the image. The enhanced reconstruction features are features obtained by decoding the third stored bit stream, which include details and texture information that make the image clearer and richer. The all-zero reconstruction features are features obtained by decoding the fourth stored bit stream. Due to the introduction of the all-zero features, it is equivalent to there being no distortion-sensitive points in the image, and all pixels are regarded as background. These features will be used for subsequent decoding feature generation.

[0108] Step B12: Channel-concatenate the basic reconstruction feature and the enhanced reconstruction feature to generate a first decoding feature, and channel-concatenate the basic reconstruction feature and the all-zero reconstruction feature to generate a second decoding feature. The first decoding feature represents the optimal reconstruction scale, and the second decoding feature represents the worst reconstruction scale.

[0109] It should be noted that the basic reconstruction feature is channel concatenated with the enhanced reconstruction feature and the all-zero reconstruction feature, respectively. Channel concatenation is an operation of splicing different features in the channel dimension to fuse the information of different features to generate the first decoding feature and the second decoding feature. The first decoding feature refers to the feature after the fusion of the basic reconstruction feature and the enhanced reconstruction feature, represents the optimal reconstruction scale, and the optimal reconstruction scale is the reconstruction scale with the highest image quality obtained by restoring the first decoding feature. The second decoding feature refers to the feature after the fusion of the basic reconstruction feature and the all-zero reconstruction feature, represents the worst reconstruction scale, and the worst reconstruction scale refers to the reconstruction scale with the lowest image quality obtained by restoring the second decoding feature.

[0110] Step B13, if the target image is restored according to the first decoding feature, a first reconstructed image of the optimal reconstruction scale is generated.

[0111] It should be noted that the target image is restored according to the first decoding feature to generate a reconstructed image of the optimal reconstruction scale, i.e., the first reconstructed image, which has reached a high level in details, texture and overall quality.

[0112] Step B14, if the target image is restored according to the second decoding feature, a second reconstructed image of the worst reconstruction scale is generated.

[0113] It should be noted that the target image is restored according to the second decoding feature to generate a reconstructed image of the worst reconstruction scale, i.e., the second reconstructed image, which performs poorly in details, texture or overall quality and is used to evaluate the performance lower limit of the encoding and decoding method.

[0114] In this embodiment, different levels of information of the image can be obtained by decoding different feature streams, which provides rich materials for subsequent feature fusion. The basic reconstruction feature is channel concatenated with the enhanced reconstruction feature and the all-zero reconstruction feature, respectively, to fully utilize the complementarity between different features and improve the quality of the reconstructed image. At the same time, by comparing the reconstruction effects of different feature combinations, the optimal and worst reconstruction scales can be evaluated to reconstruct images of the optimal and worst two distortion levels. By comparing images of different reconstruction scales, the performance of the encoding and decoding method can be evaluated intuitively, and the image of the optimal reconstruction scale provides high-quality restoration effect for users.

[0115] In a feasible implementation, each pixel point is represented in a preset feature space. For any pixel point among the pixels, the relative distance between the pixel point and the saliency segmentation network in the preset feature space represents the absolute value of the saliency value corresponding to the pixel point. If the relative distance is smaller, it represents more saliency information contained in the pixel point, and the weight allocation value corresponding to the saliency value of the pixel point is larger. If the relative distance is larger, it represents less saliency information contained in the pixel point, and the weight allocation value corresponding to the saliency value of the pixel point is smaller.

[0116] It should be noted that each pixel is mapped to a preset feature space and becomes a coordinate point in the space. This feature space is artificially designed to capture various features of pixel points in the image, such as color, texture, shape, etc. When the saliency value is positive, it is predicted as a salient area. If it is negative, it is predicted as a background area. For each coordinate point, its relative distance from the saliency segmentation network in the feature space is calculated. The relative distance reflects the degree of proximity between the pixel represented by the coordinate point and the saliency area defined by the saliency segmentation network. Based on the size of the relative distance, a weight value is assigned to each pixel point. The smaller the relative distance, the more saliency information the pixel point contains, so the assigned weight value is also larger. The larger the relative distance, the less saliency information the pixel point contains, and the smaller the assigned weight value.

[0117] For example, to help understand the technical concept or technical principle of this application, please refer to Figure 4 , Figure 4 A geometric diagram of saliency segmentation is provided. In a preset feature space, Φ is the saliency segmentation network. Coordinate points above Φ are represented as positive and predicted as saliency areas. Coordinate points below Φ are represented as negative and predicted as background areas. For each coordinate point, the relative distance from the saliency segmentation network in the feature space represents the distance between the corresponding pixel point and the interface in the feature space after nonlinear transformation. The smaller the relative distance, the more saliency information the pixel point contains, and therefore the larger the assigned weight value, so that it is allocated more bits to retain more saliency information. Conversely, the larger the relative distance, the less saliency information the pixel point contains, and the smaller the assigned weight value, resulting in fewer allocated bits.

[0118] In this embodiment, by introducing a preset feature space and a saliency segmentation network, the saliency information in the image can be captured more accurately, and different weights and numbers of bits are assigned to different pixels based on this information, thereby improving the efficiency and quality of encoding, so that the saliency information in the image can be better preserved and transmitted under a limited bit rate.

[0119] For example, to help understand the technical concept or technical principle of this application, please refer to Figure 5 and Figure 6 , Figure 5 The overall encoding flowchart is provided, Figure 6 The model training flowchart is provided.

[0120] As Figure 5 shown, the input target image I is calculated by the calculation formula of the saliency value to obtain the saliency value of each point in the target image, and the saliency value is normalized to obtain the saliency probability. The saliency probability is applied to the hyperbolic cosine roll-off activation function to calculate the weight distribution value of each point, and a bit weight distribution map is generated The bit weight distribution map is three-dimensionally expanded to generate a three-dimensional mask At the same time, in the process of encoding the target image I, the original latent feature y is obtained, and the original latent feature y is divided into the original enhanced feature f e and the original basic feature f b in the channel dimension, that is, y = [f b , f e ], the three-dimensional mask is element-wise multiplied with the original latent feature y to obtain the encoded latent feature , wherein the encoded latent feature includes the first basic feature and the first enhanced feature , that is, The encoded latent feature is subjected to the coding and decoding of the double-scale entropy module (which includes the hyper-prior entropy model and the full decomposition entropy model) to determine the decoded feature , and the decoded feature is subjected to the decoding operation to reproduce the reconstructed image of the target image I

[0121] As Figure 6 shown, when the encoding network is trained, the input image x is analyzed by the main encoding network g a to generate the original enhanced feature f e and the original basic feature f b . The full zero feature is introduced for the original enhanced feature f e , and after the noise adding module is used to add noise to generate the noise-added full zero feature and the noise-added enhanced feature , the arithmetic coding is performed in combination with the full decomposition entropy model, the AE is the process of arithmetic coding, and the fourth storage bit stream Bits_f e0 and the third storage bit stream Bits_f e are generated, respectively. At the same time, the hyper-prior encoding network h a is introduced for the original basic feature fb The noise is added to generate the hyper-prior noise basis feature z b Then, arithmetic coding is performed to generate the fifth storage bit stream Bits_z b The fifth storage bit stream Bits_z b Arithmetic decoding is performed, AD is the process of arithmetic decoding, and the result of arithmetic decoding is input to the hyper-prior decoding network h s , and the decoding result is fed back to the original basis feature f b , to assist the encoding and decoding process of the original basis feature f b , to assist the execution of the encoding and decoding process of the original basis feature f b Arithmetic decoding is performed on the second storage bit stream Bits_f The fourth storage bit stream Bits_f e generated from the original enhancement feature f e0 and the third storage bit stream Bits_f e Arithmetic decoding is performed, respectively, to generate all-zero reconstructed feature and enhanced reconstructed feature The basis reconstructed feature is respectively concatenated with the all-zero reconstructed feature and the enhanced reconstructed feature , to generate the first decoded feature and the second decoded feature The first decoded feature and the second decoded feature are input to the main decoding network g s , and the optimal reconstructed scale reconstructed image and the worst reconstructed scale reconstructed image are output after analysis. The input image x, the optimal reconstructed scale reconstructed image and the worst reconstructed scale reconstructed image are input into the mean square error loss function, the mean square error is calculated, and the encoding parameters in the encoding network are adjusted accordingly, so that the performance of the encoding network reaches the optimal.

[0122] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the significant segmentation-oriented depth image coding method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0123] The present application also provides a significant segmentation-oriented depth image coding device, please refer to Figure 7 The significant segmentation-oriented depth image coding device comprises:

[0124] The probability obtaining module 10 is configured to obtain a saliency value of each pixel point in the target image, and determine a saliency probability distribution of the target image according to the saliency value of each pixel point.

[0125] The weight calculation module 20 is configured to calculate a weight distribution value of each pixel point based on a preset activation function and the saliency probability distribution.

[0126] The encoding module 30 is configured to perform channel pruning on the original latent feature of the target image according to the weight distribution value, to obtain an encoded latent feature, and encode the encoded latent feature based on a preset encoding network to obtain an encoding result.

[0127] Optionally, the encoding module 30 is further configured to:

[0128] obtain the original latent feature of the target image through a preset encoder;

[0129] generate a bit distribution weight map based on the weight distribution value, and expand the bit distribution weight map into a three-dimensional mask;

[0130] perform element point multiplication on the three-dimensional mask and the original latent feature, to prune a neural network channel with a weight distribution value lower than a preset weight distribution threshold, to obtain the encoded latent feature.

[0131] Optionally, the depth image encoding device for saliency segmentation further comprises a training module 40, and the training module 40 is further configured to:

[0132] for any one of the preset training sample images in the training phase, perform feature extraction on the preset training sample image to obtain a training encoded latent feature corresponding to the preset training sample image, wherein the training encoded latent feature comprises a first basic feature and a first enhanced feature;

[0133] encode the training encoded latent feature based on an initial encoding network to obtain a training encoding result;

[0134] decode and restore the training encoding result based on an initial decoding network to obtain a training reconstruction image;

[0135] update network parameters in the initial encoding network and the initial decoding network based on a mean square error between the training reconstruction image and the preset training sample image;

[0136] after a preset training condition is reached, the initial encoding network is taken as the preset encoding network, and the initial decoding network is taken as a preset decoding network for restoring the encoding result.

[0137] Optionally, the initial coding network includes a super priori entropy model and a full decomposition entropy model, and the training module 40 is further used to:

[0138] Performing entropy coding on the first basic feature by using a super prior entropy model to generate a second storage bit stream;

[0139] performing entropy coding on the first enhanced feature by using a full decomposition entropy model to generate a third storage bitstream;

[0140] An all-zero feature is introduced, and the first enhanced feature is entropy encoded in combination with the full decomposition entropy model to generate a fourth storage bit stream, wherein the dimension of the all-zero feature is the same as that of the first enhanced feature.

[0141] Optionally, the training module 40 is further configured to:

[0142] Decoding the second stored bit stream, the third stored bit stream, and the fourth stored bit stream to generate a basic reconstruction feature, an enhanced reconstruction feature, and an all-zero reconstruction feature, respectively;

[0143] The basic reconstruction feature and the enhanced reconstruction feature are channel-concatenated to generate a first decoding feature, and the basic reconstruction feature and the all-zero reconstruction feature are channel-concatenated to generate a second decoding feature. The first decoding feature represents the optimal reconstruction scale, and the second decoding feature represents the worst reconstruction scale;

[0144] If the target image is restored according to the first decoding feature, a first reconstructed image with an optimal reconstruction scale is generated;

[0145] If the target image is restored according to the second decoding feature, a second reconstructed image with the worst reconstruction scale is generated.

[0146] Optionally, each pixel point is represented in a preset feature space. For any pixel point among the pixels, the relative distance between the pixel point and the saliency segmentation network in the preset feature space represents the absolute value of the saliency value corresponding to the pixel point. If the relative distance is smaller, it represents more saliency information contained in the pixel point, and the weight allocation value corresponding to the saliency value of the pixel point is larger. If the relative distance is larger, it represents less saliency information contained in the pixel point, and the weight allocation value corresponding to the saliency value of the pixel point is smaller.

[0147] The depth image encoding device for saliency segmentation provided in this application utilizes the depth image encoding method for saliency segmentation in the above-mentioned embodiments, and can address the technical issue of poor performance of depth encoding in downstream saliency detection tasks. Compared to the prior art, the beneficial effects of the depth image encoding device for saliency segmentation provided in this application are the same as those of the depth image encoding method for saliency segmentation provided in the above-mentioned embodiments. Other technical features of the depth image encoding device for saliency segmentation are the same as those disclosed in the above-mentioned embodiments and are not further elaborated here.

[0148] The present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the depth image encoding method for saliency segmentation in the above-mentioned embodiment 1.

[0149] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, PADs (Portable Application Description: tablet computers), etc., and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0150] like Figure 8As shown, the electronic device can include a processing device 1001 (e.g., a central processor, a graphics processor, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or loaded from a storage device 1003 into a random access memory (RAM) 1004. Various programs and data required for operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate wirelessly or wired with other devices to exchange data. Although the electronic device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0151] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0152] The electronic device provided in the present application adopts the depth image coding method for saliency segmentation in the above-mentioned embodiments, which can solve the technical problem of low performance of depth coding in downstream saliency detection tasks. Compared with the prior art, the electronic device provided in the present application has the same beneficial effects as the depth image coding method for saliency segmentation provided in the above-mentioned embodiments, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0153] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0154] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0155] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the depth image encoding method for saliency segmentation in the above-mentioned embodiment.

[0156] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0157] The computer-readable storage medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0158] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the saliency partition oriented depth image coding device to: acquire a saliency value of each pixel point in a target image, determine a saliency probability of the saliency value of each pixel point; calculate a weight distribution value of each saliency probability based on a preset activation function; determine a coding latent feature of the target image according to each weight distribution value, and code the coding latent feature, wherein for any one of the weight distribution values, the weight distribution value is related to a corresponding distribution proportion in a first storage bit stream of the corresponding pixel point, and the greater the weight distribution value, the higher the corresponding distribution proportion in the first storage bit stream.

[0159] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0160] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or in the reverse order, depending on the functionality involved. It is also noted that each block in the block diagrams and / or flow diagrams and combinations of blocks in the block diagrams and / or flow diagrams can be implemented by special-purpose hardware-based systems that perform the specified functions or operations, or combinations of special-purpose hardware and computer instructions.

[0161] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0162] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the above-mentioned depth image coding method for saliency segmentation. Compared with the prior art, the computer readable storage medium provided by the present application can solve the technical problem of low performance of depth coding in the downstream saliency detection task. The beneficial effects of the computer readable storage medium provided by the present application are the same as those of the depth image coding method for saliency segmentation provided by the above-mentioned embodiments, and will not be repeated here.

[0163] The present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the depth image coding method for saliency segmentation as described above are implemented.

[0164] The computer program product provided by the present application can solve the technical problem of low performance of depth coding in the downstream saliency detection task. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the depth image coding method for saliency segmentation provided by the above-mentioned embodiments, and will not be repeated here.

[0165] The above-mentioned is only some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation, direct / indirect application in other related technical fields based on the technical concept of the present application, and the content of the present application specification and drawings are included in the patent protection scope of the present application.

Claims

1. A deep image coding method for saliency segmentation, characterized in that: The depth image coding method for saliency segmentation includes: Obtaining a saliency value of each pixel in a target image, and determining a saliency probability distribution of the target image according to the saliency value of each pixel; Calculating the weight distribution value of each pixel based on a preset activation function and the significance probability distribution; Performing channel cropping on original latent features of the target image according to the weight distribution values ​​to obtain encoded latent features, and encoding the encoded latent features based on a preset encoding network to obtain an encoding result, wherein, for any one of the weight distribution values, the weight distribution value is related to a distribution ratio of a corresponding pixel point in a first stored bitstream, and a larger weight distribution value indicates a higher corresponding distribution ratio in the first stored bitstream; The step of performing channel cropping on the original potential features of the target image according to the weight distribution value to obtain the encoded potential features includes: Obtaining original potential features of the target image through a preset encoder; generating a bit allocation weight map based on the weight allocation values, and expanding the bit allocation weight map into a three-dimensional mask; Performing element-wise multiplication of the three-dimensional mask and the original latent feature to clip the neural network channels whose weight distribution values ​​are lower than a preset weight distribution threshold to obtain the encoded latent feature; The depth image encoding method for saliency segmentation further includes: For any preset training sample image in the training phase, feature extraction is performed on the preset training sample image to obtain a training coding latent feature corresponding to the preset training sample image, wherein the training coding latent feature includes a first basic feature and a first enhanced feature; Encoding the training coding potential features based on the initial coding network to obtain a training coding result; Decoding and restoring the training encoding result based on the initial decoding network to obtain a training reconstructed image; updating network parameters in the initial encoding network and the initial decoding network based on a mean square error between the training reconstructed image and the preset training sample image; After a preset training condition is reached, the initial encoding network is used as the preset encoding network, and the initial decoding network is used as the preset decoding network for restoring the encoding result; The initial coding network includes a super priori entropy model and a full decomposition entropy model, and the step of encoding the training coding potential features based on the initial coding network to obtain a training coding result includes: Performing entropy coding on the first basic feature by using the super a priori entropy model to generate a second storage bitstream; Performing entropy coding on the first enhanced feature by using the full decomposition entropy model to generate a third storage bitstream; An all-zero feature is introduced, and entropy coding is performed on the first enhanced feature in combination with a full decomposition entropy model to generate a fourth storage bitstream, wherein the dimension of the all-zero feature is the same as that of the first enhanced feature.

2. The depth image coding method for saliency segmentation according to claim 1, wherein The step of decoding and restoring the training encoding result based on the initial decoding network to obtain a training reconstructed image includes: In the initial decoding network, the second stored bit stream, the third stored bit stream, and the fourth stored bit stream are decoded to generate a basic reconstruction feature, an enhanced reconstruction feature, and an all-zero reconstruction feature, respectively; Perform channel concatenation on the basic reconstruction feature and the enhanced reconstruction feature to generate a first decoding feature, perform channel concatenation on the basic reconstruction feature and the all-zero reconstruction feature to generate a second decoding feature, wherein the first decoding feature represents an optimal reconstruction scale, and the second decoding feature represents a worst reconstruction scale; If the target image is restored according to the first decoding feature, a first reconstructed image with an optimal reconstruction scale is generated; If the target image is restored according to the second decoding feature, a second reconstructed image with the worst reconstruction scale is generated.

3. The depth image coding method for saliency segmentation according to claim 1, wherein Each pixel point is represented in a preset feature space. For any pixel point among the pixel points, the relative distance between the pixel point and the saliency segmentation network in the preset feature space represents the absolute value of the saliency value corresponding to the pixel point. If the relative distance is smaller, it represents that the pixel point contains more saliency information and the saliency value of the pixel point corresponds to a larger weight allocation value. If the relative distance is larger, it represents that the pixel point contains less saliency information and the saliency value of the pixel point corresponds to a smaller weight allocation value.

4. A depth image encoding device for saliency segmentation, characterized in that: The depth image encoding device for saliency segmentation includes: A probability acquisition module is used to obtain the saliency value of each pixel in the target image and determine the saliency probability distribution of the target image according to the saliency value of each pixel; A weight calculation module, configured to calculate a weight distribution value for each pixel based on a preset activation function and the significance probability distribution; The encoding module is used to perform channel cropping on the original potential features of the target image according to the weight allocation value to obtain encoded potential features, and encode the encoded potential features based on a preset encoding network to obtain an encoding result, wherein, for any one of the weight allocation values, the weight allocation value is related to the allocation ratio of the corresponding pixel point in the first stored bit stream, and the larger the weight allocation value, the higher the corresponding allocation ratio in the first stored bit stream; wherein, the step of performing channel cropping on the original potential features of the target image according to the weight allocation value to obtain the encoded potential features includes: obtaining the original potential features of the target image through a preset encoder; generating a bit allocation weight map based on the weight allocation value, and expanding the bit allocation weight map into a three-dimensional mask; performing element-wise multiplication of the three-dimensional mask with the original potential features to crop the neural network channels whose weight allocation values ​​are lower than a preset weight allocation threshold to obtain the encoded potential features; the deep image encoding method for saliency segmentation also includes: for any one of the preset training sample images in the training stage, performing feature extraction on the preset training sample image to obtain a training encoding corresponding to the preset training sample image Latent features, wherein the training coding latent features include a first basic feature and a first enhanced feature; encoding the training coding latent features based on an initial coding network to obtain a training coding result; decoding and restoring the training coding result based on an initial decoding network to obtain a training reconstructed image; updating the network parameters in the initial coding network and the initial decoding network based on the mean square error between the training reconstructed image and the preset training sample image; after meeting the preset training conditions, using the initial coding network as the preset coding network and the initial decoding network as the preset decoding network for restoring the coding result; the initial coding network includes a super priori entropy model and a full decomposition entropy model, and the step of encoding the training coding latent features based on the initial coding network to obtain the training coding result includes: entropy encoding the first basic feature through the super priori entropy model to generate a second storage bit stream; entropy encoding the first enhanced feature through the full decomposition entropy model to generate a third storage bit stream; introducing an all-zero feature, entropy encoding the first enhanced feature in combination with the full decomposition entropy model to generate a fourth storage bit stream, wherein the dimension of the all-zero feature is the same as that of the first enhanced feature.

5. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the deep image coding method for saliency segmentation according to any one of claims 1 to 3.

6. A storage medium, characterized in that The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the depth image coding method for saliency segmentation according to any one of claims 1 to 3 are implemented.

7. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the deep image coding method for saliency segmentation according to any one of claims 1 to 3 are implemented.

Images