Underground video coding method and device

Abstract

The invention relates to an underground video coding method and device, and the method comprises the steps: obtaining to-be-coded video data based on the perception capability evaluation and video reconstruction of an underground video stream; obtaining a time-space domain saliency model based on the human eye visual perception structure; and carrying out bit allocation and code rate allocation on the to-be-coded video data based on the time-space domain saliency model, and determining coded data corresponding to the underground video stream. The underground video is subjected to lossless reconstruction based on a human visual perception mechanism, so that the visual redundancy is removed, the code rate is reduced, and the underground bandwidth is relieved; bit allocation and code rate allocation are performed on the to-be-coded video data through the time-space domain saliency model, so that the coding efficiency and the visual experience degree are remarkably improved, and the method has wide application value.

Description

Technical Field

[0001] The present application relates to the field of image recognition design and application technology, and in particular to a method and device for downhole video encoding. Background Art

[0002] Underground video monitoring in the existing technology can promote coal mine safety production and improve production efficiency. It is an important means of underground engineering safety supervision. The transmission of coal mine video data requires more underground transmission bandwidth. Therefore, it is particularly important to reduce the transmission volume before the transmission of coal mine video data. Traditional video coding technology usually samples a large amount of information first, and then removes the redundant information of the video information through a complex compression algorithm. However, in the special environment of the underground, these technologies often cannot achieve the ideal coding effect. Video coding based on visual perception usually divides the visual perception coding combined with human visual perception characteristics into two methods: preprocessing and embedded coding, depending on whether the internal structure of the encoder has changed. The preprocessing method starts from the visual perception mechanism and considers the spatial masking effect, brightness and contrast sensitivity, attention and other characteristics to preprocess the encoded video. Embedded coding improves the coding performance by modifying the internal structure of the encoder. However, the existing visual perception coding does not fully consider the visual perception characteristics of the human eye. This method focuses on improving the performance of the encoder and ignores the subjective visual experience of the decoded video image, resulting in excessive visual redundancy, uneven bit rate distribution and poor subjective visual perception effect.

[0003] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because they are described in the background technology section of the present application. Summary of the invention

[0004] The purpose of the present application is to provide a method and device for downhole video encoding, so as to solve the problem in the prior art of ignoring the subjective visual experience of the decoded video image, resulting in excessive visual redundancy, uneven bit rate distribution and poor subjective visual perception effect.

[0005] In order to solve the above problems, a downhole video encoding method involved in this application adopts the following technical solutions:

[0006] Based on the perception capability evaluation and video reconstruction of the underground video stream, the video data to be encoded is obtained;

[0007] Based on the human visual perception structure, obtain the spatiotemporal saliency model;

[0008] Based on the spatiotemporal saliency model, bit allocation and code rate allocation are performed on the video data to be encoded to determine the encoding data corresponding to the downhole video stream.

[0009] In order to solve the above problems, a downhole video encoding device involved in this application adopts the following technical solutions:

[0010] A first acquisition module, the first acquisition module is used to acquire the video data to be encoded based on the perception ability evaluation and video reconstruction of the downhole video stream;

[0011] A second acquisition module, the second acquisition module is used to acquire a spatiotemporal domain saliency model based on a human visual perception structure;

[0012] The encoding module is used to perform bit allocation and code rate allocation on the video data to be encoded based on the spatiotemporal saliency model, and determine the encoding data corresponding to the downhole video stream.

[0013] In order to solve the above problems, an electronic device involved in the present application adopts the following technical solution: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the downhole video encoding method involved in the present application.

[0014] In order to solve the above problems, a non-temporary computer-readable storage medium involved in the present application adopts the following technical solution: when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is able to execute the downhole video encoding method involved in the present application.

[0015] In order to solve the above problems, a computer program product involved in the present application adopts the following technical solution: it includes a computer program, and the computer program implements the downhole video encoding method involved in the present application when executed by a processor in a communication device.

[0016] The beneficial effects of this application are as follows:

[0017] Based on the human visual perception mechanism, the underground video is losslessly reconstructed to remove visual redundancy, reduce bit rate, and alleviate underground bandwidth. The bit allocation and bit rate allocation of the encoded video data are performed through the spatiotemporal saliency model, which significantly improves the encoding efficiency and visual experience, and has wide application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the embodiment of the present application, the following is a brief introduction to the drawings required for use in the embodiment:

[0019] Figure 1 A schematic diagram of a downhole video encoding method provided in an embodiment of the present application;

[0020] Figure 2A schematic diagram of a second downhole video encoding method flow chart provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of a third downhole video encoding method provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the structure of a downhole video encoding device provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application;

[0024] Figure 6 It is a schematic diagram of the structure of another electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the technical purpose, technical solution and beneficial effects of the present application clearer, the technical solution of the present application is further described below in conjunction with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0026] The downhole video encoding method and device according to the embodiment of the present application are described below with reference to the accompanying drawings.

[0027] Figure 1 The following is a flow chart of a method for downhole video encoding provided by an embodiment of the present application. Figure 1 As shown, the downhole video encoding method includes but is not limited to the following steps:

[0028] S101, obtaining video data to be encoded based on perception capability evaluation and video reconstruction of the underground video stream.

[0029] In a feasible implementation, the perception capability assessment mainly involves quality analysis of the initial frame image data in the underground video stream to determine whether it meets the requirements of subsequent video reconstruction and encoding, which usually includes image clarity assessment, noise level assessment, motion detection and tracking, etc. Image clarity assessment is to evaluate whether the image clarity meets the requirements by calculating indicators such as image resolution and contrast. For underground environments, the image clarity may be affected by factors such as lighting conditions and dust, so special attention should be paid. Noise level assessment is because underground video streams may contain various noises, such as electronic noise and environmental noise. By evaluating the noise level, the purity of the image data can be understood, which in turn provides a basis for subsequent filtering, denoising, and other processing. Since there are a large number of dynamic objects in underground video streams, such as miners, mechanical equipment, etc., motion detection and tracking can help understand the dynamic information in the video and provide a basis for key frame selection, motion compensation, and other subsequent video reconstruction and encoding.

[0030] Video reconstruction is a process of processing the initial frame image data to restore or enhance the video information based on the results of perception ability assessment. It usually includes image repair and enhancement, video frame interpolation, video compression and encoding, etc. For image quality problems in underground video streams, such as blur and noise, image repair and enhancement technology can be used to process them. For example, deep learning algorithms such as convolutional neural networks can be used to perform super-resolution reconstruction and denoising on images to improve image quality. In underground video streams, due to low frame rate or transmission delay, the video may not be smooth. At this time, video frame interpolation technology can be used to insert new frames between adjacent frames to improve the smoothness and frame rate of the video. After video reconstruction, the video data is usually compressed and encoded for storage and transmission. For underground environments, due to bandwidth and energy limitations, efficient video compression and encoding algorithms need to be used. For example, advanced video encoding standards such as H.264 and H.265 can be used to efficiently compress and encode video data.

[0031] After completing the perception ability assessment and video reconstruction, the video data to be encoded can be obtained. This process usually includes data preprocessing, encoder selection and configuration, encoding process, etc. Before sending the video data to the encoder, some preprocessing work is required, such as format conversion, color space adjustment, etc., which helps to ensure that the video data meets the input requirements of the encoder. According to the special needs of the downhole environment and the characteristics of the video data, select a suitable video encoder and configure it accordingly. For example, you can choose an encoder that supports features such as efficient compression and low power consumption to adapt to the limitations of the downhole environment. Send the preprocessed video data to the encoder for encoding. During the encoding process, you can adjust the encoding parameters as needed, such as bit rate, frame rate, resolution, etc., to optimize the encoding effect and performance.

[0032] S102, based on the visual perception structure of the human eye, a spatiotemporal domain saliency model is obtained.

[0033] In a feasible implementation, the human eye is a complex organ, mainly composed of the eyeball, ocular appendages, and visual pathways. These structures work together to enable humans to see the world clearly. Among them, the retina is the photosensitive layer of the eye, containing photoreceptor cells that can convert light signals into neural signals. These signals are then transmitted to the brain for processing to form the images and visual perceptions seen by humans. Using the color, brightness, texture and other features of the image, combined with the sensitivity of the human eye to these features, the spatial domain saliency in the image can be calculated by algorithms such as Fourier transform and center-edge difference. The motion information in the downhole video stream can be extracted using algorithms such as optical flow method and frame difference method, and the temporal domain saliency can be calculated in combination with the sensitivity of the human eye to moving objects. The spatial domain saliency and the temporal domain saliency are fused to form a spatiotemporal feature vector, and a spatiotemporal domain saliency model is obtained, wherein the weight distribution between different features and the sensitivity of the human eye to different features can be considered in the fusion process. Then, according to the spatiotemporal feature vector, machine learning classifiers, regressors and other models can be used to calculate the saliency. It should be noted that the higher the saliency value, the higher the visual importance of the region in vision. Then the spatiotemporal saliency model is optimized through cross-validation, grid search and other methods to improve the accuracy and robustness of the model.

[0034] S103, performing bit allocation and bit rate allocation on the video data to be encoded based on the spatiotemporal saliency model, and determining the encoding data corresponding to the downhole video stream.

[0035] In a feasible implementation, the region corresponding to the video data to be encoded is divided into saliency levels according to the saliency value obtained by the spatiotemporal saliency model. For example, the region can be divided into two categories of high saliency and low saliency based on a specific saliency value as a threshold, and then more bits are allocated to the high saliency region to ensure the image quality of these regions; fewer bits are allocated to the low saliency region to save coding resources. A bit allocation algorithm based on rate-distortion optimization can also be used to dynamically adjust the bit allocation according to the relationship between the saliency value and the coding distortion. The bit rate allocation of the region is determined according to the overall bit rate requirements of the downhole video stream and the saliency value of the region corresponding to the video data to be encoded. For the high saliency region, the bit rate can be appropriately increased to ensure the quality and details of the video; for the low saliency region, the bit rate can be appropriately reduced to save bandwidth and storage space.

[0036] Next, according to the characteristics and application scenarios of the downhole video stream, select a suitable video coding standard, such as H.264 / AVC or H.265 / HEVC. The downhole video stream is encoded using the selected coding standard and bit allocation and bit rate allocation strategies. During the encoding process, the coding parameters, such as quantization parameters, frame rate, etc., can be adjusted as needed to optimize the coding performance and image quality. After the encoding is completed, the obtained coded data is stored or transmitted, and the coded data can also be further processed and analyzed as needed, such as decoding and display. It should be noted that the performance evaluation of the encoded video stream is also included, such as image quality, bit rate, coding efficiency and other indicators; then, according to the evaluation results, the coding strategy is adjusted and optimized to meet the actual application requirements. Due to the complex and changeable downhole environment, there may be problems such as insufficient lighting and severe noise interference. Therefore, when constructing the spatiotemporal saliency model and performing encoding operations, the influence of these factors should be fully considered.

[0037] In summary, the downhole video is losslessly reconstructed based on the human visual perception mechanism, thereby removing visual redundancy, reducing the bit rate, and alleviating the downhole bandwidth; the bit allocation and bit rate allocation of the encoded video data are performed through the spatiotemporal saliency model, which significantly improves the coding efficiency and visual experience, and has wide application value.

[0038] Figure 2 This is a schematic diagram of the second downhole video encoding method provided in the embodiment of the present application. Figure 2 As shown, the downhole video encoding method includes but is not limited to the following steps:

[0039] S201, obtaining corresponding coefficient energy values ​​based on performing Kaluni-Löf transform on the downhole video stream.

[0040] In a feasible implementation, the Karhunen-Loeve transform, full name Karhunen-Loeve Transform, referred to as KLT, is a transformation based on statistical characteristics. KLT is an orthogonal transformation that transforms the original data into a new set of coordinate systems so that the components of the transformed data in the new coordinate system are independent of each other (i.e., decorrelation) and can be arranged in sequence according to the energy size (i.e., variance size). KLT performs best in the sense of mean square error. It can retain the information of the original data to the maximum extent while removing redundancy and noise. Specifically, for an N×N digital image or a set of vector data, KLT solves the covariance matrix of the data, finds the eigenvectors and eigenvalues ​​of the matrix, and then uses these eigenvectors to form a transformation matrix to transform the original data into a new coordinate system. The components of the transformed data in the new coordinate system are the projections of the original data in the direction of the eigenvectors. These components are independent of each other and are arranged according to the energy size.

[0041] For a given downhole video stream, multiple initial frame image data are extracted. Specifically, as an example, according to a preset first pixel value, the downhole video stream is decomposed into initial images I with a pixel size of M×N, frame by frame, where M and N are both positive integers greater than 1, and M×N is the preset first pixel value.

[0042] According to the preset second pixel value, each initial image I is then divided into a plurality of initial frame image data that do not overlap each other and are all square. For example, the size of the initial frame image data is K. 1 / 2 ×K 1 / 2 , K is a positive integer greater than 1. Then, multiple initial frame image data are combined to form a new first matrix A, where A = [I1, I2, ···, I S ], where is a K×1 matrix, S represents the total number of initial frame image data. The initial frame image data is usually set according to actual needs, which will not be repeated here.

[0043] Next, we obtain the covariance matrix corresponding to the first matrix A. The expression of the covariance matrix is ​​as follows:

[0044]

[0045] in, Represents the average row vector of the first matrix A.

[0046] Then calculate the multiple eigenvalues ​​of the covariance matrix and the multiple eigenvectors P corresponding to the eigenvalues N , where N is equal to 1, 2, ···, K; the eigenvector P N is a K×1 matrix. N Arrange in descending order based on the size of the eigenvalues ​​to obtain the second matrix P, where P = [P1, P2, . . . , P K ].

[0047] Then, based on the second matrix, the downhole video stream is subjected to a Kaluna-Lof transform to obtain a plurality of coefficient energy values. It should be noted that, in this embodiment, the initial image I corresponding to the downhole video stream is subjected to a Kaluna-Lof transform to obtain a plurality of coefficient energies. Specifically, the expression for the Kaluna-Lof transform of the initial image I is as follows:

[0048] Y=P T *I

[0049] Where Y=[y1,y2,···,y k ] T , is the coefficient matrix obtained by KLT; y k =(P K )T ×I,y k is the kth spectral component of the coefficient matrix. Next, calculate the spectral component y k KLT coefficient energy E n :

[0050]

[0051] Normalize the KLT coefficient to get the coefficient energy Ep n , coefficient energy Ep n The expression is as follows:

[0052]

[0053] It should be noted that there are K coefficient energies.

[0054] For further introduction to step S201, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0055] S202, obtaining the video data to be encoded according to the mapping relationship between the coefficient energy value and the visual perception structure of the human eye.

[0056] In a feasible implementation, a critical threshold corresponding to the human visual perception structure is determined according to the human visual perception structure. Then, a lossless perception threshold is determined according to a mapping between the critical threshold and the coefficient energy value. Then, based on an inverse Kalunas-Love transform of the lossless perception threshold, the video data to be encoded is obtained.

[0057] It should be noted that due to the complexity of human eye perception, it is impossible to accurately calculate the threshold information through objective data. In order to obtain the lossless perception threshold more reasonably, this embodiment can determine the critical threshold of each subject by analyzing the subjective feelings of the subject in combination with objective quantification. Specifically, a comprehensive evaluation can be made through indicators such as the subject's subjective score, the degree of eye fatigue of the subject, the peak signal-to-noise ratio of the objective picture, the structural similarity index of the picture, and the root mean square error of the picture. When the objective quantitative indicators are not much different, the critical threshold is determined by combining the subjective feelings of the subject.

[0058] As an example, 50 subjects can be selected, and the initial image I can be segmented by adjusting the parameters to obtain the initial frame image data; then the first matrix A is constructed; the covariance matrix corresponding to the first matrix A is obtained; the second matrix P is obtained; the initial image I is transformed by Kalunas-Löf transform to obtain the spectral component; K coefficient energies are calculated; the K coefficient energies Epn are reconstructed to obtain the KLT image; then the subjects subjectively feel the difference between the initial image I and the KLT image, and when the subjects do not perceive any difference (that is, the minimum coefficient energy), the test stops. At this time, the spectral component is characterized as a preliminary critical threshold, which is symbolized by Lx K , which means that the xth subject believes that there is no perceptual difference between the image reconstructed using K spectral components and the initial image I. Finally, the initial critical thresholds obtained by all subjects are weighted averaged to determine the critical threshold L of all subjects.

[0059] After obtaining the critical threshold L, the lossless perception threshold C for the initial image I is determined by mapping the critical threshold to the coefficient energy value. The specific operation process is as follows:

[0060]

[0061] in, It is expressed as the average value of coefficient energy accumulated from all initial images I when obtaining the lossless perceptual threshold; When the above relationship is satisfied, it indicates that the KTL image can restore all the macroscopic structures and sufficient microscopic details of the initial image I.

[0062] Next, we use the C spectral components to obtain the reconstructed image I by inverse Kalunas-Love transform. (C) The specific operations are as follows:

[0063] I (C) =P*Y (C)

[0064]

[0065] Among them, Y (C) is the reconstructed KLT coefficient matrix; I (C) represents the image reconstructed considering C spectral components. Since C is the lossless perception threshold, the image reconstructed using C spectral components can also be considered as lossless. Then I (C) Synthesize the video data to be encoded.

[0066] For further introduction to step S202, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0067] S203, based on the visual perception structure of human eyes, a spatiotemporal domain saliency model is obtained.

[0068] In a feasible implementation, the initial spatial domain saliency of the initial frame image data is obtained based on the inverse probability mass function calculation of the initial frame image data. Specifically, the co-occurrence matrix corresponding to each initial frame image data is obtained; all the co-occurrence counts of the co-occurrence matrix are counted to obtain the corresponding co-occurrence histogram; the co-occurrence histogram is normalized according to the overall background discontinuity characteristics of the initial frame image data and the characteristics of the significant area in the human eye visual perception structure to obtain the probability mass function; according to the inverse probability mass function calculation of the initial frame image data, the initial spatial domain saliency of the initial frame image data is obtained.

[0069] As an example, visual saliency detection is a common method for extracting salient regions of an image, which can automatically detect target regions that attract visual attention in unknown scenes. The co-occurrence matrix is ​​a matrix used to describe the spatial distribution characteristics of image grayscale (i.e., texture characteristics). First, the co-occurrence matrix H is defined as follows:

[0070] H=[h(e,f)],1≤e,f≤K

[0071] Among them, the co-occurrence matrix H is a K×K square matrix, the quantization level K=20 is set, the neighborhood radius is 4, the pixel with gray value e is taken as the center origin, and all pixel values ​​in the window with a neighborhood radius of 4 are traversed. If there is a pixel with gray value f, the pixel is called co-occurrence, all co-occurrence counts are counted, and then the histogram h(e,f) is obtained. Then, by using the overall background discontinuity characteristics of the initial frame image data and the salient area characteristics in the human eye visual perception structure, the probability mass function G(e,f) is obtained by normalizing the histogram. The initial spatial domain saliency S1(o) of pixel o can be calculated by the inverse probability mass function calculation inv G, and the expression is as follows:

[0072] S1(o)=∑invG(y o(i，j) ,y r(i，j) )

[0073] Among them, Y o(i , j) Represents the brightness component of pixel o; Y r(i , j) Represents the brightness component of the neighboring pixels of pixel o.

[0074] After obtaining the initial spatial domain saliency of the initial frame image data, the spatial domain saliency to be encoded can be obtained based on the conversion of the initial spatial domain saliency by the coding tree unit. It should be noted that the spatial domain saliency S1(o) of pixel o is the frame-level saliency, and the coding tree unit (Coding Tree Unit, abbreviated as CTU) can cut the input image into units of equal size for efficient video encoding, that is, to provide more accurate bit rate redistribution.

[0075] Assume that the number of CTUs is equal to the quantization level K of the co-occurrence matrix H (the number of CTUs should be set according to actual needs). The spatial domain saliency of the Kth CTU to be encoded is defined as Svs(K), and the expression of Svs(K) is as follows:

[0076]

[0077] Where d represents the size of CTU, and S1(o) represents the initial spatial domain saliency of the Kth CTU pixel o. Since the human eye has different sensitivities to different regions, the spatial domain saliency Svs(K) to be encoded is normalized to obtain the normalized spatial domain saliency W1(K), where W1(K) is expressed as follows:

[0078]

[0079] It should be noted that if the value of the normalized spatial domain saliency W1(K) is larger, it means that the image area is more likely to be noticed by the human eye and the tolerable distortion is smaller, which means that more bit rate is needed to compress and encode it.

[0080] On the other hand, the initial frame image data is recursively divided based on the coding tree unit to obtain the time domain saliency to be encoded. Further, the initial frame image data is recursively divided based on the coding tree unit to obtain the first coding unit. Based on the absolute error and operation of the brightness component between the first coding unit and the preset second coding unit, the time domain saliency to be encoded is obtained. It should be noted that the motion characteristics of the video can be used to describe the sensitivity of human visual perception to different visual contents. Since the coding tree unit (i.e., CTU) can cut the initial frame image data into units of equal size (i.e., the first coding unit), the cutting process can be characterized as a recursive division operation. The absolute error (Sum of AbsoluteDifference, SAD) of the brightness component between the first coding unit and the preset second coding unit (as a reference) can be used to characterize the degree of motion of the first coding unit. If the SAD is larger, it indicates that the first coding unit moves more violently.

[0081] Furthermore, the brightness component Y of the pixel o can be o(i, j) and the brightness component Y of the neighboring pixels of pixel o r(i , j) To calculate the initial time domain saliency S2(o), the expression of S2(o) is as follows:

[0082]

[0083] Assume that the number of first coding units is equal to the quantization level K of the co-occurrence matrix H (the number of first coding units should be set according to actual needs). The time domain saliency to be encoded of the Kth first coding unit is defined as Tvs(K), and the expression of Tvs(K) is as follows:

[0084]

[0085] Where d represents the size of CTU, and S2(o) represents the initial time domain saliency of the pixel point o of the Kth first coding unit. Similar to the spatial domain saliency to be encoded, the time domain saliency to be encoded Tvs(K) is normalized to obtain the normalized time domain saliency W2(K), where W2(K) is expressed as follows:

[0086]

[0087] It should be noted that if the value of the normalized temporal saliency W2(K) is larger, it means that the image area is more likely to be noticed by the human eye and the tolerable distortion is smaller, which means that more bit rate is needed to compress and encode it.

[0088] Next, based on the fusion of the spatial domain saliency to be encoded and the temporal domain saliency to be encoded, the spatiotemporal domain saliency model is obtained. It should be noted that the spatiotemporal domain saliency model is composed of the fused spatiotemporal domain saliency, and the fused spatiotemporal domain saliency can be represented as W12(K), where the expression of W12(K) is as follows:

[0089] W12(K)=θ1×W1(K)+θ2×W2(K)+θ3×W1(K)×W2(K)

[0090] Among them, θ1, θ2, and θ3 are all weight component parameters; there is no competitive relationship between the spatial domain saliency to be encoded and the temporal domain saliency to be encoded, and the weight component parameters of the two can be the same, and for θ3, it can be fitted through experiments. As an example, θ3 can be (θ1 2 +θ2 2 ) / 2.

[0091] For further introduction to step S203, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0092] S204, performing bit allocation and bit rate allocation on the video data to be encoded based on the spatiotemporal saliency model, and determining the encoding data corresponding to the downhole video stream.

[0093] In a feasible implementation, the target number of bits is obtained by allocating bits of the encoded data according to the spatiotemporal saliency model. It should be noted that, assuming that the number of CTUs is equal to the quantization level K of the co-occurrence matrix H, the target number of bits of the Kth CTU T CTU The expression of (K) is as follows:

[0094]

[0095] The proportionality coefficient is obtained by the ratio of the fused spatiotemporal saliency of the encoded CTU at the current moment to the fused spatiotemporal saliency of the remaining unencoded CTUs.

[0096] After obtaining the target bit number, the target bit number is allocated with a bit rate according to the Lagrangian factor to obtain the coded data corresponding to the downhole video stream. It should be noted that the rate-distortion function in the form of a hyperbola can reflect the relationship between the video bit rate and distortion. The expression of the rate-distortion function D is as follows:

[0097] D(bit rate) = U*bit rate -V

[0098] Among them, U and V are model parameters related to the content characteristics of the underground video stream. Then the Lagrangian factor λ in the spatiotemporal saliency model can be expressed as:

[0099]

[0100] Among them, α and β are model parameters related to the content characteristics of the downhole video stream.

[0101] Furthermore, according to the human visual attention mechanism, the Lagrangian factor of the Kth CTU can be further refined as

[0102] λ VPS (K), where λ VPS (K) can be expressed as:

[0103]

[0104] It should be noted that in order to ensure the stability of video quality in the time domain and reduce fluctuations, the quantization parameters between adjacent CTUs can be constrained, where the constraint condition can be expressed as:

[0105]

[0106] Among them, currCTU represents the relevant parameters of the current CTU, and AdjoinCTU represents the relevant parameters of the CTU adjacent to the current CTU. Both currCTU and AdjoinCTU are adjusted according to specific application scenarios. The detailed parameter adjustment process is not repeated here.

[0107] When encoding, it is usually implemented based on the quantization parameter QP and the Lagrangian factor λ VPS The relationship between them is as follows:

[0108] QP=δ1*ln(λ VPS )+δ2

[0109] Among them, δ1 and δ2 are both adjusted by the specific Lagrangian factor λ VPS The specific acquisition process will not be described here.

[0110] Then, according to the quantization parameter, the encoding data corresponding to the downhole video stream is determined.

[0111] For further introduction to step S204, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0112] In summary, the downhole video is losslessly reconstructed based on the human visual perception mechanism, thereby removing visual redundancy, reducing the bit rate, and alleviating the downhole bandwidth; the bit allocation and bit rate allocation of the encoded video data are performed through the spatiotemporal saliency model, which significantly improves the coding efficiency and visual experience, and has wide application value.

[0113] Figure 3 The third downhole video encoding method provided in the embodiment of the present application is a flow chart. Figure 3 As shown, the downhole video encoding method includes but is not limited to the following steps:

[0114] S301, obtaining downhole video stream.

[0115] For further introduction to step S301, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0116] S302, based on disassembling the downhole video stream, obtaining a plurality of initial frame image data.

[0117] For further introduction to step S302, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0118] S303, obtaining a plurality of coefficient energy values ​​based on performing Kaluni-Löf transform on the initial frame image data.

[0119] For further introduction to step S303, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0120] S304: Determine a lossless perception threshold corresponding to each initial frame image data according to the visual perception structure of the human eye.

[0121] For further introduction to step S304, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0122] S305 , obtaining to-be-encoded video data based on performing an inverse Kaluni-Löf transform on the lossless perceptual threshold.

[0123] For further introduction to step S305, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0124] S306, based on the inverse probability mass function calculation of the initial frame image data, the spatial domain saliency to be encoded is obtained; based on the coding tree unit, the initial frame image data is recursively divided to obtain the time domain saliency to be encoded.

[0125] For further introduction to step S306, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0126] S307, acquiring a spatiotemporal saliency model based on fusing the spatial domain saliency to be encoded and the temporal domain saliency to be encoded.

[0127] For further introduction to step S307, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0128] S308, performing bit allocation on the video data to be encoded based on the spatiotemporal saliency model to obtain a target number of bits.

[0129] For further introduction to step S308, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0130] S309, based on the bit rate allocation for the target bit number, determine the encoding data corresponding to the downhole video stream.

[0131] For further introduction to step S309, please refer to the relevant contents in the above embodiment, which will not be repeated here.

[0132] In summary, the downhole video is losslessly reconstructed based on the human visual perception mechanism, thereby removing visual redundancy, reducing the bit rate, and alleviating the downhole bandwidth; the bit allocation and bit rate allocation of the encoded video data are performed through the spatiotemporal saliency model, which significantly improves the coding efficiency and visual experience, and has wide application value.

[0133] Figure 4 This is a schematic diagram of the structure of a downhole video encoding device provided in an embodiment of the present application. Figure 4 As shown, the downhole video encoding device 400 includes:

[0134] A first acquisition module 401, the first acquisition module 401 is used to acquire the video data to be encoded based on the perception capability evaluation and video reconstruction of the downhole video stream;

[0135] A second acquisition module 402, the second acquisition module 402 is used to acquire a spatiotemporal domain saliency model based on a human visual perception structure;

[0136] The encoding module 403 is used to perform bit allocation and code rate allocation on the video data to be encoded based on the spatiotemporal saliency model, and determine the encoding data corresponding to the downhole video stream.

[0137] Figure 5 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0138] like Figure 5 As shown, the electronic device 500 includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a memory 506 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processor 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0139] The following components are connected to the I / O interface 505: a memory 506 including a hard disk, etc.; and a communication part 507 including a network interface card such as a LAN (Local Area Network) card, a modem, etc., and the communication part 507 performs communication processing via a network such as the Internet; a drive 508 is also connected to the I / O interface 505 as needed.

[0140] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 507. When the computer program is executed by the processor 501, the above-mentioned functions defined in the method of the present application are executed.

[0141] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by the processor 501 of the electronic device 500 to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0142] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0143] Figure 6 It is a schematic diagram of the structure of another electronic device provided according to an embodiment of the present application. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application. Figure 6 As shown, the electronic device 600 includes a processor 601 and a memory 602. The memory 602 is used to store program codes, and the processor 601 is connected to the memory 602 to read the program codes from the memory 602 to implement the downhole video encoding method in the above embodiment.

[0144] Optionally, the number of processors 601 may be one or more.

[0145] Optionally, the electronic device may further include an interface 603, and the number of the interface 603 may be multiple. The interface 603 may be connected to an application program and may receive data from an external device such as a sensor.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate but not limit the technical solutions of the present application. Any equivalent substitutions of the present application and any modifications or partial substitutions that do not depart from the spirit and scope of the present application shall be included in the scope of protection of the claims of the present application.

Claims

1. A method for downhole video encoding, characterized in that: include: Based on the perception capability evaluation and video reconstruction of the underground video stream, the video data to be encoded is obtained; Based on the human visual perception structure, obtain the spatiotemporal saliency model; Based on the spatiotemporal saliency model, bit allocation and code rate allocation are performed on the video data to be encoded, and encoding data corresponding to the downhole video stream is determined.

2. The method according to claim 1, characterized in that The method of obtaining the video data to be encoded based on the perception capability evaluation and video reconstruction of the downhole video stream includes: Based on the Kaluna-Löf transform of the downhole video stream, the corresponding coefficient energy value is obtained; According to the visual perception structure of the human eye, determining the critical threshold corresponding to the visual perception structure of the human eye; Determining a lossless perception threshold according to a mapping between the critical threshold and the coefficient energy value; Based on performing an inverse Kaluni-Löf transform on the lossless perceptual threshold, video data to be encoded is obtained.

3. The method according to claim 2, characterized in that The method is based on performing Kaluna-Löf transform on the downhole video stream to obtain multiple coefficient energy values, including: Extracting a plurality of initial frame image data from a downhole video stream; Constructing a first matrix corresponding to a plurality of the initial frame image data; Obtaining a covariance matrix corresponding to the first matrix; Determine a plurality of eigenvalues ​​of the covariance matrix and a plurality of eigenvectors corresponding to the eigenvalues; Arrange the plurality of eigenvectors in descending order according to the eigenvalues ​​to obtain a second matrix; Based on the second matrix, a Kaluni-Löf transform is performed on the downhole video stream to obtain multiple coefficient energies.

4. The method according to claim 3, characterized in that The obtaining of a plurality of initial frame image data corresponding to the downhole video stream comprises: Decomposing the downhole video stream according to a preset first pixel value to obtain a plurality of initial images; According to the preset second pixel value, each of the initial images is decomposed into a plurality of initial frame image data which are non-overlapping and all are square.

5. The method according to claim 3, characterized in that: The method of performing a Kaluna-Löf transform on the downhole video stream based on the second matrix to obtain multiple coefficient energies includes: Obtaining a coefficient matrix based on performing a Kaluni-Love transform on the second matrix; Based on performing spectral component calculation and normalization operations on the coefficient matrix, a plurality of coefficient energy values ​​are obtained.

6. The method according to claim 1, characterized in that The method of obtaining a spatiotemporal saliency model based on the human visual perception structure includes: Based on performing an inverse probability mass function calculation on the initial frame image data, obtaining an initial spatial domain saliency of the initial frame image data; Based on the conversion of the initial spatial domain saliency by the coding tree unit, obtaining the spatial domain saliency to be coded; Recursively dividing the initial frame image data based on the coding tree unit to obtain the time domain saliency to be coded; Based on fusing the spatial domain saliency to be encoded and the temporal domain saliency to be encoded, a spatiotemporal domain saliency model is obtained.

7. The method according to claim 6, characterized in that The step of obtaining the initial spatial domain saliency of the initial frame image data based on performing an inverse probability mass function calculation on the initial frame image data comprises: Obtain the co-occurrence matrix corresponding to each initial frame image data; Counting all the co-occurrence counts of the co-occurrence matrix to obtain a corresponding co-occurrence histogram; Normalizing the co-occurrence histogram according to the overall background discontinuity characteristics of the initial frame image data and the significant region characteristics in the human eye visual perception structure to obtain a probability mass function; According to the inverse probability mass function calculation performed on the initial frame image data, the initial spatial domain saliency of the initial frame image data is obtained.

8. The method according to claim 6, characterized in that The recursively dividing the initial frame image data based on the coding tree unit to obtain the time domain saliency to be coded includes: Recursively dividing the initial frame image data based on the coding tree unit to obtain a first coding unit; Based on performing an absolute error sum operation on the brightness component between the first encoding unit and a preset second encoding unit, the time domain saliency to be encoded is obtained.

9. The method according to claim 1, characterized in that: The step of performing bit allocation and code rate allocation on the video data to be encoded based on the spatiotemporal saliency model to determine the encoded data corresponding to the downhole video stream includes: Obtaining a target number of bits by performing bit allocation on the data to be encoded according to the spatiotemporal saliency model; According to the Lagrangian factor, the target bit number is allocated with a bit rate to obtain the encoded data corresponding to the downhole video stream.

10. A downhole video encoding device, characterized in that: include: A first acquisition module, the first acquisition module is used to acquire the video data to be encoded based on the perception ability evaluation and video reconstruction of the downhole video stream; A second acquisition module, the second acquisition module is used to acquire a spatiotemporal domain saliency model based on a human visual perception structure; The encoding module is used to perform bit allocation and code rate allocation on the video data to be encoded based on the spatiotemporal saliency model, and determine the encoding data corresponding to the downhole video stream.

Images