Panoramic video transmission method and apparatus, storage medium, and electronic device

Abstract

The application discloses a panoramic video transmission method and device, a storage medium and an electronic device. The method comprises the following steps: receiving a video stream from a sending terminal device, and decomposing the video stream into an image set; dividing the images in the image set into different types of image regions according to the user's visual field, wherein the different types of regions correspond to different visual fields; determining a target type region in the different image regions, encoding the target type region, dividing the encoded target type region into blocks, and obtaining a plurality of block images; compressing the plurality of block images using different quantization steps to obtain a target block image; and sending the target block image and other image regions to a receiving terminal device. The application solves the technical problem of unreasonable occupation of channels and waste of resources in real-time transmission of panoramic video caused by uniform quantization step transmission of panoramic video.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding, and more specifically, to a method, apparatus, storage medium, and electronic device for transmitting panoramic video. Background Technology

[0002] In the era of 5G communication and gigabit optical networks, more and more people are using virtual reality (VR) technology for entertainment and work. Panoramic video live streaming is an important manifestation of VR technology. Compared to traditional video live streaming, panoramic video has higher resolution and frame rate, requiring the transmission of more data and consuming more bandwidth resources during transmission. The encoding and decoding methods used in the compression, transmission, and reception of panoramic video directly affect the user's viewing experience.

[0003] Currently, panoramic video live streaming typically uses uniform quantization step size for transmission. However, due to different channel transmission conditions for different users, and different transmission conditions for the same channel at different times, encoding the content according to a constant compression processing unit and encoding method will result in unreasonable channel occupation and waste of resources. The disadvantages of using uniform quantization step size for transmission are even more prominent in the scenario of real-time transmission of panoramic video.

[0004] Furthermore, for higher resolution panoramic videos, using the same H.264 encoding scheme as planar videos will drastically increase the number of processing units, leading to increased redundant information and reduced compression efficiency.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a method, apparatus, storage medium, and electronic device for transmitting panoramic video, to at least solve the technical problem of unreasonable channel occupation and resource waste during real-time transmission of panoramic video caused by using uniform quantization step size for transmitting panoramic video.

[0007] According to one aspect of the embodiments of this application, a method for transmitting panoramic video is provided, comprising: receiving a video stream from a transmitting device and decomposing the video stream into an image set; dividing the images in the image set into regions according to the user's field of view to obtain different types of image regions, wherein the different types of regions correspond to different fields of view; determining target type regions in the different image regions, encoding the target type regions, dividing the encoded target type regions into blocks to obtain multiple block images, wherein the multiple block images have different human visual sensitivities; compressing the multiple block images using different quantization step sizes to obtain a target block image; and sending the target block image and other image regions to a receiving device.

[0008] Optionally, the different types of image regions include: a first type region, a second type region, and a third type region, wherein the field of view of the first type region, the second type region, and the third type region increases sequentially.

[0009] Optionally, before encoding the target type region, the method further includes: performing discrete cosine transform encoding on the first type region, the second type region, and the third type region, and filtering the discrete cosine coefficients; determining different thresholds corresponding to the first type region, the second type region, and the third type region according to the algorithm for minimum perceptible distortion by the human eye; determining redundant regions in the first type region, the second type region, and the third type region according to the relationship between the discrete cosine coefficients and the thresholds; and clearing redundant regions.

[0010] Optionally, the encoded target type region is divided into blocks, including: weighting the AC coefficient energy using the HVS visual function to obtain the judgment coefficient, wherein the AC coefficient energy is obtained by discrete cosine transform encoding; and dividing the encoded target type region into blocks according to the judgment coefficient, wherein the target type region is a first type region.

[0011] Optionally, compression is performed on multiple block images using different quantization step sizes, including: establishing a normalization function based on the judgment coefficient and the average value of the total AC coefficient as variables for different first-type region blocks; determining the quantization step size for different first-type region blocks based on the numerical range defined by the normalization function; selecting the quantization step size based on the sensitivity of the human eye to different first-type region blocks; and compressing different first-type region blocks based on the quantization step size.

[0012] Optionally, the quantization step size is selected based on the sensitivity of the human eye to different first-type region blocks, including: selecting a first quantization step size for first-type region blocks corresponding to highly sensitive regions of the human eye; and selecting a second quantization step size for first-type region blocks corresponding to low-sensitive regions of the human eye, wherein the first quantization step size is smaller than the second quantization step size.

[0013] Optionally, sending the target block image, the second type region, and the third type region to the receiving device includes: determining the transmission priority of the first type region, the second type region, and the third type region, wherein the priority of the first type region, the second type region, and the third type region decreases sequentially; and sending the target block image, the second type region, and the third type region to the receiving device according to the transmission priority.

[0014] Optionally, determining the transmission priority of the first type area, the second type area, and the third type area includes: determining the target network bandwidth and target field of view of the receiving device during the live broadcast of the video stream; and determining the transmission priority of the first type area, the second type area, and the third type area based on the target network bandwidth and target field of view of the receiving device.

[0015] Optionally, sending the target block image and other image regions to the receiving device includes: compressing the second type region according to a first preset step size to obtain a target second type region; compressing the third type region according to a second preset step size to obtain a target third type region; and sending the target block image, the second type region, and the third type region to the receiving device.

[0016] According to another aspect of the embodiments of this application, an apparatus for transmitting panoramic video is also provided, comprising: a decomposition module for receiving a video stream from a transmitting device and decomposing the video stream into an image set; a segmentation module for dividing the images in the image set into regions according to the user's field of view to obtain different types of image regions, wherein the different types of regions correspond to different fields of view; an encoding module for determining target type regions in different image regions, encoding the target type regions, and dividing the encoded target type regions into blocks to obtain multiple block images, wherein the multiple block images have different human visual sensitivities; a compression module for compressing the multiple block images using different quantization step sizes to obtain target block images; and a transmission module for transmitting the target block images and other image regions to a receiving device.

[0017] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, comprising: the storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute any one of the methods for transmitting panoramic video.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement any of the methods for transmitting panoramic video.

[0019] In this embodiment, a non-uniform step size encoding method is used for different types of image regions to encode the video stream. The video stream is received from the transmitting device and decomposed into image sets. The images in the image sets are divided into regions according to the user's field of view, resulting in different types of image regions, each with a different field of view. Target type regions are determined within these different image regions, encoded, and then divided into blocks to obtain multiple block images. These block images are compressed using different quantization step sizes to obtain target block images. The target block images and other image regions are then sent to the receiving device. This achieves reduced latency and stuttering during panoramic video live streaming, thus meeting the processing requirements for transmitting massive amounts of panoramic video data. It improves the efficiency of panoramic video data transmission and the playback quality of panoramic videos. Combined with the characteristics of human visual discrimination, it improves the human visual experience of panoramic videos and flexibly adjusts the transmission order of different video regions. This solves the technical problems of unreasonable channel occupation and resource waste caused by using uniform quantization step size for panoramic video transmission in real-time. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a flowchart illustrating a method for transmitting panoramic video according to an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating an optional method for transmitting panoramic video according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a device structure for transmitting panoramic video according to an embodiment of this application;

[0024] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of this application is shown. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] To facilitate a better understanding by those skilled in the art, the technical terms or some nouns that may be involved in this application are explained below in the relevant embodiments:

[0028] The human visual resolution characteristic (HVS) is an important feature of the human visual system that forms vision by perceiving light. HVS plays an important role in the subjective optimization of video coding. The main characteristics of human visual resolution are as follows: (1) In terms of spatial frequency domain, the human eye is a low-pass linear system. Since the human eye pupil has a certain geometric size and optical aberration, and the visual cells have a certain size, the resolution of the human eye is limited, and the human eye is not sensitive to high resolution. (2) The human eye's response to brightness has a logarithmic nonlinear property to achieve the dynamic range of brightness. In areas with high average brightness, the human eye is not sensitive to grayscale errors. (3) The spatial resolution of the human eye for brightness signals is greater than that for chrominance signals. (4) Due to the regulation of the nervous system, the human eye has the characteristics of a bandpass linear system, and the human eye will produce an edge enhancement sensation: lateral inhibition effect. (5) The human eye can easily perceive changes in edge position, but the human eye is not sensitive to grayscale errors at the edges. (6) The visual masking effect of the human eye is a local effect, which is affected by background illumination, texture complexity and signal frequency. Regions with different local characteristics can alter signal strength without being detected by the human eye.

[0029] The DCT transformation is the Discrete Cosine Transform. The DCT transformation belongs to a type of Fourier transform and is commonly used for lossy data compression of signals and images (including pictures and videos). The DCT divides an image into small blocks composed of different frequencies and then performs quantization. During the quantization process, high-frequency components are discarded, and the remaining low-frequency components are saved for subsequent image reconstruction. Essentially, the discrete cosine transform requires a set of N similar data points. After the transformation, N dissimilar data points are returned. Its characteristic is that the energy is compressed into only M coefficients, where M < N. Among them, the DCT compresses the energy of the matrix into the first element, which is called the direct current (DC) coefficient, and the remaining coefficients are called alternating current (AC) coefficients.

[0030] JND is the Just Noticeable Difference, which is used to measure the sensitivity of the human eye to distortions in different regions of an image and is mostly used in image / video coding, digital watermarking, and image quality evaluation based on visual characteristics. Currently, multiple JND models have been proposed, mainly divided into two categories: pixel-domain based JND models and DCT-domain based JND models.

[0031] MPC is the Model Predictive Control algorithm. The working mechanism of MPC is described as follows: At each sampling moment, according to the currently obtained measurement information, an online solution to a finite-time open-loop optimization problem is carried out, and the first element of the obtained control sequence is applied to the controlled object. At the next sampling moment, the above process is repeated: using the new measurement value as the initial condition for predicting the future dynamics of the system at this time, refreshing the optimization problem and re-solving it.

[0032] According to an embodiment of the present application, a method embodiment for transmitting panoramic videos is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0033] Figure 1 This is a method for transmitting panoramic videos according to an embodiment of the present application, as Figure 1 shown. This method includes the following steps:

[0034] Step S102: Receive the video stream from the sending-end device and decompose the video stream into an image set;

[0035] It can be understood that the video stream is decomposed into images frame by frame, and different video stream extraction speeds can be set for decomposition.

[0036] Step S104: Divide the images in the image set according to the user's field of view to obtain different types of image regions, where different types of regions correspond to different fields of view;

[0037] It should be noted that before dividing the images in the image set into regions, the image states in the image set are determined. The image states include initial states and non-initial states. The initial state refers to the state in which the image has not been predicted based on the user's field of view, while the non-initial state refers to the state in which the image has been predicted based on the user's field of view.

[0038] It is understandable that different types of areas are divided based on the human eye's field of vision combined with the panoramic video projection format.

[0039] For example, an image can be divided into regions according to different user field of view, where the user field of view includes the horizontal field of view, the vertical field of view, and the diagonal field of view.

[0040] Step S106: Determine the target type regions in different image regions, encode the target type regions, divide the encoded target type regions into blocks to obtain multiple block images, wherein the human visual sensitivity of the multiple block images is different.

[0041] Step S108: Compress multiple block images using different quantization step sizes to obtain the target block image;

[0042] Step S110: Send the target block image and other image regions to the receiving device.

[0043] In this embodiment, a non-uniform step size encoding method is used for different types of image regions to encode the video stream. The video stream is received from the transmitting device and decomposed into image sets. The images in the image sets are divided into regions according to the user's field of view, resulting in different types of image regions, each with a different field of view. Target type regions are determined within these different image regions, encoded, and then divided into blocks to obtain multiple block images. These block images are compressed using different quantization step sizes to obtain target block images. The target block images and other image regions are then sent to the receiving device. This achieves reduced latency and stuttering during panoramic video live streaming, thus meeting the processing requirements for transmitting massive amounts of panoramic video data. It improves the efficiency of panoramic video data transmission and the playback quality of panoramic videos. Combined with the characteristics of human visual discrimination, it improves the human visual experience of panoramic videos and flexibly adjusts the transmission order of different video regions. This solves the technical problems of unreasonable channel occupation and resource waste caused by using uniform quantization step size for panoramic video transmission in real-time.

[0044] In some optional embodiments of this application, different types of image regions include: a first type region, a second type region, and a third type region, wherein the field of view of the first type region, the second type region, and the third type region increases sequentially.

[0045] In an exemplary embodiment of this application, before encoding the target type region, the method further includes: performing discrete cosine transform encoding on the first type region, the second type region, and the third type region, and filtering the discrete cosine coefficients; determining different thresholds corresponding to the first type region, the second type region, and the third type region according to the minimum perceptible distortion algorithm; determining redundant regions in the first type region, the second type region, and the third type region according to the relationship between the discrete cosine coefficients and the thresholds; and clearing redundant regions.

[0046] For example, the discrete cosine coefficients are filtered by a preprocessing filter, and the thresholds for different types of image regions are determined according to the algorithm for minimum perceptible distortion by the human eye. When the discrete cosine coefficients are less than or equal to the threshold, redundant areas of the image region are removed.

[0047] It should be noted that redundant areas include areas corresponding to visual elements whose visual perception changes little, including but not limited to color, contrast, and resolution.

[0048] Understandably, panoramic video has a much larger data volume than planar video. By combining it with an algorithm that minimizes the perceptible distortion to the human eye to eliminate image redundancy, the problem of slow transmission rate caused by the need to transmit massive amounts of data when transmitting panoramic video is solved.

[0049] As an optional implementation, the encoded target type region is divided into blocks, including: weighting the AC coefficient energy using the HVS visual function to obtain the judgment coefficient, wherein the AC coefficient energy is obtained by discrete cosine transform encoding; and dividing the encoded target type region into blocks according to the judgment coefficient, wherein the target type region is a first type region.

[0050] In an exemplary embodiment of this application, compression of multiple block images using different quantization step sizes includes: establishing a normalization function with the judgment coefficient and the average value of the total AC coefficient as variables for different first-type region blocks; determining the quantization step size for different first-type region blocks according to the numerical range defined by the normalization function; selecting the quantization step size according to the sensitivity of the human eye to different first-type region blocks; and compressing the different first-type region blocks according to the quantization step size.

[0051] Understandably, using different quantization step sizes for compression ensures high coding efficiency while balancing compression quality, effectively improving quantization accuracy, and enhancing the visual experience of panoramic videos under high compression ratios.

[0052] In some optional embodiments of this application, the quantization step size is selected according to the sensitivity of the human eye to different first-type region blocks, including: selecting a first quantization step size for first-type region blocks corresponding to highly sensitive regions of the human eye; and selecting a second quantization step size for first-type region blocks corresponding to low-sensitive regions of the human eye, wherein the first quantization step size is smaller than the second quantization step size.

[0053] As an optional implementation, sending the target block image, the second type region, and the third type region to the receiving device includes: determining the transmission priority of the first type region, the second type region, and the third type region, wherein the priority of the first type region, the second type region, and the third type region decreases sequentially; and sending the target block image, the second type region, and the third type region to the receiving device according to the transmission priority.

[0054] Optionally, determining the transmission priority of the first type area, the second type area, and the third type area includes: determining the target network bandwidth and target field of view of the receiving device during the live broadcast of the video stream; and determining the transmission priority of the first type area, the second type area, and the third type area based on the target network bandwidth and target field of view of the receiving device.

[0055] For example, when the target network bandwidth of the receiving device is large during the current time period, the first type of area is sent to the receiving device first, and then the second type of area and the third type of area are sent to the receiving device in sequence; when the target network bandwidth of the receiving device is small during the current time period, the target block image is sent to the receiving device first, until the target network bandwidth is restored, and the transmission order of the second type of area and the third type of area is dynamically adjusted.

[0056] For example, if the target field of view of the receiving device is the first type of area during the current time period, the first type of area will be sent to the receiving device first. The transmission order of the first type of area, the second type of area, and the third type of area will be dynamically adjusted according to the target field of view of the receiving device during the current time period.

[0057] Optionally, sending the target block image and other image regions to the receiving device includes: compressing the second type region according to a first preset step size to obtain a target second type region; compressing the third type region according to a second preset step size to obtain a target third type region; and sending the target block image, the second type region, and the third type region to the receiving device.

[0058] To facilitate a better understanding of the technical solutions of this application by those skilled in the art, a specific embodiment will now be described.

[0059] Figure 2This is a flowchart illustrating an optional method for transmitting panoramic video according to an embodiment of this application, as shown below. Figure 2 As shown, the process mainly includes the following steps:

[0060] (1) Decompose the panoramic video stream output from the sending end into a panoramic image set, and divide the panoramic images into user main, middle and secondary regions according to the method of uniform division in the initial state and division according to the user's predicted field of view (FOV) in the non-initial state.

[0061] (2) Then perform Discrete Cosine Transform (DCT) on each region separately;

[0062] (3) Based on the human visual discrimination characteristics HVS, the DCT coefficients are filtered by setting different thresholds for each region using the Just Noticeable Distortion (JND) method to remove image redundancy.

[0063] (4) Encode the main region of the image based on the user's field of view (FOV) and the human visual discrimination characteristics (HVS);

[0064] (5) The human eye is more sensitive to noise in the smooth area of ​​an image, image edge contours, low frequency and other information. Combining Rayleigh energy theorem, the block judgment coefficient is used as the value after the AC coefficient energy is weighted by the HVS visual function to perform different types of block classification operations.

[0065] (6) For blocks with different classifications, different normalization functions are established with the block judgment coefficient and the average value of the total communication coefficient as variables, so as to carry out self-regulation;

[0066] (7) Based on the function calculation results, set the quantization step size of the adaptive processing unit for each block, and then compress each block separately;

[0067] It should be noted that a highly precise quantitative method is used for areas of high sensitivity in the human eye, while a less precise quantitative method is used for areas of low sensitivity in the human eye.

[0068] Understandably, compression can improve the transmission quality of panoramic images.

[0069] (8) After compression, in order to improve the utilization of limited bandwidth, the Model Predictive Control (MPC) algorithm is used to predict the user's bandwidth and FOV at the next moment during the live broadcast, determine the transmission priority of each block and each region, prioritize the transmission stability of high-level blocks, and appropriately sacrifice the transmission quality of low-level blocks in combination with real-time bandwidth during transmission.

[0070] Understandably, transmitting different image regions according to priority helps reduce latency on the user end during live streaming and ensures smooth playback of panoramic videos.

[0071] It is noteworthy that this application combines panoramic video stream decomposition technology, image transformation redundancy removal technology, human visual characteristic analysis technology, adaptive quantization technology, bandwidth and field-of-view prediction technology to design a panoramic video adaptive quantization encoding and transmission method, and applies it to real-time live streaming scenarios: Panoramic video is decomposed into image frames, and redundancy is eliminated after dividing the area according to the user's predicted field of view. The main region blocks are classified and compressed using an adaptive quantization mechanism based on human visual characteristics. Combined with real-time bandwidth during transmission, this effectively improves the quality of panoramic video transmission, optimizes the user's experience with VR technology, and enables low-latency, smooth panoramic video transmission. In scenarios such as one-to-one communication live streaming, one-to-many training, and demonstration live streaming, it provides a smooth panoramic viewing experience for individual and enterprise users, contributing to the transformation from traditional planar live streaming to innovative panoramic video display modes.

[0072] This application has a high degree of integration with panoramic video. It divides the panoramic region according to the user's predicted FOV, which is more conducive to handling the massive data processing needs in the scenario of panoramic video encoding and transmission. The method of this application is suitable for live streaming scenarios. It eliminates redundancy during encoding and predicts the user's bandwidth at the next moment during transmission, giving priority to ensuring the transmission stability of high-level blocks. Compared with the direct transmission method, it can effectively improve transmission efficiency and reduce the latency when users watch panoramic live streams. The method of this application is flexible. It combines HVS and adaptively sets the quantization step size of the processing unit. Compared with the uniform step size setting, it allows the human eye-sensitive blocks to retain more detailed information.

[0073] This application reduces the amount of data to be transmitted. The data volume of panoramic video is much larger than that of planar video. This invention combines the quantitative measurement characteristics of the human eye and uses JND to further sparse the DCT coefficients, eliminate image redundancy, and solve the slow speed problem caused by the large amount of data when processing panoramic video. This application balances coding efficiency and compression quality. By performing block classification on the image and introducing adaptive quantization coefficients based on HVS, the perceptual quantization step size enables the panoramic image to maintain high coding efficiency while balancing compression quality, effectively improving quantization accuracy and improving the human visual experience of panoramic video under high compression ratios. This application makes reasonable use of resources. By using the MPC algorithm to predict user bandwidth and FOV, combined with priority mode, it effectively solves the problem of decreased panoramic video reception quality caused by user resource allocation and network bandwidth fluctuations under different network conditions.

[0074] Figure 3 This is a schematic diagram of a device structure for transmitting panoramic video according to an embodiment of this application, as shown below. Figure 3 As shown, the device includes:

[0075] The decomposition module 30 is used to receive the video stream from the transmitting device and decompose the video stream into a set of images;

[0076] The segmentation module 32 is used to segment the images in the image set into regions according to the user's field of view, so as to obtain different types of image regions, wherein the different types of regions correspond to different fields of view.

[0077] Encoding module 34 is used to determine target type regions in different image regions, encode the target type regions, divide the encoded target type regions into blocks to obtain multiple block images, wherein the human visual sensitivity of the multiple block images is different;

[0078] Compression module 36 is used to compress multiple block images using different quantization step sizes to obtain a target block image;

[0079] The transmitting module 38 is used to transmit the target block image and other image regions to the receiving device.

[0080] In this device, the decomposition module 30 receives the video stream from the transmitting device and decomposes it into an image set; the segmentation module 32 divides the images in the image set into regions according to the user's field of view, obtaining different types of image regions, each with a different field of view; the encoding module 34 determines the target type region in different image regions, encodes the target type region, and divides the encoded target type region into blocks to obtain multiple block images, each with different human visual sensitivity; the compression module 36 compresses the multiple block images using different quantization steps to obtain the target block image; and the transmission module 38 transmits the target block image and other image regions to the receiving device. This achieves the goal of reducing latency and stuttering during panoramic video live streaming, thereby meeting the processing requirements for transmitting massive panoramic video data, improving the efficiency of panoramic video data transmission and the playback quality of panoramic video, improving the human visual experience of panoramic video by combining the characteristics of human visual discrimination, and flexibly adjusting the transmission order of different video regions. This solves the technical problem of unreasonable channel occupation and resource waste during real-time transmission of panoramic video caused by using uniform quantization steps.

[0081] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is running, it controls the device where the non-volatile storage medium is located to execute any one of the panoramic video transmission methods.

[0082] Specifically, the aforementioned storage medium is used to store program instructions for the following functions, thereby implementing the following functions:

[0083] The system receives a video stream from a transmitting device and decomposes it into a set of images. Based on the user's field of view, the images in the set are divided into regions, resulting in different types of image regions, each with a different corresponding field of view. Target type regions are identified within these different image regions, encoded, and then divided into blocks to obtain multiple block images, each with different human visual sensitivities. These block images are compressed using different quantization steps to obtain a target block image. Finally, the target block image and other image regions are sent to the receiving device.

[0084] Optionally, in this embodiment, the storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of the storage medium include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0085] In an exemplary embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the panoramic video transmission method described above.

[0086] Optionally, when executed by a processor, the computer program may perform the following steps:

[0087] The system receives a video stream from a transmitting device and decomposes it into a set of images. Based on the user's field of view, the images in the set are divided into regions, resulting in different types of image regions, each with a different corresponding field of view. Target type regions are identified within these different image regions, encoded, and then divided into blocks to obtain multiple block images, each with different human visual sensitivities. These block images are compressed using different quantization steps to obtain a target block image. Finally, the target block image and other image regions are sent to the receiving device.

[0088] An electronic device is provided according to an embodiment of this application, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-described methods for transmitting panoramic video.

[0089] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0090] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0091] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0092] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0093] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the panoramic video transmission method. For example, in some embodiments, the panoramic video transmission method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the panoramic video transmission method described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the panoramic video transmission method by any other suitable means (e.g., by means of firmware).

[0094] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0096] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0097] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0098] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0099] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0100] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0101] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0106] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for transmitting panoramic video, characterized in that, include: Receive a video stream from a transmitting device and decompose the video stream into a set of images; The images in the image set are divided into regions according to the user's field of view to obtain different types of image regions. The different types of image regions correspond to different fields of view. The different types of image regions include: a first type of region, a second type of region, and a third type of region. The field of view of the first type of region, the second type of region, and the third type of region increases sequentially. Discrete cosine transform encoding is performed on the first type region, the second type region, and the third type region, and the discrete cosine coefficients are filtered; different thresholds corresponding to the first type region, the second type region, and the third type region are determined according to the algorithm for minimum perceptible distortion of the human eye; redundant regions in the first type region, the second type region, and the third type region are determined according to the relationship between the discrete cosine coefficients and the thresholds; and the redundant regions are cleared. The target type regions in the different image regions are determined, the target type regions are encoded, and the encoded target type regions are divided into blocks to obtain multiple block images, wherein the multiple block images have different human visual sensitivities. The multiple block images are compressed using different quantization step sizes to obtain the target block image; The target block image and other image regions are sent to the receiving device.

2. The method according to claim 1, characterized in that, The encoded target type region is divided into blocks, including: The energy of the AC coefficients is weighted by the HVS visual function to obtain the judgment coefficients, wherein the energy of the AC coefficients is obtained by discrete cosine transform encoding; The encoded target type region is divided into blocks according to the judgment coefficient, wherein the target type region is a first type region.

3. The method according to claim 2, characterized in that, Compression of the multiple image blocks using different quantization step sizes includes: Based on different first-type region blocks, a normalization function is established using the judgment coefficient and the average value of the total communication coefficient as variables; The quantization step size of the different first-type region blocks is determined based on the numerical range defined by the normalization function. The quantization step size is selected based on the sensitivity of the human eye to the different first-type region blocks, and the different first-type region blocks are compressed according to the quantization step size.

4. The method according to claim 3, characterized in that, The quantization step size is selected based on the sensitivity of the human eye to the different first-type region blocks, including: For the first type of region block corresponding to the highly sensitive area of ​​the human eye, select the first quantization step size; For the first type of region block corresponding to the low-sensitivity area of ​​the human eye, a second quantization step size is selected, wherein the first quantization step size is smaller than the second quantization step size.

5. The method according to claim 1, characterized in that, Sending the target block image, the second type region, and the third type region to the receiving device includes: The transmission priorities of the first type region, the second type region, and the third type region are determined, wherein the priorities of the first type region, the second type region, and the third type region decrease sequentially. The target block image, the second type region, and the third type region are sent to the receiving device according to the transmission priority.

6. The method according to claim 5, characterized in that, Determining the transmission priority of the first type region, the second type region, and the third type region includes: Determine the target network bandwidth and target field of view of the receiving device during the live broadcast of the video stream; The transmission priorities of the first type region, the second type region, and the third type region are determined based on the target network bandwidth and target field of view of the receiving device.

7. The method according to claim 1, characterized in that, Sending the target block image and other image regions to the receiving device includes: The second type region is compressed according to the first preset step size to obtain the target second type region; The third type region is compressed according to the second preset step size to obtain the target third type region; The target block image, the second type region, and the third type region are sent to the receiving device.

8. A device for transmitting panoramic video, characterized in that, include: A decomposition module is used to receive a video stream from a transmitting device and decompose the video stream into a set of images; A segmentation module is used to segment the images in the image set into different types of image regions based on the user's field of view. Each type of region corresponds to a different field of view. These different types of image regions include: a first type region, a second type region, and a third type region, with the field of view increasing sequentially from the first type region to the second type region and the third type region. The module performs discrete cosine transform encoding on the first type region, the second type region, and the third type region, and filters the discrete cosine coefficients. It determines different thresholds for each of the first type region, the second type region, and the third type region based on the minimum perceptible distortion algorithm for the human eye. It determines redundant regions in the first type region, the second type region, and the third type region based on the relationship between the discrete cosine coefficients and the thresholds. Finally, it removes the redundant regions. The encoding module is used to determine the target type region in the different image regions, encode the target type region, divide the encoded target type region into blocks to obtain multiple block images, wherein the multiple block images have different human visual sensitivities; A compression module is used to compress the multiple block images using different quantization step sizes to obtain a target block image; The sending module is used to send the target block image and other image regions to the receiving device.

9. A non-volatile storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the storage medium to perform the panoramic video transmission method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method for transmitting panoramic video as described in any one of claims 1 to 7.

Images