Image Transmission Method, Image Display and Processing Device, and Image Transmission System

Abstract

The present disclosure relates to an image transmission method, an image display and processing device, and an image transmission system, and relates to the technical field of image transmission. On the side of the image processing device, the left-eye image data and the right-eye image data corresponding to the same original image frame in the target video are segmented into a plurality of sub-images, and a plurality of groups of image data corresponding one-to-one to the plurality of sub-images are generated, and the plurality of groups of image data are sent to the image display device through a plurality of sending threads; each group of image data includes a sub-image, a first sequence number and a second sequence number corresponding to the sub-image, and the image display device combines the sub-images in the plurality of groups of image data according to the first sequence number and the second sequence number in the plurality of groups of image data to obtain the left-eye image data and the right-eye image data corresponding to the same original image frame, and finally plays the left-eye image corresponding to the left-eye image data and the right-eye image corresponding to the right-eye image data.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of image transmission technology, and more particularly, to an image transmission method, an image display and processing device, and an image transmission system. Background Art

[0002] A head-mounted display device (HMD) is a display device that can be worn on a user's head and can achieve different effects such as virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0003] Generally, the HMD performs binocular rendering. That is, the HMD renders a left-eye image corresponding to the user's left eye and a right-eye image corresponding to the user's right eye at the same time, and the images of the left-eye image and the right-eye image are of the same moment. In this case, a three-dimensional stereoscopic image with depth of field and layering can be formed in the user's mind, making the experience of virtual reality more realistic.

[0004] Currently, the HMD receives left-eye image data and right-eye image data from a personal computer (PC) end through two independent transmission channels, respectively, in order to render the left-eye image according to the left-eye image data and render the right-eye image according to the right-eye image data, realizing the synchronous transmission of the left-eye image data and the right-eye image data. In actual situations, the time for the PC end to transmit the left-eye image data and the right-eye image data to the HMD is relatively long, and the transmission efficiency is low. Summary of the Invention

[0005] An object of the present disclosure is to provide a new technical solution for an image transmission method, an image display and processing device, and an image transmission system.

[0006] According to a first aspect of the present disclosure, an image transmission method is provided, which is applied to an image display device. The method includes: receiving multiple sets of image data sent by an image processing device through multiple sending threads; each set of image data includes a sub-image, and its corresponding first sequence number and second sequence number. The sub-image is any one of multiple sub-images obtained by processing after segmenting target data. The target data includes left-eye image data and right-eye image data corresponding to any target image frame among multiple image frames of a target video. The first sequence number reflects the playing order of the target image frame among multiple image frames, and the second sequence number reflects the arrangement order of each sub-image among multiple sub-images; according to the first sequence number and the second sequence number in the multiple sets of image data, combining multiple sub-images corresponding to the same image frame in the multiple sets of image data to obtain left-eye image data and right-eye image data; synchronously playing the left-eye image corresponding to the left-eye image data and the right-eye image corresponding to the right-eye image data.

[0007] Optionally, combining multiple sub-images corresponding to the same image frame in the multiple sets of image data to obtain left-eye image data and right-eye image data includes: determining multiple sub-images corresponding to the same image frame according to the first sequence number in the multiple sets of image data; according to the second sequence number of the multiple sub-images corresponding to the same image frame, sequentially storing the multiple sub-images corresponding to the same image frame into a preset buffer area, and correspondingly obtaining left-eye image data and right-eye image data.

[0008] Optionally, receiving multiple sets of image data sent by an image processing device through multiple sending threads includes: receiving multiple target RTP packets corresponding one-to-one to the multiple sets of image data sent by the image processing device through multiple sending threads; wherein, a set of image data is encapsulated in the target RTP packet.

[0009] Optionally, the image data further includes: a preset image data identifier; receiving multiple target RTP packets corresponding one-to-one to the multiple sets of image data sent by the image processing device through multiple sending threads includes: receiving an RTP packet; in the case that the RTP packet includes the preset image data identifier, determining that the RTP packet is a target RTP packet.

[0010] According to a second aspect of the present disclosure, there is provided an image transmission method, which is applied to an image processing device. The method includes: performing segmentation processing on target data to obtain a plurality of sub-images; wherein the target data includes left-eye image data and right-eye image data corresponding to any target image frame among a plurality of image frames of a target video; determining a first sequence number and a second sequence number corresponding to each sub-image; wherein the first sequence number reflects the playing order of the target image frame among the plurality of image frames, and the second sequence number reflects the arrangement order of each sub-image among the plurality of sub-images; generating multiple sets of image data corresponding one-to-one to the plurality of sub-images according to the determination result; each set of image data includes: a sub-image, and its corresponding first sequence number and second sequence number; sending the multiple sets of image data to an image display device through a plurality of sending threads, so that the image display device combines the sub-images in the multiple sets of image data according to the first sequence number and the second sequence number in the multiple sets of image data to obtain the left-eye image data and the right-eye image data.

[0011] Optionally, generating multiple sets of image data corresponding one-to-one to the plurality of sub-images according to the determination result includes: encapsulating the sub-image, the first sequence number corresponding to the sub-image, and the second sequence number in each set of image data in a target RTP data packet to obtain a plurality of target RTP data packets; sending the multiple sets of image data to an image display device through a plurality of sending threads includes: determining the number of target RTP data packets corresponding to each sending thread according to the number of the plurality of sending threads and the number of the plurality of target RTP data packets; allocating the plurality of target RTP data packets to the plurality of sending threads for sending according to the number of target RTP data packets corresponding to each sending thread.

[0012] Optionally, the image data further includes: a preset image data identifier, so that when the image display device receives a plurality of RTP data packets sent by the image processing device through a plurality of sending threads, it determines the RTP data packet including the image data identifier as the target RTP data packet.

[0013] According to a third aspect of the present disclosure, there is provided an image display device, including: an image processing device is provided, including a memory and a processor, the memory is used for storing a computer program; the processor is used for executing the computer program to implement the method of the first aspect described above.

[0014] According to a fourth aspect of the present disclosure, there is provided an image processing device, including a memory and a processor, the memory is used for storing a computer program; the processor is used for executing the computer program to implement the method of the second aspect described above.

[0015] According to a fifth aspect of the present disclosure, there is provided an image transmission system, including: the image display device of the third aspect described above and the image processing device of the fourth aspect described above.

[0016] According to a sixth aspect of the present disclosure, there is also provided a computer-readable storage medium storing a computer program thereon, and when the computer program is executed by a processor, the method according to the first aspect or the second aspect of the present disclosure is implemented.

[0017] One beneficial effect of the embodiments of the present disclosure is that on the side of the image display device, it is possible to receive multiple sets of image data sent by the image processing device through multiple sending threads; each set of image data includes a sub-image, and its corresponding first sequence number and second sequence number. The sub-image is any one of the multiple sub-images obtained by segmenting and processing the target data. The target data includes the left-eye image data and the right-eye image data corresponding to any target image frame among the multiple image frames of the target video. The first sequence number reflects the playing order of the target image frame among the multiple image frames, and the second sequence number reflects the arrangement order of each sub-image among the multiple sub-images. Then, according to the first sequence number and the second sequence number in the multiple sets of image data, the multiple sub-images corresponding to the same image frame in the multiple sets of image data are combined and processed to obtain the left-eye image data and the right-eye image data, and then the left-eye image corresponding to the left-eye image data and the right-eye image corresponding to the right-eye image data are played synchronously. It can be seen that the embodiments of the present disclosure can transmit the left-eye image data and the right-eye image data through multiple sending threads, thereby effectively shortening the time for the image processing device to transmit image data to the image display device and improving the image transmission efficiency. In this process, the embodiments of the present disclosure can play the corresponding left-eye image and right-eye image synchronously after obtaining the left-eye image data and the right-eye image data, ensuring the synchronization of the display of the left-eye image and the right-eye image.

[0018] Through the following detailed description of the exemplary embodiments of the present specification with reference to the accompanying drawings, other features and advantages of the present specification will become clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings incorporated in the specification and constituting a part of the specification illustrate the embodiments of the present specification and, together with the description thereof, are used to explain the principles of the present specification.

[0020] Figure 1 is a schematic diagram of image data processing of a PC and a VR device in the related art;

[0021] Figure 2 is a flowchart of a method for an image transmission method according to some embodiments of the present disclosure;

[0022] Figure 3A is a schematic diagram of cutting the left-eye image data and the right-eye image data of the same frame of the original image according to some embodiments of the present disclosure;

[0023] Figure 3BAnother schematic diagram of cutting the left-eye image data and right-eye image data of the same frame of original image according to some embodiments of the present disclosure;

[0024] Figure 3C Schematic diagram of the relationship between the entire data packet that requires multiple sending threads for transmission at one time and multiple target RTP data according to some embodiments of the present disclosure;

[0025] Figure 4 Schematic diagram of the format of the header of a target RTP data packet according to some embodiments of the present disclosure;

[0026] Figure 5 Schematic diagram of the format of the extension header of a target RTP data packet according to some embodiments of the present disclosure;

[0027] Figure 6 Method flowchart of another image transmission method according to some embodiments of the present disclosure;

[0028] Figure 7 Schematic diagram of the sub-image cache arrangement in multiple target RTP data packets in a preset buffer according to some embodiments of the present disclosure;

[0029] Figure 8 Functional block diagram of an image transmission device according to some embodiments of the present disclosure;

[0030] Figure 9 Functional block diagram of another image transmission device according to some embodiments of the present disclosure;

[0031] Figure 10 Schematic diagram of the hardware structure of an image processing device according to some embodiments of the present disclosure;

[0032] Figure 11 Schematic diagram of the hardware structure of an image display device according to some embodiments of the present disclosure;

[0033] Figure 12 Schematic diagram of an image transmission system according to some embodiments of the present disclosure. Detailed implementation manners

[0034] Now, various exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that: unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present invention.

[0035] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way a limitation on the present invention or its application or use.

[0036] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods, and devices should be considered as part of the specification.

[0037] In all the examples shown and discussed herein, any specific values should be construed as merely exemplary and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

[0038] It should be noted that like reference numerals and letters denote like items in the following figures, and thus, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0039] Currently, VR devices (such as HMDs) can perform binocular rendering so that a three-dimensional stereoscopic image with depth of field and a sense of hierarchy can be formed in the user's mind, making the user's virtual reality experience more realistic.

[0040] As Figure 1 shown, the VR device can be connected to a personal computer (PC) in a wired or wireless manner. On the PC side, the PC runs an application and plays the application screen. For example, the PC runs a video application to play a video screen, or the PC runs a game application to play a game screen, etc. In this case, the PC captures the application screen and processes the same frame of the application screen into a left-eye image corresponding to the left eye and a right-eye image corresponding to the right eye. The PC has two transmission channels, a left-eye data channel and a right-eye data channel (such as Figure 1 the dual channels of the PC in Figure 1 ). Encoders are respectively provided in the two transmission channels for encoding the data (such as

[0041] the dual-channel encoding in Figure 1 ).

[0042] The VR device also has two transmission channels, a left-eye data channel and a right-eye data channel (such as Figure 1Dual channels in VR devices in the VR device). Similar to PC, a decoder and a renderer are respectively provided in the left eye data channel and the right eye data channel of the VR device. The left eye data channel of the VR device receives the encoded left eye image data from the PC end, and transmits the encoded left eye image data through the left eye data channel. After the encoded left eye image data is transmitted to the decoder corresponding to the left eye data channel of the VR device, it is decoded by the decoder to obtain the left eye image data. The left eye image data continues to be transmitted in the left eye data channel of the VR device. After being transmitted to the renderer of the left eye data channel of the VR device, it is rendered by the renderer and displayed on the display screen of the VR device. The right eye data channel of the VR device receives the encoded right eye image data from the PC end. The processing method of the encoded right eye image data in the VR device is similar to the above-mentioned processing method of the encoded left eye image data. The specific details can be referred to the above-mentioned processing process of the left eye image data, which will not be repeated here.

[0043] For users wearing VR devices, the user's left eye sees the left eye image displayed on the VR device display, and the user's right eye sees the right eye image displayed on the VR device display. Through dual-channel decoding and dual-channel rendering in the VR device, a three-dimensional image with depth of field and layering can be formed in the user's brain, making the user's experience of virtual reality more realistic.

[0044] At present, in the process of transmitting image data from PC to VR device, a single thread is used to transmit the above image data. In the case of a large amount of image frame data, such as an intra-frame coding frame (i.e., I frame, which retains the entire frame) or an instantaneous decoding refresh (IDR) frame in the image frame, it takes a long time to transmit the above image data using a single thread, and the transmission efficiency is low.

[0045] In view of the technical problems existing in the above implementations, the present disclosure provides an image transmission method, an image display and processing device, and an image transmission system to effectively solve the problem of low image data transmission efficiency in the process of transmitting image data to a binocular rendering image display device.

[0046] Hereinafter, various embodiments and examples according to the present invention are described with reference to the accompanying drawings.

[0047] <Method Example>

[0048] Figure 2 1 is a schematic diagram of a method flow of an image transmission method according to some embodiments of the present disclosure. The method embodiment can be implemented by an image processing device. The image display device can be, for example, a PC.

[0049] like Figure 2As shown in the figure, the image transmission method includes the following steps S210 to S250:

[0050] Step S210: Perform segmentation processing on the target data to obtain multiple sub-images; where the target data includes the left-eye image data and the right-eye image data corresponding to any target image frame among multiple image frames of the target video.

[0051] The left-eye image data is the image data corresponding to the left-eye image of the target image frame, and the right-eye image data is the image data corresponding to the right-eye image of the target image frame.

[0052] Exemplarily, as Figure 3A shown, the left-eye image data can be segmented first, and then the right-eye image data can be segmented to obtain multiple sub-images.

[0053] Exemplarily, as Figure 3B shown, the right-eye image data can be segmented first, and then the left-eye image data can be segmented to obtain multiple sub-images.

[0054] Exemplarily, due to the limitation of the network Maximum Transmission Unit (MTU) (the MTU is 1500 bytes), the left-eye image data and the right-eye image data corresponding to the target image frame are segmented to obtain multiple sub-images with smaller data lengths, and the data length of each sub-image is less than or equal to 1400 bytes.

[0055] Step S220: Determine the first sequence number and the second sequence number corresponding to each sub-image; where the first sequence number reflects the playback order of the target image frame among multiple image frames, and the second sequence number reflects the arrangement order of each sub-image among multiple sub-images.

[0056] The multiple image frames in the target video can be numbered in sequence according to the playback order, and the number of each image frame is the first sequence number corresponding to that image frame. The number of the target image frame is the first sequence number corresponding to the target image frame.

[0057] Exemplarily, set the first sequence number as frameIndex(0) to frameIndex(g). Where frameIndex(0) = 0, frameIndex(1) = 1, frameIndex(2) = 2... frameIndex(g) = g, g is a natural number and g ≥ 0. If the playback order of the target image frame among the multiple image frames in the target video is 5, the first sequence number of the target image frame is 5.

[0058] Multiple sub-images are obtained by segmenting a target image frame. The serial number of each sub-image in the multiple sub-images is the first serial number of the target image frame. When the first serial numbers of the multiple sub-images are the same, the multiple sub-images come from the same image frame.

[0059] When determining the second serial number of each sub-image, whether the left-eye image data is segmented first and then the right-eye image data is segmented, or the right-eye image data is segmented first and then the left-eye image data is segmented, the order in which the multiple sub-images are obtained is used as the arrangement order of the multiple sub-images, and the multiple sub-images are numbered according to this arrangement order to obtain the second serial numbers corresponding to the multiple sub-images.

[0060] It should be noted that after the multiple sub-images are combined in the above arrangement order, the above left-eye image data and right-eye image data can be obtained.

[0061] Exemplarily, the second serial number is set to packetIndexInframe(0) to packetIndexInframe(k). Among them, packetIndexInframe(0) = 0, packetIndexInframe(1) = 1, packetIndexInframe(2) = 2... packetIndexInframe(k) = k. k is a natural number and k > 1. If the arrangement order of a sub-image in the multiple sub-images is 6, the second serial number of this sub-image is 6.

[0062] Step S230: According to the determination result, generate multiple groups of image data corresponding one-to-one to the multiple sub-images; each group of image data includes: a sub-image, and its corresponding first serial number and second serial number.

[0063] In some embodiments, each group of image data can be encapsulated in a target Real-time Transport Protocol (RTP) data packet to obtain multiple target RTP data packets. Each target RTP data packet corresponds to a group of image data.

[0064] Exemplarily, for any sub-image in the multiple sub-images, the sub-image and its corresponding first serial number and second serial number can be written into a target RTP data packet. The format of the target RTP data packet is the format of an RTP data packet.

[0065] Exemplarily, the format of the header of each target RTP data packet is as Figure 4 shown. V is the version number of the RTP protocol, occupying 2 bits, such as Figure 4The current protocol version number of the RTP packet shown is 2. P is the padding flag, which takes 1 bit. If P = 1, one or more additional octets are padded at the end of the message, and they are not part of the payload. X is the extension flag, which takes 1 bit. If X = 1, there is an extension header following the RTP header. CC is the contributing source (CSRC) counter, which takes 4 bits and indicates the number of CSRC identifiers. M is the marker, which takes 1 bit. Different payloads have different meanings. For video, it marks the end of a frame; for audio, it marks the start of a session. PT is the payload type, which takes 7 bits and is used to describe the type of payload in the RTP packet. For example, in Chinese, it is Global System for Mobile Communications (GSM) audio, Joint Photographic Experts Group (JPEM) image, etc. In streaming media, it is mostly used to distinguish audio streams and video streams, which facilitates parsing by the client. SN (i.e., Figure 4 the sequence number in) is the sequence number, which takes 16 bits. After the sender sends each RTP packet, the value of this field is incremented by 1. The receiver can detect packet loss and restore the packet sequence from this field. The initial value of the sequence number is random. Figure 4 In, "timestamp" is the timestamp, which takes 32 bits and records the sampling moment of the first byte of data in the packet. It is indispensable for removing jitter and achieving synchronization. SSRC (i.e., Figure 4 the "synchronization source (SSRC) identifier" in) is the synchronization source identifier, which takes 32 bits. The synchronization source refers to the source of the RTP packet stream. There cannot be two identical SSRC values in the same RTP packet session. CSRC (i.e., Figure 4 the "contributing source (CSRC) identifiers" in) is the contributing source identifier. Each CSRC identifier takes 32 bits, and there can be 0 to 15 of them. Each CSRC identifies all the contributing sources included in the payload of the RTP message. Among them, the header of the target RTP packet fixedly occupies 12 bytes.

[0066] The first sequence number and the second sequence number corresponding to the sub-image can be written into the extension header of the target RTP packet (such as the Figure 5 "header extension" part shown), and the format of the extension header of the target RTP packet is as shown in Figure 5As shown, it includes the defined by profile field and the length. The defined by profile and the length together occupy 4 bytes, and the first sequence number and the second sequence number each occupy 4 bytes. In this case, the total length of the RTP protocol field is 12 + 4 + 4 + 4 = 24 bytes. The MTU is 1500 bytes. To ensure that each target RTP packet can be normally transmitted in the network, the length of the sub-image within each target RTP data is set to 1400 bytes. In this case, the total length of a target RTP packet is 1400 + 24 = 1424 bytes, ensuring that each target RTP packet can be normally transmitted in the network.

[0067] Exemplarily, for the entire packet that needs to be transmitted by multiple sending threads at one time, the entire packet may include the above-mentioned multiple target RTP packets (such as Figure 3C the target RTP packet RTP[0], the target RTP packet RTP[1], the target RTP packet RTP[2]...). The relationship between the multiple target RTP packets and the entire packet is as Figure 3C shown.

[0068] Of course, it can be understood that a format similar to the data format in the above-mentioned target RTP packet can also be used to set each group of image data. The specific format of each group of image data is not limited in the embodiments of the present disclosure.

[0069] Step S240: Send multiple groups of image data to the image display device through multiple sending threads, so that the image display device combines the sub-images in the multiple groups of image data according to the first sequence number and the second sequence number in the multiple groups of image data to obtain the left-eye image data and the right-eye image data.

[0070] Exemplarily, each sending thread can send one group or w groups of image data; where w is a natural number and w>1.

[0071] In some embodiments, when the sub-image in each group of image data, the first sequence number and the second sequence number corresponding to the sub-image are encapsulated in a target RTP packet, the number of target RTP packets corresponding to each sending thread can be determined according to the number of multiple sending threads and the number of multiple target RTP packets, and then according to the number of target RTP packets corresponding to each sending thread, the multiple target RTP packets are allocated to multiple sending threads for sending.

[0072] Exemplarily, the process of allocating multiple target RTP data packets to multiple sending threads for sending is as follows: The number of multiple sending threads is M, where M is a natural number and M > 1. The number of multiple target RTP data packets is N, where N is a natural number and N > 1. Then, the number of target RTP data packets sent by each sending thread is (N / M). Serial numbers can be set for the N target RTP data packets to obtain N target RTP data packets with serial numbers from 0 to N. In this case, the first sending thread sends the target RTP data packets with serial numbers from 0 to (N / M), the second sending thread sends the target RTP data packets with serial numbers from [(N / M) + 1] to [2*(N / M)]... and so on. In the case where N is not an integer multiple of M, the number of target RTP data packets sent by the last sending thread is less than that sent by other sending threads. After the M threads send the N target RTP data packets, they temporarily enter the sleep state and are awakened to send the multiple target RTP data packets when sending multiple target RTP data packets next time.

[0073] In the embodiments of the present disclosure, the left-eye image data and the right-eye image data corresponding to the target image frame in the target video can be segmented into multiple sub-images with smaller data lengths, and then each sub-image, the first serial number and the second serial number corresponding to the sub-image are used as a group of image data to obtain multiple groups of image data. The multiple groups of image data are sent through multiple sending threads, thereby effectively shortening the time for the image processing device to transmit the image data to the image display device. Especially for video frames with a large amount of data, the time required to transmit the video frame can be greatly shortened, and the image transmission efficiency can be improved.

[0074] In some embodiments, the image data further includes: a preset image data identifier, so that when the image display device receives multiple RTP data packets sent by the image processing device through multiple sending threads, the RTP data packets including the image data identifier are determined as target RTP data packets.

[0075] Figure 6 It is a schematic flowchart of an image transmission method according to some embodiments of the present disclosure. The method embodiments can be implemented by an image display device. The image display device can be, for example, a VR device such as an HMD. The image display device can have one transmission channel.

[0076] As Figure 6 shown, the image transmission method includes the following steps S610 to S630:

[0077] Step S610: Receive multiple groups of image data sent by the image processing device through multiple sending threads; wherein, the image data includes sub-images, the first serial number and the second serial number corresponding to the sub-images.

[0078] The process of the image processing device and the multiple sets of image data transmitted through multiple sending threads can be referred to the corresponding descriptions in the above embodiments, which will not be elaborated here.

[0079] In step S610, multiple receiving threads can be used to correspondingly receive the multiple sets of image data transmitted by the multiple sending threads. The multiple receiving threads correspond one by one to the multiple sending threads.

[0080] Each set of image data includes a sub-image, and its corresponding first sequence number and second sequence number. The sub-image is any one of the multiple sub-images obtained by processing the segmented target data. The target data includes the left-eye image data and the right-eye image data corresponding to any target image frame among the multiple image frames of the target video. The first sequence number reflects the playing order of the target image frame among the multiple image frames, and the second sequence number reflects the arrangement order of each sub-image among the multiple sub-images.

[0081] The obtaining methods of the first sequence number and the second sequence number can be referred to the corresponding descriptions in the above embodiments, which will not be elaborated here.

[0082] In some embodiments, receiving the multiple sets of image data transmitted by the image processing device through multiple sending threads can be: receiving multiple target RTP data packets corresponding one by one to the multiple sets of image data transmitted by the image processing device through multiple sending threads; wherein, each target RTP data packet encapsulates a set of image data.

[0083] Step S620: According to the first sequence number and the second sequence number in the multiple sets of image data, combine the multiple sub-images corresponding to the same image frame in the multiple sets of image data to obtain the left-eye image data and the right-eye image data.

[0084] The first sequence number reflects the playing order of the target image frame among the multiple image frames of the target video. In this case, the first sequence numbers of the left-eye image data and the right-eye image data corresponding to the same image frame are the same.

[0085] In some embodiments, the multiple sub-images corresponding to the same image frame can be determined according to the first sequence number in the multiple sets of image data, and then according to the second sequence numbers of the multiple sub-images corresponding to the same image frame, the multiple sub-images are sequentially stored in a preset buffer area to correspondingly obtain the left-eye image data and the right-eye image data.

[0086] In some embodiments, when receiving multiple target RTP packets corresponding one-to-one to multiple groups of image data sent by an image processing device through multiple sending threads, the first sequence number and the second sequence number are stored at the position of the extension header of the target RTP packet. Therefore, the first sequence number and the second sequence number of the extension header of the target RTP packet are read first. After determining multiple target RTP packets corresponding to the same image frame according to the first sequence number, the sub-images in the multiple target RTP packets are sequentially stored in a preset buffer according to the second sequence number in the multiple target RTP packets. As Figure 7 shown, the sub-images in the multiple target RTP packets are arranged in sequence according to the second sequence number as RTP[0], RTP[1]... RTP[packetIndexInframe(K - 1)], RTP[packetIndexInframe(K)]. The second sequence numbers RTP[packetIndexInframe(K - 1)] and RTP[packetIndexInframe(K)] corresponding to two adjacent sub-images are adjacent, and the starting positions of adjacent sub-images differ by the data length of one sub-image; where K is a natural number and k ≥ K ≥ 0.

[0087] Step S630: Synchronously play the left eye image corresponding to the left eye image data and the right eye image corresponding to the right eye image data.

[0088] The left eye image data and the right eye image data corresponding to the same original image frame are transmitted to a decoder for decoding, and the decoded left eye image data and right eye image data are rendered synchronously, so that the left eye image corresponding to the left eye image data and the right eye image corresponding to the right eye image data can be displayed synchronously.

[0089] In some embodiments, the image data further includes: a preset image data identifier. In this case, receiving multiple target RTP packets corresponding one-to-one to multiple groups of image data sent by an image processing device through multiple sending threads includes: receiving RTP packets; and determining that an RTP packet is a target RTP packet when the RTP packet includes the preset image data identifier.

[0090] One beneficial effect of the embodiments of the present disclosure is that on the side of the image display device, it is capable of receiving multiple sets of image data sent by the image processing device through multiple sending threads; each set of image data includes a sub-image, and its corresponding first sequence number and second sequence number. The sub-image is any one of the multiple sub-images obtained by processing the segmented target data. The target data includes the left-eye image data and the right-eye image data corresponding to any target image frame among the multiple image frames of the target video. The first sequence number reflects the playing order of the target image frame among the multiple image frames, and the second sequence number reflects the arrangement order of each sub-image among the multiple sub-images. Then, according to the first sequence number and the second sequence number in the multiple sets of image data, the multiple sub-images corresponding to the same image frame in the multiple sets of image data are combined and processed to obtain the left-eye image data and the right-eye image data, and then the left-eye image corresponding to the left-eye image data and the right-eye image corresponding to the right-eye image data are played synchronously. Thus, it can be seen that the embodiments of the present disclosure can transmit the left-eye image data and the right-eye image data through multiple sending threads, thereby effectively shortening the time for the image processing device to transmit the image data to the image display device and improving the image transmission efficiency. In this process, the embodiments of the present disclosure can play the corresponding left-eye image and right-eye image synchronously after obtaining the left-eye image data and the right-eye image data, ensuring the synchronization of the display of the left-eye image and the right-eye image.

[0091] <Device Embodiment>

[0092] Figure 8 is a functional block diagram of an image transmission device provided according to some embodiments of the present disclosure. The image transmission device is applied to an image processing device. As Figure 8 shown, the image transmission device 80 includes:

[0093] A segmentation module 81, configured to perform segmentation processing on target data to obtain multiple sub-images; wherein, the target data includes the left-eye image data and the right-eye image data corresponding to any target image frame among the multiple image frames of the target video.

[0094] A determination module 82, configured to determine the first sequence number and the second sequence number corresponding to each sub-image; wherein, the first sequence number reflects the playing order of the target image frame among the multiple image frames, and the second sequence number reflects the arrangement order of each sub-image among the multiple sub-images.

[0095] A generation module 83, configured to generate multiple sets of image data corresponding one-to-one to the multiple sub-images according to the determination result; each set of image data includes: a sub-image, and its corresponding first sequence number and second sequence number.

[0096] A sending module 84, configured to send multiple groups of image data to an image display device through multiple sending threads, so that the image display device combines sub-images in the multiple groups of image data according to a first sequence number and a second sequence number in the multiple groups of image data to obtain left-eye image data and right-eye image data.

[0097] For the implementation processes of the functions of the modules of the image transmission device 80, reference may be made to the corresponding descriptions in the foregoing method embodiments, which will not be elaborated herein.

[0098] The image processing device may be a PC, for example.

[0099] Figure 9 is a functional block diagram of an image transmission device provided according to some embodiments of the present disclosure. The image transmission device is applied to an image display device. As Figure 9 shown, the image transmission device 90 includes:

[0100] A receiving module 91, configured to receive multiple groups of image data sent by an image processing device through multiple sending threads; wherein each group of image data includes a sub-image, and its corresponding first sequence number and second sequence number, and the sub-image is any one of multiple sub-images obtained by processing after segmenting target data, the target data includes left-eye image data and right-eye image data corresponding to any target image frame among multiple image frames of a target video, the first sequence number reflects the playing order of the target image frame among the multiple image frames, and the second sequence number reflects the arrangement order of each sub-image among the multiple sub-images.

[0101] A processing module 92, configured to combine multiple sub-images corresponding to the same image frame in the multiple groups of image data according to the first sequence number and the second sequence number in the multiple groups of image data to obtain left-eye image data and right-eye image data.

[0102] A playing module 93, configured to synchronously play a left-eye image corresponding to the left-eye image data and a right-eye image corresponding to the right-eye image data.

[0103] For the implementation processes of the functions of the modules of the image transmission device 90, reference may be made to the corresponding descriptions in the foregoing method embodiments, which will not be elaborated herein.

[0104] The image display device may be a VR device such as an HMD, for example.

[0105] Figure 10 is a schematic hardware structure diagram of an image processing device according to some embodiments of the present disclosure.

[0106] As Figure 10As shown, the image processing device 1000 includes a processor 1010 and a memory 1020. The memory 1020 is used to store executable computer programs, and the processor 1010 is used to execute the methods of any of the above method embodiments under the control of the computer programs.

[0107] The image processing device 1000 may be a PC.

[0108] Each module of the above image transmission device 80 may be implemented by the processor 1010 in this embodiment executing the computer programs stored in the memory 1020, or may be implemented by other circuit structures, which is not limited herein.

[0109] Figure 11 It is a schematic diagram of the hardware structure of an image display device according to some embodiments of the present disclosure.

[0110] As Figure 11 As shown, the image display device 1100 includes a processor 1110 and a memory 1120. The memory 1120 is used to store executable computer programs, and the processor 1110 is used to execute the methods of any of the above method embodiments under the control of the computer programs.

[0111] The image display device 1100 may be a VR device such as an HMD.

[0112] Each module of the above image transmission device 90 may be implemented by the processor 1110 in this embodiment executing the computer programs stored in the memory 1120, or may be implemented by other circuit structures, which is not limited herein.

[0113] <System Embodiment>

[0114] Figure 12 It is a schematic diagram of an image transmission system according to some embodiments of the present disclosure. As Figure 12 As shown, the image transmission system 1200 includes an image processing device 1000 and an image display device 1100.

[0115] <Computer-readable Storage Medium Embodiment>

[0116] This embodiment provides a computer-readable storage medium, in which executable commands are stored. When the executable commands are executed by a processor, the methods described in any method embodiment of this specification are executed (for example, the methods corresponding to steps S210 to S240 or the methods corresponding to steps S610 to S630 above).

[0117] One or more embodiments of this specification can be a system, method, and / or computer program product. The computer program product can include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement various aspects of this specification.

[0118] A computer-readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing. The computer-readable storage medium as used herein is not construed as being a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0119] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0120] The computer program instructions for performing the operations of the embodiments of this specification may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine - related instructions, microcode, firmware instructions, state - setting data, or source code or object code written in any combination of one or more programming languages. The programming languages include object - oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer - readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand - alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, by using the state information of the computer - readable program instructions to customize an electronic circuit, such as a programmable logic circuit, a field - programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer - readable program instructions to implement various aspects of this specification.

[0121] Aspects of this specification are described herein with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer - readable program instructions.

[0122] These computer - readable program instructions can be provided to a processor of a general - purpose computer, a special - purpose computer, or other programmable data - processing apparatus to produce a machine such that, when the instructions are executed by the processor of the computer or other programmable data - processing apparatus, a device is produced that implements the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer - readable program instructions can also be stored in a computer - readable storage medium, and these instructions cause a computer, a programmable data - processing apparatus, and / or other devices to work in a particular manner. Thus, the computer - readable medium storing the instructions includes a manufacture that includes instructions for implementing various aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0123] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, causing a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process such that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0124] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present specification. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of an instruction, and the module, segment of a program, or part of an instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the figures. For example, two consecutive blocks may in fact be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block of the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions. As is well known to those of ordinary skill in the art, implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

[0125] The embodiments of the present specification have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of technologies in the market, or to enable other ordinary skill in the art in the technical field to understand the embodiments disclosed herein. The scope of the present application is defined by the appended claims.

Claims

1. An image transmission method, characterized in that, Applied to an image display device, the image display device having a transmission channel; the method includes: Receiving multiple sets of image data sent by an image processing device through multiple sending threads; each set of image data includes a sub-image, and its corresponding first sequence number and second sequence number. The sub-image is any one of multiple sub-images obtained by processing after segmenting target data. The target data includes left-eye image data and right-eye image data corresponding to any target image frame among multiple image frames of a target video. The first sequence number reflects the playback order of the target image frame among the multiple image frames, and the second sequence number reflects the arrangement order of each sub-image among the multiple sub-images. Wherein, when segmenting the target data, the left-eye image data and the right-eye image data are segmented in sequence; According to the first sequence number and the second sequence number in the multiple sets of image data, combining multiple sub-images corresponding to the same image frame in the multiple sets of image data to obtain the left-eye image data and the right-eye image data; Synchronously playing the left-eye image corresponding to the left-eye image data and the right-eye image corresponding to the right-eye image data, and the left-eye image and the right-eye image are synchronously displayed through synchronous decoding and synchronous rendering; Wherein, the combining multiple sub-images corresponding to the same image frame in the multiple sets of image data according to the first sequence number and the second sequence number in the multiple sets of image data to obtain the left-eye image data and the right-eye image data includes: Determining multiple sub-images corresponding to the same image frame according to the first sequence number in the multiple sets of image data; According to the second sequence number of multiple sub-images corresponding to the same image frame, sequentially storing the multiple sub-images corresponding to the same image frame into a preset buffer area to correspondingly obtain the left-eye image data and the right-eye image data.

2. The method according to claim 1, wherein The receiving multiple sets of image data sent by an image processing device through multiple sending threads includes: Receiving multiple target RTP data packets corresponding one-to-one to the multiple sets of image data sent by an image processing device through multiple sending threads; wherein, a set of image data is encapsulated in the target RTP data packet.

3. The method according to claim 2, characterized in that, The image data further includes: a preset image data identifier; The receiving multiple target RTP data packets corresponding one-to-one to the multiple sets of image data sent by an image processing device through multiple sending threads includes: Receiving an RTP data packet; When the RTP data packet includes the preset image data identifier, determining that the RTP data packet is the target RTP data packet.

4. An image transmission method, characterized in that, Applied to an image processing device, the image display device having a transmission channel; the method includes: Performing segmentation processing on target data to obtain multiple sub-images; wherein, the target data includes left-eye image data and right-eye image data corresponding to any target image frame among multiple image frames of a target video; wherein, when segmenting the target data, the left-eye image data and the right-eye image data are segmented in sequence; Determine a first sequence number and a second sequence number corresponding to each sub-image; wherein, the first sequence number reflects the playback order of the target image frame among the multiple image frames, and the second sequence number reflects the arrangement order of each sub-image among the multiple sub-images; Generate multiple groups of image data corresponding one-to-one to the multiple sub-images according to the determination result; each group of image data includes: A sub-image, and its corresponding first sequence number and second sequence number; Send the multiple groups of image data to an image display device through multiple sending threads, so that the image display device combines and processes the sub-images in the multiple groups of image data according to the first sequence number and the second sequence number in the multiple groups of image data to obtain the left-eye image data and the right-eye image data; Among them, the combining and processing of the multiple sub-images corresponding to the same image frame in the multiple groups of image data according to the first sequence number and the second sequence number in the multiple groups of image data to obtain the left-eye image data and the right-eye image data includes: Determine multiple sub-images corresponding to the same image frame according to the first sequence number in the multiple groups of image data; According to the second sequence number of the multiple sub-images corresponding to the same image frame, sequentially store the multiple sub-images corresponding to the same image frame into a preset buffer area to correspondingly obtain the left-eye image data and the right-eye image data.

5. The method according to claim 4, characterized in that, The generating multiple groups of image data corresponding one-to-one to the multiple sub-images according to the determination result includes: Encapsulate the sub-image, the first sequence number and the second sequence number corresponding to the sub-image in each group of image data in a target RTP data packet to obtain multiple target RTP data packets; The sending the multiple groups of image data to an image display device through multiple sending threads includes: Determine the number of target RTP data packets corresponding to each sending thread according to the number of the multiple sending threads and the number of the multiple target RTP data packets; Allocate the multiple target RTP data packets to the multiple sending threads for sending according to the number of target RTP data packets corresponding to each sending thread.

6. The method according to claim 4, characterized in that, The image data further includes: a preset image data identifier, so that when the image display device receives multiple RTP data packets sent by the image processing device through multiple sending threads, it determines the RTP data packet including the image data identifier as the target RTP data packet.

7. An image display device, characterized in that, Comprising a memory and a processor, the memory is used for storing a computer program; the processor is used for executing the computer program to implement the method according to any one of claims 1-3.

8. An image processing apparatus, characterized in that, Comprising a memory and a processor, the memory is used for storing a computer program; the processor is used for executing the computer program to implement the method according to any one of claims 4-6.

9. An image transmission system, characterized in that, Comprising the image display device according to claim 7 and the image processing device according to claim 8.

Images