Transmission device, reception device, and transmission / reception system

Abstract

The present invention flexibly and effectively transmits information.　In the present invention, a generation unit generates a lane stream in which a plurality of lane frames including a first lane frame are continuous, and a transmission unit transmits the lane stream. The first lane frame includes a plurality of lanes different from each other. Of these lanes, a first lane includes first division information among a plurality of pieces of division information different from each other, and a second lane frame includes second division information. The first lane frame includes grouping information for indicating that the first division information and the second division information were obtained by dividing the first information.

Description

Transmitting device, receiving device, and transmitting / receiving system

[0001] The present technology relates to a transmitting device, a receiving device, and a transmitting / receiving system, and more particularly to a transmitting device or the like that effectively transmits information.

[0002] It is possible to transmit information (data) via an interface. For example, an entertainment system is known in which a server on a network renders an image according to the user's viewpoint, transmits the image to the user via the network, and displays the image on the user's display device, such as a head-mounted display. In this case, shortening the motion-to-photon latency (the time it takes from action to display) is important for enhancing the entertainment value.

[0003] Furthermore, for example, Patent Document 1 discloses a technique in which a single screen is divided into a plurality of regions to generate partial images, and the partial images are transmitted in a predetermined order.

[0004] Japanese Patent Application Publication No. 2001-275025

[0005] The purpose of this technology is to transmit information flexibly and effectively.

[0006] The concept of the present technology is a transmitting device including: a generating unit that generates a lane stream in which a plurality of lane frames including a first lane frame are consecutive; and a transmitting unit that transmits the lane stream, wherein the first lane frame includes a plurality of lanes that are different from each other, the plurality of lanes include a first lane and a second lane, the first lane includes first division information among a plurality of division information that are different from each other, the second lane includes second division information among the plurality of division information, and the first lane frame includes grouping information for indicating that the first division information and the second division information are obtained by dividing the first information.

[0007] Another concept of the present technology is a receiving device including a receiving unit that receives a continuous lane stream of multiple lane frames including a first lane frame, wherein the first lane frame includes multiple lanes that are different from each other, the multiple lanes include a first lane and a second lane, the first lane includes first division information among multiple division information that are different from each other, the second lane includes second division information among the multiple division information, the first lane frame includes grouping information indicating that the first division information and the second division information are obtained by dividing the first information, and further including a processing unit that combines the multiple division information based on the grouping information.

[0008] Another concept of the present technology is a transmission / reception system having a transmitting device and a receiving device, wherein the transmitting device comprises: a generating unit that generates a lane stream in which a plurality of lane frames including a first lane frame are successive; and a transmitting unit that transmits the lane stream, wherein the first lane frame includes a plurality of lanes that are different from each other, wherein the plurality of lanes include a first lane and a second lane, wherein the first lane includes first division information among a plurality of division information that are different from each other, wherein the second lane includes second division information among the plurality of division information, and wherein the first lane frame includes grouping information for indicating that the first division information and the second division information are obtained by dividing first information, and the receiving device comprises: a receiving unit that receives the lane stream; and a processing unit that combines the plurality of division information based on the grouping information.

[0009] Illustrative embodiments of the present technology will be described with reference to the following drawings. FIGS. 1A to 1C are diagrams illustrating an example of a transmission format in the present technology. FIG. 1A shows an example of the configuration of a lane stream, FIG. 1B shows an example of the configuration of a lane frame in 1x speed transmission, and FIG. 1C shows an example of the configuration of a lane frame in Nx speed transmission. FIGS. 2A to 2B are diagrams illustrating lane division transmission. FIG. 2A shows an example of the configuration of a lane frame in simple transmission, and FIG. 2B shows an example of the configuration of a lane frame in lane division transmission. FIGS. 3A to 3B are diagrams illustrating time division transmission. FIG. 3A shows an example of the configuration of a lane frame in simple transmission, and FIG. 3B shows an example of the configuration of a lane frame in time division transmission. FIGS. 4A to 4B are diagrams illustrating combination division transmission. FIG. 4A shows an example of the configuration of a lane frame in simple transmission, and FIG. 4B shows an example of the configuration of a lane frame in combination division transmission. FIGS. 5A and 5B are diagrams illustrating another example of combination division information transmission. FIG. 5A shows an example of the configuration of a lane frame in simple transmission, and FIG. 5B shows an example of the configuration of a lane frame in combination division transmission. FIGS. 6A and 6B are diagrams illustrating another example of time-division information transmission (quadruple-speed transmission). FIG. 6A shows an example of the configuration of a lane frame in simple transmission, and FIG. 6B shows an example of the configuration of a lane frame in time-division transmission. FIGS. 7A and 7B are diagrams illustrating video transmission based on lane-division transmission. FIG. 7A shows an example of the configuration of a lane frame in simple transmission, and FIG. 7B shows an example of the configuration of a lane frame in lane-division transmission. FIGS. 8A and 8B are diagrams illustrating video transmission based on time-division transmission. FIG. 8A shows an example of the configuration of a lane frame in simple transmission, and FIG. 8B shows an example of the configuration of a lane frame in time-division division transmission. FIGS. 9A and 9B are diagrams illustrating video transmission based on combination division transmission. FIG. 9A shows an example of the configuration of a lane frame in simple transmission, and FIG. 9B shows an example of the configuration of a lane frame in combination division transmission. FIGS. 10A and 10B are diagrams illustrating another example of combination division transmission. FIG. 10A shows an example of the configuration of a lane frame in simple transmission, and FIG. 10B shows an example of the configuration of a lane frame in combination division transmission.Figures 11A and 11B are diagrams illustrating another example of time-division transmission (quadruple-speed transmission). Figure 11A shows an example of the configuration of a lane frame in simple transmission, and Figure 11B shows an example of the configuration of a lane frame in time-division transmission. Figure 12 shows an example of the bit field configuration of a header unit. Figure 13 is a diagram illustrating each field of the header unit. Figures 14A to 14C are diagrams illustrating the physical lane number, lane group number, and logical lane number. Figure 14A shows an example of value assignment for the physical lane number, Figure 14B shows an example of value assignment for the lane group number, and Figure 14C shows an example of value assignment for the logical lane number. Figures 15A and 15B are diagrams illustrating the program number and category. Figure 15A shows an example of value assignment for the program number, and Figure 15B shows an example of value assignment for the category. Figure 16 shows an example of the bit field configuration of an MVTU. Figure 17 is a diagram illustrating each field of an MVTU. Figures 18A to 18C are diagrams illustrating the stream number, video format match flag, and data reduction flag. FIG. 18A shows an example of value assignment for a stream number, FIG. 18B shows an example of value assignment for a video format etc. match flag, and FIG. 18C shows an example of value assignment for a data reduction flag. FIGS. 19A to 19C are diagrams for explaining data reduction levels, data reduction methods, and data division methods. FIG. 19A shows an example of value assignment for a data reduction level, FIG. 19B shows an example of value assignment for a data reduction method, and FIG. 19C shows an example of value assignment for a data division method. FIG. 20 shows an example of data reduction mode. FIGS. 21A and 21B are diagrams for explaining SASU. FIG. 21A shows an example of the bit field configuration of SASU, and FIG. 21B is a diagram for explaining each field of SASU. FIGS. 22A and 22B are diagrams for explaining how to specify PLN, LGN, LLN, PGN, STN, and OPT. Fig. 22A shows an example of the configuration of a lane frame, and Fig. 22B shows an example of the values ​​of PLN, LGN, LLN, PGN, STN, and OPT. Figs. 23A and 23B are diagrams for explaining the MPU. Fig. 23A shows an example of the bit field configuration of the MPU, and Fig. 23B is a diagram for explaining each field of the MPU.FIG. 24 shows an example of the bit field configuration of an MPU including information such as video frame rate and resolution. FIG. 25 is a diagram for explaining each field of an MPU including information such as video frame rate and resolution. FIGS. 26A to 26C are diagrams for explaining a transmission / reception system that uses the transmission format of the present technology. FIG. 26A shows an example of the configuration of a transmission / reception system, FIG. 26B shows an example of the configuration of a server, and FIG. 26C shows an example of the configuration of a user device. FIG. 27 shows a switching sequence between a normal transmission mode and an N-times speed transmission mode. FIGS. 28A and 28B are diagrams for explaining motion-to-photon latency. FIG. 28A shows an example of the screen configuration of a transmitted video, and FIG. 28B shows an example of the configuration of a lane stream. FIG. 29 shows another example of the configuration of a lane stream. FIG. 30 shows another example of the configuration of a lane stream. FIG. 31 is a block diagram showing an example of the hardware configuration of a computer.

[0010] Below, modes for carrying out the invention (hereinafter referred to as "embodiments") will be described. The description will be given in the following order: 1. Transmission format example 2. Information transmission example 2-1. Video transmission example 2-2. Other transmission examples 3. Header unit 4. MVTU 5. SASU 6. Main field setting example 7. MPU 8. Transmission / reception system example 9. Motion-to-photon latency 10. Transmission priority 11. Computer hardware configuration example 12. Modified example

[0011] "1. Example of Transmission Format" An example of a transmission format in the present technology will be described. Fig. 1A shows an exemplary structure of a lane stream transmitted in one lane (channel). A lane stream is a sequence of multiple lane frames.

[0012] 1B shows an example of the structure of a lane frame in regular-speed transmission. A lane frame is a series of multiple units, for example, 6000 units. Each unit is, for example, 18 bytes long. For example, the beginning of a lane frame is a header unit in which header information is placed, and each subsequent unit is a payload unit in which information to be transmitted can be inserted.

[0013] Even during Nx-speed transmission (N is a positive real number other than 1), a lane frame contains the same number of units (6000) as during 1x-speed transmission. Figure 1C shows an example of the configuration of a lane frame during 4x-speed transmission. During 4x-speed transmission, four lane frames can be transmitted in the time it takes to transmit one lane frame during 1x-speed transmission.

[0014] "2. Examples of Information Transmission" Examples of information transmission using the lane stream of the transmission format described above include simple transmission, which simply transmits one piece of information on one lane, lane-division multiplexing transmission, time-division multiplexing transmission, and combination multiplexing transmission, which combines lane division and time division.

[0015] First, lane division transmission, that is, information transmission in which one piece of information is transmitted using a plurality of lanes, will be described.

[0016] 2A shows an example of the structure of a lane frame in simple transmission in which one piece of information is transmitted over only one lane. In this example, a header is followed by a payload 1, and the remaining part is used as padding.

[0017] FIG. 2B shows an example of the configuration of a lane frame for each lane in lane-division transmission in which one piece of information is arranged and transmitted across four lanes. Payloads 1-1 to 1-4 are divided information obtained by dividing payload 1 into four parts. In this example, in the lane frames for the first to fourth lanes, payloads 1-1 to 1-4 are arranged following the header, with the remaining part being used as padding. Furthermore, in the lane frame for each lane, grouping information (GRPI) indicating that the payload is divided information is inserted before the payload. Note that if the header includes, for example, information about the lane, the contents of the header for each lane may differ. Furthermore, the grouping information may be inserted after the payload, or padding may be inserted between the header and the grouping information and / or between the grouping information and the payload.

[0018] Transmitting one piece of information using multiple lanes in this way enables efficient transmission in terms of the actual transmission speed. Based on the grouping information, the receiving side can easily recognize that the payload inserted into the lane frame of each lane is split information and combine it into one piece of information for processing. While the case where payload 1 fits into one lane frame has been described here, it is also possible for payload 1 to span multiple lane frames. In such a case, the free space (padding) in FIG. 2B can be filled with information (part of payload 1) that would have been sent in the next lane frame. According to a simple calculation ignoring the overhead of the grouping information, for example, payload 1, which requires 1,000 lane frames in simple transmission, can be transmitted in 250 lane frames in lane-split transmission using four lanes.

[0019] Next, time division transmission, that is, information transmission in which a plurality of pieces of information are transmitted in one lane, will be described.

[0020] 3A shows an example of the configuration of each lane frame in a simple transmission in which four pieces of information are transmitted using four lanes. In this example, in the lane frames of the first to fourth lanes, payloads 1 to 4 are placed following the header, and the remaining part is used as padding.

[0021] 3B shows an example of the configuration of a lane frame in time-division transmission in which four pieces of information are transmitted over one lane. In this example, payloads 1 to 4 are arranged following a header, with the remaining portion being padding. Separating information (SEPI) for separating (distinguishing) the payloads from one another is inserted between each of payloads 1 to 4. Separating information for separating the header and payload 1 may be inserted before payload 1. Padding may also be inserted before and / or after the payload.

[0022] By transmitting multiple pieces of information over a single lane in this way, lane frames can be used efficiently. The receiving side can easily separate, extract, and process the multiple pieces of information based on the separating information. While we have described intraframe time-division multiplexing transmission, in which the payload switches within a lane frame, interframe time-division multiplexing transmission, in which the payload switches between lane frames, is also possible. In such cases, the payloads can be distinguished by the header without relying on the separating information.

[0023] Next, combined division transmission, that is, information transmission that combines lane division and time division, will be described.

[0024] 4A shows an example of the configuration of each lane frame in a simple transmission in which four pieces of information are transmitted using four lanes. In this example, in the lane frame of each of lanes 1 to 4, payloads 1 to 4 are placed following the header, and the remaining part is used as padding.

[0025] 4B shows an example of the configuration of a lane frame for each lane in a combined transmission in which four pieces of information are arranged and transmitted on four lanes. Payloads 1-1 to 1-4 are divided pieces of information obtained by dividing payload 1 into four. Similarly, payloads 2-1 to 2-4 are divided pieces of information obtained by dividing payload 2 into four, payloads 3-1 to 3-4 are divided pieces of information obtained by dividing payload 3 into four, and payloads 4-1 to 4-4 are divided pieces of information obtained by dividing payload 4 into four. In other words, payloads X-1 to X-Y are divided pieces of information obtained by dividing payload X into Y pieces.

[0026] In this example, in the lane frame of the first lane, payload 1-1, payload 2-1, payload 3-1, and payload 4-1 are arranged in this order following the header, with padding arranged after each payload. It is also possible to arrange the padding arranged after each payload X-1 together after payload 4-1, etc. In this case, transmission of payloads 1-1 to 4-1 in the lane frame will be completed more quickly. However, depending on the processing capability of the receiving side, it may be better to make the padding between payloads longer than a predetermined length (minimum inter-payload space) because payload switching processing, etc., takes time. This also applies to the lane frames of the second to fourth lanes. In this example, grouping information 1 is inserted before payload 1-1, grouping information 2 and separating information are inserted before payload 2-1, grouping information 3 and separating information are inserted before payload 3-1, and grouping information 4 and separating information are inserted before payload 4-1.

[0027] In the lane frames of the second to fourth lanes, as in the first lane, payloads 1-Y to 4-Y are arranged in order in the lane frame of the Yth lane. Furthermore, grouping information 1 is inserted before payload 1-Y, grouping information 2 and separating information are inserted before payload 2-Y, grouping information 3 and separating information are inserted before payload 3-Y, and grouping information 4 and separating information are inserted before payload 4-Y. However, because the lane allocation of transmitting one payload using the first to fourth lanes is common to payloads 1 to 4, grouping information 2 to 4 may be omitted and only grouping information 1 may be transmitted.

[0028] By dividing multiple pieces of information into multiple lanes and transmitting them in this way, flexible information transmission becomes possible. The receiving side can easily separate, extract, combine, and process the transmitted information appropriately based on the grouping information and separating information. While intraframe combination multiplexing transmission, which combines lane division and intraframe time division, has been described here, interframe combination multiplexing transmission, which combines lane division and interframe time division, is also possible. In such cases, the payload can be distinguished by the header without relying on separating information.

[0029] Next, another example of information transmission using combination division will be described.

[0030] 5A shows an example of the configuration of each lane frame in a simple transmission in which four pieces of information are transmitted using four lanes. Unlike the example in FIG. 4A, the sizes of payloads 1 to 4 are not the same, but rather payload 4 > payload 1 > payload 3 > payload 2.

[0031] 5B shows an example of the lane frame configuration for each lane in a combined transmission in which four pieces of information are arranged and transmitted across four lanes. Payloads 1-1 to 1-3 are divided pieces of information obtained by dividing Payload 1 into three, and Payloads 3-1 and 3-2 are divided pieces of information obtained by dividing Payload 3 into two.

[0032] In this example, in the lane frame of the first lane, payload 1-1 and payload 3-1 are arranged in this order following the header, with the remaining portion being used as padding. Furthermore, grouping information 1 is inserted before payload 1-1, and grouping information 3 and separating information are inserted before payload 3-1. Similarly, in the lane frame of the second lane, payload 1-2 and payload 3-2 are arranged in this order following the header, with the remaining portion being used as padding. Furthermore, grouping information 1 is inserted before payload 1-2, and grouping information 3 and separating information are inserted before payload 3-2.

[0033] On the other hand, in the lane frame of the third lane, payloads 1-3 and payload 2 are arranged in this order following the header, with the remaining portion being padding. Furthermore, grouping information 1 is inserted before payloads 1-3, and separating information is inserted before payload 2. In the lane frame of the fourth lane, payload 4 is arranged following the header, with the remaining portion being padding. Unlike the example of FIG. 4B described above, the number of divisions for each payload is not the same, and some payloads (payload 2, payload 4) are transmitted without being divided. Note that the transmission start and end times for payloads X-1 to X-Y are the same, but one or both may be different. For example, the transmission order for the third lane may be as follows: header, payload 2, grouping information 1 and separating information, payloads 1-3, and padding, while the first and second lanes remain the same.

[0034] Even if the size of the information or the number of divisions is different in this way, the same effect as the example of FIG. 4B can be obtained.

[0035] Next, information transmission using double-speed transmission will be described.

[0036] 6A shows an example of the configuration of each lane frame in simple transmission in which four pieces of information are transmitted using four lanes at 1x speed. In this example, in the lane frame of each of lanes 1 to 4, payloads 1 to 4 are placed following the header, and the remaining part is used as padding.

[0037] 6B shows an example of the lane frame configuration in time-division transmission, in which four pieces of information are arranged in one lane and transmitted at quadruple speed. In this example, payloads 1 to 4 are arranged in order, including padding, following the header. Separating information is inserted before each of payloads 2 to 4 from the second onwards.

[0038] By utilizing double-speed transmission in this way, it is possible to use lane frames efficiently.

[0039] 2-1. Video Transmission Example An example of transmission in which the information to be transmitted is specifically video data (video stream) will be described. First, lane division transmission will be described.

[0040] 7A shows an example of the configuration of a lane frame in simple transmission in which video data as one piece of information is transmitted using only one lane. Since this example corresponds to the example of FIG. 2A described above, a description of common points will be omitted. In this example, metadata (Meta) and video data (Video) 1 are arranged as Payload 1 in FIG. 2A. The video data may be baseband video data or may be video data encoded using H.264 / AVC, H.265 / HEVC, or the like. Since one video frame typically does not fit into one lane frame, one video frame is transmitted using multiple lane frames. Periodic transmission is assumed, such that when transmission of a video frame is completed, transmission of the next video frame begins from the next lane frame.

[0041] The metadata inserted before the video data includes the video format data, time code data, etc. The metadata may be inserted after the video data, or padding may be inserted between the header and the metadata and / or between the metadata and the video data.

[0042] 7B shows an example of the configuration of lane frames for each lane in lane-division transmission in which video data as a single piece of information is transmitted using four lanes. This example corresponds to the example of FIG. 2B described above, and therefore common points will not be described again. Video data 1-1 to 1-4 are divided video data obtained by dividing video data 1 into four parts, and are the division information in the example of FIG. 2B. For example, video data 1 is video data for one screen (one frame), and video data 1-1 to 1-4 are divided video data corresponding to the divided screens obtained by dividing the screen into four equal parts (half vertically and half horizontally).

[0043] In this example, metadata and video data 1-1 to 1-4 are arranged as payloads 1-1 to 1-4 in Fig. 2B. The header of the Yth lane includes, as part of the header information, grouping information indicating that video data 1-Y is divided video data of video data 1. That is, the grouping information in Fig. 2B is included in the header in this example.

[0044] The header unit, the payload unit into which video format data is inserted, and the payload unit into which time code data is inserted will be described in detail later. Note that if the metadata for each lane is the same, it is sufficient that the metadata is transmitted using at least one lane. Furthermore, the metadata may be transmitted, for example, for each video frame, and does not necessarily have to be transmitted for each lane frame.

[0045] Next, time division transmission will be described.

[0046] 8A shows an example of the configuration of each lane frame in a simple transmission in which four pieces of video data as four pieces of information are transmitted using four lanes. This example corresponds to the example of FIG. 3A described above, and therefore a description of common points will be omitted. In this example, metadata X and video data X are arranged as payload X in FIG. 3A.

[0047] 8B shows an example of the configuration of a lane frame in time-division transmission in which four pieces of video data as four pieces of information are transmitted in one lane. This example corresponds to the example of FIG. 3B described above, and therefore common points will not be described again. In this example, metadata X and video data X are arranged as payload X in FIG. 3B.

[0048] At the beginning of metadata 2 to 4, MVTUs (Multiplex Video Transfer Units) 2 to 4 are inserted as separating information in Fig. 3B. MVTU also indicates that the information placed in the following payload unit is video data. Note that, like metadata 2 to 4, MVTU1 may be inserted at the beginning of metadata 1. Also, MVTUX may be separated from metadata X, and may be transmitted in the order of MVTUX, video data X, and metadata X, with padding etc. appropriately interspersed. Details of MVTU will be described later.

[0049] Next, the information transmission of the combination division will be explained.

[0050] 9A shows an example of the configuration of each lane frame in a simple transmission in which four pieces of video data as four pieces of information are transmitted using four lanes. This example corresponds to the example of FIG. 4A described above, and therefore a description of common points will be omitted. In this example, metadata X and video data X are arranged as payload X in FIG. 4A.

[0051] 9B shows an example of the configuration of a lane frame for each lane in a combined transmission in which four pieces of video data as four pieces of information are arranged in four lanes for transmission. This example corresponds to the example of FIG. 4B described above, and therefore a description of common points will be omitted. In this example, metadata X and video data X-Y are arranged as payload X-Y in FIG. 4B. Video data X-1 to X-Y are divided video data obtained by dividing video data X into Y parts.

[0052] MVTUs 2 to 4 are inserted at the beginning of metadata 2-Y to 4-Y as separating information in FIG. 4B. As with the example in FIG. 7B, grouping information 1 in FIG. 4B is included in the header in this example. Also, grouping information 2 to 4 in FIG. 4B are included in MVTUs 2 to 4 in this example. However, because the lane assignment is common to video data 1 to 4, grouping information 2 to 4 may also be included in the header.

[0053] Next, another example of information transmission using combination division will be described.

[0054] 10A shows an example of the configuration of each lane frame in a simple transmission in which five pieces of video data are transmitted using five lanes. Unlike the example in FIG. 9A, the capacity of video data 1 to 5 is not the same, with video data 1 and 3 > video data 4 > video data 2 and 5.

[0055] 10B shows an example of the lane frame configuration for each lane in a combined transmission in which five pieces of video data as five pieces of information are arranged in four lanes and transmitted. Video data 1-1 to 1-3 are divided video data obtained by dividing video data 1 into three pieces.

[0056] Similarly, video data 3-1 to 3-2 are split video data obtained by dividing video data 3 in two, and video data 4-1 to 4-2 are split video data obtained by dividing video data 4 in two. That is, video data X-1 to XY are split video data obtained by dividing video data X in Y parts.

[0057] Next, information transmission using double-speed transmission will be described.

[0058] 11A shows an example of the configuration of each lane frame in a simple transmission in which four pieces of video data as four pieces of information are transmitted using four lanes at 1x speed. This example corresponds to the example of FIG. 6A described above, and therefore a description of common points will be omitted. In this example, metadata X and video data X are arranged as payload X in FIG. 6A.

[0059] 11B shows an example of the configuration of a lane frame in time division transmission in which four pieces of video data as four pieces of information are arranged in one lane and transmitted at quadruple speed. This example corresponds to the example of FIG. 6B described above.

[0060] In this way, the same effect can be obtained even when the information to be transmitted is specifically video data.

[0061] "2-2. Other Transmission Examples" Furthermore, similar information transmission is possible even when the information to be transmitted includes data other than video data (such as USB (Universal Serial Bus) data, Ethernet (registered trademark) data, audio data, etc.). In this case, instead of dividing the screen, each piece of data can be divided using an appropriate method, and metadata appropriate for each piece of data can be used. For data other than video data, for example, a Supplementary Data Area Start Symbol Unit (SASU) can be used instead of MVTU, and details will be described later.

[0062] "3. Header Unit" The header unit will now be described in detail. Figure 12 shows an exemplary bit field structure of a header unit. A 64-bit fixed pattern of HDR63 to HDR0 is placed in bytes 0 to 7 of the header unit as synchronization information (see Figure 13). In Figure 12, an example of the value of the fixed pattern is shown in parentheses.

[0063] The eighth byte of the header unit contains an 8-bit value of PLN7 to PLN0 indicating the physical lane number (PLN) (see FIG. 13). FIG. 14A shows an example of the value assignment of PLN7 to PLN0.

[0064] The ninth byte of the header unit contains an 8-bit value, LGN7 to LGN0, which indicates the lane group number (LGN) used to group the divided streams when a single stream of information (such as video data or USB data) is divided into multiple lanes (see Figure 13). Divided streams with the same LGN assigned belong to the same stream. In this way, the LGN can be used as the grouping information mentioned above. Figure 14B shows an example of the value assignment of LGN7 to LGN0. When transmitting a single stream without dividing it, LGN7 to LGN0 are set to "0x00".

[0065] The 10th byte of the header unit contains an 8-bit value of LLN7 to LLN0 indicating the Logical Lane Number (LLN) (see FIG. 13). FIG. 14C shows an example of the value assignment of LLN7 to LLN0.

[0066] The 7th to 3rd bits of the 11th byte of the header unit contain 5-bit values ​​PGN4 to PGN0 that indicate the program number (PGN) assigned to each stream (see Figure 13). Figure 15A shows an example of the value assignment of PGN4 to PGN0.

[0067] Bits 1 to 0 of byte 11 of the header unit contain two-bit values ​​SCA1 to SCA0 indicating the category (Stream Category (SCA)) of the information being transmitted (see FIG. 13). FIG. 15B shows an example of value assignment for SCA1 to SCA0. For example, "0b00" indicates that the information is sub-data containing only data other than video data, and "0b01" indicates that the information is main data containing only video data. Furthermore, "0b11" indicates that the information contains a mixture of main data (video data) and sub-data (metadata).

[0068] The 17th byte of the header unit contains an 8-bit fixed pattern, ENU (End of Unit), to declare the end of the header unit. For example, ENU is "0xFD (0b11111101)." The second bit of the 11th byte and bytes 12 to 16 of the header unit are reserved (Rsvd), and the value of each bit is set to "0" as shown in parentheses.

[0069] 4. MVTU Details of the MVTU will be described. Fig. 16 shows an example of the bit field configuration of the MVTU. As described above, the MVTU is inserted into the payload unit in association with video data.

[0070] The 0th and 1st bytes of the MVTU contain 8-bit fixed patterns MVT0 and MVT1, respectively, which serve as identifiers indicating that the unit is an MVTU (see FIG. 17). For example, MVT0 is set to "0xFB (0b11111011)" and MVT1 is set to "0xC3 (0b11000011)". MVT0 and MVT1 can be used as the separating information described above.

[0071] The second byte of MVTU contains an 8-bit value, STN7 to STN0, indicating the Stream Number (STN) of the video stream (see Figure 17). STN corresponds to the PGN mentioned above (including cases where they match), and can be used as the grouping information mentioned above. Figure 18A shows an example of value assignment for STN7 to STN0.

[0072] The seventh bit of the third byte of the MVTU contains a flag SMT indicating whether the video format, color space, etc. match with the video data transmitted first in the lane frame (see FIG. 17). As shown in FIG. 18B, "0" indicates that the video format, etc. do not match, and "1" indicates that they match. If they do not match, for example, a Metadata Packet Unit (MPU) specifying the video format, etc. may be inserted in the payload unit following the MVTU, or default values ​​may be applied.

[0073] The seventh bit of the fourth byte of the MVTU contains a flag THN indicating whether data reduction (thinning-out) has been performed (see FIG. 17). As shown in FIG. 18C, "0" indicates that no reduction has been performed, and "1" indicates that reduction has been performed.

[0074] The fifth to third bits of the fourth byte of the MVTU are assigned three-bit values ​​THQ2 to THQ0 that indicate the reduction level (see FIG. 17). FIG. 19A shows an example of the value assignment of THQ2 to THQ0.

[0075] The 2nd to 0th bits of the 4th byte of the MVTU are filled with 3-bit values ​​THM2 to THM0, which indicate the reduction method (see FIG. 17). FIG. 19B shows an example of the allocation of THM2 to THM0 values. Usable reduction methods include, for example, frame rate reduction, resolution reduction, color space conversion, and color depth reduction.

[0076] In this way, the data reduction mode is transmitted to the receiving side through a combination of THN, THM2 to THM0, and THQ2 to THQ0. FIG. 20 shows examples of data reduction modes. Frame rate reduction may be implemented by simple frame skipping, or by more complex conversion that takes into account neighboring frames, etc. Resolution reduction may be implemented by simple line skipping, or by more complex conversion that takes into account neighboring lines, etc. Furthermore, a reserved area may be used to define other reduction modes (reduction method and reduction level), or to define data reduction using a combination of multiple reduction methods. For example, a non-uniform color depth reduction may be defined, such as reducing R and B to 5 bits each and G to 6 bits. Furthermore, a value indicating that a reduction method has been transmitted prior to transmission of video data, a value indicating that reduction definition data has been inserted before or after the MVTU, or a value indicating that the reduction mode is specified by another MVTU with the same STN, may be defined.

[0077] The header unit does not include a field for specifying the data reduction mode. This is because it is assumed that the first video data of a lane frame, which is sent with the highest priority in terms of time, is transmitted without data reduction. However, by inserting an MVTU as necessary, data reduction of the first video data can also be supported.

[0078] The fifth byte of MVTU contains 8-bit values ​​OPT7 to OPT0 that indicate how the video data is divided (see FIG. 17). FIG. 19C shows an example of the value allocation of OPT7 to OPT0.

[0079] For example, "0x00" indicates that the video data is not divided, "0x01" indicates that the video data corresponds to a split screen obtained by dividing the screen into two equal halves, top and bottom, "0x02" indicates that the video data corresponds to a split screen obtained by dividing the screen into two equal halves, left and right, and "0x03" indicates that the video data corresponds to a split screen obtained by dividing the screen into four equal parts. Also, "0xFD" indicates that the division method has been transmitted prior to the transmission of the video data, "0xFE" indicates that division definition data has been inserted before or after the MVTU, and "0xFF" indicates that the division method is specified by OPT7 to OPT0 of another MVTU having the same STN.

[0080] The 17th byte of the MVTU contains an 8-bit fixed pattern of ENU to declare the end of the MVTU. Note that bits 6 to 0 of the 3rd byte, bit 6 of the 4th byte, and bytes 6 to 16 of the MVTU are reserved.

[0081] 5. SASU Details of the SASU will be explained. Fig. 21A shows an example of the bit field configuration of the SASU. As described above, the SASU is inserted into the payload unit in association with data other than video data.

[0082] The 0th and 1st bytes of the SASU contain 8-bit fixed patterns SAS0 and SAS1, respectively, which serve as identifiers that indicate that this unit is a SASU (see FIG. 21B). For example, SAS0 is "0xFB (0b11111011)" and SAS1 is "0x01 (0b00000001)." SAS0 and SAS1 can be used as the separating information described above.

[0083] The second byte of SASU contains 8-bit values ​​STN7 to STN0, similar to the second byte of MVTU (see FIG. 21B).

[0084] The third byte of the SASU contains an 8-bit value, ATR7 to ATR0, that indicates the attribute of the data stream (see Figure 21B). For example, "0x01" indicates USB data, "0x02" indicates Ethernet data, "0x03" indicates I2C data, and "0xFF" indicates other data.

[0085] The 17th byte of the SASU contains an 8-bit fixed pattern of the ENU to declare the end of the SASU. Bytes 4 to 16 of the SASU are reserved.

[0086] 6. Example of Setting Main Fields FIG. 22B shows an example of values ​​for PLN, LGN, LLN, PGN, STN, and OPT corresponding to the example of FIG. 22A. The PLN of the header unit of each lane is set to a value corresponding to that lane. For example, the PLN of header unit A of the first lane is set to "0x00," indicating physical lane #1 (see FIG. 14A). Similarly, the PLNs of header units B to E of the second to fifth lanes are set to "0x01" to "0x04," indicating physical lanes #2 to #5. Furthermore, logical lanes #1 to #5 are assigned corresponding to physical lanes #1 to #5, and the LLNs of header units A to E are set to "0x00" to "0x04" (see FIG. 14C). Note that, for example, if physical lane #1 becomes unusable due to a physical failure, unused physical lane #6 can be assigned to logical lane #1 instead of physical lane #1. Even when such a change is made, the LLN does not change.

[0087] A common value is set for the LGN in the header units (header units A and E) corresponding to the divided video data of video data 1 (video data 1-1 to 1-2). For example, the LGN is set to "0x01," indicating lane group #1 (see FIG. 14B). A common value is also set for the PGN, "0x00," indicating program #1 (see FIG. 15A). Using this common LGN or PGN as a key, the receiving side can extract video data 1-1 to 1-2 from multiple pieces of data in the lane frame. Whether video data 1-1 is data from the first lane or the fifth lane (whether video data 1-2 is data from the first lane or the fifth lane) can be determined, for example, based on the LLN. That is, since the LLNs of the header units corresponding to the first lane and the fifth lane are "0x00" and "0x04," the data from the first lane, which has the smaller LLN, is video data 1-1, and the data from the fifth lane, which has the larger LLN, is video data 1-2. On the receiving side, for example, video data 1 can be obtained by combining video data 1-1 and 1-2 based on an agreement that video data 1-1 corresponds to the left half of the screen and video data 1-2 corresponds to the right half of the screen.

[0088] Similarly, a common value is set for the LGN in the header units (header units B and C) corresponding to the divided video data of video data 2 (video data 2-1 to 2-2). For example, the LGN is set to "0x02," which indicates lane group #2 (see FIG. 14B). A common value is also set for the PGN, "0x01," which indicates program #2 (see FIG. 15A). Using this common LGN or PGN as a key, the receiving side can extract video data 2-1 to 2-2 from multiple pieces of data within the lane frame. Since the LLNs of the header units corresponding to the second and third lanes are "0x01" and "0x02," the data in the second lane, which has the smaller LLN, is designated as video data 2-1, and the data in the third lane, which has the larger LLN, is designated as video data 2-2. The receiving side can obtain video data 2 by combining video data 2-1 to 2-2 based on the agreement that, for example, video data 2-1 corresponds to the upper half of the screen and video data 2-2 corresponds to the lower half of the screen.

[0089] In the header unit (header unit D) corresponding to the USB data 3 transmitted without division, the LGN is set to "0x00" indicating that division has not been performed, and the PGN is set to, for example, "0x03." The receiving side can recognize from the value of LGN that combining of the USB data 3 is not necessary.

[0090] A common value is set for the STN in the MVTUs (MVTUs 4A, 4B, and 4D) corresponding to the divided video data (video data 4-1 to 4-3) of video data 4. For example, the STN is set to "0x03," which indicates stream #4 (see FIG. 18A). Using this common STN as a key, the receiving side can extract video data 4-1 to 4-3 from multiple pieces of data in the lane frame. Whether video data 4-1 is data for the first, second, or fourth lane can be determined, for example, based on the LLN. That is, since the LLNs of the header units corresponding to the first, second, and fourth lanes are "0x00," "0x01," and "0x03," the data for the first lane, which has the smallest LLN, is video data 4-1, the data for the second lane, which has the second smallest LLN, is video data 4-2, and the data for the fourth lane, which has the largest LLN, is video data 4-3.

[0091] For example, the OPT value of the MVTU (MVTU4A) corresponding to video data 4-1 is set to "0xFD," indicating that the division method has been transmitted in advance (see FIG. 19C). Furthermore, the MVTUs (MVTU4B, 4D) corresponding to video data 4-2 to 4-3 are set to "0xFF," indicating that the division method has been specified by the OPT of another MVTU (MVTU4A) having the same STN ("0x03," indicating stream #4) (see FIG. 19C). On the receiving side, video data 4 can be obtained by combining video data 4-1 to 4-3 based on a prior agreement, for example, that video data 4-1 corresponds to the left of the screen, video data 4-2 corresponds to the center of the screen, and video data 4-3 corresponds to the right of the screen.

[0092] Similarly, a common value is set for the STN in the SASUs (SASUs 5C and 5E) corresponding to the divided USB data (USB data 5-1 to 5-2) of USB data 5. For example, the STN is set to "0x04," indicating stream #5 (see FIG. 18A). Using this common STN as a key, the receiving side can extract USB data 5-1 to 5-2 from the multiple data in the lane frame. Since the LLNs of the header units corresponding to the third and fifth lanes are "0x02" and "0x04," the data in the third lane, which has the smaller LLN, is USB data 5-1, and the data in the fifth lane, which has the larger LLN, is USB data 5-2. For example, the receiving side can combine USB data 4-1 to 4-2 in this order to obtain USB data 5.

[0093] In the MVTU (MVTU6C) corresponding to the video data 6 transmitted without division, the STN is set to "0x05" indicating stream #6 (see FIG. 18A), and the OPT is set to "0x00" indicating that division has not been performed (see FIG. 19C). The receiving side can recognize from the value of OPT that synthesis of the video data 6 is not necessary.

[0094] Similarly, in a SASU (SASU7E) corresponding to Ethernet data 7 transmitted without segmentation, STN is set to "0x06" (see FIG. 18A), which indicates stream #7, and OPT is set to "0x00" (see FIG. 19C), which indicates that segmentation has not been performed. The receiving side can recognize from the value of OPT that combining of Ethernet data 7 is not necessary.

[0095] 7. MPU The MPU will be described in detail below. Fig. 23A shows an example of the bit field configuration of the MPU.

[0096] The 0th and 1st bytes of the MPU contain 8-bit fixed patterns MDS0 and MDS2, respectively, which serve as identifiers that indicate that this unit is an MPU (see FIG. 23B). For example, MDS0 is set to "0xFB (0b11111011)" and MDS1 is set to "0x05 (0b00000101)".

[0097] The second byte of the MPU contains an 8-bit value of MPID7 to MPID0 indicating a metadata ID for identifying the metadata included in the MPU (see Figure 23B).

[0098] For example, an MPU with MPID7 to MPID0 set to "0x11" contains information about color gamut representations such as xvYCC and BT.2020. MPUs with MPID7 to MPID0 set to "0x20" and "0x21" correspond to ISRC1 packets and ISRC2 packets, respectively. MPUs with MPID7 to MPID0 set to "0x81" handle vendor-specific information.

[0099] The metadata itself is located in bytes 5 to 16 of the MPU.

[0100] The 17th byte of the MPU contains an 8-bit fixed pattern of ENU to declare the end of the MPU. The 3rd and 4th bytes of the MPU are reserved.

[0101] 24 shows an example of the bit field configuration of an MPU in which MPID7 to MPID0 are "0x10." The body of this MPU contains information such as the video frame rate and resolution (see FIG. 25).

[0102] The body of an MPU with MPID7 to MPID0 set to "0x14" contains information such as the video frame time code. Because the MPU body can store up to 96 bits (12 bytes) of data (see FIG. 23A), an 80-bit SMPTE time code can be used as the time code format. SMPTE time code expresses time in seconds using hours, minutes, and seconds, with subseconds expressed as frame numbers. However, because frame numbers are specified as integers between 0 and 29 (with the ones digit expressed as a 4-bit binary-coded decimal number and the tens digit expressed as a 2-bit number), they are not suitable for high-frame-rate video exceeding 30 fps. Therefore, the bits allocated to frame numbers may be expanded by using free space (binary groups) within the SMPTE time code or free space within the MPU body (e.g., bytes 15 to 16 of the MPU).

[0103] For example, by using the lower two bits of binary group 1 in addition to the two bits originally assigned, the tens digit of the frame number can be expressed in binary-coded decimal, making it possible to specify frame numbers from 0 to 99. Furthermore, by using the upper two bits of binary group 1 to express the hundreds digit of the frame number, it becomes possible to specify frame numbers from 0 to 299.

[0104] It should be noted that various fields may be defined in the free space, not limited to frame numbers. For example, fields corresponding to PGN and STN may be defined in binary group 2.

[0105] 26A shows a configuration example of a transmission / reception system 10 that transmits data using the transmission format of the present technology. In this transmission / reception system 10, a server 100 and a user device 200 are connected via a wired or wireless network 300. A head-mounted display (HMD) 400 worn by a user is connected to the user device 200.

[0106] 26B shows an example configuration of the server 100. The server 100 includes a control unit 101, a storage unit 102, a signal processing unit 103, and a network interface 104. The control unit 101 controls the operation of each unit of the server 100. The signal processing unit 103 constitutes a generation unit that arranges transmitted information into lane frames and generates a lane stream, which is a series of lane frames. In particular, if the transmitted information includes video data, the signal processing unit 103 may perform video rendering.

[0107] The network interface 104 is an interface for communicating with the user device 200. The lane stream generated by the signal processing unit 103 is transmitted from the network interface 104 to the user device 200 via the network 300. Therefore, the network interface 104 constitutes a transmitting unit.

[0108] 26C shows an example configuration of the user device 200. The user device 200 includes a control unit 201, a network interface 202, a storage unit 203, a signal processing unit 204, a USB interface 205, and a high-speed interface 206. The control unit 201 controls the operation of each unit of the user device 200. The network interface 202 is an interface for communicating with the server 100. The lane stream sent from the server 100 via the network 300 is received by the network interface 202. Therefore, the network interface 202 constitutes a receiving unit.

[0109] The storage unit 203 stores the lane stream sent from the server 100 or each piece of data extracted from the lane stream. The signal processing unit 204 constitutes a processing unit that processes each lane frame included in the lane stream sent from the server 100. In particular, when the transmitted information includes video data, the signal processing unit 204 processes the video data to generate image data for display. The USB interface 205 is an interface for acquiring viewpoint information (described later) from the HMD 400. The high-speed interface 206 is a high-speed interface for transmitting video (described later) to the HMD 400. Note that the high-speed interface 206 may be used to perform bidirectional communication between the user device 200 and the HMD 400, and the user device 200 may acquire viewpoint information from the HMD 400 as USB data using the high-speed interface 206 instead of the USB interface 205.

[0110] The user device 200 acquires viewpoint information from the HMD 400, including information such as the position of the viewpoint and the moving speed of the user wearing the HMD 400. Note that the viewpoint information can be acquired in the HMD 400 by using techniques such as head movement detection and eye tracking. The user device 200 sends this viewpoint information to the server 100 via the network 300.

[0111] The server 100 performs rendering based on the viewpoint information sent from the user device to generate display video data for each video frame. The server 100 then sends this display video data for each video frame to the user device 200 via the network 300. The user device 200 sends the display video data to the HMD 400, causing the HMD 400 to display an image.

[0112] The user equipment 200 may be capable of supporting only the regular-speed transmission mode or may be capable of supporting both the regular transmission mode and the Nx-speed transmission mode. Figure 27 shows a switching sequence between the regular-speed transmission mode and the Nx-speed transmission mode performed between the server 100 and the user equipment 200. This sequence is primarily performed prior to data transmission, but may be re-executed if an error is detected during data transmission. Furthermore, although this sequence is intended to determine (adjust) the transmission rate, the lane frame length and / or the minimum inter-payload spacing may also be determined (adjusted) using a procedure similar to this sequence.

[0113] In step ST1, the server 100 transmits an Nx-speed receiving capability information request to the user device 200. Next, in step ST2, the user device 200 receives the Nx-speed receiving capability information request sent from the server 100. Next, in step ST3, the user device 200 transmits its own Nx-speed receiving capability information to the server 100. This Nx-speed receiving capability information includes, for example, a list of receivable speed values ​​and ranges. Next, in step ST4, the server 100 receives the Nx-speed receiving capability information sent from the user device 200. However, steps ST1 and ST2 may be omitted, and the switching sequence may be initiated by the user device 200 spontaneously transmitting the Nx-speed receiving capability information in step ST3. Furthermore, the server 100 may acquire N-times speed reception compatibility information of the user device 200 from an external database or its own storage unit 102, based on device information of the user device 200 and / or the HMD 400, etc.

[0114] Next, in step ST5, the server 100 determines whether the user device 200 is capable of N-times speed reception based on the N-times speed reception support information sent from the user device 200. If it is determined that the user device 200 is capable of N-times speed reception, the server 100 determines in step ST6, based on the N-times speed reception support information sent from the user device 200, whether the server 100 itself can transmit data at a speed that the user device 200 can receive.

[0115] When the user device 200 determines that N-times speed reception is not possible, or when the server 100 determines that it cannot transmit data at any of the speeds that the user device 200 can receive, the server 100 selects the normal transmission mode in step ST7. Next, in step ST8, the server 100 transmits transmission mode information indicating that the normal transmission mode has been selected to the user device 200, starts transmitting data in the normal transmission mode, and then ends the switching sequence in step ST9.

[0116] Furthermore, when the server 100 starts transmitting data in the normal transmission mode in step ST8, the user device 200 starts receiving data in the normal mode in step ST10, and then ends the switching sequence in step ST11.

[0117] Furthermore, when the server 100 determines in step ST6 that it can transmit data at any of the speeds that the user device 200 can receive, the server 100 transmits transmission mode information indicating the selected speed to the user device 200 in step ST12. The selected speed is, for example, the maximum value or the minimum value suitable for the expected data communication among the speeds that both the user device 200 and the server 100 can support.

[0118] Next, in step ST13, the user device 200 receives the transmission mode information sent from the server 100. Next, in step ST14, the user device 200 sets itself to a state where it is ready to receive in the selected transmission mode, and sends Nx-speed receiving ready information to the server 100.

[0119] Next, in step ST15, the server 100 receives the N-times speed reception capability information sent from the user device 200. Next, in step ST16, the server 100 transmits test data at the selected speed to the user device 200. Next, in step ST17, the user device 200 receives the test data sent from the server 100.

[0120] Next, in step ST18, the user device 200 determines whether the test data has been received correctly. If it is determined that the test data has been received correctly, the user device 200 transmits successful receipt information to the server 100 in step ST19. On the other hand, if it is determined that the test data has not been received correctly, the user device 200 transmits erroneous receipt information to the server 100 in step ST20, and returns the user device 200 to a state where it can receive data in the normal transmission mode.

[0121] Next, in step ST21, the server 100 determines whether or not it has received successful reception information from the user device 200. If it determines that it has received successful reception information, the server 100 starts transmitting data in the N-times speed transmission mode of the selected speed in step ST22, and then ends the switching sequence in step ST9.

[0122] Also, when the server 100 starts transmitting data in N-times speed transmission mode in step ST22, the user device 200 starts receiving data in N-times speed transmission mode in step ST23, and then ends the switching sequence in step ST11.

[0123] Furthermore, when the server 100 determines in step ST21 that it has not received successful reception information, i.e., that it has received reception error information, it selects the normal transmission mode in step ST7. Next, in step ST8, the server 100 transmits transmission mode information indicating that the normal transmission mode has been selected to the user device 200, starts transmitting data in the normal transmission mode, and then ends the switching sequence in step ST9. However, when it determines in step ST21 that it has received reception error information, it may return to step ST12 instead of proceeding to step ST7 and repeat steps ST12 to ST21 while selecting another speed, such as a slower speed including when N is 1 or less. Furthermore, steps ST14 to ST15 and / or steps ST16 to ST21 may be omitted. Furthermore, it may be possible to omit steps ST1 to ST6 and ST12 to ST15 and start the switching sequence by transmitting test data at a speed appropriate for the data to be transmitted in step ST16.

[0124] Also, when the server 100 starts transmitting data in the normal transmission mode in step ST8, the user device 200 starts receiving data in the normal transmission mode in step ST10, and then ends the switching sequence in step ST11.

[0125] 9. Motion-to-Photon Latency Motion-to-photon latency in the transmission / reception system 10 will be described using three examples. In all three examples, the server 100 renders, as images for the HMD 400, one video frame at a time, a video (Video C) near the center of the user's field of view and videos (Video U, Video D, Video L, and Video R) at the top, bottom, left, and right, as shown in FIG. 28A . At least one of these five rendered video data is transmitted from the server 100 to the HMD 400. Different PGNs (STNs) are assigned to these five videos. For simplicity's sake, it is assumed that the HMD 400 starts receiving a video frame one video frame period after the server 100 starts rendering the video frame, and then completes reception of the video frame one video frame period after that. At the same time, the HMD 400 starts displaying the video frame, and then the HMD 400 is able to switch video frames one video frame period after that.

[0126] First, a first example will be described with reference to FIG. 28B . For simplicity, FIG. 28B only shows which video frame is rendered, transmitted, and displayed, without illustrating details at the lane frame level. Therefore, the "frame" in the figure refers to a video frame, not a lane frame. In the first example, only one of the five pieces of video data rendered by the server 100 is transmitted to the HMD 400. This video transmission can be achieved by simple transmission using one lane (see FIG. 7A ), but it can also be achieved by, for example, lane-split transmission using multiple lanes (see FIG. 7B ). To calculate the motion-to-photon latency, consider a situation in which the user turns right while the center video (Center) is being transmitted, and the right video (Right) is required. Specifically, assume that the user turns right when the display on the HMD 400 switches from video frame 1 (Frame #1) to video frame 2. For simplicity, it is assumed that information that the user has turned right is transmitted to the server 100 with a delay sufficiently small relative to the video frame period. Specifically, this information arrives at the server 100 while the server 100 is rendering video frame 4. In response to this information, the server 100 switches the video to be transmitted from the center video to the right video starting from video frame 5. The HMD 400 receives video frame 5 while displaying video frame 4. Simultaneously with the completion of reception of video frame 5 (one video frame period after the start of display of video frame 4), the HMD 400 starts displaying video frame 5. In this way, it takes three frames, from video frame 2 to video frame 5, from when the user turns to the right until the right video is displayed on the HMD 400, so the motion-to-photon latency in this example is three frames.

[0127] Next, a second example will be described with reference to FIG. 29 . The difference from the first example is that in the second example, two of the five pieces of video data rendered by the server 100 are simultaneously transmitted to the HMD 400. Specifically, the two transmitted videos are a center video and a right video. This video transmission can be achieved by simple transmission using two lanes, but it can also be achieved by lane-split transmission using, for example, four lanes. As in the first example, assume that the user turns to the right when the display on the HMD 400 switches from video frame 1 to video frame 2. Unlike the first example, the right video is transmitted from the server 100. Therefore, it is only necessary to switch the video displayed on the HMD 400 from the center video to the right video from the next video frame (video frame 3). Therefore, the motion-to-photon latency in this example is one frame, from video frame 2 to video frame 3, which is smaller than that in the first example. If a video that has not been transmitted from the server 100 (any of the upper, lower, or left video) becomes necessary, the motion-to-photon latency becomes 3 frames, as in the first example.

[0128] Next, a third example will be described with reference to FIG. 30 . The difference from the second example is that in the third example, two videos are transmitted to the user device 200 in a time-division manner. This video transmission can be achieved, for example, by inter-lane time-division transmission using a single double-speed lane or inter-lane combined transmission using multiple lanes. As with the first and second examples, assume that the user turns to the right when the display on the HMD 400 switches from video frame 1 to video frame 2. As with the second example, the motion-to-photon latency in this example is one frame, which is smaller than that of the first example. Note that if the HMD 400 supports a refresh rate twice the video frame rate, the motion-to-photon latency could be 0.5 frames. Furthermore, even if the left video that has not been transmitted from the server 100 becomes necessary later, the motion-to-photon latency is two frames, which is smaller than that of the first example.

[0129] "10. Transmission Priority" As mentioned above, there are cases where motion-to-photon latency can be reduced by transmitting multiple videos. Ideally, it would be desirable to transmit all five videos, for example, using a sufficient number of lanes, but such transmission is not always possible. When the number of videos that can be transmitted is less than five, it is necessary to determine which videos to transmit, and whether videos are being transmitted when they become necessary can affect motion-to-photon latency.

[0130] The transmission priority for determining which video to transmit may be determined based on, for example, the predicted frequency with which a certain video will be required. That is, in accordance with the tendency for eye movement from side to side to be more frequent than from top to bottom, the transmission priority may be set as follows: center video > left and right video > top and bottom video. The transmission priority may also be determined based on viewpoint information (position vector and / or velocity vector). For example, if it is detected that the user's viewpoint is located in the upper left or moving toward the upper left at a certain point in time, the transmission priority of the left video and the top video may be increased, and / or the transmission priority of the right video and the bottom video may be decreased. Furthermore, because a user would not notice if a video frame with little change (motion) were substituted with a previous video frame, the transmission priority may also be determined based on the amount of change (motion) between video frames. The transmission priority may be determined based on at least one of these multiple factors.

[0131] When determining which videos to transmit based on their transmission priorities, there are options such as not transmitting videos with low transmission priorities or determining the frequency of transmission according to the transmission priorities. For example, if the number of videos available for transmission is three, only the videos with the highest transmission priorities may be transmitted, or the videos with the highest transmission priorities may be transmitted every time while the videos with the lowest transmission priorities may be transmitted according to round-robin scheduling or weighted round-robin scheduling based on the transmission priorities, or five videos may be transmitted according to round-robin scheduling or weighted round-robin scheduling based on the transmission priorities.

[0132] When round robin scheduling or weighted round robin scheduling is used, the frame rate reduction of Fig. 19B is substantially applied, but other data reduction methods may also be used. For example, if the number of videos available for transmission is three, the videos with the highest two transmission priorities may be transmitted without data reduction, while the videos with the third and fourth transmission priorities may each have their data reduced by half for transmission; alternatively, the videos with the highest two transmission priorities may be transmitted without data reduction, while the video with the third transmission priority may have its data reduced by half, and the videos with the fourth and fifth priorities may each have their data reduced by a quarter for transmission.

[0133] Note that a specific video, such as the video in the center, may be always transmitted regardless of transmission priority. This may be achieved by ranking five videos including the specific video, and restoring the specific video if it falls out of the ranking range. Alternatively, the specific video may be fixed to a specific rank, such as the top or bottom, and ranking may be performed excluding the specific video.

[0134] The transmission / reception system to which the present technology can be applied is not limited to the above-described transmission / reception system 10. For example, a configuration in which the HMD 400 is a display device such as a projector, a television receiver, a tablet, or a smartphone is also conceivable. Also, for example, a configuration in which, instead of user viewpoint information, information from a sensor that captures the movement of the user's arms, legs, or the like is sent from a smartphone or the like to the server 100, and the server 100 renders video data based on the information is also conceivable.

[0135] 11. Example of Hardware Configuration of Computer FIG. 31 is a block diagram showing an example of the hardware configuration of a computer 600 that functions as the server 100 or the user device 200 described above.

[0136] The computer 600 includes a CPU 601, a ROM 602, a RAM 603, a bus 604, an input / output interface 605, an input unit 606, an output unit 607, a storage unit 608, a drive 609, a connection port 610, and a communication unit 611. Note that the hardware configuration shown here is an example, and some of the components may be omitted. Furthermore, the computer 600 may include components other than those shown here.

[0137] The CPU 601 functions as, for example, an arithmetic processing device or a control device, and controls the overall operation or part of the operation of each component based on various programs recorded in the ROM 602 , the RAM 603 , the storage unit 608 , or the removable recording medium 701 .

[0138] The ROM 602 is a means for storing programs to be read into the CPU 601, data to be used for calculations, etc. The RAM 603 temporarily or permanently stores, for example, the programs to be read into the CPU 601 and various parameters that change as appropriate when the programs are executed.

[0139] The CPU 601, ROM 602, and RAM 603 are connected to one another via a bus 604. On the other hand, various components are connected to the bus 604 via an input / output interface 605.

[0140] The input unit 606 may be, for example, a mouse, a keyboard, a touch panel, a button, a switch, a lever, etc. Furthermore, the input unit 606 may also be a remote controller (hereinafter referred to as a remote control) that is capable of transmitting control signals using infrared rays or other radio waves.

[0141] The output unit 607 is a device capable of visually or audibly notifying the user of acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, or a facsimile.

[0142] The storage unit 608 is a device for storing various types of data. For example, the storage unit 608 may be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.

[0143] The drive 609 is a device that reads information recorded on a removable recording medium 701 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable recording medium 701 .

[0144] The removable recording medium 701 is, for example, a DVD, a Blu-ray (registered trademark) disc, an HD DVD, various semiconductor storage media, etc. Of course, the removable recording medium 701 may also be, for example, an IC card equipped with a contactless IC chip, an electronic device, etc.

[0145] The connection port 610 is a port for connecting an external device 502, such as a Universal Serial Bus (USB) port, a High-Definition Multimedia Interface (HDMI) port, an IEEE 1394 port, a Small Computer System Interface (SCSI), an RS-232C port, or an optical audio terminal. Note that "HDMI" is a registered trademark. The external device 702 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0146] The communication unit 611 is a communication device for connecting to the network 703, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0147] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0148] As described above, in the present technology, a plurality of lane frames are used to generate and transmit a continuous lane stream, and when the lane frame contains a plurality of pieces of information in a time-division manner, it contains first information before each piece of information from the second onwards, indicating that the information is different from the previous information, and when it contains divided information obtained by dividing one piece of information into multiple pieces, it contains second information before the divided information, indicating that the divided information is divided, making it possible to transmit information effectively.

[0149] In addition, this technology transmits video data from multiple screens, including the screen where the user's viewpoint is located and a predetermined number of surrounding screens, using one or multiple lanes, making it possible to reduce the delay time of "Motion-to-photon."

[0150] "12. Modifications" Note that while preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person with ordinary skill in the technical field of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0151] In the examples of FIG. 2B or 4B , grouping information relating to payloads 1-1 to 1-4 is transmitted on each of the four lanes, but it is also possible to transmit grouping information on some of the four lanes. For example, when transmitting grouping information only on the first lane, the grouping information only needs to indicate that payload 1-1 is divided information of payload 1 and that payloads 1-2 to 1-4 are also divided information of payload 1. Such grouping information may be transmitted on lanes other than the four lanes. Also, for example, it is possible to transmit grouping information indicating that payload 1-1 is divided information of payload 1 on the second lane, and transmit grouping information indicating that payload 1-2 is divided information of payload 1 on the first lane. Furthermore, for example, it is also possible to transmit grouping information indicating that payload 1-1 is divided information of payload 1 on the second lane, and transmit grouping information indicating that payloads 1-2 to 1-4 are divided information of payload 1 on lanes other than the four lanes.

[0152] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0153] The present technology may also be configured as follows: (1) A transmitting device including: a generating unit that generates a lane stream in which a plurality of lane frames including a first lane frame are consecutive; and a transmitting unit that transmits the lane stream, wherein the first lane frame includes a plurality of lanes that are different from each other, the plurality of lanes including a first lane and a second lane, the first lane includes first division information among a plurality of division information that are different from each other, the second lane includes second division information among the plurality of division information, and the first lane frame includes grouping information indicating that the first division information and the second division information are obtained by dividing the first information. (2) The transmitting device described in (1), wherein the lane stream includes second information, third information, and separating information that separates the second information and the third information. (3) The transmitting device described in (2), wherein the second information is the first division information, and the third information is different from any of the plurality of division information and is included in the first lane. (4) The transmitting device according to (2), wherein the second information and the third information are included in different or the same lane frames among the plurality of lane frames. (5) The transmitting device according to any of (1) to (4), wherein each of the plurality of lane frames includes a leading header unit and a plurality of payload units following the header unit. (6) The transmitting device according to any of (1) to (5), wherein the grouping information includes first grouping information indicating that the first division information is obtained by dividing the first information, and second grouping information indicating that the second division information is obtained by dividing the first information. (7) The transmitting device according to (6), wherein the first grouping information is included in the first lane, and the second grouping information is included in the second lane. (8) The transmitting device according to (7), wherein the first grouping information is arranged before the first division information, and the second grouping information is arranged before the second division information.(9) The transmitting device according to any one of (1) to (8), wherein the number of the plurality of pieces of division information can be determined from the grouping information. (10) The transmitting device according to any one of (5) to (9), wherein the header unit of the first lane frame includes the grouping information. (11) The transmitting device according to any one of (5) to (10), wherein the header unit includes physical lane information or logical lane information. (12) The transmitting device according to any one of (1) to (11), wherein the first lane frame includes category information for indicating a category of the first information. (13) The transmitting device according to any one of (1) to (12), wherein the first information is first video data. (14) The transmitting device according to (13), wherein the first lane frame includes metadata related to the first video data. (15) The transmitting device according to (14), wherein the metadata includes at least one of video format information, time code information, data reduction information, and screen division information of the first video data. (15) The transmitting device according to any one of (1) to (14), wherein the lane stream includes attribute information or vendor-specific information. (16) The transmitting device according to any one of (13) to (15), wherein the first video data is data of a first frame of a first video, the first division information is data of a first region of the first frame, and the second division information is data of a second region of the first frame that is different from the first region.(17) The transmission device according to (17), wherein the plurality of lane frames include a second lane frame different from the first lane frame, the first lane frame includes second video data, the second lane frame includes third video data and fourth video data, the second video data is data of a frame of a second video different from the first video that is synchronized with the first frame, the third video data is data of a second frame of the first video that is different from the first frame, and the fourth video data is data of a frame of a third video different from the first video that is synchronized with the second frame, and the generation unit determines whether the third video is the same as the second video based on priority information. (18) The transmission device according to (17), wherein the priority information includes viewpoint information received from a head-mounted display. (19) A receiving device comprising: a receiving unit that receives a continuous lane stream of multiple lane frames including a first lane frame; the first lane frame includes multiple lanes that are different from each other; the multiple lanes include a first lane and a second lane; the first lane includes first division information from multiple division information that are different from each other; the second lane includes second division information from the multiple division information; the first lane frame includes grouping information indicating that the first division information and the second division information are obtained by dividing the first information; and a processing unit that combines the multiple division information based on the grouping information.(20) A transmission / reception system having a transmitting device and a receiving device, wherein the transmitting device comprises: a generating unit that generates a lane stream of a series of multiple lane frames including a first lane frame; and a transmitting unit that transmits the lane stream, wherein the first lane frame comprises multiple lanes that are different from each other, the multiple lanes include a first lane and a second lane, the first lane comprises first division information from multiple division information that are different from each other, the second lane comprises second division information from the multiple division information, and the first lane frame comprises grouping information indicating that the first division information and the second division information are obtained by dividing the first information, and the receiving device comprises: a receiving unit that receives the lane stream; and a processing unit that combines the multiple division information based on the grouping information.

[0154] 10: Transmission / reception system 100: Server 101: Control unit 102: Storage unit 103: Signal processing unit 104: Network interface 200: User device 201: Control unit 202: Network interface 203: Storage unit 204: Signal processing unit 205: USB interface 206: High-speed interface 300: Network 400: HMD

Claims

1. A transmitting device comprising: a generating unit that generates a lane stream consisting of a series of multiple lane frames including a first lane frame; and a transmitting unit that transmits the lane stream, wherein the first lane frame includes a plurality of lanes that are different from each other, the plurality of lanes including a first lane and a second lane, the first lane includes first division information from a plurality of different division information, the second lane includes second division information from the plurality of division information, and the first lane frame includes grouping information indicating that the first division information and the second division information are obtained by dividing the first information.

2. The transmitting device according to claim 1, wherein the lane stream includes second information, third information, and separating information for separating the second information from the third information.

3. The transmitting device according to claim 2, wherein the second information is the first division information, and the third information is different from any of the plurality of division information and is included in the first lane.

4. The transmitting device according to claim 2, wherein the second information and the third information are included in different lane frames among the plurality of lane frames.

5. The transmitting device according to claim 1, wherein each of the plurality of lane frames includes a header unit at the beginning and a plurality of payload units following the header unit.

6. The transmitting device of claim 1, wherein the grouping information includes first grouping information for indicating that the first division information was obtained by dividing the first information, and second grouping information for indicating that the second division information was obtained by dividing the first information, the first grouping information being included in the first lane, and the second grouping information being included in the second lane.

7. The transmitting device according to claim 6, wherein the first grouping information is arranged before the first division information, and the second grouping information is arranged before the second division information.

8. The transmitting device according to claim 1, wherein the number of the plurality of pieces of divided information can be determined from the grouping information.

9. The transmitting device according to claim 5, wherein the header unit of the first lane frame includes the grouping information.

10. The transmitting device according to claim 5, wherein the header unit includes physical lane information and logical lane information.

11. The transmitting device according to claim 1, wherein the first lane frame includes category information for indicating a category of the first information.

12. The transmitting device according to claim 1, wherein the first information is first video data.

13. The transmitting device of claim 12, wherein the first lane frame includes metadata related to the first video data.

14. The transmitting device according to claim 13, wherein the metadata includes video format information or time code information of the first video data.

15. The transmitting device according to claim 13, wherein the metadata includes data reduction information for the first video data.

16. A transmitting device as described in claim 12, wherein the first video data is data of a first frame of a first video, the first division information is data of a first region of the first frame, and the second division information is data of a second region of the first frame that is different from the first region.

17. The transmitting device described in claim 16, wherein the plurality of lane frames include a second lane frame different from the first lane frame, the first lane frame includes second video data, the second lane frame includes third video data and fourth video data, the second video data is data of a frame of a second video different from the first video that is synchronized with the first frame, the third video data is data of a second frame of the first video that is different from the first frame, and the fourth video data is data of a frame of a third video different from the first video that is synchronized with the second frame, and the generating unit determines whether the third video is the same as the second video based on priority information.

18. The transmitting device according to claim 17, wherein the priority information includes viewpoint information received from a head-mounted display.

19. A receiving device comprising: a receiving unit that receives a continuous lane stream of multiple lane frames including a first lane frame; the first lane frame includes multiple lanes that are different from each other; the multiple lanes include a first lane and a second lane; the first lane includes first division information from multiple division information that are different from each other; the second lane includes second division information from the multiple division information; the first lane frame includes grouping information indicating that the first division information and the second division information are obtained by dividing the first information; and a processing unit that combines the multiple division information based on the grouping information.

20. A transmission and reception system having a transmitting device and a receiving device, wherein the transmitting device comprises a generating unit that generates a lane stream consisting of a series of lane frames including a first lane frame, and a transmitting unit that transmits the lane stream, wherein the lane frames include a plurality of lanes that are different from each other, the plurality of lanes including a first lane and a second lane, the first lane includes first division information from a plurality of different division information, the second lane includes second division information from the plurality of division information, and the first lane frame includes grouping information indicating that the first division information and the second division information are obtained by dividing the first information, and the receiving device comprises a receiving unit that receives the lane stream, and a processing unit that combines the plurality of division information based on the grouping information.

Images