Transmitter, processing system, control method for transmitter, program

Abstract

This technology provides the ability to transmit data more effectively even when communication conditions deteriorate. [Solution] A transmitting device that transmits first data to a receiving device includes: a first generation means that generates packets for each of the multiple data obtained by dividing the first data by adding a packet header to each of the multiple data; a first transmitting means that transmits each packet generated by the first generation means to the receiving device; and a first control means that controls the packet size, which is the size of each packet, according to the communication state between the transmitting device and the receiving device. The first control means sets the packet size to a first size if the communication state is a first state, and sets the packet size to a second size, which is larger than the first size, if the communication state is a second state that is worse than the first state.

Description

Technical Field

[0005]

[0001] The present invention relates to a transmission device, a processing system, a control method for a transmission device, and a program.

Background Art

[0002] In recent years, as a technology for seamlessly integrating the real world and the virtual world in real time, a composite reality technology, so-called MR (Mixed Reality) technology, is known. As one of the MR technologies, there is an MR system using a video see-through type HMD (Head Mounted Display; hereinafter referred to as "HMD" as necessary).

[0003] In the MR system, the imaging unit of the HMD images a subject that substantially coincides with the subject observed from the pupil position of the user wearing the HMD. Then, the image processing device generates a display image in which CG (Computer Graphics) is superimposed on the captured image, and the display unit of the HMD displays the display image. Thereby, the MR system can allow the user to experience the MR space.

[0004] In an MR system using a video see-through type HMD, it is important to ensure the user's field of view and not give the user a sense of discomfort even when an error occurs in communication between the HMD and the image processing device. In particular, in a wireless MR system that employs a wireless method for image transmission, the communication state may deteriorate due to an increase in the distance between devices, shielding by obstacles, or interference with other wireless devices. Due to the deterioration of the communication state, a decrease in the received radio wave intensity or an increase in the occurrence frequency of errors may occur, and reception of the captured image and the display image may become impossible.

[0005] Patent Document 1 describes a technique of reducing the packet size and increasing the number of data transmissions when the radio wave condition deteriorates. By this, communication with high reliability and high throughput is realized.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2001-197144 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] However, in Patent Document 1, the packet size is reduced when radio wave conditions deteriorate, which increases the total amount of packet headers added to the actual data, and also increases the number of transmissions. Therefore, when attempting to transmit data for a single image, the total amount of data transmitted increases. Consequently, it is not suitable for streaming transmission where real-time performance is prioritized, such as image transmission in MR systems.

[0008] Therefore, the present invention aims to provide a technology that enables more appropriate data transmission even when communication conditions deteriorate. [Means for solving the problem]

[0009] One aspect of the present invention is, A transmitting device that transmits first data to a receiving device, A first generation means that generates packets for each of the multiple data obtained by dividing the first data, by adding a packet header to each of the multiple data, A first transmission means that transmits each packet generated by the first generation means to the receiving device, A first control means controls the packet size, which is the size of each packet, according to the communication status between the transmitting device and the receiving device. It has, The first control means sets the packet size to a first size if the communication state is a first state, and sets the packet size to a second size which is larger than the first size if the communication state is a second state which is worse than the first state. This is a transmitting device characterized by the following features. [Effects of the Invention]

[0010] According to the present invention, data can be transmitted more appropriately even when communication conditions deteriorate. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram showing the configuration of the MR system according to Embodiment 1. [Figure 2] This is a diagram illustrating the generation of a composite image according to Embodiment 1. [Figure 3] This is a functional configuration diagram of the HMD and image processing device according to Embodiment 1. [Figure 4] This is a diagram illustrating a UDP packet according to Embodiment 1. [Figure 5] This diagram illustrates the control of packet size according to Embodiment 1. [Figure 6] This is a flowchart of the packet size control process according to Embodiment 1. [Figure 7] This diagram illustrates the image data correction process according to Embodiment 1. [Figure 8] This diagram illustrates the image data correction process according to Embodiment 1. [Figure 9] This diagram illustrates the image data correction process according to Embodiment 1. [Figure 10] This diagram illustrates the image data correction process according to Embodiment 1. [Figure 11] This diagram illustrates the image data correction process according to Embodiment 1. [Figure 12] This is a functional configuration diagram of the HMD and image processing device according to Embodiment 2. [Figure 13] This is a flowchart of the packet size control process according to Embodiment 2. [Figure 14] This is a flowchart of the packet size control process according to Embodiment 2. [Figure 15]It is a hardware configuration diagram of the HMD and the image processing apparatus according to Embodiment 3.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0013] <Embodiment 1> Using FIG. 1, a configuration example of the MR system (processing system) according to Embodiment 1 will be described. As shown in FIG. 1, the wireless MR system includes an HMD 101, a control device 102, and a computer device 103.

[0014] The HMD 101 is an example of a head-mounted display device. The HMD 101 acquires a captured image of the real space. The HMD 101 measures (measurement processing) the position and orientation of the HMD 101. The HMD 101 has a display unit that displays a composite image (an image obtained by synthesizing a CG image of a virtual space generated by the computer device 103 and the captured image) representing the composite reality space.

[0015] In FIG. 1, the interface (interface for performing wireless communication) with the control device 102 is shown as a separate body from the HMD 101, but it may be configured integrally with the HMD 101. As the wireless communication method, a method for configuring a small-scale network such as WLAN (Wireless Local Area Network) or WPAN (Wireless Personal Area Network) is assumed.

[0016] ​The HMD101 transmits the captured image and / or information about its position and orientation to the control device 102. The HMD101 also receives a CG image (a CG image generated by the computer device 103 based on the captured image and / or position and orientation) or a composite image from the control device 102. Furthermore, the HMD101 displays a composite image created by combining the CG image and the captured image. As a result, a composite image representing a mixed reality space is presented in front of the eyes of the user wearing the HMD101 on their head.

[0017] In Figure 1, the HMD 101 and the control device 102 are connected wirelessly. However, the connection between the HMD 101 and the control device 102 is not limited to wireless; it may also be implemented via a wired connection, or a combination of wireless and wired connections. In other words, the connection configuration between the HMD 101 and the control device 102 is not limited to a specific configuration.

[0018] The HMD101 may operate using power supplied from its internal battery, or it may operate using power supplied externally via a power cable. In other words, there is no specific method by which power is supplied to the HMD101.

[0019] The control device 102 performs various image processing operations (resolution conversion, color space conversion, distortion correction of the optical system of the HMD 101's imaging unit, encoding, etc.) on the captured image transmitted from the HMD 101. The control device 102 then transmits the processed captured image and / or the position and orientation information transmitted from the HMD 101 to the computer device 103. The control device 102 also performs similar image processing on the CG image or composite image transmitted from the computer device 103, and then transmits the processed image to the HMD 101.

[0020] The computer device 103 determines the position and orientation of the HMD 101 (the position and orientation of the imaging unit of the HMD 101) based on the captured image and / or position and orientation information of the HMD 101 received from the control device 102. The computer device 103 generates a computer graphics (CG) image, which is an image of the virtual space as seen from the determined position and orientation. Furthermore, the computer device 103 generates a composite image representing the mixed reality space by combining the CG image of the virtual space with the captured image. Then, the computer device 103 transmits the generated CG image or composite image to the control device 102.

[0021] Here, we will explain the process of generating a composite image using Figure 2. The captured image 201 contains markers 202 that are artificially placed in real space. In Figure 2, the number of markers is set to 1 for simplicity of explanation, but in reality, the captured image 201 may contain multiple markers. Furthermore, SLAM (Simultaneous Localization and Mapping) using natural feature points in the image is also used. When estimating position and orientation using localization and mapping, it is not necessary to place markers. The computer device 103 extracts markers 202 or natural feature points contained in the captured image 201. Based on the extracted markers 202 or natural feature points and / or position and orientation information received from the control device 102, the computer device 103 determines the position and orientation of the HMD 101. The computer device 103 then generates a CG image 203, which is an image of the virtual space viewed from the determined position and orientation. The CG image 203 includes a virtual object 204.

[0022] Then, the HMD 101 or computer device 103 generates a composite image 205 representing the mixed reality space by combining the captured image 201 and the CG image 203. 101 displays the composite image 205. When compositing the captured image and the CG image, information regarding depth in three-dimensional space or information regarding the transparency of the virtual object may be used. If such information is used, it is possible to generate a composite image that considers the front-to-back relationship between real and virtual objects, or a composite image in which the virtual object is represented in a semi-transparent state.

[0023] In Figure 1, the computer device 103 and the control device 102 are shown as separate devices, but the computer device 103 and the control device 102 may be integrated. Embodiment 1 describes a configuration in which the computer device 103 and the control device 102 are integrated. Hereafter, the device in which the computer device 103 and the control device 102 are integrated will be referred to as the image processing device 104.

[0024] Next, using the block diagram in Figure 3, we will explain examples of the functional configurations of the receiving device HMD101 and the transmitting device image processing device 104. In Figure 3, the HMD101 combines the captured image and the CG image. The HMD101 includes an imaging unit 301, an attitude sensor 302, a display unit 303, a processing unit 304, an interface (I / F) 305, a detection unit 306, a correction unit 307, a synthesis unit 308, and a processing unit 309.

[0025] The imaging unit 301 acquires images by capturing images that are used for compositing with CG images of the virtual space. The imaging unit 301 has an imaging unit for the left eye and an imaging unit for the right eye. The imaging unit for the left eye captures the real space corresponding to the left eye of the user of the HMD 101. The left eye imaging unit outputs an image (captured image) for each frame in the moving image. The imaging unit for the right eye captures the real space corresponding to the right eye of the user. The right eye imaging unit outputs an image (captured image) for each frame in the moving image. In other words, the imaging unit 301 acquires captured images as stereo images with parallax that approximately coincides with the positions of the left and right eyes of the user of the HMD 101. In the case of an HMD for an MR system, it is preferable that the central optical axis of the imaging range of the imaging unit be positioned so as to approximately coincide with the user's line of sight.

[0026] Each of the imaging units, one for the left eye and the other for the right eye, comprises an optical system and an imaging device. Light incident from the outside world enters the imaging device via the optical system. The imaging device outputs an image corresponding to the incident light as an captured image.

[0027] The attitude sensor 302 measures various data necessary to determine the position and orientation of the HMD 101. Based on the measured data, the attitude sensor 302 detects the orientation of the HMD 101 and outputs orientation information. The attitude sensor 302 is implemented using magnetic sensors (including geomagnetic sensors), ultrasonic sensors, acceleration sensors, angular velocity sensors, etc.

[0028] The display unit 303 has a display unit for the right eye and a display unit for the left eye. The image for the left eye, representing the mixed reality space, is displayed on the left eye display unit. The image for the right eye, representing the mixed reality space, is displayed on the right eye display unit. Both the left eye display unit and the right eye display unit have a display optical system and a display element. The display optical system may be an eccentric optical system such as a free-form prism, or a normal coaxial optical system or an optical system with a zoom mechanism. For the display element, for example, a small liquid crystal display, an organic EL display, or a retinal scanning type device using MEMS (Micro Electro Mechanical Systems) can be used. Light from the image displayed on the display element enters the user's eye of the HMD 101 via the display optical system.

[0029] The processing unit 304 performs various image processing operations on the captured image acquired by the imaging unit 301.

[0030] Interface 305 receives attitude information output from attitude sensor 302 and processes it in an image processing device. It transmits to 104. Interface 305 receives CG images from image processing device 104 to be used for generating "composite images to be displayed on display unit 303". Interface 305 is a transmitting and receiving unit that transmits and receives other control signals, including setting information for each device.

[0031] The detection unit 306 detects the "communication status between the HMD 101 and the image processing device 104 (communication status from the image processing device 104 to the HMD 101)." The communication status is based on various measured values, such as the received radio wave strength (strength of the received radio waves), throughput, frame rate, and bit error rate (BER), or a combination of multiple measured values, during data transmission. Specifically, for example, the lower the received radio wave strength, throughput, or frame rate, the worse the communication status. For example, the higher the bit error rate of the data received by the HMD 101, the worse the communication status. The detection unit 306 also detects error portions contained in the CG image data received from the image processing device 104.

[0032] The correction unit 307 performs correction processing on the error portion of the CG image data based on the error portion detection result by the detection unit 306.

[0033] The synthesis unit 308 generates a composite image by combining the captured image acquired by the imaging unit 301 with the CG image output from the correction unit 307. The synthesis unit 308 may perform chroma keying and alpha blending, among other things. Furthermore, by utilizing depth information, more advanced synthesis processing may be performed that takes into account the front-to-back relationship of objects in the captured image and the CG image.

[0034] The processing unit 309 performs various image processing operations on the composite image generated by the synthesis unit 308. Image processing operations include, for example, correcting individual variations in the display devices or display optical systems that constitute the display unit 303. Specifically, image processing operations include offset and gain adjustment operations, pixel defect correction, or distortion correction operations for the display optical system.

[0035] Next, the image processing device 104 will be described. The image processing device 104 includes an interface (I / F) 310, an image generation unit 311, a content DB (database) 312, a drawing unit 313, and a control unit 314.

[0036] Interface 310 receives attitude information output from attitude sensor 302 from HMD 101. Interface 310 transmits CG images to HMD 101. Interface 310 is a transmitting and receiving unit that transmits and receives other control signals, including setting information for each device.

[0037] The image generation unit 311 determines the position and orientation of the left eye imaging unit and the right eye imaging unit based on the orientation information received from the HMD 101. Since the "process for determining the position and orientation of the imaging unit based on the position and orientation measured by the sensor" is well known, a description of this technology will be omitted.

[0038] Content DB312 stores various data necessary for rendering images of the virtual space (hereinafter referred to as "virtual space data"). The virtual space data includes, for example, data that defines each virtual object that makes up the virtual space (for example, data that defines the geometric shape, color, texture, or position and orientation of the virtual object). The virtual space data also includes, for example, data that defines the light sources placed in the virtual space (for example, data that defines the type or position and orientation of the light sources).

[0039] The drawing unit 313 generates (constructs) a CG image using the virtual space data stored in the content DB 312. Here, the drawing unit 313 uses the left eye data obtained by the image generation unit 311. The drawing unit 313 draws a CG image L representing the virtual space as seen from the position and orientation of the imaging unit. The drawing unit 313 also draws a CG image R representing the virtual space as seen from the position and orientation of the imaging unit for the right eye, as determined by the image generation unit 311.

[0040] Based on the detection result of the communication status between the HMD 101 and the image processing device 104, the control unit 314 controls the packet size (size of each packet) when transmitting the CG image drawn by the drawing unit 313. The control unit 314 transmits data (packets) according to the packet size to the HMD 101 via the interface 310.

[0041] In addition to the components mentioned above, the MR system may also include, for example, a control unit for configuring each device in either the HMD 101 or the image processing device 104, or an input unit for receiving configuration information from an external source.

[0042] Next, the packet configuration during image transmission according to Embodiment 1 will be explained using Figure 4. Figure 4 is a diagram illustrating the packet configuration of UDP (User Datagram Protocol), a type of communication protocol that is commonly used in image transmission where real-time performance is important. Although the explanation will focus on the case using UDP, TCP (Transmission Control Protocol) may be used instead.

[0043] A UDP packet consists of a header and a payload, which is the actual data being transmitted. The header includes the source port number (the port number indicating the data source), the destination port number (the port number indicating the destination), the packet length, and a checksum. Regardless of the payload size, the header size is fixed at 8 bytes. For example, when transmitting 50 megabytes of image data, if the payload size is 32 bytes, the total amount of data transmitted, including the header, is 50M × (40 / 32) = 62.5 MB. If the payload size is changed to 128 bytes, the total amount of data transmitted, including the header, becomes 50M × (136 / 128) = 53.125 MB, which is a 15% reduction in data volume compared to when the payload size is 32 bytes.

[0044] Next, with reference to Figures 5A to 5D, the control of packet size according to radio wave conditions in Embodiment 1 will be explained.

[0045] Figure 5A is a graph showing the received radio wave intensity (=communication state) measured by the detection unit 306 of the HMD101 on the vertical axis and the passage of time on the horizontal axis. Figure 5B shows image data 503 with a packet size of size S1 when the moving average 502 of the received radio wave intensity 501 is equal to or greater than intensity P1. Figure 5C shows image data 504 with a packet size of size S2 (S2>S1) when the moving average 502 is less than intensity P1 and equal to or greater than intensity P2. Figure 5D shows image data 505 with a packet size of size S3 (S3>S2) when the moving average 502 is less than intensity P2. In this way, when the communication state deteriorates, by increasing the packet size and transmitting the data, the total amount of transmitted data, including the header, can be reduced even when sending image data of the same size, as explained using Figure 4. Note that the packet size may be determined to be inversely proportional to the moving average 502.

[0046] Figures 5A to 5D show control that increases the packet size when the received signal strength decreases. However, it is also possible to similarly increase the packet size when throughput decreases, frame rate decreases, or bit error rate increases. Furthermore, if it is possible to control the transmitted signal strength at the interface 310 of the image processing device 104 according to detection results such as received signal strength, then control The unit 314 may control the packet size according to the transmitted radio wave strength.

[0047] Referring to the flowchart in Figure 6, the processing flow for packet size control in the MR system according to Embodiment 1 will be explained. Figure 6 illustrates packet size control using received radio wave strength.

[0048] In step S601, the detection unit 306 of the HMD 101 measures the received radio wave strength in order to detect the communication status between the HMD 101 and the image processing device 104.

[0049] In step 602, the detection unit 306 transmits the measured received radio wave intensity information to the image processing device 104 via the interface 305.

[0050] In step 603, the control unit 314 of the image processing device 104 receives information on the received radio wave strength measured by the detection unit 306 of the HMD 101 via the interface 310. In Embodiment 1, the HMD 101 and the image processing device 104 can send and receive data from each other. For this reason, instead of the radio wave strength measured by the detection unit 306 of the HMD 101 (= received radio wave strength), the radio wave strength received by the image processing device 104 from the HMD 101 may be used. This is because the radio wave strength received by the HMD 101 from the image processing device 104 and the radio wave strength received by the image processing device 104 from the HMD 101 are correlated with each other.

[0051] In step 604, the control unit 314 calculates a moving average of the received signal strength. The moving average of the received signal strength indicates the communication status between the HMD 101 and the image processing device 104 (specifically, the communication status from the image processing device 104 to the HMD 101). Here, the control unit 314 performs the calculation of the moving average, but the HMD 101 (for example, the detection unit 306) may perform the calculation of the moving average and then transmit the calculation result to the image processing device 104.

[0052] In step 605, the control unit 314 determines whether the calculated moving average of the received radio wave intensity is equal to or greater than intensity P1. If it is determined that the moving average of the received radio wave intensity is equal to or greater than intensity P1, the "communication state between HMD 101 and image processing device 104 (the state of communication from image processing device 104 to HMD 101)" is determined to be better than the first state, and the process proceeds to step S606. On the other hand, if it is determined that the moving average of the received radio wave intensity is less than intensity P1, the "communication state between HMD 101 and image processing device 104" is determined to be worse than the first state, and the process proceeds to step S607.

[0053] In step S606, the control unit 314 sets the packet size of the image data to size S1.

[0054] In step S607, the control unit 314 determines whether the calculated moving average of the received signal strength is greater than or equal to strength P2 (less than strength P1 and greater than or equal to strength P2). If it is determined that the moving average of the received signal strength is greater than or equal to strength P2, the communication state between the HMD 101 and the image processing device 104 is determined to be better than the second state, and the process proceeds to step S608. On the other hand, if it is determined that the moving average of the received signal strength is less than strength P2, the communication state between the HMD 101 and the image processing device 104 is determined to be worse than the second state, and the process proceeds to step S609. The second state is a worse communication state than the first state.

[0055] In step S608, the control unit 314 sets the packet size of the image data to size S2. Size S2 is larger than size S1.

[0056] In step S609, the control unit 314 sets the packet size of the image data to size S3. Size S3 is larger than size S2.

[0057] In step S610, the control unit 314 divides the image data into multiple data files according to the set packet size. The control unit 314, acting as a packet generation unit, generates a packet by adding a packet header to each of the divided data files.

[0058] In step S611, the control unit 314 transmits each generated packet to the HMD 101 via the interface 310 of the image processing device 104.

[0059] In step S612, the interface 305 of the HMD 101 receives each packet generated by the control unit 314 of the image processing device 104.

[0060] Next, Figures 7 to 11 will be used to explain the image data correction process according to the packet size.

[0061] Figure 7 illustrates a case where the image of the frame immediately preceding the frame in which an error occurred (detected) (hereinafter referred to as the "error image") is used as the image used to correct the error image (hereinafter referred to as the "correction image").

[0062] In the left-eye image stream 701L, which consists of images from each frame for the left eye, if an error image 702 occurs due to a transmission error, the image of the frame preceding the error image 702 in the same left-eye image stream 701L is used as a correction image 703. This corrects the error image 702 based on the correction image 703. All or part of the correction image 703 may be used in place of the entirety of the error image 702 or a portion of the error area. Alternatively, the correction image 703 may be shifted and / or deformed based on the change in displacement measured by the attitude sensor 302 before being used in place of the error image 702.

[0063] Such correction processing is used, for example, when the packet size of the image data is larger than a predetermined size, and the amount of displacement (displacement between frames) of the posture measurement result by the posture sensor 302 is smaller than a predetermined amount. In this case, at least a portion of the error image 702, including the erroneous part, is completely replaced by the area of ​​the correction image 703, so the erroneous part is not included in the replaced image. Also, since the change in the posture of the HMD 101 is small, the parallax error as a stereo image is also small. As a result, the stereo effect is not impaired, and it is possible to provide the user with a display image that is less jarring. In particular, when there is no movement from the image of the previous frame, an image with less jarring can be displayed. Here, we have shown the case in which an error occurs in the left eye image stream 701L, but when an error image occurs in the right eye image stream 701R, correction processing can be performed using the image of the previous frame in the same right eye image stream 701R.

[0064] Figure 8 illustrates the case where the current frame image from the image stream on the eye side opposite to the error image is used as the correction image.

[0065] In the left eye image stream 801L, when an error image 802 occurs due to a transmission error, the current frame image in the right eye image stream 801R, which is opposite to the error image 802, is used as the correction image 803 to correct the error image 802. The error image 802 and the correction image 803 are images that were captured at the same time, although they have parallax. All or part of the correction image 803 corrects all or part of the error in the error image 802. It is used in place of a portion of the area. Alternatively, the entire correction image 803 or a portion of the area including the error may be shifted and / or transformed based on the parallax information of the left-eye display unit and the right-eye display unit, and then replaced.

[0066] Such correction processing is used, for example, when the packet size of the image data is larger than a predetermined size, and the amount of displacement (displacement between frames) of the posture measurement result by the posture sensor 302 is larger than a predetermined amount. In this case, at least a portion of the area including the error is completely replaced with the area of ​​the correction image, so the error is not included in the image. Also, because the change in the posture of the HMD 101 is large, the amount of correction is less than if the correction processing were performed from the image of the previous frame of the left eye image stream. Therefore, it is possible to provide the user with a display image that is less unnatural. Here, we have shown the case in which an error occurs in the left eye image stream 801L, but if an error image occurs in the right eye image stream 801R, the same correction processing can be performed using the current frame image in the opposite left eye image stream 801L.

[0067] The correction unit 307 may also select a correction image from "the image of the frame before the error image" and "the image of the current frame of the image stream on the eye side opposite to the error image" based on at least one of the packet size and the amount of displacement of the HMD 101's posture.

[0068] Figures 9A to 9C illustrate three types of correction processes for error images using correction images.

[0069] Figure 9A shows a correction process in which the entire error image 902 is replaced with a correction image 901. Such a correction process is used, for example, when the packet size of the image data is larger than a certain size, and the error portions 903a to 903d occur over a wide and sparse area of ​​the error image 902. In this case, replacing the entire error image with a correction image prevents the overall uniformity of the corrected image from being compromised, making it possible to provide the user with a display image that is less jarring.

[0070] Figure 9B shows a correction process in which only the erroneous portion 906 in the error image 905 is replaced with the area corresponding to the erroneous portion 906 in the correction image 904. Similarly, Figure 9C shows a correction process in which only the erroneous portion 909 in the error image 908 is replaced with the area corresponding to the erroneous portion 909 in the correction image 907. Such correction processes shown in Figures 9B and 9C are used, for example, when the packet size of the image data is larger than a certain size, but the erroneous portion occurs only in a part of the error image. In this case, since the area replaced by the correction image is small in the entire error image, an image close to the original image can be obtained. Therefore, it is possible to display an image that is less jarring to the user.

[0071] Furthermore, the error image correction process that can be used in Embodiment 1 is not limited to the method described above. Examples of other correction processes are described below with reference to Figures 10 and 11.

[0072] Figure 10 illustrates the concept of a correction process that reconstructs an error pixel (hereinafter referred to as an "error pixel") from surrounding pixels.

[0073] In this correction process, the error pixel 1003 contained in the error portion 1002 of the error image 1001 is corrected using the surrounding pixel group 1004 of the error pixel 1003. One such correction method is to use the average of the pixel values ​​of the eight pixels surrounding the error pixel 1003 as the pixel value of the error pixel 1003. Here, only two adjacent pixels in either the vertical or horizontal direction are used as correction pixels, and the average pixel value of the correction pixels is used to correct the error pixel 1003. By using the corrected / corrected pixel values, it is possible to reduce the burden of correction (interpolation) processing.

[0074] Figure 10 illustrates the case where there is one error pixel, but even when multiple adjacent pixels are affected by errors, the surrounding pixels of those error pixels can be used as correction pixels to perform the correction process. For example, one possible method is to weight the pixel value of each correction pixel according to the distance between the error pixel to be corrected and each correction pixel, and use the average of these weights as the corrected pixel value of the error pixel. In Embodiment 1, in addition to the correction method described above, it is also possible to apply a method of interpolating error pixels from surrounding pixels.

[0075] Figure 11 shows a correction process that uses the same data as the lines above and below the error portion. In this correction process, the line one level above line 1102, which contains the error portion in the error image 1101, is used as the correction line 1103 to perform the correction. Here, the data of line 1102, which contains the error portion, is simply replaced with the data of the correction line 1103, which is one level above it. That is, line 1102 and the correction line 1103 after the correction process have the same data. It is also possible to use the line one level below line 1102, which is adjacent to line 1102, as the correction line 1103, instead of the line one level above line 1102, which contains the error portion. Alternatively, the data of line 1102 may be replaced with the average of the correction lines, which are the lines above and below line 1102, which contains the error portion.

[0076] The correction process described using Figures 10 and 11 is used, for example, when the packet size of the image data is smaller than a certain size. In this case, the original image data can be used for all areas except for the error pixels. Furthermore, because the correction processing range is very narrow, the processing load is low, and any errors in the processing result are not very noticeable. Therefore, it is possible to provide users with display images that are less jarring.

[0077] Therefore, the correction unit 307 may select a correction process to actually execute from among the five types of correction processes (correction methods) shown in Figures 9A to 9C, Figure 10, and Figure 11, based on the packet size and the arrangement (distribution) of the error portion. For example, if the packet size is size S1, the correction unit 307 selects the correction process shown in Figure 11 as the correction process to actually execute. For example, if the packet size is size S2, the correction unit 307 selects the correction process shown in Figure 10 as the correction process to actually execute. For example, if the packet size is size S3 and the distribution of the error portion is higher than a predetermined value, the correction unit 307 selects the correction process shown in Figure 9A as the correction process to actually execute. For example, if the packet size is size S3 and the distribution of the error portion is lower than a predetermined value, the correction unit 307 selects the correction process shown in Figure 9B or Figure 9C as the correction process to actually execute.

[0078] Thus, in Embodiment 1, packet sizes may be classified into three stages, and the error image correction method may differ depending on the packet size. However, packet sizes may be further subdivided. It is also possible to control the system to perform a different correction method depending on each packet size than described above.

[0079] According to Embodiment 1, when radio wave conditions deteriorate, the image processing device 104 increases the packet size of the image data for transmission. This reduces the total amount of transmitted data, including the header, even when sending image data of the same size. Other methods for reducing the amount of transmitted data include reducing the resolution or frame rate, or reducing color information. Additionally, when performing image compression using a codec, methods such as increasing the image compression ratio can be used. Compared to these methods, the method of Embodiment 1 has the advantage of not significantly degrading the quality of the image data itself.

[0080] Furthermore, the method for correcting errors that occur (or are detected) in the image data is switched based on the packet size of the image data. This makes it possible to provide users with a display image that is less jarring even when errors occur in the image data due to transmission errors.

[0081] <Embodiment 2> In subsequent embodiments, including Embodiment 2, the differences from Embodiment 1 will be described, and unless otherwise specified below, they will be the same as Embodiment 1. Embodiment 1 described a configuration in which one image stream is transmitted between the HMD 101 and the image processing device 104. Embodiment 2 describes a configuration in which multiple image streams are transmitted between the HMD 101 and the image processing device 104.

[0082] Examples of the functional configurations of the HMD 101 and the image processing device 104 according to Embodiment 2 will be explained using the block diagram in Figure 12. Figure 12 shows an example configuration when the image processing device 104 combines an captured image and a CG image.

[0083] First, let's describe the HMD101. In addition to the configuration shown in Figure 3, the HMD101 also includes an imaging unit 1201, a processing unit 1202, and a control unit 1203.

[0084] The imaging unit 1201 has multiple imaging units for capturing images of real space used for alignment. The imaging unit 1201 acquires captured images as stereo images with parallax. Each imaging unit captures a moving image of real space and outputs an image (captured image) of each frame in the moving image. Each imaging unit of the imaging unit 1201 has an optical system and an imaging device. Light incident from the outside world enters the imaging device through the optical system. The imaging device outputs an image corresponding to the incident light as a captured image.

[0085] The imaging units 301 and 1201 may use either a rolling shutter image sensor or a global shutter image sensor, taking into consideration various factors such as pixel count, image quality, noise, sensor size, power consumption, and cost. Depending on the application, the imaging units 301 and 1201 can use a combination of rolling shutter and global shutter image sensors. For example, the imaging unit 301, which acquires an image to be composited with an image in a virtual space, uses a rolling shutter image sensor that can acquire higher quality images. The imaging unit 1201, which acquires an image used for alignment, uses a global shutter image sensor that does not produce image blur. Image blur is a phenomenon that occurs due to the operating principle of the rolling shutter system, in which exposure processing is started sequentially for each line in the scanning direction. Specifically, image blur is known as a phenomenon in which, due to a time lag in the exposure timing of each line, the subject is deformed and recorded as blurred when the imaging unit or the subject moves during the exposure time. In a global shutter system, exposure processing is performed simultaneously on all lines, so there is no time lag in the exposure timing of each line, and no image blurring occurs.

[0086] The processing unit 1202 performs various image processing operations on the captured image acquired by the imaging unit 1201.

[0087] Based on the detection result of the "communication status between the HMD 101 and the image processing device 104," the control unit 1203 controls the packet size when transmitting the captured images acquired by the imaging unit 301 and the imaging unit 1201. Based on the control of the packet size, the control unit 1203 transmits the captured images to the image processing device 104 via the interface 305.

[0088] Next, the image processing device 104 will be described. In addition to the configuration shown in Figure 3, the image processing device 104 has a detection unit 1204, a correction unit 1205, a processing unit 1206, and a synthesis unit 1207. .

[0089] Interface 310 receives captured images acquired by the imaging unit 301, captured images acquired by the imaging unit 1201, and attitude information output from the attitude sensor 302 from the HMD 101. Interface 310 transmits a composite image to the HMD 101 for display on the display unit 303. Interface 310 also transmits and receives other control signals, including setting information for each device.

[0090] The detection unit 1204 detects the "communication status between the HMD 101 and the image processing device 104." The communication status is based on various measured values ​​such as received signal strength, throughput, frame rate, and bit error rate (BER), or a combination of multiple measured values. The detection unit 1204 also detects error portions contained in the data of the captured image received from the HMD 101.

[0091] The correction unit 1205 performs error correction processing on the error portion of the captured image data based on the detection result of the error portion.

[0092] The processing unit 1206 extracts (recognizes) markers or natural feature points from the image captured for the left eye and the image captured for the right eye.

[0093] The compositing unit 1207 generates a composite image by combining the captured image acquired by the imaging unit 301 with the CG image output from the rendering unit 313. The compositing unit 1207 may perform chroma keying and alpha blending, among other things. Furthermore, by utilizing depth information, more advanced compositing processing may be performed that takes into account the front-to-back relationship of objects in the captured image and the CG image.

[0094] Based on the detection result of the communication status between the HMD 101 and the image processing device 104, the control unit 314 controls the packet size when transmitting the composite image generated by the synthesis unit 1207. Based on the controlled packet size, the control unit 314 transmits the composite image to the HMD 101 via the interface 310.

[0095] In addition to the components mentioned above, the MR system may also include, for example, a control unit for configuring each device in either the HMD 101 or the image processing device 104, or an input unit for receiving configuration information from an external source.

[0096] Next, the control of packet size in the MR system according to Embodiment 2 will be described with reference to the flowcharts in Figures 13 and 14. Figure 13 shows the flow of control for the packet size of the captured image data generated by the control unit 1203 of the HMD 101.

[0097] In step S1301, the detection unit 1204 of the image processing device 104 measures the received radio wave strength to detect the "communication state between the HMD 101 and the image processing device 104".

[0098] In step 1302, the detection unit 1204 transmits the measured received radio wave strength information to the HMD 101 via the interface 310.

[0099] In step 1303, the control unit 1203 of the HMD 101 receives information on the measured received radio wave strength via the interface 305.

[0100] In step 1304, the control unit 1203 calculates a moving average of the received radio wave intensity. The moving average of the received radio wave intensity is calculated based on the "communication status between the HMD 101 and the image processing device 104 (specifically)." This shows the communication status from HMD101 to image processing device 104. Here, the control unit 1203 of HMD101 performs the moving average calculation process, but it is also possible for the image processing device 104 (for example, the detection unit 1204) to perform the moving average calculation process and then send the calculation result to HMD101.

[0101] In step 1305, the control unit 1203 determines whether the calculated moving average of the received radio wave intensity is equal to or greater than intensity P1. If it is determined that the moving average of the received radio wave intensity is equal to or greater than intensity P1, the "communication state between HMD 101 and image processing device 104 (the state of communication from HMD 101 to image processing device 104)" is determined to be better than the first state, and the process proceeds to step S1306. On the other hand, if it is determined that the moving average of the received radio wave intensity is less than intensity P1, the "communication state between HMD 101 and image processing device 104" is determined to be worse than the first state, and the process proceeds to step S1307.

[0102] In step S1306, the control unit 1203 sets the packet size (imaging packet size) of the captured image data (hereinafter referred to as "imaging image data") to size S1.

[0103] In step S1307, the control unit 1203 determines whether the calculated moving average of the received signal strength is equal to or greater than signal strength P2. If it is determined that the moving average of the received signal strength is equal to or greater than signal strength P2, the "communication state between HMD 101 and image processing device 104" is determined to be better than the second state, and the process proceeds to step S1308. On the other hand, if it is determined that the moving average of the received signal strength is less than signal strength P2, the "communication state between HMD 101 and image processing device 104" is determined to be worse than the second state, and the process proceeds to step S1309. The second state is a worse communication state than the first state.

[0104] In step S1308, the control unit 1203 sets the packet size of the captured image data to size S2.

[0105] In step S1309, the control unit 1203 sets the packet size of the captured image data to size S3.

[0106] In step S1310, the control unit 1203 divides the captured image data according to the set packet size to generate multiple data. The control unit 1203 then generates a packet (imaging packet) by adding a packet header to each of the divided data.

[0107] In step S1311, the control unit 1203 transmits each packet of captured image data to the image processing device 104 via the interface 305 of the HMD 101.

[0108] In step S1312, the interface 310 of the image processing device 104 receives each packet of captured image data.

[0109] Figure 14 shows the control flow of the packet size of the composite image (display image) data (hereinafter referred to as "composite image data") in the control unit 314 of the image processing device 104.

[0110] In step S1401, the detection unit 306 of the HMD 101 measures the received radio wave strength in order to detect the "communication status between the HMD 101 and the image processing device 104".

[0111] In step S1402, the detection unit 306 transmits the measured received radio wave intensity to the image processing device 104 via the interface 305.

[0112] In step S1403, the control unit 314 of the image processing device 104 controls interface 3 Information on the received radio wave strength is received via 10.

[0113] In step S1404, the control unit 314 calculates a moving average of the received radio wave intensity. The moving average of the received radio wave intensity indicates the "communication status between the HMD 101 and the image processing device 104 (specifically, the communication status from the image processing device 104 to the HMD 101)." Here, the control unit 314 of the image processing device 104 performs the calculation of the moving average, but it is also possible for the HMD 101 (for example, the detection unit 306) to perform the calculation of the moving average and then transmit the calculation result to the image processing device 104.

[0114] In step S1405, the control unit 314 determines whether the calculated moving average of the received radio wave intensity is equal to or greater than intensity P1. If it is determined that the moving average of the received radio wave intensity is equal to or greater than intensity P1, the "communication state between HMD 101 and image processing device 104 (the state of communication from image processing device 104 to HMD 101)" is determined to be better than the first state, and the process proceeds to step S1406. On the other hand, if it is determined that the moving average of the received radio wave intensity is less than intensity P1, the "communication state between HMD 101 and image processing device 104" is determined to be worse than the second state, and the process proceeds to step S1407.

[0115] In step S1406, the control unit 314 sets the packet size of the composite image data (composite packet size) to size S1.

[0116] In step S1407, the control unit 314 determines whether the calculated moving average of the received radio wave intensity is equal to or greater than intensity P2. If it is determined that the moving average of the received radio wave intensity is equal to or greater than intensity P2, the "communication state between HMD 101 and image processing device 104" is determined to be better than the second state, and the process proceeds to step S1408. On the other hand, if it is determined that the moving average of the received radio wave intensity is less than intensity P2, the "communication state between HMD 101 and image processing device 104" is determined to be worse than the second state, and the process proceeds to step S1410.

[0117] In step S1408, the control unit 314 determines whether the packet size of the captured image data is less than or equal to size S2. If it is determined that the packet size of the captured image data is less than or equal to size S2, the process proceeds to step S1409. On the other hand, if it is determined that the packet size of the captured image data is greater than size S2, the process proceeds to step S1406.

[0118] In step S1409, the control unit 314 sets the packet size of the composite image data (composite packet size) to size S2.

[0119] In step S1410, the control unit 314 determines whether the packet size of the captured image data is size S3. If it is determined that the packet size of the captured image data is size S3, the process proceeds to step S1411. On the other hand, if it is determined that the packet size of the captured image data is not size S3, the process proceeds to step S1409.

[0120] According to steps S1408 and S1410, the control unit 314 controls the packet size of the composite image data to be as small as possible based on the packet size of the captured image data. This allows the control unit 314 to transmit the composite image data with higher reliability.

[0121] In step S1411, the control unit 314 sets the packet size of the composite image data to size S3.

[0122] In step S1412, the control unit 314 sets the packet size of the configured composite image data. The composite image data is divided into multiple data sets according to the sequence. The control unit 314 generates packets by adding a packet header to each of the divided data sets.

[0123] In step S1413, the control unit 314 transmits each packet of the composite image data to the HMD 101 via the interface 310 of the image processing device 104.

[0124] In step S1414, the interface 305 of the HMD101 receives each packet of the composite image data.

[0125] The correction processing of image data according to the packet size in the correction unit 307 and correction unit 1205 according to Embodiment 2 is the same as in Embodiment 1, so a description will be omitted.

[0126] As described above, in the MR system of Embodiment 2, the control of packet size when radio wave conditions deteriorate can be made differently for multiple image streams. Figures 13 and 14 describe a method for controlling the packet size of captured image data before that of composite image data, but the reverse is also possible. Furthermore, it is possible to control the packet size differently between the captured image used for compositing with CG images and the captured image used for alignment. In addition, the intensity P1, intensity P2, size S1, size S2, and size S3 do not have to be common values ​​in the flowcharts shown in Figures 13 and 14. The intensity P1, intensity P2, size S1, size S2, and size S3 may be different for each image data transmitted.

[0127] In this way, by controlling packet sizes differently across multiple image streams when radio wave conditions deteriorate, the amount of transmitted data can be reduced more flexibly throughout the entire system. Therefore, it is possible to provide users with display images that are less jarring even when radio wave conditions deteriorate.

[0128] <Embodiment 3> Each functional unit in the HMD101 and image processing device 104 shown in Figures 3 and 12 may be implemented in hardware, or some of the functional units may be implemented in software (computer program).

[0129] If some functional units are implemented in software, the HMD101 may have the imaging unit 301, imaging unit 1201, attitude sensor 302, display unit 303, and interface 305 implemented in hardware, while the remaining functional units are implemented in software. In this case, the software is stored in the memory of the HMD101. The processor of the HMD101 executes the software to realize the functions of the corresponding functional units.

[0130] An example of the hardware configuration of the HMD101 will be explained using the block diagram shown in Figure 15A. The HMD101 includes a processor 1510, RAM 1520, non-volatile memory 1530, imaging unit 1540, attitude sensor 1550, display unit 1560, interface 1570, and bus 1580.

[0131] The processor 1510 performs various processes using computer programs and data stored in the RAM 1520. In this way, the processor 1510 controls the overall operation of the HMD 101 and executes or controls the aforementioned processes performed by the HMD 101.

[0132] RAM1520 is a computer program loaded from non-volatile memory 1530. The RAM 1520 also has an area for storing data. The RAM 1520 has an area for storing data received from the image processing device 104 via the interface 1570. Furthermore, the RAM 1520 has a work area used by the processor 1510 when performing various processes. In this way, the RAM 1520 can provide various areas as appropriate.

[0133] The non-volatile memory 1530 stores computer programs and data that cause the processor 1510 to execute or control the operation of the HMD 101 described above. The computer programs include computer programs that cause the CPU 1501 to execute the functions of the functional parts of the HMD 101 (excluding the imaging unit 301 and imaging unit 1201, attitude sensor 302, display unit 303, and interface 305). The computer programs and data stored in the non-volatile memory 1530 are loaded into the RAM 1520 as appropriate according to the control of the processor 1510 and used for processing by the processor 1510.

[0134] The imaging unit 1540 includes the imaging units 301 and 1201 described above. The attitude sensor 1550 includes the attitude sensor 302 described above. The display unit 1560 includes the display unit 303 described above. The interface 1570 includes the interface 305 described above. The processor 1510, RAM 1520, non-volatile memory 1530, imaging unit 1540, attitude sensor 1550, display unit 1560, and interface 1570 are all connected to the bus 1580. Note that the configuration shown in Figure 15A is an example of a configuration applicable to the HMD 101 and can be modified as appropriate.

[0135] Furthermore, the image processing device 104 can be any computer device capable of executing software corresponding to each functional unit (excluding the interface 310 and content DB 312). An example of a hardware configuration of a computer device applicable to the image processing device 104 will be explained using the block diagram in Figure 15B. The image processing device 104 has a CPU 1501, RAM 1502, ROM 1503, operation unit 1504, display unit 1505, external storage device 1506, interface (I / F) 1507, and bus 1508.

[0136] The CPU 1501 executes various processes using computer programs and data stored in the RAM 1502 or ROM 1503. In this way, the CPU 1501 controls the overall operation of the image processing device 104 and executes or controls the aforementioned processes performed by the image processing device 104.

[0137] RAM 1502 has an area for storing computer programs or data loaded from ROM 1503 or external storage device 1506. RAM 1502 also has an area for storing data received from HMD 101 via interface 1507. Furthermore, RAM 1502 has a work area used by CPU 1501 when executing various processes. In this way, RAM 1502 can provide various areas as appropriate. ROM 1503 stores setting data and startup programs for the image processing device 104, etc.

[0138] The control unit 1504 is a user interface such as a keyboard, mouse, or touch panel. The control unit 1504 inputs various instructions to the CPU 1501 in response to user operations.

[0139] The display unit 1505 includes an LCD screen or a touch panel screen. The display unit 1505 displays the processing results from the CPU 1501 as images or characters. The display unit 1505 may also be a projection device such as a projector that projects images or characters.

[0140] The external storage device 1506 is a large-capacity information storage device such as a hard disk drive. The OS (operating system) is stored in the external storage device 1506. The external storage device 1506 also stores computer programs and data for the CPU 1501 to execute the functions of each functional unit of the image processing device 104 shown in Figure 3 (excluding the interface 310 and content DB 312). The content DB 312 is also included in the external storage device 1506.

[0141] Computer programs and data stored in the external storage device 1506 are loaded into the RAM 1502 as appropriate, according to the control of the CPU 1501, and used for processing by the CPU 1501.

[0142] Interface 1507 is a communication interface for data communication with HMD 101. Interface 1507 functions as the interface 310 described above. In other words, the image processing device 104 performs data communication with HMD 101 via interface 1507.

[0143] The CPU 1501, RAM 1502, ROM 1503, operation unit 1504, display unit 1505, external storage device 1506, and interface 1507 are all connected to the bus 1508. Note that the configuration shown in Figure 15B is just one example of a configuration applicable to the image processing device 104, and can be modified or altered as appropriate.

[0144] <Embodiment 4> The configuration of the MR system shown in Figures 3 and 12 is just one example. For example, the processes described above, which are performed by the HMD 101, may be divided and executed by multiple devices, or the processes described above, which are performed by the image processing device 104, may be divided and executed by multiple devices.

[0145] Alternatively, instead of a head-mounted display device, a portable device having an imaging unit, an attitude sensor, and a display unit, such as a smartphone, may be used. Furthermore, such a portable device may be added to the MR system in addition to the head-mounted display device. In such cases, the image processing device 104 generates an image representing the mixed reality space according to the position and orientation of the head-mounted display device and delivers it to the head-mounted display device. The display device generates an image representing the mixed reality space according to the position and orientation of the portable device and delivers it to the portable device. The method for generating the image representing the mixed reality space is as described in the above embodiment.

[0146] Furthermore, in the above embodiment, the attitude sensor 302 was described as being provided by the HMD 101. However, this is not the only option; for example, the necessary information may be obtained from images captured by cameras installed around the user of the HMD 101.

[0147] Furthermore, while each embodiment describes an example in which image data is transmitted and received between two devices, for example, instead of image data, "audio data," "distance data (distance map)," or "data representing scent" may be transmitted and received. In each embodiment, controlling the "packet size" is equivalent to controlling the "number of divisions of the image data when generating a packet." This is because as the packet size increases, the number of divisions of the image data decreases. Specifically, as shown in Figures 5B to 5D, as the packet size increases to sizes S1, S2, and S3, the number of divisions of the image data decreases. In other words, if the communication state between the HMD 101 and the image processing device 104 is in a first state, the number of divisions of the image data may be set to a first number. On the other hand, if the communication state is in a second state which is worse than the first state, the number of divisions of the image data may be set to a second number which is less than the first number. The correction method when an error occurs in the image data may also be determined (selected) based on the number of divisions of the image data. At this time, "the packet size is larger than the first size" The phrase "when it is large" can be rephrased as "when the number of divisions of the image data is smaller than the number of divisions corresponding to the first size." Similarly, the phrase "when the packet size is smaller than the first size" can be rephrased as "when the number of divisions of the image data is larger than the number of divisions corresponding to the first size."

[0148] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be rephrased as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2" may be rephrased as "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, as long as no contradiction arises, "greater than or equal to A" may be rephrased as "greater than (higher; longer; more) than A," and "less than or equal to A" may be rephrased as "less than (lower; shorter; fewer) than A." And "greater than (higher; longer; more) than A" may be rephrased as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be rephrased as "less than or equal to A."

[0149] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0150] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0151] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0152] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0153] The above-disclosed embodiments include the following configurations, methods, and programs. (Composition 1) A transmitting device that transmits first data to a receiving device, A first generation means that generates packets for each of the multiple data obtained by dividing the first data, by adding a packet header to each of the multiple data, A first transmission means that transmits each packet generated by the first generation means to the receiving device, A first control means controls the packet size, which is the size of each packet, according to the communication status between the transmitting device and the receiving device. It has, The first control means sets the packet size to a first size if the communication state is a first state, and sets the packet size to a second size which is larger than the first size if the communication state is a second state which is worse than the first state. A transmitting device characterized by the following features. (Configuration 2) The aforementioned communication status is based on radio wave strength, throughput, or frame rate. A transmitting device according to configuration 1, characterized in that it is a transmitting device. (Composition 3) The aforementioned communication state is based on the bit error rate of the data received by the receiving device. A transmitting device according to configuration 1 or 2, characterized by the above. (Composition 4) A processing system comprising a transmitting device described in any of configurations 1 to 3, and a receiving device that receives the first data transmitted from the transmitting device, The receiving device has correction means for correcting errors when an error is detected in the received first data. A processing system characterized by the following: (Composition 5) The first data mentioned above is image data, The aforementioned error occurred in a pixel of the image data. The correction means determines a method for correcting the error based on the packet size. The processing system according to configuration 4, characterized in that... (Composition 6) The first data mentioned above is video image data, If the packet size is larger than the third size, the correction means corrects the error based on the image of the frame immediately preceding the frame in which the error was detected. The processing system according to configuration 5, characterized in that it is a processing system according to configuration 5. (Composition 7) The first data includes data for a first image and a second image that have parallax with respect to each other. If the packet size is larger than the third size, and the error is detected in the first image, the correction means corrects the error based on the second image. The processing system according to configuration 5, characterized in that it is a processing system according to configuration 5. (Composition 8) If the packet size is smaller than the third size, the correction means corrects the error based on the pixels surrounding the pixel where the error was detected. A processing system according to any one of configurations 5 to 7, characterized by the above. (Composition 9) If the packet size is smaller than the third size, the correction means corrects the error based on lines adjacent to the line containing the pixel in which the error was detected. A processing system according to any one of configurations 5 to 7, characterized by the above. (Composition 10) The first transmitting means transmits a plurality of image data including the first data to the receiving device. The first control means causes each of the plurality of image data to have a different packet size according to the communication state. A processing system according to any one of configurations 4 to 9, characterized by the above. (Composition 11) The receiving device is, A second generation means that generates packets for each of the multiple data obtained by dividing the second data, by adding a packet header to each of the multiple data, A second transmission means for transmitting each packet generated by the second generation means to the transmission device, A second control means controls the size of each packet generated by the second generation means according to the communication state between the transmitting device and the receiving device, Having, A processing system according to any one of configurations 4 to 10, characterized by the above. (Composition 12) A transmitting device that transmits first data to a receiving device, A first generation means that generates packets for each of the multiple data obtained by dividing the first data, by adding a packet header to each of the multiple data, A first transmission means that transmits each packet generated by the first generation means to the receiving device, A first control means controls the number of divisions when dividing the first data into the plurality of data, according to the communication state between the transmitting device and the receiving device, It has, The first control means sets the number of divisions to a first number if the communication state is a first state, and sets the number of divisions to a second number which is less than the first number if the communication state is a second state which is worse than the first state. A transmitting device characterized by the following features. (method) A control method for a transmitting device that transmits first data to a receiving device, A first generation step involves generating packets for each of the multiple data obtained by dividing the first data, by adding a packet header to each of the multiple data, A first transmission step in which each packet generated in the first generation step is transmitted to the receiving device, A first control step controls the packet size, which is the size of each packet, according to the communication status between the transmitting device and the receiving device. It has, In the first control step, if the communication state is a first state, the packet size is set to a first size; if the communication state is a second state which is worse than the first state, the packet size is set to a second size which is larger than the first size. A control method for a transmitting device, characterized by the following: (program) A program for causing a computer to function as one of the means of a transmitting device described in any of configurations 1 to 3. [Explanation of Symbols]

[0154] 101: HMD (receiving device), 104: Image processing device (transmitting device), 314: Control Unit (Packet Generation Unit), 310: Interface (Transmission Unit)

Claims

1. A transmitting device that transmits first data to a receiving device, A first generation means that generates packets for each of the multiple data obtained by dividing the first data, by adding a packet header to each of the multiple data, A first transmission means that transmits each packet generated by the first generation means to the receiving device, A first control means controls the packet size, which is the size of each packet, according to the communication state between the transmitting device and the receiving device. It has, The first control means sets the packet size to a first size if the communication state is a first state, and sets the packet size to a second size which is larger than the first size if the communication state is a second state which is worse than the first state. A transmitting device characterized by the following features.

2. The aforementioned communication status is based on radio wave strength, throughput, or frame rate. The transmitting device according to feature 1.

3. The aforementioned communication state is based on the bit error rate of the data received by the receiving device. The transmitting device according to feature 1.

4. A processing system comprising a transmitting device according to any one of claims 1 to 3, and a receiving device for receiving the first data transmitted from the transmitting device, The receiving device has correction means for correcting errors when an error is detected in the received first data. A processing system characterized by the following:

5. The first data is image data, The aforementioned error occurred in a pixel of the image data. The correction means determines a method for correcting the error based on the packet size. The processing system according to feature 4.

6. The first data mentioned above is video image data, If the packet size is larger than the third size, the correction means corrects the error based on the image of the frame immediately preceding the frame in which the error was detected. The processing system according to claim 5.

7. The first data includes data for a first image and a second image that have parallax with respect to each other. If the packet size is larger than the third size, and the error is detected in the first image, the correction means corrects the error based on the second image. The processing system according to claim 5.

8. If the packet size is smaller than the third size, the correction means corrects the error based on the pixels surrounding the pixel where the error was detected. The processing system according to claim 5.

9. If the packet size is smaller than the third size, the correction means corrects the error based on lines adjacent to the line containing the pixel in which the error was detected. The processing system according to claim 5.

10. The first transmission means transmits a plurality of image data, including the first data, to the receiving device. The first control means causes each of the plurality of image data to have a different packet size according to the communication state. The processing system according to feature 4.

11. The receiving device is, A second generation means generates packets for each of the multiple data obtained by dividing the second data, by adding a packet header to each of the multiple data, A second transmission means for transmitting each packet generated by the second generation means to the transmission device, A second control means controls the size of each packet generated by the second generation means according to the communication state between the transmitting device and the receiving device, Having, The processing system according to feature 4.

12. A transmitting device that transmits first data to a receiving device, A first generation means that generates packets for each of the multiple data obtained by dividing the first data, by adding a packet header to each of the multiple data, A first transmission means that transmits each packet generated by the first generation means to the receiving device, A first control means controls the number of divisions when dividing the first data into the plurality of data, according to the communication state between the transmitting device and the receiving device, It has, The first control means sets the number of divisions to a first number if the communication state is a first state, and sets the number of divisions to a second number which is less than the first number if the communication state is a second state which is worse than the first state. A transmitting device characterized by the following features.

13. A control method for a transmitting device that transmits first data to a receiving device, A first generation step involves generating packets for each of the multiple data obtained by dividing the first data, by adding a packet header to each of the multiple data, A first transmission step in which each packet generated in the first generation step is transmitted to the receiving device, A first control step controls the packet size, which is the size of each packet, according to the communication status between the transmitting device and the receiving device. It has, In the first control step, if the communication state is a first state, the packet size is set to a first size; if the communication state is a second state which is worse than the first state, the packet size is set to a second size which is larger than the first size. A control method for a transmitting device, characterized by the following:

14. A program for causing a computer to function as one of the means of a transmitting device according to any one of claims 1 to 3.

Images