Transmitting device and receiving device

Abstract

To reduce the pixel data size of a depth image while suppressing deterioration in the resolution of the depth image and reduction in a depth range.SOLUTION: A transmission device 20 includes a communication unit 23 that communicates with a reception device 30, and a controller 21 for acquiring a depth image obtained by measuring the distance to a subject within a measurement range using at least one depth camera 26, referring to the acquired depth image to specify a window corresponding to the position of the subject from among a plurality of windows which are set within the measurement range and aligned in a depth direction, sending a window identifier for identifying the specified window to the reception device 30 via the communication unit 23, converting the measured distance into a relative distance inside the window for each pixel of the depth image, and transmitting an acquired conversion value as pixel data to the reception device 30 via the communication unit 23.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a transmission device, a program, and a reception device.

Background Art

[0002] Patent Document 1 discloses an apparatus that displays a composite image obtained by superimposing a two-dimensional map image of depth information of a subject generated by a depth camera module on an image generated by a camera module for photography.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When it is necessary to reduce the pixel data size of a depth image due to circumstances such as transmitting a composite image via HDMI (registered trademark), there may arise a problem that the resolution of the depth image decreases or the depth range narrows. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface.

[0005] An object of the present disclosure is to reduce the pixel data size of a depth image while suppressing a decrease in the resolution of the depth image and a reduction in the depth range.

Means for Solving the Problems

[0006] The transmission device according to the present disclosure is a communication unit that communicates with a reception device, Obtain a depth image obtained by measuring the distance to a subject within the measurement range using at least one depth camera, and with reference to the obtained depth image, identify a window corresponding to the position of the subject from among a plurality of windows arranged in the depth direction and set within the measurement range, and transmit a window identifier for identifying the identified window to the receiving device via the communication unit. Also, for each pixel of the depth image, convert the measured distance into a relative distance within the window, and transmit the obtained conversion value as pixel data to the receiving device via the communication unit, and a control unit comprises.

[0007] The program according to the present disclosure causes a computer that communicates with a receiving device to obtain a depth image obtained by measuring the distance to a subject within the measurement range using at least one depth camera, with reference to the obtained depth image, identify a window corresponding to the position of the subject from among a plurality of windows arranged in the depth direction and set within the measurement range, transmit a window identifier for identifying the identified window to the receiving device, for each pixel of the depth image, convert the measured distance into a relative distance within the window, and for each pixel of the depth image, transmit the obtained conversion value as pixel data to the receiving device. to execute operations including these.

[0008] The receiving device according to the present disclosure comprises a communication unit that communicates with a transmitting device that obtains a depth image obtained by measuring the distance to a subject within the measurement range using at least one depth camera, Receive a window identifier for identifying a window corresponding to the position of the subject among a plurality of windows arranged in the depth direction and set within the measurement range from the transmission device via the communication unit. For each pixel of the depth image, receive, as pixel data, a conversion value obtained by converting the measured distance into a relative distance within the window from the transmission device via the communication unit. Identify the window corresponding to the received window identifier from among the plurality of windows, and for each pixel of the depth image, convert the relative distance within the identified window indicated by the received pixel data into the measured distance, thereby restoring the depth image, and a control unit comprises.

Advantages of the Invention

[0009] According to the present disclosure, it is possible to reduce the pixel data size of the depth image while suppressing a decrease in the resolution of the depth image and a reduction in the depth range.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0012] In each figure, the same or corresponding parts are denoted by the same reference numerals. In the description of the present embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0013] Referring to FIG. 1, the configuration of the communication system 10 according to the present embodiment will be described.

[0014] The communication system 10 includes a transmission device 20 and a reception device 30. In the present embodiment, the transmission device 20 and the reception device 30 communicate with each other via a network 40. However, they may communicate with each other only via a cable without using other devices, or communicate with each other wirelessly.

[0015] The transmission device 20 is a terminal device used by the first user U1. The transmission device 20 is, for example, a general-purpose computer such as a PC, or a dedicated computer specialized for specific calculations. "PC" is an abbreviation for personal computer.

[0016] The reception device 30 is a terminal device used by the second user U2. The reception device 30 is, for example, a general-purpose computer such as a PC, or a dedicated computer specialized for specific calculations.

[0017] The network 40 includes the Internet, at least one WAN, at least one MAN, or any combination thereof. "WAN" is an abbreviation for wide area network. "MAN" is an abbreviation for metropolitan area network. The network 40 may include at least one wireless network, at least one optical network, or any combination thereof. The wireless network is, for example, an ad hoc network, a cellular network, a wireless LAN, a satellite communication network, or a terrestrial microwave network. "LAN" is an abbreviation for local area network.

[0018] Referring to FIGS. 1 and 2, the outline of the present embodiment will be described.

[0019] The transmitting device 20 acquires a depth image Di. The depth image Di is an image obtained by measuring the distance to a subject within a measurement range 28 by at least one depth camera 26. The depth camera 26 is provided in the transmitting device 20 in this embodiment, but may be connected to the transmitting device 20 as an external input device. As the connection interface, an interface corresponding to a standard such as USB or Bluetooth (registered trademark) can be used. "USB" is an abbreviation for Universal Serial Bus. As shown in FIG. 2, a plurality of windows W1, ···, Wn are set within the measurement range 28. The plurality of windows W1, ···, Wn are arranged in the depth direction Z. That is, the plurality of windows W1, ···, Wn are arranged at positions with different depths. The number n of windows is an arbitrary integer of 2 or more, such as 30.

[0020] The transmitting device 20 refers to the depth image Di and identifies a window Wj corresponding to the position of the subject from among the plurality of windows W1, ···, Wn. The transmitting device 20 transmits a window identifier for identifying the window Wj to the receiving device 30. When the receiving device 30 receives the window identifier from the transmitting device 20, it identifies the window Wj corresponding to the received window identifier from among the plurality of windows W1, ···, Wn.

[0021] For each pixel of the depth image Di, the transmitting device 20 converts the measured distance into a relative distance within the window Wj, and transmits the obtained conversion value to the receiving device 30 as pixel data. The receiving device 30 receives the pixel data from the transmitting device 20 for each pixel of the depth image Di, and restores the depth image Di by converting the relative distance within the window Wj indicated by the received pixel data into the measured distance.

[0022] According to this embodiment, it is possible to reduce the pixel data size of the depth image Di while suppressing a decrease in the resolution of the depth image Di and a reduction in the measurement range 28.

[0023] A plurality of windows W1, ···, Wn are set such that at least two windows overlap each other. That is, the plurality of windows W1, ···, Wn are arranged such that there are two or more windows at at least one depth position. In the present embodiment, the plurality of windows W1, ···, Wn are set to overlap each other with a constant width corresponding to the size of the subject in the depth direction Z. For example, the subject is the first user U1, and the constant width is 0.2 meters or more and 0.8 meters or less. In the example shown in FIG. 2, the constant width is 0.5 meters, and the width of each window in the depth direction Z is 1 meter. That is, in the example shown in FIG. 2, the window W1 is set in the range of 0 meter or more and 1 meter or less from the position of the depth camera 26 in the depth direction Z. The window W2 is set in the range of 0.5 meter or more and 1.5 meters or less from the position of the depth camera 26. The window W3 is set in the range of 1 meter or more and 2 meters or less from the position of the depth camera 26. In FIG. 2, the windows W2 and W3 are shifted horizontally to show that the windows W1 and W2 overlap each other and the windows W2 and W3 overlap each other. However, the windows W2 and W3 are actually set within the measurement range 28 in the same manner as the window W1.

[0024] As an example, assuming that the first user U1 is within a range of 0.7 meters or more and 1.2 meters or less from the position of the depth camera 26 in the depth direction Z, in the example shown in FIG. 2, the window W2 corresponds to the position of the first user U1. Therefore, the transmission device 20 refers to the depth image Di and recognizes that the window W2 is the window Wj corresponding to the position of the subject. The transmission device 20 transmits the window identifier "2" for identifying the window W2 to the reception device 30, and for each pixel of the depth image Di, converts the measured distance into a relative distance within the window W2, and transmits the obtained conversion value to the reception device 30 as pixel data. When the reception device 30 receives the window identifier "2" from the transmission device 20, it recognizes that the window W2 is the window Wj corresponding to the position of the subject. The reception device 30 receives pixel data for each pixel of the depth image Di, and restores the depth image Di by converting the relative distance within the window W2 indicated by the received pixel data into the measured distance.

[0025] In this embodiment, the transmitting device 20 synthesizes the RGB image Ri and the depth image Di. As a result, a composite image 11 as shown in FIG. 3 is obtained. The RGB image Ri is an image of a subject photographed by at least one RGB camera 27. In this embodiment, the RGB camera 27 is provided in the transmitting device 20, but it may be connected to the transmitting device 20 as an external input device. As the connection interface, an interface compatible with a standard such as USB or Bluetooth (registered trademark) can be used. The transmitting device 20 transmits the obtained composite image 11 to the receiving device 30. The receiving device 30 receives the composite image 11 from the transmitting device 20 via the network 40. The receiving device 30 displays the received composite image 11 on the display 36 for the second user U2. The display 36 is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescent. In this embodiment, the display 36 is provided in the receiving device 30, but it may be connected to the receiving device 30 as an external output device. As the connection interface, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0026] In the example shown in FIG. 3, the RGB images R1, R2, R3 and the depth images D1, D2, D3 are synthesized. The RGB images R1, R2, R3 are images of the first user U1 photographed by RGB cameras arranged in front of, to the left of, and to the right of the first user U1, respectively. The depth images D1, D2, D3 are images obtained by measuring the distances to the first user U1 by three depth cameras arranged in front of, to the left of, and to the right of the first user U1, respectively.

[0027] For example, it is conceivable to transmit the composite image 11 from the transmission device 20 via HDMI (registered trademark). As a method of transmitting the HDMI (registered trademark) signal via the network 40, known methods such as HDMI (registered trademark) over IP or HDMI (registered trademark) over Ethernet (registered trademark) can be used. "IP" is an abbreviation for Internet Protocol. Generally, pixel data of an RGB image is 8 bits for each of the RGB color components, that is, a total of 24-bit data. In HDMI (registered trademark), 8-bit data is transmitted for each RGB channel. Since the pixel data of the depth image is 16-bit data, if this data is to be transmitted as it is via HDMI (registered trademark), it is conceivable to divide the data into upper 8 bits and lower 8 bits and transmit it using any two of the three RGB channels. However, if this is done, the image will deteriorate significantly when a bit error occurs in the channel that transmits the upper 8 bits. It is also possible to reduce the pixel data size of the depth image to 8 bits and transmit the data using one channel, but in that case, the resolution of the depth image has to be lowered or the range of measurable distances, that is, the depth range, has to be narrowed. For example, when the width in the depth direction Z of the measurement range 28 is 100 meters, if the pixel data size of 16 bits is reduced to 8 bits by narrowing the measurement range 28, the width in the depth direction Z of the measurement range 28 will be reduced to about 0.4 meters.

[0028] In this embodiment, the space in front of the depth camera 26 is divided into a plurality of overlapping windows W1, ···, Wn of a certain size, and the measured distance of each pixel is replaced with the relative distance within the corresponding window Wj and expressed, thereby reducing the pixel data size of the depth image Di. Therefore, when transmitting the composite image 11 via HDMI (registered trademark), the pixel data of the depth image Di can be transmitted in one channel while suppressing a decrease in the resolution of the depth image Di and a reduction in the measurement range 28. The window identifier of the window Wj may be embedded in the empty area at the lower right corner of the composite image 11 shown in FIG. 3 and transmitted in the same channel as the pixel data of the depth image Di, or may be transmitted via a communication route different from that of the pixel data of the depth image Di.

[0029] Referring to FIG. 1, the configuration of the transmission device 20 according to this embodiment will be described.

[0030] The transmission device 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, and an output unit 25.

[0031] The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or a GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 21 executes processing related to the operation of the transmission device 20 while controlling each unit of the transmission device 20.

[0032] The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. The magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 22 functions as, for example, a main memory device, an auxiliary memory device, or a cache memory. The storage unit 22 stores data used for the operation of the transmission device 20 and data obtained by the operation of the transmission device 20.

[0033] The communication unit 23 includes at least one communication interface. The communication interface is, for example, an interface corresponding to HDMI (registered trademark), an interface corresponding to a wired LAN communication standard such as Ethernet (registered trademark), a wireless LAN communication standard such as IEEE802.11, or an interface corresponding to a mobile communication standard such as LTE, 4G standard, or 5G standard. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 23 communicates with the receiving device 30. In this embodiment, the communication unit 23 communicates with the receiving device 30 by a communication method that transmits data of each pixel of an image in a separate channel for each color element, such as HDMI (registered trademark). The communication unit 23 receives data used for the operation of the transmitting device 20 and transmits data obtained by the operation of the transmitting device 20.

[0034] The input unit 24 includes a depth camera 26, an RGB camera 27, and at least one other input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen provided integrally with a display, or a microphone. The input unit 24 receives an operation for inputting data used for the operation of the transmitting device 20. Instead of being provided in the transmitting device 20, the input unit 24 may be connected to the transmitting device 20 as an external input device. As the connection interface, an interface corresponding to a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0035] The output unit 25 includes at least one output interface. The output interface is, for example, a display or a speaker. The output unit 25 outputs data obtained by the operation of the transmission device 20. Instead of being provided in the transmission device 20, the output unit 25 may be connected to the transmission device 20 as an external output device. As the connection interface, an interface compliant with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0036] The functions of the transmission device 20 are realized by executing the program according to this embodiment on a processor as the control unit 21. That is, the functions of the transmission device 20 are realized by software. The program causes a computer to execute the operations of the transmission device 20, thereby causing the computer to function as the transmission device 20. That is, the computer functions as the transmission device 20 by executing the operations of the transmission device 20 according to the program.

[0037] The program can be stored in a non-transitory computer-readable medium. The non-transitory computer-readable medium is, for example, a flash memory, a magnetic recording device, an optical disc, a magneto-optical recording medium, or a ROM. The distribution of the program is performed, for example, by selling, transferring, or lending a portable medium such as an SD card, a DVD, or a CD-ROM storing the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be stored in the storage of a server and transferred from the server to other computers to distribute the program. The program may be provided as a program product.

[0038] A computer stores, for example, a program stored in a portable medium or a program transferred from a server in a main memory device once. Then, the computer reads the program stored in the main memory device with a processor and executes processing according to the read program with the processor. The computer may directly read a program from a portable medium and execute processing according to the program. The computer may sequentially execute processing according to the received program each time a program is transferred from a server to the computer. Processing may be executed by a so-called ASP type service that realizes functions only by execution instructions and result acquisition without transferring a program from a server to a computer. "ASP" is an abbreviation for application service provider. A program includes information for use in processing by an electronic computer and things conforming to the program. For example, data that is not a direct instruction to a computer but has a property that defines the processing of the computer corresponds to "things conforming to the program".

[0039] Some or all of the functions of the transmission device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. That is, some or all of the functions of the transmission device 20 may be realized by hardware.

[0040] With reference to FIG. 1, the configuration of the receiving device 30 according to the present embodiment will be described.

[0041] The receiving device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, and an output unit 35.

[0042] The control unit 31 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. The programmable circuit is, for example, an FPGA. The dedicated circuit is, for example, an ASIC. While controlling each part of the receiving device 30, the control unit 31 executes processing related to the operation of the receiving device 30.

[0043] The storage unit 32 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, ROM, or flash memory. The RAM is, for example, an SRAM or DRAM. The ROM is, for example, an EEPROM. The flash memory is, for example, an SSD. The magnetic memory is, for example, an HDD. The storage unit 32 functions as, for example, a main memory device, an auxiliary storage device, or a cache memory. Stored in the storage unit 32 are data used for the operation of the receiving device 30 and data obtained by the operation of the receiving device 30.

[0044] The communication unit 33 includes at least one communication interface. The communication interface is, for example, an interface corresponding to HDMI (registered trademark), an interface corresponding to a wired LAN communication standard such as Ethernet (registered trademark), a wireless LAN communication standard such as IEEE802.11, or an interface corresponding to a mobile communication standard such as LTE, 4G standard, or 5G standard. The communication unit 33 communicates with the transmitting device 20. In this embodiment, the communication unit 33 communicates with the transmitting device 20 by a communication method that transmits data of each pixel of an image in separate channels for each color element, such as HDMI (registered trademark). The communication unit 33 receives data used for the operation of the receiving device 30 and transmits data obtained by the operation of the receiving device 30.

[0045] The input unit 34 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen provided integrally with a display, a camera, or a microphone. The camera is, for example, an RGB camera or a combination of an RGB camera and a depth camera. The input unit 34 receives an operation for inputting data used for the operation of the receiving device 30. Instead of being provided in the receiving device 30, the input unit 34 may be connected to the receiving device 30 as an external input device. As the connection interface, an interface corresponding to a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0046] The output unit 35 includes a display 36 and at least one other output interface. The output interface is, for example, a speaker. The output unit 35 outputs data obtained by the operation of the receiving device 30. Instead of being provided in the receiving device 30, the output unit 35 may be connected to the receiving device 30 as an external output device. As the connection interface, an interface corresponding to a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0047] The functions of the receiving device 30 are realized by executing another program according to the present embodiment by a processor as the control unit 31. That is, the functions of the receiving device 30 are realized by software. The program causes a computer to execute the operation of the receiving device 30, thereby causing the computer to function as the receiving device 30. That is, the computer functions as the receiving device 30 by executing the operation of the receiving device 30 according to the program.

[0048] Some or all of the functions of the receiving device 30 may be realized by a programmable circuit or a dedicated circuit as the control unit 31. That is, some or all of the functions of the receiving device 30 may be realized by hardware.

[0049] Referring to FIGS. 4 and 5, the operation of the communication system 10 according to this embodiment will be described. This operation corresponds to the communication method according to this embodiment.

[0050] FIG. 4 shows the operation of the transmission device 20.

[0051] The processes from step S201 to step S206 are repeatedly executed, for example, while the first user U1 and the second user U2 communicate with each other in the virtual three-dimensional space using the transmission device 20 and the reception device 30, respectively.

[0052] In step S201, the control unit 21 acquires a depth image Di via at least one depth camera 26. In this embodiment, the control unit 21 acquires the depth image Di and captures an RGB image Ri using at least one RGB camera 27. The control unit 21 synthesizes the acquired depth image Di and the captured RGB image Ri, and stores the obtained synthesized image 11 in the storage unit 22. The control unit 21 may acquire the voice of the first user U1 via a microphone as the input unit 24. In the example shown in FIG. 3, the control unit 21 acquires depth images D1, D2, and D3 via three depth cameras arranged in front of, to the left of, and to the right of the first user U1, and captures RGB images R1, R2, and R3 using RGB cameras arranged in front of, to the left of, and to the right of the first user U1. The control unit 21 synthesizes the acquired depth images D1, D2, and D3 and the captured RGB images R1, R2, and R3. As a result, a synthesized image 11 as shown in FIG. 3 is obtained.

[0053] In step S202, the control unit 21 refers to the depth image Di acquired in step S201, and identifies a window Wj corresponding to the position of the subject from among a plurality of windows W1, ···, Wn as shown in FIG. 2. The control unit 21 stores in the storage unit 22 a window identifier for identifying the identified window Wj. The control unit 21 may remove the background from the depth image Di acquired in step S201. In the present embodiment, the control unit 21 may remove the background from both the depth image Di and the RGB image Ri included in the composite image 11 stored in the storage unit 22. In the example shown in FIG. 3, for each of the depth images D1, D2, D3 included in the composite image 11 obtained in step S201, the control unit 21 detects the position of the first user U1, and stores in the storage unit 22 the window identifier of the window corresponding to the detected position.

[0054] In step S203, the control unit 21 checks whether depth encoding has been completed for all pixels of the depth image Di acquired in step S201. If depth encoding has been completed for all pixels of the depth image Di, the process of step S206 is executed. If depth encoding has not been completed for one or more pixels of the depth image Di, the process of step S204 is executed. In the example shown in FIG. 3, if depth encoding has been completed for all pixels of the depth images D1, D2, D3 included in the composite image 11 obtained in step S201, the process of step S206 is executed. If depth encoding has not been completed for one or more pixels of the depth images D1, D2, D3, the process of step S204 is executed.

[0055] In step S204, the control unit 21 selects one pixel of the depth image Di acquired in step S201 for which depth encoding has not been completed. In the example shown in FIG. 3, the control unit 21 selects one pixel of any of the depth images D1, D2, D3 included in the composite image 11 obtained in step S201 for which depth encoding has not been completed.

[0056] In step S205, the control unit 21 performs depth encoding by converting the measured distance for the pixel selected in step S204 into a relative distance within the window Wj specified in step S202. The control unit 21 stores the obtained conversion value in the storage unit 22 as encoded pixel data. For example, assuming that the measured distance for the pixel selected in step S204 is 0.7 meters and the window Wj specified in step S202 is window W2, in the example shown in FIG. 2, the control unit 21 subtracts the minimum distance of 0.5 meters of window W2 from the measured distance of 0.7 meters to calculate a relative distance of 0.2 meters within window W2. The control unit 21 stores data representing the obtained calculated value in 8 bits in the storage unit 22 as encoded pixel data. In the present embodiment, the control unit 21 overwrites the pixel data of the corresponding pixel of the depth image Di included in the composite image 11 stored in the storage unit 22 with the encoded pixel data. In the example shown in FIG. 3, the control unit 21 converts the measured distance for the pixel selected in step S204 in any of the depth images D1, D2, D3 included in the composite image 11 obtained in step S201 into a relative distance within the window corresponding to the window identifier stored in the storage unit 22, and overwrites and stores the obtained conversion value in the storage unit 22 as encoded pixel data. After step S205, the process of step S203 is executed again.

[0057] In step S206, the control unit 21 transmits the window identifier stored in the storage unit 22 to the receiving device 30 via the communication unit 23, and for each pixel of the depth image Di, transmits the pixel data stored in the storage unit 22 to the receiving device 30 via the communication unit 23. In the present embodiment, the control unit 21 transmits at least the composite image 11 stored in the storage unit 22 to the receiving device 30 by a communication method that transmits data of each pixel of an image in separate channels for each color component, such as HDMI (registered trademark). The transmitted composite image 11 includes encoded pixel data instead of the original pixel data for each pixel of the depth image Di. The control unit 21 transmits this encoded pixel data to the receiving device 30 using any one of the channels used in the above-described communication method. When the control unit 21 acquires the voice of the first user U1 in step S201, the acquired voice is also transmitted to the receiving device 30 via the communication unit 23. In the example shown in FIG. 3, the control unit 21 embeds the window identifier stored in the storage unit 22 in the empty area of the composite image 11 obtained in step S201, and then transmits the composite image 11 to the receiving device 30 via HDMI (registered trademark). Specifically, the control unit 21 transmits the data of each pixel of the RGB images R1, R2, and R3 included in the composite image 11 to the receiving device 30 using each of the three RGB channels for each RGB color component. For each of the depth images D1, D2, and D3 included in the composite image 11, the control unit 21 transmits the encoded pixel data and the window identifier to the receiving device 30 using any one of the three RGB channels.

[0058] FIG. 5 shows the operation of the receiving device 30.

[0059] The processes from step S301 to step S306 are repeatedly executed, for example, while the first user U1 and the second user U2 communicate with each other in the virtual three-dimensional space using the transmitting device 20 and the receiving device 30, respectively.

[0060] In step S301, the control unit 31 receives a window identifier from the transmission device 20 via the communication unit 33, and for each pixel of the depth image Di, receives pixel data from the transmission device 20 via the communication unit 33. In the present embodiment, the control unit 31 receives at least the composite image 11 from the transmission device 20 by the above-described communication method, and stores the received composite image 11 in the storage unit 32. The received composite image 11 includes encoded pixel data instead of the original pixel data for each pixel of the depth image Di. The control unit 31 receives this encoded pixel data from the transmission device 20 on any one of the channels used in the above-described communication method. The control unit 31 may receive the voice of the first user U1 from the transmission device 20 via the communication unit 33. In the example shown in FIG. 3, the control unit 31 receives the composite image 11 from the transmission device 20 by HDMI (registered trademark) in a form in which the window identifier is embedded in the empty area of the composite image 11. Specifically, the control unit 31 receives the data of each pixel of the RGB images R1, R2, R3 included in the composite image 11 from the transmission device 20 on each of the three RGB channels for each RGB color element. For each of the depth images D1, D2, D3 included in the composite image 11, the control unit 31 receives the encoded pixel data and the window identifier from the transmission device 20 on any one of the three RGB channels.

[0061] In step S302, the control unit 31 identifies the window Wj corresponding to the window identifier received in step S301 from among the plurality of windows W1, ···, Wn as shown in FIG. 2. In the example shown in FIG. 3, the control unit 31 identifies the window corresponding to the window identifier received in step S301 for each of the depth images D1, D2, D3 included in the composite image 11 received in step S301.

[0062] In step S303, the control unit 31 checks whether depth decoding has been completed for all pixels of the depth image Di received in step S301, that is, whether the depth image Di has been restored. If depth decoding has been completed for all pixels of the depth image Di, the process of step S306 is executed. If depth decoding has not been completed for one or more pixels of the depth image Di, the process of step S304 is executed. In the example shown in FIG. 3, if depth decoding has been completed for all pixels of the depth images D1, D2, and D3 included in the composite image 11 received in step S301, that is, if all of the depth images D1, D2, and D3 have been restored, the process of step S306 is executed. If depth decoding has not been completed for one or more pixels of the depth images D1, D2, and D3, that is, if any one of the depth images D1, D2, and D3 has not been restored, the process of step S304 is executed.

[0063] In step S304, the control unit 31 selects one pixel of the depth image Di received in step S301 for which depth decoding has not been completed. In the example shown in FIG. 3, the control unit 31 selects one pixel of any of the depth images D1, D2, and D3 included in the composite image 11 received in step S301 for which depth decoding has not been completed.

[0064] In step S305, the control unit 31 performs depth decoding by converting the relative distance within the window Wj specified in step S302, indicated by the pixel data received in step S301, for the pixel selected in step S304, into the measured distance. The control unit 31 stores the obtained conversion value in the storage unit 32 as decoded pixel data. For example, if the window Wj specified in step S302 is window W2, and for the pixel selected in step S304, the relative distance within window W2 indicated by the pixel data received in step S301 is 0.2 meters, then in the example shown in FIG. 2, the control unit 31 adds the minimum distance of 0.5 meters of window W2 to the relative distance of 0.2 meters within window W2 to calculate the measured distance of 0.7 meters. The control unit 31 stores data representing the obtained calculated value in 16 bits in the storage unit 32 as decoded pixel data. In this embodiment, the control unit 31 overwrites the pixel data of the corresponding pixel of the depth image Di included in the composite image 11 stored in the storage unit 32 with the decoded pixel data. In the example shown in FIG. 3, for any of the depth images D1, D2, D3 included in the composite image 11 received in step S301, for the pixel selected in step S304, the control unit 31 converts the relative distance within the window specified in step S302 into the measured distance, and stores the obtained conversion value in the storage unit 32 as decoded pixel data. After step S305, the process of step S303 is executed again.

[0065] In step S306, the control unit 31 displays the depth image Di restored from step S303 to step S305 on the display 36 for the second user U2. In the present embodiment, the control unit 31 displays the composite image 11 stored in the storage unit 32 on the display 36 in such a manner that the depth image Di received in step S301 is replaced with the one restored from step S303 to step S305. When the control unit 31 receives the voice of the first user U1 in step S301, the received voice is output from the speaker as the output unit 35. In the example shown in FIG. 3, for each pixel of the depth images D1, D2, and D3, the control unit 31 displays the composite image 11 received in step S301 in such a way that the pixel data decoded from step S303 to step S305 is displayed instead of the encoded pixel data.

[0066] According to the present embodiment, by the operation as described above, it is possible to reduce the pixel data size of the depth image Di without significantly reducing the resolution of the depth image Di and without narrowing the depth range. For example, when the distance measured by the depth camera 26 is represented by 16 bits, by setting a plurality of windows W1, ···, Wn so that the relative distance within each window can be represented by 8 bits, it becomes possible to transmit the pixel data of the depth image Di through one channel of HDMI (registered trademark). Furthermore, according to the present embodiment, by transmitting the composite image 11, it becomes possible to generate and display a real hologram of the subject in the virtual three-dimensional space.

[0067] As a modification example of this embodiment, the same pixel data of the depth image Di may be transmitted from the transmission device 20 using two or all three of the RGB three channels, and the received data may be averaged by the receiving device 30. In this modification example, for each pixel of the depth image Di, the control unit 21 of the transmission device 20 transmits the encoded pixel data to the receiving device 30 using any one of the channels used in the above communication method, and transmits the same data to the receiving device 30 using one or more of the remaining channels used in the communication method. For each pixel of the depth image Di, the control unit 31 of the receiving device 30 receives the encoded pixel data from the transmission device 20 not only using any one of the channels used in the communication method, but also using one or more of the remaining channels used in the communication method. The control unit 31 restores the depth image Di by averaging the conversion values obtained for each channel for each pixel of the depth image Di. For example, for each of the depth images D1, D2, and D3 included in the composite image 11, the control unit 21 of the transmission device 20 transmits the encoded pixel data to the receiving device 30 using all three RGB channels. For each of the depth images D1, D2, and D3 included in the composite image 11, the control unit 31 of the receiving device 30 receives the encoded pixel data from the transmission device 20 using all three RGB channels. For each of the depth images D1, D2, and D3, each time a pixel is selected, the control unit 31 converts the value of the relative distance received as pixel data for each channel into the value of the measured distance, divides the total value of the obtained values by the number of channels 3, and stores the obtained average value in the storage unit 32 as the decoded pixel data.

[0068] According to this modification example, even if a bit error occurs in any channel, its influence can be reduced. As a result, image degradation can be suppressed.

[0069] As another modification example of this embodiment, the YCbCr color format may be used instead of the RGB color format.

[0070] Examples of some embodiments of the present disclosure are illustrated below. However, note that the embodiments of the present disclosure are not limited to these. [Appendix 1] A communication unit that communicates with a receiving device, A depth image obtained by measuring the distance to a subject within a measurement range by at least one depth camera is acquired. With reference to the acquired depth image, from among a plurality of windows arranged in the depth direction and set within the measurement range, a window corresponding to the position of the subject is specified, and an identifier for identifying the specified window is transmitted to the receiving device via the communication unit. For each pixel of the depth image, the measured distance is converted into a relative distance within the window, and the obtained conversion value is transmitted to the receiving device via the communication unit as pixel data, together with a control unit A transmitting device comprising the same. [Appendix 2] The transmitting device according to Appendix 1, wherein at least two of the plurality of windows overlap each other. [Appendix 3] The communication unit communicates with the receiving device by a communication method that transmits data of each pixel of an image in a separate channel for each color element, The transmitting device according to Appendix 1 or Appendix 2, wherein the control unit transmits the pixel data to the receiving device through any one of the channels used in the communication method. [Appendix 4] The transmitting device according to Appendix 3, wherein the control unit transmits the same data as the pixel data to the receiving device through one or more of the remaining channels used in the communication method. [Appendix 5] The transmitting device according to any one of Appendices 1 to 4, wherein the plurality of windows overlap each other with a constant width corresponding to the size of the subject in the depth direction. [Appendix 6] The subject is a user of the transmitting device, The transmitting device according to Appendix 5, wherein the constant width is 0.2 meters or more and 0.8 meters or less. [Appendix 7] On a computer that communicates with a receiving device, Obtaining a depth image obtained by measuring the distance to a subject within a measurement range using at least one depth camera; Referring to the obtained depth image, identifying a window corresponding to the position of the subject from among a plurality of windows arranged in the depth direction and set within the measurement range; Transmitting a window identifier for identifying the identified window to the receiving device; For each pixel of the depth image, converting the measured distance into a relative distance within the window; For each pixel of the depth image, transmitting the obtained conversion value as pixel data to the receiving device A program for executing operations including the above. [Appendix 8] A communication unit that communicates with a transmission device that obtains a depth image obtained by measuring the distance to a subject within a measurement range using at least one depth camera; Among a plurality of windows arranged in the depth direction and set within the measurement range, receiving a window identifier for identifying a window corresponding to the position of the subject from the transmission device via the communication unit, and for each pixel of the depth image, receiving, as pixel data, a conversion value obtained by converting the measured distance into a relative distance within the window from the transmission device via the communication unit, identifying a window corresponding to the received window identifier from among the plurality of windows, and for each pixel of the depth image, converting the relative distance within the identified window indicated by the received pixel data into the measured distance to restore the depth image. A control unit; A receiving device comprising the above. [Appendix 9] The communication unit communicates with the transmission device by a communication method that transmits data of each pixel of an image in separate channels for each color component, The control unit receives the pixel data from the transmission device through any one of the channels used in the communication method. The receiving device according to Appendix 8. [Appendix 10] The control unit receives the pixel data via one or more of the remaining channels used in the communication method from the transmission device, and restores the depth image by averaging the conversion values obtained for each channel for each pixel of the depth image. The receiving device according to Supplementary Note 9.

[0071] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks described in the block diagram may be integrated, or one block may be divided. Instead of executing two or more steps described in the flowchart in time series according to the description, they may be executed in parallel or in a different order according to the processing capabilities of the device that executes each step, or as necessary. In addition, changes can be made without departing from the spirit of the present disclosure.

Explanation of Reference Numerals

[0072] 10 Communication system 11 Composite image 20 Transmission device 21 Control unit 22 Storage unit 23 Communication unit 24 Input unit 25 Output unit 26 Depth camera 27 RGB camera 28 Measurement range 30 Receiving device 31 Control unit 32 Storage unit 33 Communication unit 34 Input unit 35 Output unit 36 Display 40 Network

Claims

1. A communication unit that communicates with a receiving device, Obtains a depth image obtained by measuring the distance to a subject within a measurement range using at least one depth camera, refers to the obtained depth image, and from a plurality of windows arranged in the depth direction and overlapping each other with a constant width in the depth direction within the measurement range, identifies a single window corresponding to the position of the subject, and transmits an identifier of the identified single window to the receiving device via the communication unit. For each pixel of the depth image, the measured distance is converted into a relative distance within the single window, and the obtained conversion value is used as pixel data and transmitted to the receiving device via the communication unit, and a control unit comprising, The communication unit communicates with the receiving device by a communication method that transmits data of each pixel of an image in separate channels for each color element, The control unit transmits the pixel data to the receiving device through any one of the channels used in the communication method, and also transmits the pixel data to the receiving device through one or more of the remaining channels used in the communication method, The receiving device restores the depth image by averaging the pixel data received for each channel for each pixel of the depth image. A transmitting device.

2. The transmitting device according to claim 1, wherein the constant width is predetermined according to a standard size of the subject in the depth direction.

3. The subject is a person, The transmitting device according to claim 2, wherein the constant width is 0.2 meters or more and 0.8 meters or less.

4. A communication unit that communicates with a transmitting device that obtains a depth image obtained by measuring the distance to a subject within a measurement range using at least one depth camera, Receive a window identifier that identifies a single window corresponding to the position of the subject among a plurality of windows arranged in the depth direction, having a constant width in the depth direction, and overlapping each other within the measurement range, from the transmission device via the communication unit. For each pixel of the depth image, receive, as pixel data, a conversion value obtained by converting the measured distance into a relative distance within the single window, from the transmission device via the communication unit. Identify a single window corresponding to the received window identifier from among the plurality of windows, and for each pixel of the depth image, convert the relative distance within the identified single window indicated by the received pixel data into the measured distance, thereby restoring the depth image, a control unit comprising The communication unit communicates with the transmission device by a communication method that transmits data of each pixel of the image in separate channels for each color component The control unit receives the pixel data from the transmission device through any one of the channels used in the communication method, and also receives the pixel data from the transmission device through the remaining one or more of the channels used in the communication method. For each pixel of the depth image, the control unit averages the pixel data received for each channel to restore the depth image, a receiving device

Images