Program and Communication Method

Abstract

To enable a user to show an object in front of the user to a person whom the user is performing communication using a video even if a background removal function is enabled.SOLUTION: A program causes a computer that removes a background from a first image taken in a first space 11 where a first user 13 is present and generates a second image to be displayed to a second user present in a second space separate from the first space 11, upon detecting a cue of the first user 13, to perform operations including scanning an object 15 in the first space 11 and transmitting an image of the scanned object 15 along with the second image through a network.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a program and a communication method.

Background Art

[0002] Patent Document 1 discloses a system that extracts the background in a video conference stream, separates it from the user, and replaces the separated background with the background received from another user's system or a pre-recorded background.

[0003] In the system disclosed in Patent Document 2, a first terminal transmits a moving image captured by a camera to a second terminal. The second terminal displays the received moving image on a display and causes the user to write an instruction for the moving image. The second terminal transmits the written instruction image and a still image obtained by trimming the moving image within the range corresponding to the instruction image to the first terminal. The first terminal extracts the portion of the moving image captured by the camera that matches the still image while performing projective transformation or pose transformation on the received still image. The first terminal displays on the display a moving image in which an instruction image subjected to the same projective transformation or pose transformation as the still image is superimposed on the extracted portion.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a conventional system, even if a user tries to show an object in front of them to the other party in a video conference, the object may be regarded as part of the background and removed.

[0006] An object of the present disclosure is to enable a user to show an object in front of him / her to a communication partner using video even when a background removal function is enabled.

Means for Solving the Problem

[0007] A program according to the present disclosure causes a computer that removes a background from a first image captured in a first space where a first user is located and generates a second image to be displayed toward a second user in a second space separate from the first space, when detecting a signal from the first user, to scan an object in the first space, and to transmit an image of the scanned object via a network together with the second image. to execute operations including the above.

[0008] Another program according to the present disclosure causes a terminal device that removes a background from a first image captured in a first space where a first user is located and generates a second image, when detecting a signal from the first user, to scan an object in the first space, and to receive, via a network, an image of the object scanned by the terminal device together with the second image, and to display the received image of the object together with the second image toward a second user in a second space separate from the first space. to execute operations including the above on a computer.

[0009] A communication method according to the present disclosure includes a terminal device that removes a background from a first image captured in a first space where a first user is located and generates a second image to be displayed toward a second user in a second space separate from the first space, when detecting a signal from the first user, to scan an object in the first space, and to transmit, from the terminal device, an image of the scanned object via a network together with the second image. including the above.

Advantages of the Invention

[0010] According to the present disclosure, even when the background removal function is enabled for a communication partner with whom the user is communicating using video, it is possible to show an object in front of the eyes.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0012] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0013] In each figure, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0014] With reference to FIGS. 1 to 3, the configuration of a communication system 10 according to this embodiment will be described.

[0015] The communication system 10 includes a server device 20, a first terminal device 30, and a second terminal device 40. The server device 20 can communicate with the first terminal device 30 and the second terminal device 40 via a network 50. The first terminal device 30 may also be able to communicate with the second terminal device 40 via the network 50.

[0016] The server device 20 is installed in a facility such as a data center and is operated by a service provider. The server device 20 is a server computer belonging to a cloud computing system or other computing system.

[0017] The first terminal device 30 is installed in a first space 11 where the first user 13 is located and is used by the first user 13. The first space 11 is, for example, the home or office of the first user 13. The first terminal device 30 is, for example, a general-purpose computer such as a PC, a dedicated computer specialized for specific calculations, or a mobile device such as a mobile phone, smartphone, or tablet. "PC" is an abbreviation for personal computer.

[0018] The second terminal device 40 is installed in a second space 12, which is separate from the first space 11 and where the second user 14 is located, and is used by the second user 14. The second space 12 is, for example, the home or office of the second user 14. The second terminal device 40 is, for example, a general-purpose computer such as a PC, a dedicated computer specialized for specific calculations, or a mobile device such as a mobile phone, smartphone, or tablet.

[0019] The network 50 includes the Internet, at least one WAN, at least one MAN, or any combination thereof. "WAN" is an abbreviation for wide area network. "MAN" is an abbreviation for metropolitan area network. The network 50 may include at least one wireless network, at least one optical network, or any combination thereof. The wireless network may be, for example, an ad hoc network, a cellular network, a wireless LAN, a satellite communication network, or a terrestrial microwave network. "LAN" is an abbreviation for local area network.

[0020] Referring to FIGS. 1 to 3, the outline of this embodiment will be described.

[0021] The first terminal device 30 is a terminal device that removes the background from the first image M1 captured in the first space 11 and generates a second image M2 to be displayed toward the second user 14 in the second space 12. When the first terminal device 30 detects a signal from the first user 13, it scans the object 15 in the first space 11. The first terminal device 30 transmits the scanned image M3 of the object 15 together with the second image M2 via the network 50. The object 15 is not limited to a model of a vehicle as shown in FIG. 2, and may be any physical object such as a memo or other document, or food or other merchandise.

[0022] The second terminal device 40 receives the image M3 of the object 15 scanned by the first terminal device 30 together with the second image M2 via the network 50. The second terminal device 40 displays the received image M3 of the object 15 together with the second image M2 toward the second user 14.

[0023] According to this embodiment, even if the background removal function is enabled for the person with whom the first user 13 is communicating using video, the object 15 in front of the user can be shown. For example, when the first user 13 tries to show the object 15 in front of him / her to the other party in a video conference, it is possible to avoid the object 15 being regarded as part of the background and removed.

[0024] In this embodiment, the first terminal device 30 identifies the location on the object 15 that the first user 13 is focusing on. The first terminal device 30 transmits the attention information A1 indicating the identified location via the network 50.

[0025] The second terminal device 40 receives the attention information A1 via the network 50. The second terminal device 40 displays a mark 16 at the position corresponding to the location indicated by the received attention information A1 in the image M3 of the object 15.

[0026] According to this embodiment, the first user 13 can easily convey the location on which the first user 13 is focusing to the other party with whom the first user 13 is communicating using video. For example, when the first user 13 wants to convey to the other party the location on the object 15 that the first user 13 is focusing on during a video conference, there is no need to write down the location.

[0027] In the example of FIG. 2, the first terminal device 30 uses the camera 36 to capture, as the first image M1, the video of the first space 11 including the first user 13 during the web conference. The first terminal device 30 transmits, as the second image M2, the video obtained by removing the background from the captured video to the server device 20 via the network 50. The first terminal device 30 receives, via the network 50, the video of the second space 12 including the second user 14 during the web conference, or the video obtained by removing the background from the video. The first terminal device 30 displays the received video on the display 37. When the first terminal device 30 detects a signal from the first user 13, it uses the camera 36 to scan, as the object 15 in the first space 11, the model of the vehicle that the first user 13 is focusing on. For example, when the first user 13 holds the model of the vehicle with both hands, places it in the peripheral area of both hands, looks at it, or points at it, it is considered that the first user 13 is focusing on the model of the vehicle. The first terminal device 30 transmits, as the image M3 of the object 15, the image of the scanned model of the vehicle to the server device 20 via the network 50.

[0028] The server device 20 receives, via the network 50, the video obtained by removing the background from the video of the first space 11 from the first terminal device 30, appropriately processes the received video, and then transmits it to the second terminal device 40 via the network 50. The server device 20 receives, via the network 50, the video of the second space 12 or the video obtained by removing the background from the video, appropriately processes the received video, and then transmits it to the first terminal device 30 via the network 50. When the server device 20 receives an image of a vehicle model from the first terminal device 30 via the network 50, it appropriately processes the received image and then transmits it to the second terminal device 40 via the network 50.

[0029] In the example of FIG. 3, the second terminal device 40 uses the camera 46 to capture the video of the second space 12 including the second user 14 during the web conference. The second terminal device 40 transmits, via the network 50, the captured video or the video obtained by removing the background from the captured video to the server device 20. The second terminal device 40 receives, via the network 50, the video obtained by removing the background from the video of the first space 11 including the first user 13 during the web conference from the server device 20. The second terminal device 40 displays the received video on the display 47. When the second terminal device 40 receives an image of a vehicle model from the server device 20 via the network 50, it displays the received image on the display 47.

[0030] According to the examples of FIGS. 2 and 3, in a web conference, the presenter can easily display an object 15 in front of him / her to the other party without performing a special operation with the web conferencing tool.

[0031] In the example of FIG. 2, the first terminal device 30 uses the camera 36 to detect the face orientation, line of sight, or finger orientation of the first user 13, and recognizes where the first user 13 is looking at the vehicle model. The first terminal device 30 notifies the server device 20, via the network 50, of where the first user 13 is looking at the vehicle model as the attention information A1.

[0032] When the server device 20 is notified from the first terminal device 30 via the network 50 of where the first user 13 is looking at the model of the vehicle, the server device 20 notifies the second terminal device 40 via the network 50 of where the first user 13 is looking at the model of the vehicle.

[0033] In the example of FIG. 3, when the second terminal device 40 is notified from the server device 20 via the network 50 of where the first user 13 is looking at the model of the vehicle, the second terminal device 40 displays a dashed-line frame as the mark 16 on the image of the model of the vehicle displayed on the display 47 so that the second user 14 can know where the first user 13 is looking at the model of the vehicle.

[0034] According to the examples of FIGS. 2 and 3, in a web conference, it is possible to easily make the other party recognize where the presenter is explaining the object 15 being shared with the other party without performing a special operation with the web conference tool.

[0035] The image M3 of the object 15 may be a 2D image or a 3D image. When the image M3 of the object 15 is a 3D image, as shown in FIG. 4, the second terminal device 40 may rotate or move the image M3 of the object 15 and display the position corresponding to the location indicated by the received attention information A1 of the image M3 of the object 15 toward the second user 14. According to such a modification example, even without displaying the mark 16, it is possible to easily convey to the other party with whom the first user 13 is communicating using video the location on which the first user 13 is focusing.

[0036] In the example of FIG. 4, when the second terminal device 40 is notified from the server device 20 via the network 50 of where the first user 13 is looking at the model of the vehicle, the second terminal device 40 rotates or moves the image of the model of the vehicle displayed on the display 47 so that the location on which the first user 13 is focusing on the model of the vehicle can be seen by the second user 14.

[0037] The image M3 of the object 15 scanned by the first terminal device 30 may be displayed not only on the second terminal device 40 but also on the first terminal device 30. In such a modification, the first terminal device 30 displays the scanned image M3 of the object 15 toward the first user 13.

[0038] For example, the first terminal device 30 displays an image of a vehicle model on the display 37.

[0039] As a further modification, the first terminal device 30 may identify the location on the image M3 of the object 15 being displayed on the display 37 that the first user 13 is focusing on, and transmit attention information A2 indicating the identified location via the network 50. In such a modification, the second terminal device 40 receives the attention information A2 via the network 50. The second terminal device 40 displays a mark at the position corresponding to the location on the image M3 of the object 15 indicated by the received attention information A2. Alternatively, when the image M3 of the object 15 is a three-dimensional image, the second terminal device 40 may rotate or move the image M3 of the object 15 and display the position corresponding to the location on the image M3 of the object 15 indicated by the received attention information A2 toward the second user 14.

[0040] For example, the first terminal device 30 uses the camera 36 to detect the orientation of the face, line of sight, or finger orientation of the first user 13, and recognizes where the first user 13 is focusing on the image of the vehicle model. The first terminal device 30 notifies the server device 20 via the network 50 of where the first user 13 is focusing on the image of the vehicle model as the attention information A2.

[0041] When the server device 20 is notified from the first terminal device 30 via the network 50 of where the first user 13 is focusing on the image of the vehicle model, the server device 20 notifies the second terminal device 40 via the network 50 of where the first user 13 is focusing on the image of the vehicle model.

[0042] When the second terminal device 40 is notified from the server device 20 via the network 50 of where the first user 13 is looking at the image of the vehicle model, it displays a broken-line frame as a mark on the image of the vehicle model displayed on the display 47 so that the second user 14 can know where the first user 13 is looking at the image of the vehicle model. Alternatively, when the second terminal device 40 is notified from the server device 20 via the network 50 of where the first user 13 is looking at the image of the vehicle model, it may rotate or move the image of the vehicle model displayed on the display 47 so that the portion of the image of the vehicle model that the first user 13 is looking at can be seen by the second user 14.

[0043] As yet another alternative modification, the first terminal device 30 may receive, via the network 50, attention information A3 indicating the portion of the image M3 of the object 15 that the second user 14 is looking at, and display a mark at a position corresponding to the portion indicated by the received attention information A3 on the image M3 of the object 15 displayed on the display 37. Alternatively, when the image M3 of the object 15 is a three-dimensional image, the first terminal device 30 may receive the attention information A3 via the network 50, and rotate or move the image M3 of the object 15 displayed on the display 37 so that the position corresponding to the portion indicated by the received attention information A3 of the image M3 of the object 15 is displayed facing the first user 13.

[0044] For example, the second terminal device 40 uses the camera 46 to detect the orientation of the face, line of sight, or finger orientation of the second user 14, and recognizes where the second user 14 is looking at the image of the vehicle model. The second terminal device 40 notifies the server device 20 via the network 50 of where the second user 14 is looking at the image of the vehicle model as the attention information A3.

[0045] When the server device 20 is notified from the second terminal device 40 via the network 50 of where the second user 14 is looking at the image of the vehicle model, it notifies the first terminal device 30 via the network 50 of where the second user 14 is looking at the image of the vehicle model.

[0046] When the first terminal device 30 is notified from the server device 20 via the network 50 of where the second user 14 is looking at the image of the vehicle model, the first terminal device 30 displays a broken-line frame as a mark on the image of the vehicle model being displayed on the display 37 so that the first user 13 can know where the second user 14 is looking at the image of the vehicle model. Alternatively, when the first terminal device 30 is notified from the server device 20 via the network 50 of where the second user 14 is looking at the image of the vehicle model, the first terminal device 30 may rotate or move the image of the vehicle model being displayed on the display 37 so that the first user 13 can see the location where the second user 14 is looking at the image of the vehicle model.

[0047] According to this example, in a web conference, it is possible to easily let the other party recognize where the participant is looking at the object 15 shared with the other party without performing a special operation with the web conference tool.

[0048] With reference to FIG. 1, the configuration of the first terminal device 30 according to the present embodiment will be described.

[0049] The first terminal device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, and an output unit 35.

[0050] The control unit 31 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. While controlling each part of the first terminal device 30, the control unit 31 executes processes related to the operation of the first terminal device 30.

[0051] The storage unit 32 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, ROM, or flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. The magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 32 functions as, for example, a main memory device, an auxiliary storage device, or a cache memory. Data used for the operation of the first terminal device 30 and data obtained by the operation of the first terminal device 30 are stored in the storage unit 32.

[0052] The communication unit 33 includes at least one communication interface. The communication interface is, for example, an interface corresponding to a wired LAN communication standard such as Ethernet (registered trademark), a wireless LAN communication standard such as IEEE 802.11, or a mobile communication standard such as LTE, 4G standard, or 5G standard. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 33 communicates with the server device 20. The communication unit 33 may also communicate with the second terminal device 40. The communication unit 33 receives data used for the operation of the first terminal device 30 and transmits data obtained by the operation of the first terminal device 30.

[0053] The input unit 34 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen provided integrally with the display 37, a camera 36, LiDAR, or a microphone. "LiDAR" is an abbreviation for light detection and ranging. The camera 36 is, for example, an RGB camera or a combination of an RGB camera and a depth camera. The input unit 34 receives an operation for inputting data used for the operation of the first terminal device 30. Instead of being provided in the first terminal device 30, the input unit 34 may be connected to the first terminal device 30 as an external input device. As the connection interface, an interface corresponding to a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface.

[0054] The output unit 35 includes at least one output interface. The output interface is, for example, a display 37 or a speaker. The display 37 is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescent. The output unit 35 outputs data obtained by the operation of the first terminal device 30. Instead of being provided in the first terminal device 30, the output unit 35 may be connected to the first terminal device 30 as an external output device. As the connection interface, an interface corresponding to a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0055] The functions of the first terminal device 30 are realized by executing the program according to the present embodiment with a processor as the control unit 31. That is, the functions of the first terminal device 30 are realized by software. The program causes the computer to execute the operations of the first terminal device 30, thereby causing the computer to function as the first terminal device 30. That is, the computer functions as the first terminal device 30 by executing the operations of the first terminal device 30 according to the program.

[0056] The program can be stored in a non-transitory computer-readable medium. The non-transitory computer-readable medium is, for example, a flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. The distribution of the program is performed, for example, by selling, transferring, or lending a portable medium such as an SD card, a DVD, or a CD-ROM storing the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be stored in the storage of a server and transferred from the server to another computer to distribute the program. The program may be provided as a program product.

[0057] The computer stores, for example, a program stored in a portable medium or a program transferred from a server in the main memory device once. Then, the computer reads the program stored in the main memory device with the processor and executes processing according to the read program with the processor. The computer may directly read the program from the portable medium and execute processing according to the program. The computer may sequentially execute processing according to the received program each time a program is transferred from the server to the computer. The processing may be executed by a so-called ASP-type service that realizes the function only by execution instructions and result acquisition. "ASP" is an abbreviation for application service provider. The program includes information for use in processing by an electronic computer and things conforming to the program. For example, data that is not a direct instruction to the computer but has a nature of defining the processing of the computer corresponds to "things conforming to the program".

[0058] Some or all of the functions of the first terminal device 30 may be realized by a programmable circuit or a dedicated circuit as the control unit 31. That is, some or all of the functions of the first terminal device 30 may be realized by hardware.

[0059] Referring to FIG. 1, the configuration of the second terminal device 40 according to the present embodiment will be described.

[0060] The second terminal device 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, and an output unit 45.

[0061] The control unit 41 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. The programmable circuit is, for example, an FPGA. The dedicated circuit is, for example, an ASIC. The control unit 41 executes processing related to the operation of the second terminal device 40 while controlling each part of the second terminal device 40.

[0062] The storage unit 42 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, ROM, or flash memory. The RAM is, for example, an SRAM or DRAM. The ROM is, for example, an EEPROM. The flash memory is, for example, an SSD. The magnetic memory is, for example, an HDD. The storage unit 42 functions as, for example, a main memory device, an auxiliary storage device, or a cache memory. The storage unit 42 stores data used for the operation of the second terminal device 40 and data obtained by the operation of the second terminal device 40.

[0063] The communication unit 43 includes at least one communication interface. The communication interface is, for example, an interface corresponding to a wired LAN communication standard such as Ethernet (registered trademark), a wireless LAN communication standard such as IEEE 802.11, or an interface corresponding to a mobile communication standard such as LTE, 4G standard, or 5G standard. The communication unit 43 communicates with the server device 20. The communication unit 43 may also communicate with the first terminal device 30. The communication unit 43 receives data used for the operation of the second terminal device 40 and transmits data obtained by the operation of the second terminal device 40.

[0064] The input unit 44 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen provided integrally with the display 47, a camera 46, a LiDAR, or a microphone. The camera 46 is, for example, an RGB camera or a combination of an RGB camera and a depth camera. The input unit 44 receives an operation for inputting data used for the operation of the second terminal device 40. Instead of being provided in the second terminal device 40, the input unit 44 may be connected to the second terminal device 40 as an external input device. As the connection interface, an interface conforming to a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0065] The output unit 45 includes at least one output interface. The output interface is, for example, a display 47 or a speaker. The display 47 is, for example, an LCD or an organic EL display. The output unit 45 outputs data obtained by the operation of the second terminal device 40. Instead of being provided in the second terminal device 40, the output unit 45 may be connected to the second terminal device 40 as an external output device. As the connection interface, an interface conforming to a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0066] The functions of the second terminal device 40 are realized by executing another program according to the present embodiment by a processor as the control unit 41. That is, the functions of the second terminal device 40 are realized by software. The program causes a computer to execute the operation of the second terminal device 40, thereby causing the computer to function as the second terminal device 40. That is, the computer functions as the second terminal device 40 by executing the operation of the second terminal device 40 according to the program.

[0067] Some or all of the functions of the second terminal device 40 may be realized by a programmable circuit or a dedicated circuit as the control unit 41. That is, some or all of the functions of the second terminal device 40 may be realized by hardware.

[0068] Referring to FIGS. 5 and 6, the operation of the communication system 10 according to the present embodiment will be described. This operation corresponds to the communication method according to the present embodiment.

[0069] The operations shown in FIGS. 5 and 6 are repeatedly performed while the first user 13 and the second user 14 are performing video-based communication such as a web conference using the first terminal device 30 and the second terminal device 40, respectively.

[0070] FIG. 5 shows the operation of the first terminal device 30.

[0071] In step S101, the control unit 31 uses the camera 36 as the input unit 34 to capture a first image M1 in the first space 11. In step S102, the control unit 31 removes the background from the first image M1 captured in step S101 to generate a second image M2. In step S103, the control unit 31 determines whether the image M3 of the object 15 in the first space 11 has been transmitted to the server device 20. Specifically, the control unit 31 determines that the image M3 of the object 15 has not been transmitted when the process of step S107 has not been executed yet, or when the process of step S111 has also been executed after the process of step S107 has been executed. The control unit 31 determines that the image M3 of the object 15 has been transmitted when the process of step S111 has not been executed yet after the process of step S107 has been executed. When it is determined that the image M3 of the object 15 has not been transmitted, the process of step S104 is executed. When it is determined that the image M3 of the object 15 has been transmitted, the process of step S108 is executed.

[0072] In step S104, the control unit 31 can detect a signal from the first user 13 using the microphone or camera 36 as the input unit 34. This signal may include an instruction by the voice of the first user 13, such as saying "look at this", or may include an instruction by the movement of the first user 13, such as picking up the object 15, turning the head in the direction where the object 15 is located, or pointing at the object 15. The signal may be a general pattern preset on the system side or a unique pattern arbitrarily specified by the user side. If the signal is not detected, the process of step S105 is executed. If the signal is detected, the process of step S106 is executed.

[0073] In step S105, the control unit 31 transmits the second image M2 generated in step S102 to the server device 20 via the communication unit 33. Thereafter, the process of step S101 is executed again.

[0074] In step S106, the control unit 31 scans the object 15 in the first space 11 using the camera 36 as the input unit 34. Specifically, the control unit 31 detects, as the object 15 in the first space 11, an object held by the first user 13 with their hand and scans the detected object. For example, the control unit 31 uses the camera 36 to recognize both hands of the first user 13 and detects a physical object such as a model of a vehicle between the recognized hands as the object to be scanned. Alternatively, the control unit 31 may detect, as the object 15 in the first space 11, an object that the first user 13 is bringing their hand close to and scan the detected object. For example, the control unit 31 may use the camera 36 to recognize both hands of the first user 13 and detect a physical object such as a model of a vehicle around the recognized hands as the object to be scanned. Alternatively, the control unit 31 may detect, as the object 15 in the first space 11, an object that the first user 13 is looking at and scan the detected object. For example, the control unit 31 may use the camera 36 to recognize the line of sight or the face direction of the first user 13 and detect a physical object such as a model of a vehicle at the tip of the recognized line of sight or in the direction the face is facing as the object to be scanned. Alternatively, the control unit 31 may detect, as the object 15 in the first space 11, an object that the first user 13 is holding with their hand or bringing their hand close to and looking at in order to avoid sharing unintended objects and scan the detected object. Alternatively, the control unit 31 may detect, as the object 15 in the first space 11, an object that the first user 13 is pointing at and scan the detected object. For example, the control unit 31 may use the camera 36 to recognize the finger of the first user 13 and detect a physical object such as a model of a vehicle in the direction the tip of the recognized finger is pointing as the object to be scanned. The control unit 31 may prompt the first user 13 to move the object or shift their finger or hand so that the part of the detected object hidden by the finger or hand of the first user 13 can also be captured. If the control unit 31 detects two or more objects, it may ask the first user 13 which object should be scanned.In step S107, the control unit 31 transmits the image M3 of the object 15 scanned in step S106 to the server device 20 via the communication unit 33 together with the second image M2 generated in step S102. Thereafter, the process of step S101 is executed again.

[0075] In step S108, the control unit 31 can detect another signal from the first user 13 using the microphone or the camera 36 as the input unit 34. This signal may include an instruction by the voice of the first user 13, such as "delete this", or may include an instruction by the movement of the first user 13, such as moving the object 15 out of the angle of view of the camera 36. The signal may be a general-purpose pattern preset on the system side, or may be a unique pattern arbitrarily specified by the user side. If no signal is detected, the process of step S109 is executed. If a signal is detected, the process of step S111 is executed.

[0076] In step S109, the control unit 31 uses the camera 36 as the input unit 34 to specify the location on the object 15 that the first user 13 is focusing on. Specifically, the control unit 31 specifies the location on the object 15 that the first user 13 is looking at. For example, the control unit 31 uses the camera 36 to recognize the line of sight or the face direction of the first user 13, and specifies the location on the object detected in step S106 that the recognized line of sight or face is facing. Alternatively, the control unit 31 may specify the location on the object 15 that the first user 13 is pointing at. For example, the control unit 31 uses the camera 36 to recognize the finger of the first user 13, and may specify the location on the object detected in step S106 that the tip of the recognized finger is facing. In step S110, the control unit 31 transmits the attention information A1 indicating the location specified in step S109 to the server device 20 via the communication unit 33 together with the second image M2 generated in step S102. Thereafter, the process of step S101 is executed again.

[0077] In step S111, the control unit 31 transmits an object deletion request Rq that requests to stop displaying the image M3 of the object 15, together with the second image M2 generated in step S102, to the server device 20 via the communication unit 33. After that, the process of step S101 is executed again.

[0078] When the server device 20 receives the second image M2 transmitted in step S105, step S107, step S110, or step S111 from the first terminal device 30, the server device 20 appropriately processes the received second image M2 and then transmits it to the second terminal device 40. For example, the server device 20 may convert the captured image of the first user 13 included in the second image M2 into an avatar of the first user 13 existing in the virtual three-dimensional space and then transmit it to the second terminal device 40. When the server device 20 receives the image M3 of the object 15 transmitted in step S107 from the first terminal device 30, the server device 20 appropriately processes the received image M3 of the object 15 and then transmits it to the second terminal device 40. For example, the server device 20 may convert the captured image as the image M3 of the object 15 into a three-dimensional object of the object 15 existing in the virtual three-dimensional space and then transmit it to the second terminal device 40. When the server device 20 receives the attention information A1 transmitted in step S110 from the first terminal device 30, the server device 20 appropriately processes the received attention information A1 and then transmits it to the second terminal device 40. When the server device 20 receives the object deletion request Rq transmitted in step S111 from the first terminal device 30, the server device 20 appropriately processes the received object deletion request Rq and then transmits it to the second terminal device 40.

[0079] FIG. 6 shows the operation of the second terminal device 40.

[0080] In step S201, the control unit 41 receives the second image M2 from the server device 20 via the communication unit 43. The control unit 41 may receive the image M3 of the object 15, the attention information A1, or the object deletion request Rq from the server device 20 via the communication unit 43 together with the second image M2. In step S202, the control unit 41 determines whether the image M3 of the object 15 has been received from the server device 20. Specifically, the control unit 41 determines that the image M3 of the object 15 has not been received yet, or when the object deletion request Rq has also been received after receiving the image M3 of the object 15. When the control unit 41 has not received the object deletion request Rq yet after receiving the image M3 of the object 15, it determines that the image M3 of the object 15 has been received. If it is determined that the image M3 of the object 15 has not been received, the process of step S203 is executed. If it is determined that the image M3 of the object 15 has been received, the process of step S206 is executed.

[0081] In step S203, the control unit 41 determines whether the image M3 of the object 15 was received from the server device 20 in step S201. If it is determined that the image M3 of the object 15 has not been received, the process of step S204 is executed. If it is determined that the image M3 of the object 15 has been received, the process of step S205 is executed.

[0082] In step S204, the control unit 41 displays the second image M2 received in step S201 on the display 47 as the output unit 45. That is, the control unit 41 displays the second image M2 toward the second user 14. Thereafter, the process of step S201 is executed again.

[0083] In step S205, the control unit 41 displays the image M3 of the object 15 received in step S201 on the display 47 as the output unit 45 together with the second image M2 received in step S201. That is, the control unit 41 displays the image M3 of the object 15 together with the second image M2 toward the second user 14. Thereafter, the process of step S201 is executed again.

[0084] In step S206, the control unit 41 determines whether it has received the object deletion request Rq from the server device 20 in step S201. If it is determined that the object deletion request Rq has not been received, the process of step S207 is executed. If it is determined that the object deletion request Rq has been received, the process of step S204 is executed. That is, the control unit 41 stops displaying the image M3 of the object 15 according to the object deletion request Rq received in step S201, and in step S204, only the second image M2 received in step S201 is displayed on the display 47 as the output unit 45.

[0085] In step S207, the control unit 41, in the same manner as in step S205, displays the image M3 of the object 15 received in step S203 on the display 47 as the output unit 45 together with the second image M2 received in step S201. That is, the control unit 41 displays the image M3 of the object 15 together with the second image M2 towards the second user 14. In step S208, the control unit 41 displays the mark 16 at the position corresponding to the location indicated by the attention information A1 received in step S201 in the image M3 of the object 15. For example, the control unit 41 displays a frame or a mark as the mark 16 on the image of the tangible object being displayed on the display 47 so that the second user 14 can know where the first user 13 is paying attention to a tangible object such as a model of a vehicle. After that, the process of step S201 is executed again.

[0086] With reference to FIG. 7, a modification of the operation of the second terminal device 40 will be described. Assume that the image M3 of the object 15 is a three-dimensional image.

[0087] Since the processes from step S211 to step S217 are the same as the processes from step S201 to step S207 in FIG. 6, the description thereof is omitted.

[0088] After step S207, in step S218, the control unit 41 rotates or moves the image M3 of the object 15, and displays the position corresponding to the location indicated by the attention information A1 received in step S201 of the image M3 of the object 15 toward the second user 14. For example, the control unit 41 rotates or moves the image of the physical object such as a model of a vehicle on the display 47 so that the location that the first user 13 is focusing on is visible to the second user 14. After that, the process of step S211 is executed again.

[0089] The second image M2, the image M3 of the object 15, the attention information A1, and the object deletion request Rq may be transmitted from the first terminal device 30 to the second terminal device 40. According to such a modification, the server device 20 can be omitted.

[0090] Some embodiments of the present disclosure are illustrated below. However, it should be noted that the embodiments of the present disclosure are not limited thereto. [Appendix 1] A computer that generates a second image that removes the background from a first image taken in a first space where a first user is located and is displayed toward a second user in a second space separate from the first space, when detecting a cue from the first user, scans an object in the first space, and transmits an image of the scanned object together with the second image via a network A program for executing operations including. [Appendix 2] The program according to Appendix 1, wherein the scanning includes detecting, as an object in the first space, an object held by the first user with a hand. [Appendix 3] The program according to Appendix 1, wherein the scanning includes detecting, as an object in the first space, an object that the first user is bringing a hand close to. [Appendix 4] The program according to Appendix 1, wherein the scanning includes detecting, as an object in the first space, an object that the first user is looking at. [Appendix 5] The program according to Appendix 1, wherein the scanning includes detecting an object pointed at by the first user as an object in the first space. [Appendix 6] The program according to any one of Appendices 1 to 5, wherein the cue includes an instruction by the voice of the first user. [Appendix 7] The program according to any one of Appendices 1 to 6, wherein the cue includes an instruction by the movement of the first user. [Appendix 8] The operation is identifying a location on the object that the first user is focusing on, and transmitting attention information indicating the identified location via the network to further include the program according to any one of Appendices 1 to 7. [Appendix 9] A terminal device that generates a second image by removing a background from a first image taken in a first space where a first user is located. When detecting a cue of the first user, the terminal device receives, via a network, an image of an object scanned by a terminal device that scans an object in the first space, together with the second image, and displaying the received image of the object together with the second image to a second user in a second space separate from the first space A program that causes a computer to execute an operation including. [Appendix 10] The operation further includes receiving, via the network, attention information indicating a location on the object that the first user is focusing on, and The program according to Appendix 9, wherein the displaying includes displaying a mark at a position corresponding to the location indicated by the received attention information in the image of the object. [Appendix 11] The operation further includes receiving, via the network, attention information indicating a location on the object that the first user is focusing on, and The image of the object is a three-dimensional image. The program according to appended note 9, wherein the displaying includes rotating or moving an image of the object and displaying, for the second user, a position corresponding to a location indicated by the received attention information in the image of the object, facing the second user. [Appended note 12] When a terminal device that generates a second image to be displayed for a second user in a second space separate from the first space by removing a background from a first image captured in a first space where a first user is present detects a signal from the first user, the terminal device scans an object in the first space, and transmits, from the terminal device, an image of the scanned object together with the second image via a network A communication method including the above.

[0091] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks described in the block diagrams may be integrated, or one block may be divided. Instead of executing two or more steps described in the flowchart in time series according to the description, each step may be executed in parallel or in a different order according to the processing capabilities of the device that executes each step, or as necessary. In addition, changes can be made without departing from the spirit of the present disclosure.

Explanation of reference numerals

[0092] 10 Communication system 11 First space 12 Second space 13 First user 14 Second user 15 Object 16 Mark 20 Server device 30 First terminal device 31 Control unit 32 Storage unit 33 Communication unit 34 Input unit 35 Output unit 36 Camera 37 Display 40 Second terminal device 41 Control unit 42 Memory unit 43 Communication unit 44 Input unit 45 Output unit 46 Camera 47 Display 50 Network

Claims

1. A program for causing a computer to perform operations including: removing a background from a first image of a first space including a captured image of a first user, the first image being captured in the first space where the first user is located, and generating a second image to be displayed toward a second user in a second space separate from the first space; transmitting the generated second image via a network; when detecting a signal from the first user, scanning an object in the first space; and transmitting an image of the scanned object via the network.

2. The program according to claim 1, wherein the scanning includes detecting, as an object in the first space, an object held by the first user with a hand.

3. The program according to claim 1, wherein the scanning includes detecting, as an object in the first space, an object that the first user is bringing a hand close to.

4. The program according to claim 1, wherein the scanning includes detecting, as an object in the first space, an object that the first user is looking at.

5. The program according to claim 1, wherein the scanning includes detecting, as an object in the first space, an object that the first user is pointing at.

6. The program according to claim 1, wherein the signal includes an instruction by voice of the first user.

7. The program according to claim 1, wherein the signal includes an instruction by movement of the first user.

8. The operations further include: identifying a location on the object that the first user is paying attention to; and transmitting attention information indicating the identified location via the network. The program according to claim 1.

9. A terminal device for removing a background from a first image of a first space including a captured image of a first user, the first image being captured in the first space where the first user is located, and generating a second image, and causing a computer to perform operations including: receiving, via a network, the second image generated by a terminal device that scans an object in the first space when detecting a signal from the first user; displaying the received second image toward a second user in a second space separate from the first space; receiving, via the network, an image of the object scanned by the terminal device; and displaying the received image of the object toward the second user. ​ ​ The operation further includes receiving, via the network, attention information indicating a location of the object that the first user is paying attention to. The image of the object is a three-dimensional image. The displaying includes a program of rotating or moving the image of the object and displaying the position corresponding to the location indicated by the received attention information of the image of the object toward the second user.

10. Removing a background from a first image of the first space including a captured image of the first user, which is captured in the first space where the first user is located, and transmitting, via a network, a generated second image to a second user located in a second space separate from the first space, the second image being displayed toward the second user; When the terminal device detects a signal from the first user, scanning an object in the first space; Transmitting, via the network, an image of the scanned object from the terminal device; A communication method including the above.

Images