Image processing system and image processing method

Abstract

To provide an image processing system and an image processing method that make it possible to provide visual data which is generated using a 3D model, without distributing the 3D model.SOLUTION: An image processing system comprises: a target terminal which stores a three-dimensional model that is used for generation of visual data pertaining to a target object that can be disposed in a virtual space in which an unspecified third party can participate; and a mediation device which mediates the target object in the virtual space. The mediation device comprises a device-side transmission unit which transmits, to the target terminal, address information that is used for inter-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space. The target terminal comprises: a reception unit that receives viewpoint information including a line-of-sight direction in which the target object is viewed in the virtual space; a control unit that generates the visual data pertaining to the target object in the virtual space on the basis of the viewpoint information; and a transmission unit that transmits the visual data to the partner terminal through the inter-terminal communication.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing system and an image processing method.

Background Art

[0002] In recent years, in the generation of free viewpoint videos such as 3DoF (Degree of Freedom) and 6DoF, technologies for generating free viewpoint videos on the server side (cloud rendering) have been proposed (for example, Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] As the use of virtual worlds progresses, there is also a need among users to show 3D objects to unspecified third parties. Cloud rendering solves this problem, but there is a need to avoid using cloud servers as much as possible or keep the usage to a small scale due to reasons such as high costs. In addition, due to concerns about unauthorized use of 3D objects or leakage of 3D objects by server administrators of cloud servers, it is necessary to assume the needs of users who do not want to use cloud servers in the first place.

[0005] Therefore, the present invention has been made to solve the above-described problems, and an object thereof is to provide an image processing system and an image processing method that enable the provision of visual data generated using a 3D model without distributing the 3D model.

Means for Solving the Problems

[0006] The disclosed embodiment includes a target terminal that stores a 3D model used for generating visual data related to a target object that can be placed in a virtual space where unspecified third parties can participate, and an intermediary device that mediates the target object in the virtual space. The intermediary device includes a device-side transmission unit that transmits address information used for terminal-to-terminal communication between a partner terminal that requests viewing of the target object in the virtual space and the target terminal to the target terminal. The target terminal includes a reception unit that receives viewpoint information including a line-of-sight direction for visually recognizing the target object in the virtual space, a control unit that generates visual data related to the target object in the virtual space based on the viewpoint information, and a transmission unit that transmits the visual data to the partner terminal by the terminal-to-terminal communication. It is an image processing system.

[0007] The disclosed embodiment is an image processing method used in an image processing system including a target terminal that stores a 3D model used for generating visual data related to a target object that can be placed in a virtual space where unspecified third parties can participate, and an intermediary device that mediates the target object in the virtual space. The method includes steps of: the intermediary device transmitting address information used for terminal-to-terminal communication between a partner terminal that requests viewing of the target object in the virtual space and the target terminal to the target terminal; the target terminal receiving viewpoint information including a line-of-sight direction for visually recognizing the target object in the virtual space; the target terminal generating visual data related to the target object in the virtual space based on the viewpoint information; and the target terminal transmitting the visual data to the partner terminal by the terminal-to-terminal communication.

Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing system and an image processing method that enable the provision of visual data generated using a 3D model without distributing the 3D model.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, the drawings are schematic.

[0011] [Embodiment] (Image Processing System) Hereinafter, an image processing system according to an embodiment will be described. Figure 1 is a diagram showing an image processing system 100 according to an embodiment.

[0012] As shown in FIG. 1, the image processing system 100 includes a target terminal 10, a partner terminal 20, and an intermediary server 30. The target terminal 10, the partner terminal 20, and the intermediary server 30 are connected by a network 110. Although not particularly limited, the network 110 may be constituted by the Internet. The network 110 may include a local area network, may include a mobile communication network, and may include a VPN (Virtual Private Network).

[0013] The target terminal 10 is a terminal used by a user who wants to show a target object (hereinafter, 3D object). The user may be the owner of a three-dimensional model (hereinafter, 3D model) of the 3D object. The target terminal 10 may be a personal computer, may be a smartphone, may be a tablet terminal, or may be a cloud computer. The target terminal 10 only needs to be a terminal that outputs visual data for displaying a 3D object. The target terminal 10 stores a 3D model of the 3D object. The 3D model can be arranged in a virtual space provided by a specific server. The specific server may be the intermediary server 30, or may be a server other than the intermediary server 30 (for example, a server that provides a cloud service that provides a virtual space, etc.).

[0014] Here, the visual data is not only data for displaying a simple image, but may also include data such as an image with a depth map, an image with a normal map, an uneven image, an image with a height map, a divided image, and a plurality of superimposed images. Therefore, the visual data does not have to be expressed in the data format of an image (.png,.jpg, etc.). The visual data may be converted by compression, encryption, or the like. The visual data may be used as it is, or may be used after additional processing such as writing, high resolution, and painting style conversion. The visual data is not the 3D model itself, and although the 3D model cannot be restored from the visual data alone, it only needs to be data for displaying a rendering result corresponding to a certain viewpoint of the 3D model.

[0015] In an embodiment, the target terminal 10 is an example of a target terminal that stores a three-dimensional model used for generating visual data related to a 3D object that can be placed in a virtual space where an unspecified third party can participate.

[0016] The counterpart terminal 20 is a terminal used by a viewer who requests to view a 3D object. The counterpart terminal 20 may be a personal computer, a smartphone, a tablet terminal, or a head-mounted display. The counterpart terminal 20 may have a function of displaying a 3D object based on visual data.

[0017] In an embodiment, the counterpart terminal 20 is an example of a counterpart terminal that requests to view a 3D object in a virtual space.

[0018] The mediation server 30 is a server that mediates between the counterpart terminal 20 and the target terminal 10. The mediation server 30 may provide a virtual space where an unspecified third party can participate. The above-described 3D object may be placed in the virtual space. Here, the unspecified third party may mean a third party who has not obtained permission from the user of the target terminal 10. The permission may be permission regarding the use of the 3D model of the 3D object. The permission may be an indirect permission through the operator of the mediation server 30 (for example, permission based on a contract between the operator and the user and a contract between the operator and the viewer).

[0019] For example, the mediation server 30 may recruit viewers (counterpart terminals 20) who are interested in information about the 3D object by displaying information about the 3D object in the virtual space. The information about the 3D object may be a sample of the 3D object. The sample may be a mesh with a reduced resolution of the 3D object, or an image of the 3D object. The information about the 3D object may include an explanation such as the creator of the 3D model of the 3D object.

[0020] In the embodiment, the mediation server 30 is an example of a mediation device that mediates 3D objects in a virtual space.

[0021] (Target terminal) Hereinafter, the target terminal 10 according to the embodiment will be described. FIG. 2 is a diagram showing the target terminal 10 according to the embodiment.

[0022] As shown in FIG. 2, the target terminal 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0023] The communication unit 11 may be constituted by a communication module. The communication module may be a wireless communication module compliant with standards such as IEEE802.11a / b / g / n / ac / ax, LTE, 5G, 6G, etc., or may be a wired communication module compliant with standards such as IEEE802.3.

[0024] The communication unit 11 may communicate with the partner terminal 20 and the mediation server 30. The direct communication between the target terminal 10 and the partner terminal 20 may be referred to as inter-terminal communication.

[0025] The communication unit 11 may receive address information used in inter-terminal communication from the mediation server 30. The address information may be the destination of visual data regarding the 3D object (for example, the IP address of the partner terminal 20, etc.).

[0026] The communication unit 11 may receive viewpoint information including the line-of-sight direction for visually recognizing the 3D object in the virtual space. The viewpoint information may include the viewpoint position for visually recognizing the 3D object in the virtual space. The communication unit 11 may receive the viewpoint information from the partner terminal 20.

[0027] The communication unit 11 may transmit visual data regarding the 3D object to the partner terminal 20 by inter-terminal communication. The visual data regarding the 3D object may be generated by the control unit 13 described later.

[0028] The storage unit 12 is composed of storage media such as SSD (Solid State Drive), HDD (Hard Disk Drive), and RAM (Random Access Memory), and stores various information.

[0029] The storage unit 12 stores 3D models of 3D objects. Although not particularly limited, the types of 3D models may be wireframe, surface model, solid model, etc. The 3D model may be a 3D model with or capable of having animations in appearance or shape. The 3D model may be a 3D model whose appearance or shape can be changed. The 3D model may be a program that can generate an appearance or shape even without having a pre-existing appearance or shape. The 3D model may be a program that can generate visual data corresponding to viewpoint information by interpolating image information from multiple viewpoints even without having explicit data for appearance or shape. The 3D model may be a combination of these. Although not particularly limited, the data of the 3D model may be data managed with a predetermined extension (e.g., ".obj", ".ply", ".glb", etc.).

[0030] The control unit 13 may include at least one processor. The at least one processor may be composed of a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), one or more Integrated Circuits, one or more Discrete Circuits, and combinations thereof.

[0031] The control unit 13 generates visual data regarding 3D objects in the virtual space. Specifically, the control unit 13 generates visual data regarding 3D objects based on the viewpoint information received from the counterpart terminal 20.

[0032] For example, based on the viewpoint information received from the mediation server 30 as visual data regarding the 3D object, the control unit 13 may generate visual data for displaying the 3D object as a two-dimensional image in the virtual space. Such a two-dimensional image may be referred to as a Billboard. The control unit 13 generates the visual data of the Billboard using the 3D model stored in the storage unit 12.

[0033] Under such a premise, the control unit 13 may execute the operations shown below.

[0034] The control unit 13 may omit the transmission of visual data to the first partner terminal and execute the transmission of visual data to the second partner terminal. The following options are conceivable as the first partner terminal and the second partner terminal.

[0035] In Option 1, the first partner terminal may be a partner terminal that does not satisfy a specific condition, and the second partner terminal may be a partner terminal that satisfies the specific condition. The following options are conceivable as the specific condition.

[0036] In Option 1-1, the specific condition may be winning a lottery. The lottery may be in a format where the partner terminal 20 wins when the random number assigned to the partner terminal 20 is a predetermined random number.

[0037] In Option 1-2, the specific condition may be that the order in which the 3D object enters the viewer's field of view of the partner terminal 20 in the virtual space is within a predetermined order. The order in which the 3D object enters the viewer's field of view of the partner terminal 20 may be reset at a predetermined timing (for example, the timing when a certain time has elapsed since the 3D object was placed in the virtual space, the timing selected by the user of the target terminal 10, etc.).

[0038] In Option 1-3, the specific condition may be that the billing status of the viewer of the second terminal 20 with respect to the operator of the mediation server 30 satisfies the condition. For example, the specific condition may be that billing is being performed for viewing a 3D object based on viewpoint information.

[0039] In Option 2, the first second terminal may be a second terminal that transmits a viewpoint position in the virtual space where the distance to the 3D object is farther than the threshold value, and the second second terminal may be a second terminal that transmits a viewpoint position in the virtual space where the distance to the 3D object is closer than the threshold value.

[0040] That is, for the first second terminal that transmits a viewpoint position in the virtual space where the distance to the 3D object is farther than the threshold value, it is not certain whether it is interested in the 3D object, and there is a high possibility that the 3D object may not change even based on the viewpoint information. Therefore, reduction of the load of inter-terminal communication of the target terminal 10 may be prioritized.

[0041] In the embodiment, the communication unit 11 constitutes a receiving unit that receives viewpoint information including the line-of-sight direction for visually recognizing a 3D object in the virtual space. The control unit 13 constitutes a control unit that generates visual data related to the 3D object in the virtual space based on the viewpoint information. The communication unit 11 constitutes a transmitting unit that transmits the visual data to the second terminal 20 by inter-terminal communication.

[0042] (Mediation Server) Hereinafter, the mediation server 30 according to the embodiment will be described. FIG. 3 is a diagram showing the mediation server 30 according to the embodiment.

[0043] As shown in FIG. 3, the mediation server 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

[0044] The communication unit 31 may be composed of a communication module. The communication module may be a wireless communication module compliant with standards such as IEEE802.31a / b / g / n / ac / ax, LTE, 5G, 6G, etc., or may be a wired communication module compliant with standards such as IEEE802.3.

[0045] The communication unit 31 communicates with the target terminal 10 and the partner terminal 20. The communication unit 31 may send data for displaying the virtual space to the target terminal 10 and the partner terminal 20. The communication unit 31 may receive the address information of the partner terminal 20 from the partner terminal 20 that requests viewing of the 3D object. The communication unit 31 may send the address information of the partner terminal 20 to the target terminal 10 as the address information used for communication between terminals.

[0046] The storage unit 32 is composed of a storage medium such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and stores various information.

[0047] The storage unit 32 stores data for displaying the virtual space. When membership registration of the user of the target terminal 10 is required, the storage unit 32 may manage the membership information of the user of the target terminal 10. The membership information may include the user's name, the user's handle name, the address information of the target terminal 10, etc. Similarly, when membership registration of the viewer of the partner terminal 20 is required, the storage unit 32 may manage the membership information of the viewer of the partner terminal 20. The membership information may include the viewer's name, the viewer's handle name, the address information of the partner terminal 20, etc.

[0048] The control unit 33 may include at least one processor. The at least one processor may be composed of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), one or more Integrated Circuits, one or more Discrete Circuits, and combinations thereof.

[0049] The control unit 33 may instruct the communication unit to transmit the address information of the partner terminal 20 to the target terminal 10 in response to the request of the partner terminal 20.

[0050] In the embodiment, the communication unit 31 constitutes a device-side transmission unit that transmits the address information used for inter-terminal communication between the partner terminal 20 that requests viewing of a 3D object in the virtual space and the target terminal 10 to the target terminal 10.

[0051] (Billboard) Hereinafter, the Billboard according to the embodiment will be described. FIGS. 4 to 8 are diagrams for explaining the Billboard according to the embodiment. The Billboard may be considered as an example of the above-described visual data, or may be generated based on the above-described visual data.

[0052] Here, a case where the 3D object is a human face created by CG (Computer Graphics) or the like and the human face is superimposed on the human body in the virtual space will be exemplified. The human body may be a 3D model generated in advance by CG (Computer Graphics) technology.

[0053] Also, for the sake of clarity, the viewpoint from which the viewer views the 3D object in the virtual space (hereinafter, the viewer viewpoint) will be described as an example. The viewer viewpoint is specified by the above-described viewpoint information. The viewer viewpoint may be considered as the viewpoint from which the viewer views the 3D object within the virtual space in the absolute coordinate axes that define the virtual space.

[0054] In such a case, from the viewer's perspective, the inclination of the Billboard may be changed so that the Billboard faces the viewer's perspective. That is, the inclination of the Billboard is changed based on the viewpoint information. On the other hand, from a viewpoint different from the viewer's perspective (hereinafter, another viewpoint), the inclination of the Billboard may be changed based on the relative difference between the viewer's perspective and the other viewpoint. The inclination of the Billboard may include an inclination around the vertical axis, an inclination around the horizontal axis, or an inclination around the depth axis. The inclination of the Billboard may be synchronized with the inclination changed by the viewer's perspective. That is, the inclination of the Billboard may be changed based on the relative difference between the viewer's perspective and the other viewpoint.

[0055] Under such a premise, as shown in FIGS. 4 to 8, an example of a case where a Billboard 210 including a 3D object 200 is superimposed on a torso 410 in an image from the viewer's perspective in a virtual space 400 will be illustrated.

[0056] First, as shown in FIG. 4, a case where the viewer's perspective is a perspective from which the torso 410 is viewed from the front will be described. The inclination of the Billboard 210 is changed so that the Billboard 210 faces the viewer's perspective. On the Billboard 210, a display image regarding the 3D object 200 is generated based on parameters regarding the viewer's perspective.

[0057] Second, as shown in FIG. 5, the viewer's perspective may be a perspective from which the torso 410 is viewed from approximately 30° to the left side. The inclination of the Billboard 210 is changed so that the Billboard 210 faces the viewer's perspective. On the Billboard 210, a display image regarding the 3D object 200 is generated based on parameters regarding the viewer's perspective.

[0058] Thirdly, as shown in FIG. 6, the viewer's viewpoint may be a viewpoint from which the viewer views the torso 410 from approximately 30° to the right side. The inclination of the Billboard 210 is changed so that the Billboard 210 faces the viewer's viewpoint. On the Billboard 210, the display image regarding the 3D object 200 is generated based on the parameters regarding the viewer's viewpoint.

[0059] Fourthly, from the perspective of explaining the inclination of the Billboard 210, an image that can be viewed from another viewpoint different from the viewer's viewpoint will be described. As shown in FIG. 7, when the viewer's viewpoint is a viewpoint from which the viewer views the torso 410 from the front, the other viewpoint may be a viewpoint from which the viewer views the torso 410 from approximately 30° to the right side. Since the inclination of the Billboard 210 is changed so that the Billboard 210 faces the viewer's viewpoint, in the other viewpoint, the inclination of the Billboard 210 is changed based on the relative difference between the viewer's viewpoint and the other viewpoint.

[0060] Fifthly, from the perspective of explaining the inclination of the Billboard 210, an image that can be viewed from another viewpoint different from the viewer's viewpoint will be described. As shown in FIG. 8, when the viewer's viewpoint is a viewpoint from which the viewer views the torso 410 from approximately 30° to the left side, the virtual space viewpoint may be a viewpoint from which the viewer views the torso 410 from approximately 30° to the right side. Since the inclination of the Billboard 210 is changed so that the Billboard 210 faces the viewer's viewpoint, in the other viewpoint, the inclination of the Billboard 210 is changed based on the relative difference between the viewer's viewpoint and the other viewpoint.

[0061] In FIGS. 4 to 8, for the sake of clarity of explanation, the Billboard 210 is explicitly shown in the virtual space 400, but the portion of the Billboard 210 excluding the 3D object 200 may be transparent. That is, in the portion excluding the 3D object 200, the image of the virtual space 400 may be displayed.

[0062] A billboard may be displayed for a certain viewer and does not necessarily need to be displayed for viewers other than that viewer. One billboard may be prepared for one viewer.

[0063] (Image processing method) Hereinafter, the image processing method according to the embodiment will be described. FIG. 9 is a diagram showing the image processing method according to the embodiment.

[0064] As shown in FIG. 9, in step S10, the partner terminal 20 transmits a viewing request for the 3D object to the mediation server 30.

[0065] In step S11, the mediation server 30 transmits address information used for direct inter-terminal communication between the target terminal 10 and the partner terminal 20 to the target terminal 10. The address information may be the IP address of the partner terminal 20.

[0066] In step S20, the partner terminal 20 transmits viewpoint information including the viewing direction for visually recognizing the 3D object in the virtual space to the target terminal 10.

[0067] In step S21, the target terminal 10 generates visual data regarding the 3D object in the virtual space. The target terminal 10 generates visual data based on the viewpoint information using the 3D model stored in the storage unit 12. The visual data regarding the 3D object may be visual data for displaying the 3D object as a billboard.

[0068] In step S22, the target terminal 10 transmits the visual data to the partner terminal 20 by inter-terminal communication.

[0069] (Function and effect) In the embodiment, the target terminal 10 receives the address information of the counterpart terminal 20 from the mediation server and transmits visual data to the counterpart terminal 20 via inter-terminal communication. According to such a configuration, it is possible to provide visual data generated using the 3D model without distributing the 3D model stored in the target terminal 10.

[0070] In the embodiment, the target terminal 10 may generate visual data for displaying a 3D object as a Billboard in the virtual space based on the viewpoint information transmitted from the counterpart terminal 20. According to such a configuration, it is possible to provide visual data generated using the 3D model without passing the 3D model to the counterpart terminal 20.

[0071] In the embodiment, the target terminal 10 may omit the transmission of visual data to the first counterpart terminal and execute the transmission of visual data to the second counterpart terminal. According to such a configuration, it is not necessary to transmit visual data to all the counterpart terminals 20, and the communication load of inter-terminal communication can be reduced.

[0072] [Modification Example 1] Hereinafter, Modification Example 1 of the embodiment will be described. Hereinafter, the differences from the embodiment will be mainly described.

[0073] In Modification Example 1, in order to reduce the processing load of the target terminal 10, the target terminal 10 may execute the following operations.

[0074] First, the target terminal 10 may transmit visual data based on representative viewpoint information representing two or more pieces of viewpoint information to two or more partner terminals that transmit two or more pieces of viewpoint information satisfying a first condition. The representative viewpoint information may include a representative line-of-sight direction. The representative line-of-sight direction may be a line-of-sight direction between two or more line-of-sight directions included in each of the two or more pieces of viewpoint information, may be any one of the two or more line-of-sight directions, or may be an intermediate point between the two or more line-of-sight directions. The representative viewpoint information may include a representative viewpoint position. The representative viewpoint position may be a viewpoint position between two or more viewpoint positions included in each of the two or more pieces of viewpoint information, may be any one of the two or more viewpoint positions, or may be an intermediate point between the two or more viewpoint positions.

[0075] For example, the first condition may be a condition that an angle formed by two or more line-of-sight directions included in each of the two or more pieces of viewpoint information is within a predetermined angle. The first condition may be a condition that a distance between two or more viewpoint positions included in each of the two or more pieces of viewpoint information is equal to or less than a predetermined distance.

[0076] Second, the target terminal 10 may transmit visual data based on specific viewpoint information to one or more specific partner terminals that transmit viewpoint information satisfying a second condition for the specific viewpoint information. The specific viewpoint information may include a specific line-of-sight direction. The specific viewpoint information may include a specific viewpoint position. The specific viewpoint information may be set by a user of the target terminal 10. One or more pieces of specific viewpoint information may be set as the specific viewpoint information.

[0077] For example, the second condition may be a condition that an angle formed by a line-of-sight direction included in the viewpoint information and the specific line-of-sight direction is within a predetermined angle. The second condition may be a condition that a distance between a viewpoint position included in the viewpoint information and the specific viewpoint position is equal to or less than a predetermined distance.

[0078] (Function and Effect) In Modification Example 1, the target terminal 10 may transmit visual data based on representative viewpoint information representing two or more pieces of viewpoint information to two or more partner terminals that transmit two or more pieces of viewpoint information satisfying the first condition. According to such a configuration, since a part of the visual data can be shared, the processing load on the target terminal 10 can be reduced.

[0079] In Modification Example 1, the target terminal 10 may transmit visual data based on specific viewpoint information to one or more specific partner terminals that transmit viewpoint information satisfying the second condition for the specific viewpoint information. According to such a configuration, since the generation of visual data based on the viewpoint information can be simplified, the processing load on the target terminal 10 can be reduced.

[0080] [Modification Example 2] Hereinafter, Modification Example 2 of the embodiment will be described. Hereinafter, the differences from the embodiment will be mainly described.

[0081] In the embodiment, the target terminal 10 generates visual data regarding a 3D object for the partner terminal 20. In contrast, in Modification Example 2, the mediation server 30 generates visual data regarding a 3D object for some of the partner terminals 20 instead of some of the target terminals 10. That is, a case where some of the target terminals 10 use cloud rendering will be described.

[0082] In Modification Example 2, the 3D model stored in the target terminal 10 that uses cloud rendering is also stored in the mediation server 30. The target terminal 10 that uses cloud rendering may be a target terminal 10 that has paid for the use of cloud rendering.

[0083] Hereinafter, a case where the target terminal 10 is a terminal permitted to use cloud rendering will be described.

[0084] As shown in FIG. 10, in step S30, the target terminal 10 transmits the 3D model stored in the storage unit 12 to the mediation server 30.

[0085] In step S31, the mediation server 30 stores the 3D model received from the target terminal 10.

[0086] In step S40, the partner terminal 20 transmits viewpoint information including the line-of-sight direction for visually recognizing the 3D object in the virtual space to the mediation server 30.

[0087] In step S41, the mediation server 30 generates visual data regarding the 3D object based on the viewpoint information using the 3D model stored in step S31. The visual data generated by the mediation server 30 may be visual data for displaying the 3D object as a Billboard.

[0088] In step S42, the mediation server 30 transmits the visual data regarding the 3D object to the partner terminal 20.

[0089] (Function and Effect) In Modification Example 2, for the target terminal 10 that uses cloud rendering, generation of visual data (cloud rendering) using the 3D model stored in the mediation server 30 is permitted. According to such a configuration, although it is necessary to provide the 3D model to the mediation server 30, while suppressing the unrestricted outflow of the 3D model, without the target terminal 10 preparing large computing resources, as long as the computing resources of the mediation server 30 permit, it is possible to provide visual data generated using the 3D model to as many partner terminals 20 as possible.

[0090] [Modification Example 3] Hereinafter, Modification Example 3 of the embodiment will be described. Hereinafter, the differences from the embodiment will be mainly described.

[0091] In Modification Example 3, the mediation server 30 may omit sending the address information corresponding to the first partner terminal to the target terminal and execute sending the address information corresponding to the second partner terminal to the target terminal. That is, for the first partner terminal for which the transmission of visual data is omitted, the mediation server 30 does not execute the transmission (mediation process) of the address information, and for the second partner terminal for which the transmission of visual data is executed, the mediation server 30 executes the transmission (mediation process) of the address information. As the first partner terminal and the second partner terminal, the following options can be considered as in the embodiment.

[0092] In Option 1, the first partner terminal may be a partner terminal that does not satisfy the mediation condition, and the second partner terminal may be a partner terminal that satisfies the mediation condition. The mediation condition may be the same as the specific condition described above.

[0093] In Option 2, the first partner terminal may be a partner terminal that transmits a viewpoint position in the virtual space where the distance to the 3D object is farther than the threshold, and the second partner terminal may be a partner terminal that transmits a viewpoint position in the virtual space where the distance to the 3D object is closer than the threshold.

[0094] That is, for the first partner terminal that transmits a viewpoint position in the virtual space where the distance to the 3D object is farther than the threshold, it is not certain whether it is interested in the 3D object, and it is highly possible that the 3D object does not need to change based on the viewpoint information. Therefore, reducing the load of inter-terminal communication of the target terminal 10 may be prioritized.

[0095] [Modification Example 4] Hereinafter, Modification Example 4 of the embodiment will be described. Hereinafter, the differences from the embodiment will be mainly described.

[0096] In the embodiment, the target terminal 10 may transmit visual data based on representative viewpoint information representing two or more pieces of viewpoint information to two or more partner terminals that transmit two or more pieces of viewpoint information satisfying the first condition. In Modification Example 4, the following operations may be executed as a modification of such an operation.

[0097] Specifically, at least one counterpart terminal 20 included in two or more counterpart terminals may receive visual data based on representative viewpoint information from another counterpart terminal 20 instead of receiving the visual data based on representative viewpoint information from the target terminal 10. The other counterpart terminal 20 may be a primary counterpart terminal that receives visual data based on representative viewpoint information from the target terminal 10, or may be a secondary counterpart terminal that receives visual data based on representative viewpoint information from the primary counterpart terminal.

[0098] [Modification Example 5] In the following, Modification Example 5 of the embodiment will be described. In the following, the differences from the embodiment will be mainly described.

[0099] In the embodiment, the target terminal 10 may transmit visual data based on specific viewpoint information to one or more specific counterpart terminals that transmit viewpoint information satisfying the second condition with respect to the specific viewpoint information. In Modification Example 5, the following operations may be executed as a modification of such operations.

[0100] Specifically, at least one counterpart terminal 20 included in one or more specific counterpart terminals may receive visual data based on specific viewpoint information from another counterpart terminal 20 instead of receiving the visual data based on specific viewpoint information from the target terminal 10. The other counterpart terminal 20 may be a primary counterpart terminal that receives visual data based on specific viewpoint information from the target terminal 10, or may be a secondary counterpart terminal that receives visual data based on specific viewpoint information from the primary counterpart terminal.

[0101] [Other Embodiments] Although the present invention has been described by the above disclosure, the discussions and drawings forming a part of the above disclosure should not be understood as limiting the present invention. Various alternative embodiments, examples, and operation techniques will be apparent to those skilled in the art from this disclosure.

[0102] In the above disclosure, the viewpoint information is transmitted from the partner terminal 20 to the target terminal 10 by inter-terminal communication. However, the above disclosure is not limited to this. In a case where the address information of the target terminal 10 is not provided to the partner terminal 20, the viewpoint information may be transmitted indirectly from the partner terminal 20 to the target terminal 10 via the mediation server 30.

[0103] Although not particularly mentioned in the above disclosure, the 3D model stored in the target terminal 10 may be provided to a predetermined partner terminal 20. The predetermined partner terminal 20 may be a partner terminal 20 whose use of the 3D model is permitted by the user of the target terminal 10. For example, when a request to view a 3D object is received from the partner terminal 20, the target terminal 10 may transmit the data of the 3D model (for example, data managed by extensions such as ".obj", ".ply", ".glb", etc.) to the partner terminal 20. According to such a configuration, the partner terminal 20 can view the 3D model (3D object) in the virtual space without transmitting the viewpoint information in the process after receiving the data of the 3D model. Although the 3D model will be passed to the partner terminal 20 whose use of the 3D model is permitted, an unrestricted outflow of the 3D model can be suppressed.

[0104] Although not particularly mentioned in the above disclosure, when the visual data for displaying the 3D object is not transmitted from the target terminal 10 to the partner terminal 20, the partner terminal 20 may display a sample of the 3D object. The data for displaying the sample may be transmitted from the target terminal 10 or may be transmitted from the mediation server 30. The sample may be a mesh with a reduced resolution of the 3D object or may be an image of the 3D object.

[0105] In the above disclosure, when recruiting viewers (the partner terminal 20) interested in information about a 3D object, a case was exemplified in which a sample of the 3D object was displayed in a virtual space as information about the 3D object. However, the above disclosure is not limited to this. The information about the 3D object may not be displayed in the virtual space.

[0106] Although not particularly mentioned in the above disclosure, the requirement for viewing may include cases where the mediation server 30 (system) automatically requests viewing by monitoring the viewer's viewpoint position, in addition to cases where the user actively requests. For example, when the distance between the display position of the information about the 3D object and the viewer's viewpoint position is equal to or less than a threshold value, a viewing request may be automatically executed. Alternatively, when the viewer's viewpoint position enters an area (for example, a room or a world) where the 3D object is placed in the virtual space, a viewing request may be automatically executed.

[0107] Although not particularly mentioned in the above disclosure, omitting the transmission of visual data may mean that the transmission of visual data for displaying the 3D object at a standard frame rate or resolution is not performed. For example, even in a case where the transmission of visual data is omitted, the transmission of visual data for displaying the 3D object at a frame rate lower than the standard frame rate or a resolution lower than the standard resolution may be performed.

[0108] Although not particularly mentioned in the above disclosure, a program may be provided to cause a computer to execute each process performed by the mediation server 30. Further, the program may be recorded on a computer-readable medium. By using a computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

[0109] Alternatively, a chip may be provided that includes a memory for storing a program for executing each process performed by the mediation server 30 and a processor for executing the program stored in the memory.

[0110] [Appendix] The first feature is an image processing system comprising: a target terminal that stores a three-dimensional model used for generating visual data regarding a target object that can be placed in a virtual space where an unspecified third party can participate; and a mediation device that mediates the target object in the virtual space, wherein the mediation device includes a device-side transmission unit that transmits address information used for inter-terminal communication between a partner terminal that requests viewing of the target object in the virtual space and the target terminal to the target terminal, and the target terminal includes a reception unit that receives viewpoint information including a line-of-sight direction for visually recognizing the target object in the virtual space, a control unit that generates visual data regarding the target object in the virtual space based on the viewpoint information, and a transmission unit that transmits the visual data to the partner terminal by the inter-terminal communication.

[0111] The second feature is an image processing system according to the first feature, wherein the control unit generates visual data for displaying the target object as a two-dimensional image in the virtual space as the visual data.

[0112] The third feature is an image processing system according to the first or second feature, wherein the transmission unit omits transmission of the visual data to a first partner terminal and executes transmission of the visual data to a second partner terminal.

[0113] The fourth feature is an image processing system according to the third feature, wherein the first partner terminal is the partner terminal that does not satisfy a specific condition, and the second partner terminal is the partner terminal that satisfies the specific condition.

[0114] The fifth feature is that in the third or fourth feature, the viewpoint information includes a viewpoint position for visually recognizing the target object in the virtual space, the first partner terminal is the partner terminal that transmits the viewpoint position in the virtual space where the distance to the target object is farther than a threshold value, and the second partner terminal is the partner terminal that transmits the viewpoint position in the virtual space where the distance to the target object is closer than the threshold value, which is an image processing system.

[0115] The sixth feature is that in at least any one of the first to fifth features, the transmission unit transmits the visual data based on representative viewpoint information representing the two or more pieces of viewpoint information to two or more partner terminals that transmit two or more pieces of viewpoint information satisfying a first condition, which is an image processing system.

[0116] The seventh feature is that in at least any one of the first to sixth features, the transmission unit transmits the visual data based on the specific viewpoint information to one or more specific partner terminals that transmit viewpoint information satisfying a second condition for the specific viewpoint information, which is an image processing system.

[0117] The eighth feature is that in at least any one of the first to seventh features, the device-side transmission unit omits the transmission to the target terminal of the address information corresponding to the first partner terminal and executes the transmission to the target terminal of the address information corresponding to the second partner terminal, which is an image processing system.

[0118] The ninth feature is that in the eighth feature, the first partner terminal is the partner terminal that does not satisfy the mediation condition, and the second partner terminal is the partner terminal that satisfies the mediation condition, which is an image processing system.

[0119] The tenth feature is that in the eighth or ninth feature, the viewpoint information includes the viewpoint position for visually recognizing the target object in the virtual space, the first partner terminal is the partner terminal that transmits the viewpoint position in the virtual space where the distance to the target object is farther than the threshold value, and the second partner terminal is the partner terminal that transmits the viewpoint position in the virtual space where the distance to the target object is closer than the threshold value, and it is an image processing system.

[0120] The eleventh feature is an image processing method used in an image processing system including a target terminal that stores a 3D model used for generating visual data regarding a target object that can be arranged in a virtual space where an unspecified third party can participate, and a mediation device that mediates the target object in the virtual space, the method including: a step in which the mediation device transmits address information used for inter-terminal communication between a partner terminal that requests viewing of the target object in the virtual space and the target terminal to the target terminal; a step in which the target terminal receives viewpoint information including a line-of-sight direction for visually recognizing the target object in the virtual space; a step in which the target terminal generates visual data regarding the target object in the virtual space based on the viewpoint information; and a step in which the target terminal transmits the visual data to the partner terminal by the inter-terminal communication.

Explanation of Signs

[0121] 10... Target terminal, 11... Communication unit, 12... Storage unit, 13... Control unit, 20... Partner terminal, 30... Mediation server, 31... Communication unit, 32... Storage unit, 33... Control unit, 100... Image processing system, 110... Network, 200... 3D object, 210... Billboard, 400... Virtual space, 410... Torso

Claims

1. A target terminal that stores a 3D model used for generating visual data related to a target object that can be placed in a virtual space where an unspecified third party can participate, and an intermediary device that mediates the target object in the virtual space, comprising: The intermediary device includes a device-side transmission unit that transmits address information used for terminal-to-terminal communication between a partner terminal that requests viewing of the target object in the virtual space and the target terminal to the target terminal, The target terminal, a receiving unit that receives viewpoint information including a line-of-sight direction for visually recognizing the target object in the virtual space, a control unit that generates visual data related to the target object in the virtual space based on the viewpoint information, and a transmission unit that transmits the visual data to the partner terminal by the terminal-to-terminal communication. An image processing system.

2. The control unit generates, as the visual data, visual data for displaying the target object as a two-dimensional image in the virtual space. The image processing system according to claim 1.

3. The transmission unit omits transmission of the visual data to a first partner terminal and executes transmission of the visual data to a second partner terminal. The image processing system according to claim 1.

4. The first partner terminal is the partner terminal that does not satisfy a specific condition, The second partner terminal is the partner terminal that satisfies the specific condition. The image processing system according to claim 3.

5. The viewpoint information includes a viewpoint position for visually recognizing the target object in the virtual space, The first partner terminal is the partner terminal that transmits a viewpoint position in the virtual space where the distance to the target object is farther than a threshold value, The second partner terminal is the partner terminal that transmits a viewpoint position in the virtual space where the distance to the target object is closer than the threshold value. The image processing system according to claim 3.

6. The transmission unit transmits the visual data based on representative viewpoint information representing two or more pieces of viewpoint information to two or more partner terminals that transmit two or more pieces of viewpoint information satisfying a first condition. The image processing system according to claim 1.

7. The transmission unit transmits the visual data based on the specific viewpoint information to one or more specific partner terminals that transmit viewpoint information satisfying a second condition for the specific viewpoint information. The image processing system according to claim 1.

8. The apparatus-side transmission unit omits transmission of the address information corresponding to the first partner terminal to the target terminal, and executes transmission of the address information corresponding to the second partner terminal to the target terminal. The image processing system according to claim 1.

9. The first partner terminal is the partner terminal that does not satisfy the mediation condition, The second partner terminal is the partner terminal that satisfies the mediation condition. The image processing system according to claim 8.

10. The viewpoint information includes a viewpoint position for visually recognizing the target object in the virtual space, The first partner terminal is the partner terminal that transmits the viewpoint position in the virtual space where the distance to the target object is farther than a threshold value, The second partner terminal is the partner terminal that transmits the viewpoint position in the virtual space where the distance to the target object is closer than the threshold value. The image processing system according to claim 8.

11. An image processing method used in an image processing system including a target terminal that stores a 3D model used for generating visual data related to a target object that can be arranged in a virtual space where an unspecified third party can participate, and a mediation device that mediates the target object in the virtual space, comprising: a step in which the mediation device transmits address information used for inter-terminal communication between a partner terminal that requests viewing of the target object in the virtual space and the target terminal to the target terminal; a step in which the target terminal receives viewpoint information including a line-of-sight direction for visually recognizing the target object in the virtual space; a step in which the target terminal generates visual data related to the target object in the virtual space based on the viewpoint information; a step in which the target terminal transmits the visual data to the partner terminal by the inter-terminal communication. An image processing method.

Images