Image processing system and image processing method

Abstract

The present invention provides an image processing system and an image processing method that make it possible to provide visual data which is generated using a 3D model, without distributing the 3D model. The image processing system comprises: a target terminal which stores a three-dimensional model that is used for generation of visual data pertaining to a target object that can be disposed in a virtual space in which an unspecified third party can participate; and an intermediation device which intermediates the target object in the virtual space. The intermediation device comprises a device-side transmission unit which transmits, to the target terminal, address information that is used for inter-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space. The target terminal comprises: a reception unit that receives viewpoint information including a line-of-sight direction in which the target object is viewed in the virtual space; a control unit that generates the visual data pertaining to the target object in the virtual space on the basis of the viewpoint information; and a transmission unit that transmits the visual data to the partner terminal through the inter-terminal communication.

Description

Image processing system and image processing method

[0001] The present invention relates to an image processing system and an image processing method.

[0002] In recent years, in generating free viewpoint videos such as 3DoF (Degree of Freedom) and 6DoF, a technology (cloud rendering) for generating free viewpoint videos on the server side has been proposed (for example, Patent Documents 1 and 2).

[0003] Special Publication No. 2020-515316 Publication Special Publication No. 2022-532302

[0004] As the use of virtual worlds continues to grow, there is also a need for users to be able to view 3D objects with unspecified third parties. Cloud rendering solves this problem, but due to high costs and other reasons, there is a need to avoid using cloud servers as much as possible or to keep them small. In addition, due to concerns about the server administrator of the cloud server using 3D objects for unintended purposes or the leakage of 3D objects, it is also necessary to consider the needs of users who do not want to use cloud servers in the first place.

[0005] Therefore, the present invention has been made to solve the above-mentioned problems, and aims to provide an image processing system and an image processing method that make it possible to provide visual data generated using a 3D model without distributing the 3D model.

[0006] The disclosed aspect is an image processing system comprising: a target terminal that stores a three-dimensional model used to generate visual data related to a target object that can be placed in a virtual space in which unspecified third parties can participate; and an intermediary device that mediates the target object in the virtual space, wherein the intermediary device comprises a device-side transmitting unit that transmits to the target terminal address information used for terminal-to-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space; and the target terminal comprises a receiving unit that receives viewpoint information including a line-of-sight direction for viewing the target object in the virtual space, a control unit that generates visual data related to the target object in the virtual space based on the viewpoint information, and a transmitting unit that transmits the visual data to the partner terminal via the terminal-to-terminal communication.

[0007] The disclosed aspect is an image processing method used in an image processing system including a target terminal that stores a three-dimensional model used to generate visual data regarding a target object that can be placed in a virtual space in which unspecified third parties can participate, and an intermediary device that mediates the target object in the virtual space, the image processing method including the steps of: the intermediary device transmitting to the target terminal address information used for terminal-to-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space; the target terminal receiving viewpoint information including a line-of-sight direction for viewing the target object in the virtual space; the target terminal generating visual data regarding the target object in the virtual space based on the viewpoint information; and the target terminal transmitting the visual data to the partner terminal via the terminal-to-terminal communication.

[0008] According to the present invention, it is possible to provide an image processing system and an image processing method that make it possible to provide visual data generated using a 3D model without distributing the 3D model.

[0009] FIG. 1 is a diagram showing an image processing system 100 according to an embodiment. FIG. 2 is a diagram showing a target terminal 10 according to an embodiment. FIG. 3 is a diagram showing an intermediary server 30 according to an embodiment. FIG. 4 is a diagram for explaining a billboard according to an embodiment. FIG. 5 is a diagram for explaining a billboard according to an embodiment. FIG. 6 is a diagram for explaining a billboard according to an embodiment. FIG. 7 is a diagram for explaining a billboard according to an embodiment. FIG. 8 is a diagram for explaining a billboard according to an embodiment. FIG. 9 is a diagram showing an image processing method according to an embodiment. FIG. 10 is a diagram showing an image processing method according to a second modified example.

[0010] Hereinafter, embodiments will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, the drawings are schematic.

[0011] [Embodiment] (Image Processing System) An image processing system according to an embodiment will be described below. Fig. 1 is a diagram showing an image processing system 100 according to an embodiment.

[0012] 1, the image processing system 100 includes a target terminal 10, a counterpart terminal 20, and an intermediary server 30. The target terminal 10, the counterpart terminal 20, and the intermediary server 30 are connected by a network 110. Although not particularly limited, the network 110 may be configured by the Internet. The network 110 may include a local area network, a mobile communication network, or a VPN (Virtual Private Network).

[0013] The target terminal 10 is a terminal used by a user who wants to show a target object (hereinafter, a 3D object). The user may be the owner of a three-dimensional model of the 3D object (hereinafter, a 3D model). The target terminal 10 may be a personal computer, a smartphone, a tablet terminal, or a cloud computer. The target terminal 10 may be any terminal that outputs visual data for displaying a 3D object. The target terminal 10 stores a 3D model of the 3D object. The 3D model may be placed in a virtual space provided by a specific server. The specific server may be the intermediary server 30 or a server other than the intermediary server 30 (for example, a server that provides a cloud service that provides a virtual space).

[0014] Here, visual data is not limited to data simply displaying an image, but may also include data such as an image with a depth map, an image with a normal map, an image with a concave / convex shape, an image with a height map, a segmented image, or a multi-layered image. Therefore, visual data does not have to be expressed in an image data format (.png, .jpg, etc.). Visual data may be converted by compression, encryption, or the like. Visual data may be used as is, or may be used after additional processing such as lighting, high resolution, or style conversion. Visual data is not a 3D model itself, and a 3D model cannot be reconstructed using the visual data alone. However, visual data may be data for displaying a rendering result of a 3D model corresponding to a certain viewpoint.

[0015] In the embodiment, the target terminal 10 is an example of a target terminal that stores three-dimensional models used to generate visual data related to 3D objects that can be placed in a virtual space in which unspecified third parties can participate.

[0016] The remote terminal 20 is a terminal used by a viewer who requests to view a 3D object. The remote terminal 20 may be a personal computer, a smartphone, or a tablet terminal. The remote terminal 20 may be a head-mounted display. The remote terminal 20 may have a function to display a 3D object based on visual data.

[0017] In the embodiment, the counterpart terminal 20 is an example of a counterpart terminal that requests viewing of a 3D object in a virtual space.

[0018] The intermediary server 30 is a server that mediates between the counterpart terminal 20 and the target terminal 10. The intermediary server 30 may provide a virtual space in which unspecified third parties can participate. The above-mentioned 3D objects may be placed in the virtual space. Here, the unspecified third parties may mean third parties who have not been authorized by the user of the target terminal 10. The authorization may be authorization regarding the use of the 3D model of the 3D object. The authorization may be indirect authorization via the operator of the intermediary server 30 (for example, authorization based on a contract between the operator and the user, or a contract between the operator and the viewer, etc.).

[0019] For example, the intermediary server 30 may display information about a 3D object in a virtual space to attract viewers (partner terminals 20) interested in the information about the 3D object. The information about the 3D object may be a sample of the 3D object. The sample may be a mesh with reduced resolution of the 3D object or an image of the 3D object. The information about the 3D object may include a description of the creator of the 3D model of the 3D object, etc.

[0020] In the embodiment, the intermediation server 30 is an example of an intermediation device that mediates 3D objects in a virtual space.

[0021] (Target Terminal) The target terminal 10 according to the embodiment will be described below. Fig. 2 is a diagram showing the target terminal 10 according to the embodiment.

[0022] As shown in FIG. 2, the target terminal 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0023] The communication unit 11 may be configured by a communication module. The communication module may be a wireless communication module conforming to a standard such as IEEE802.11a / b / g / n / ac / ax, LTE, 5G, or 6G, or may be a wired communication module conforming to a standard such as IEEE802.3.

[0024] The communication unit 11 may execute communication with the counterpart terminal 20 and the intermediation server 30. Direct communication between the target terminal 10 and the counterpart terminal 20 may be referred to as inter-terminal communication.

[0025] The communication unit 11 may receive address information used in terminal-to-terminal communication from the intermediary server 30. The address information may be the destination of visual data related to a 3D object (for example, the IP address of the other terminal 20).

[0026] The communication unit 11 may receive viewpoint information including a line of sight direction from which a 3D object is viewed in the virtual space. The viewpoint information may include a viewpoint position from which the 3D object is viewed in the virtual space. The communication unit 11 may receive viewpoint information from the other terminal 20.

[0027] The communication unit 11 may transmit visual data related to the 3D object to the partner terminal 20 through terminal-to-terminal communication. The visual data related to the 3D object may be generated by the control unit 13, which will be described later.

[0028] The storage unit 12 is configured with storage media such as a solid state drive (SSD), a hard disk drive (HDD), or a random access memory (RAM), and stores various information.

[0029] The storage unit 12 stores 3D models of 3D objects. While not particularly limited, the types of 3D models may include wireframes, surface models, and solid models. The 3D models may be 3D models that have or can have animations in their appearance or shape. The 3D models may be 3D models whose appearance or shape can be changed. The 3D models may be programs that can generate appearances or shapes without preserving the appearances or shapes. The 3D models may be programs that can generate visual data corresponding to viewpoint information by interpolating image information from multiple viewpoints, even without explicit data about the appearances or shapes. The 3D models may also be combinations of these. While not particularly limited, the 3D model data may be data managed with a specific extension (e.g., ".obj," ".ply," ".glb," etc.).

[0030] The control unit 13 may include at least one processor. The at least one processor may be configured by a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), one or more integrated circuits, one or more discrete circuits, or a combination thereof.

[0031] The control unit 13 generates visual data relating to a 3D object in the virtual space. Specifically, the control unit 13 generates visual data relating to the 3D object based on viewpoint information received from the partner terminal 20.

[0032] For example, the control unit 13 may generate visual data relating to a 3D object, which displays the 3D object as a two-dimensional image in a virtual space, based on viewpoint information received from the intermediary server 30. Such a two-dimensional image may be called a billboard. The control unit 13 generates visual data of the billboard using the 3D model stored in the storage unit 12.

[0033] Under such a premise, the control unit 13 may execute the following operations.

[0034] The control unit 13 may omit transmitting visual data to the first remote terminal and may transmit visual data to the second remote terminal. The following options are possible for the first remote terminal and the second remote terminal.

[0035] In option 1, the first remote terminal may be a remote terminal that does not satisfy a specific condition, and the second remote terminal may be a remote terminal that satisfies the specific condition. The following options are possible as the specific condition.

[0036] In option 1-1, the specific condition may be winning a lottery. The lottery may be in a format in which the counterpart terminal 20 wins if the random number assigned to the counterpart terminal 20 is a predetermined random number.

[0037] In option 1-2, the specific condition may be that the order in which the 3D objects enter the field of view of the viewer of the counterpart terminal 20 in the virtual space is within a predetermined order. The order in which the 3D objects enter the field of view of the viewer of the counterpart terminal 20 may be reset at a predetermined timing (for example, when a certain time has elapsed since the 3D objects were placed in the virtual space, or when selected by the user of the target terminal 10).

[0038] In options 1-3, the specific condition may be that the billing status of the viewer of the remote terminal 20 satisfies a condition with respect to the operator of the intermediary server 30. For example, the specific condition may be that billing for viewing a 3D object based on viewpoint information is being performed.

[0039] In option 2, the first remote terminal may be a remote terminal that transmits a viewpoint position that is farther away from a 3D object in the virtual space than a threshold, and the second remote terminal may be a remote terminal that transmits a viewpoint position that is closer to the 3D object in the virtual space than a threshold.

[0040] That is, for a first partner terminal that transmits a viewpoint position that is farther away from the 3D object than the threshold, it is not clear whether the first partner terminal is interested in the 3D object, and it is highly likely that the 3D object does not need to change based on the viewpoint information. Therefore, reducing the load on the inter-terminal communication of the target terminal 10 may be prioritized.

[0041] In the embodiment, the communication unit 11 constitutes a receiving unit that receives viewpoint information including a line of sight direction for viewing a 3D object in a virtual space. The control unit 13 constitutes a control unit that generates visual data related to the 3D object in the virtual space based on the viewpoint information. The communication unit 11 constitutes a transmitting unit that transmits the visual data to the other terminal 20 via terminal-to-terminal communication.

[0042] (Intermediate Server) The intermediate server 30 according to the embodiment will be described below. Fig. 3 is a diagram showing the intermediate server 30 according to the embodiment.

[0043] As shown in FIG. 3, the intermediation server 30 includes a communication unit 31 , a storage unit 32 , and a control unit 33 .

[0044] The communication unit 31 may be configured by a communication module. The communication module may be a wireless communication module conforming to a standard such as IEEE802.31a / b / g / n / ac / ax, LTE, 5G, or 6G, or may be a wired communication module conforming to a standard such as IEEE802.3.

[0045] The communication unit 31 communicates with the target terminal 10 and the counterpart terminal 20. The communication unit 31 may transmit data for displaying a virtual space to the target terminal 10 and the counterpart terminal 20. The communication unit 31 may receive address information of the counterpart terminal 20 from the counterpart terminal 20 that requests viewing of a 3D object. The communication unit 31 may transmit the address information of the counterpart terminal 20 to the target terminal 10 as address information to be used for terminal-to-terminal communication.

[0046] The storage unit 32 is configured by a storage medium such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), and stores various information.

[0047] The storage unit 32 stores data for displaying the virtual space. When membership registration of the user of the target terminal 10 is required, the storage unit 32 may manage membership information of the user of the target terminal 10. The membership information may include the user's name, the user's handle name, address information of the target terminal 10, etc. Similarly, when membership registration of the viewer of the counterpart terminal 20 is required, the storage unit 32 may manage membership information of the viewer of the counterpart terminal 20. The membership information may include the viewer's name, the viewer's handle name, address information of the counterpart terminal 20, etc.

[0048] The control unit 33 may include at least one processor. The at least one processor may be configured by a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), one or more integrated circuits, one or more discrete circuits, or a combination thereof.

[0049] The control unit 33 may instruct the communication unit to transmit the address information of the counterpart terminal 20 to the target terminal 10 in response to a request from the counterpart terminal 20 .

[0050] In the embodiment, the communication unit 31 constitutes a device-side transmission unit that transmits to the target terminal 10 address information used for inter-terminal communication between the target terminal 10 and the other terminal 20 that requests viewing of a 3D object in virtual space.

[0051] (Billboard) A billboard according to an embodiment will be described below. Figures 4 to 8 are diagrams for explaining the billboard according to an embodiment. The billboard may be considered to be an example of the visual data described above, and may be generated based on the visual data described above.

[0052] Here, the 3D object is a human face created using CG (Computer Graphics) or the like, and the case where the human face is superimposed on a human torso in a virtual space is taken as an example. The human torso may be a 3D model generated in advance using CG (Computer Graphics) technology.

[0053] For clarity of explanation, the following description will be given taking as an example the viewpoint from which a viewer views a 3D object in a virtual space (hereinafter referred to as the viewer viewpoint). The viewer viewpoint is identified by the viewpoint information described above. The viewer viewpoint may be considered as the viewpoint from which a viewer views a 3D object in a virtual space on an absolute coordinate axis that defines the virtual space.

[0054] In such a case, the tilt of the billboard may be changed from the viewer's viewpoint so that the billboard faces directly toward the viewer's viewpoint. That is, the tilt of the billboard is changed based on viewpoint information. On the other hand, from a viewpoint different from the viewer's viewpoint (hereinafter referred to as the other viewpoint), the tilt of the billboard may be changed based on the relative difference between the viewer's viewpoint and the other viewpoint. The tilt of the billboard may include a tilt around a vertical axis, a tilt around a horizontal axis, or a tilt around a depth axis. The tilt of the billboard may be synchronized with the tilt changed by the viewer's viewpoint. That is, the tilt of the billboard may be changed based on the relative difference between the viewer's viewpoint and the other viewpoint.

[0055] Under these assumptions, as shown in FIGS. 4 to 8, a case will be illustrated in which a billboard 210 including a 3D object 200 is superimposed on a body 410 in a virtual space 400 in an image from the viewer's viewpoint.

[0056] First, as shown in Fig. 4, a case will be described in which the viewer's viewpoint is a viewpoint in which the torso 410 is viewed from the front. The inclination of the billboard 210 is changed so that the billboard 210 faces the viewer's viewpoint. On the billboard 210, a display image of the 3D object 200 is generated based on parameters related to the viewer's viewpoint.

[0057] Second, as shown in Fig. 5, the viewer's viewpoint may be a viewpoint from which the torso 410 is viewed from approximately 30 degrees to the left. The inclination of the billboard 210 is changed so that the billboard 210 faces the viewer's viewpoint. On the billboard 210, a display image of the 3D object 200 is generated based on parameters related to the viewer's viewpoint.

[0058] Thirdly, as shown in Fig. 6, the viewer's viewpoint may be a viewpoint from which the torso 410 is viewed from approximately 30 degrees to the right. The inclination of the billboard 210 is changed so that the billboard 210 faces the viewer's viewpoint. On the billboard 210, a display image of the 3D object 200 is generated based on parameters related to the viewer's viewpoint.

[0059] Fourth, from the viewpoint of explaining the tilt of the billboard 210, we will explain an image that can be viewed from a viewpoint different from the viewer's viewpoint. As shown in Fig. 7 , when the viewer's viewpoint is a viewpoint from which the torso 410 is viewed from the front, the other viewpoint may be a viewpoint from which the torso 410 is viewed from approximately 30 degrees to the right. The tilt of the billboard 210 is changed so that the billboard 210 faces the viewer's viewpoint directly, and therefore, from the other viewpoint, the tilt of the billboard 210 is changed based on the relative difference between the viewer's viewpoint and the other viewpoint.

[0060] Fifth, in order to explain the tilt of the billboard 210, we will explain an image that can be viewed from a viewpoint different from the viewer's viewpoint. As shown in Fig. 8, when the viewer's viewpoint is a viewpoint from which the torso 410 is viewed from approximately 30 degrees to the left, the virtual space viewpoint may be a viewpoint from which the torso 410 is viewed from approximately 30 degrees to the right. The tilt of the billboard 210 is changed so that the billboard 210 faces the viewer's viewpoint directly, and therefore, at other viewpoints, the tilt of the billboard 210 is changed based on the relative difference between the viewer's viewpoint and the other viewpoints.

[0061] 4 to 8, for clarity of explanation, the billboard 210 is clearly shown in the virtual space 400, but the portion of the billboard 210 excluding the 3D object 200 may be transparent. In other words, an image of the virtual space 400 may be displayed in the portion excluding the 3D object 200.

[0062] A billboard may be displayed to a certain viewer and not to other viewers. One billboard may be prepared for one viewer.

[0063] (Image Processing Method) An image processing method according to the embodiment will be described below. Fig. 9 is a diagram showing the image processing method according to the embodiment.

[0064] As shown in FIG. 9, in step S10, the counterpart terminal 20 transmits a request to view a 3D object to the relay server 30.

[0065] In step S11, the intermediary server 30 transmits address information to the target terminal 10, which is used in direct terminal-to-terminal communication between the target terminal 10 and the counterpart terminal 20. The address information may be the IP address of the counterpart terminal 20.

[0066] In step S20, the counterpart terminal 20 transmits to the target terminal 10 viewpoint information including the line of sight direction in which the 3D object is viewed in the virtual space.

[0067] In step S21, the target terminal 10 generates visual data related to a 3D object in a virtual space. The target terminal 10 generates the visual data based on viewpoint information using a 3D model stored in the storage unit 12. The visual data related to the 3D object may be visual data that displays the 3D object as a billboard.

[0068] In step S22, the target terminal 10 transmits the visual data to the partner terminal 20 through terminal-to-terminal communication.

[0069] (Actions and Effects) In the embodiment, the target terminal 10 receives address information of the counterpart terminal 20 from the intermediary server, and transmits visual data to the counterpart terminal 20 through terminal-to-terminal communication. With this configuration, it is possible to provide visual data generated using a 3D model without distributing the 3D model stored in the target terminal 10.

[0070] In the embodiment, the target terminal 10 may generate visual data that displays a 3D object as a billboard in the virtual space based on viewpoint information transmitted from the counterpart terminal 20. With this configuration, it is possible to provide visual data generated using a 3D model without transferring the 3D model to the counterpart terminal 20.

[0071] In the embodiment, the target terminal 10 may omit transmitting visual data to the first counterpart terminal and transmit visual data to the second counterpart terminal. With this configuration, it is not necessary to transmit visual data to all counterpart terminals 20, and the communication load of terminal-to-terminal communication can be reduced.

[0072] [Modification 1] Modification 1 of the embodiment will be described below, focusing mainly on the differences from the embodiment.

[0073] In the first modification, in order to reduce the processing load on the target terminal 10, the target terminal 10 may perform the following operations.

[0074] First, the target terminal 10 may transmit visual data based on representative viewpoint information that represents two or more pieces of viewpoint information to two or more partner terminals that transmit two or more pieces of viewpoint information that satisfy a first condition. The representative viewpoint information may include a representative gaze direction. The representative gaze direction may be any gaze direction between two or more gaze directions included in each of the two or more pieces of viewpoint information, and may be any one of the two or more gaze directions, or may be an intermediate point between the two or more gaze directions. The representative viewpoint information may include a representative viewpoint position. The representative viewpoint position may be any gaze position between two or more viewpoint positions included in each of the two or more pieces of viewpoint information, and may be any one of the two or more viewpoint positions, or may be an intermediate point between the two or more viewpoint positions.

[0075] For example, the first condition may be a condition that the angle formed by two or more line of sight directions included in each of the two or more pieces of viewpoint information is within a predetermined angle. The first condition may be a condition that the distance between two or more viewpoint positions included in each of the two or more pieces of viewpoint information is equal to or less than a predetermined distance.

[0076] Second, the target terminal 10 may transmit visual data based on the specific viewpoint information to one or more specific counterpart terminals that transmit viewpoint information that satisfies a second condition for the specific viewpoint information. The specific viewpoint information may include a specific line of sight direction. The specific viewpoint information may include a specific viewpoint position. The specific viewpoint information may be set by the user of the target terminal 10. One or more specific viewpoint information may be set as the specific viewpoint information.

[0077] For example, the second condition may be that the angle formed by the line of sight direction included in the viewpoint information and the specific line of sight direction is within a predetermined angle, or that the distance between the viewpoint position included in the viewpoint information and the specific viewpoint position is equal to or shorter than a predetermined distance.

[0078] (Actions and Effects) In Modification Example 1, the target terminal 10 may transmit visual data based on representative viewpoint information that represents two or more pieces of viewpoint information to two or more partner terminals that transmit two or more pieces of viewpoint information that satisfy the first condition. With this configuration, part of the visual data can be shared, thereby reducing the processing load on the target terminal 10.

[0079] In Modification 1, the target terminal 10 may transmit visual data based on the specific viewpoint information to one or more specific partner terminals that transmit viewpoint information that satisfies a second condition for the specific viewpoint information. This configuration simplifies the generation of visual data based on viewpoint information, thereby reducing the processing load on the target terminal 10.

[0080] [Modification 2] Modification 2 of the embodiment will be described below, focusing mainly on the differences from the embodiment.

[0081] In the embodiment, the target terminal 10 generates visual data related to 3D objects for the counterpart terminal 20. In contrast, in Modification Example 2, the intermediary server 30 generates visual data related to 3D objects on behalf of some of the target terminals 10 for some of the counterpart terminals 20. That is, a case will be described in which some of the target terminals 10 use cloud rendering.

[0082] In the second modification, the 3D model stored in the target terminal 10 that uses cloud rendering is also stored in the intermediary server 30. The target terminal 10 that uses cloud rendering may be a target terminal 10 that has been charged for the use of cloud rendering.

[0083] The following describes a case where the target terminal 10 is a terminal that is permitted to use cloud rendering.

[0084] As shown in FIG. 10, in step S30, the target terminal 10 transmits the 3D model stored in the storage unit 12 to the intermediary server 30.

[0085] In step S31, the intermediary server 30 stores the 3D model received from the target terminal 10.

[0086] In step S40, the counterpart terminal 20 transmits to the relay server 30 viewpoint information including the line of sight direction in which the 3D object is viewed in the virtual space.

[0087] In step S41, the intermediary server 30 generates visual data for the 3D object based on the viewpoint information using the 3D model stored in step S31. The visual data generated by the intermediary server 30 may be visual data that displays the 3D object as a billboard.

[0088] In step S42, the intermediary server 30 transmits visual data relating to the 3D object to the partner terminal 20.

[0089] (Actions and Effects) In Modification Example 2, for target terminals 10 that use cloud rendering, generation of visual data (cloud rendering) using 3D models stored in the intermediary server 30 is permitted. With this configuration, although it is necessary to provide the 3D models to the intermediary server 30, it is possible to provide visual data generated using the 3D models to as many partner terminals 20 as the intermediary server 30's computational resources allow, without the target terminal 10 having to prepare large computational resources, while preventing unlimited outflow of 3D models.

[0090] [Modification 3] Modification 3 of the embodiment will be described below, focusing mainly on the differences from the embodiment.

[0091] In Modification 3, the intermediary server 30 may omit sending address information corresponding to the first remote terminal to the target terminal, and may send address information corresponding to the second remote terminal to the target terminal. That is, the intermediary server 30 does not send address information (intermediation process) for the first remote terminal for which the transmission of visual data is omitted, and sends address information (intermediation process) for the second remote terminal for which the transmission of visual data is executed. As with the embodiment, the following options are possible for the first remote terminal and the second remote terminal:

[0092] In option 1, the first opposite terminal may be a opposite terminal that does not satisfy the mediation condition, and the second opposite terminal may be a opposite terminal that satisfies the mediation condition. The mediation condition may be the same as the specific condition described above.

[0093] In option 2, the first remote terminal may be a remote terminal that transmits a viewpoint position that is farther away from a 3D object in the virtual space than a threshold, and the second remote terminal may be a remote terminal that transmits a viewpoint position that is closer to the 3D object in the virtual space than a threshold.

[0094] That is, for a first partner terminal that transmits a viewpoint position that is farther away from the 3D object than the threshold, it is not clear whether the first partner terminal is interested in the 3D object, and it is highly likely that the 3D object does not need to change based on the viewpoint information. Therefore, reducing the load on the inter-terminal communication of the target terminal 10 may be prioritized.

[0095] [Modification 4] Modification 4 of the embodiment will be described below, focusing mainly on the differences from the embodiment.

[0096] In the embodiment, the target terminal 10 may transmit visual data based on representative viewpoint information that represents the two or more viewpoint information to two or more partner terminals that transmit two or more viewpoint information that satisfy the first condition. In Modification Example 4, the operation shown below may be executed as a modification of such an operation.

[0097] Specifically, at least one of the two or more counterpart terminals 20 may receive visual data based on the representative viewpoint information from another counterpart terminal 20, instead of receiving visual data based on the representative viewpoint information from the target terminal 10. The other counterpart terminal 20 may be a primary counterpart terminal that receives visual data based on the representative viewpoint information from the target terminal 10, or may be a secondary counterpart terminal that receives visual data based on the representative viewpoint information from the primary counterpart terminal.

[0098] [Modification 5] Modification 5 of the embodiment will be described below, focusing mainly on the differences from the embodiment.

[0099] In the embodiment, the target terminal 10 may transmit visual data based on the specific viewpoint information to one or more specific counterpart terminals that transmit viewpoint information that satisfies the second condition for the specific viewpoint information. In Modification Example 5, the operation shown below may be executed as a modification of such an operation.

[0100] Specifically, at least one counterpart terminal 20 included in the one or more specific counterpart terminals may receive visual data based on the specific viewpoint information from another counterpart terminal 20, instead of receiving visual data based on the specific viewpoint information from the target terminal 10. The other counterpart terminal 20 may be a primary counterpart terminal that receives visual data based on the specific viewpoint information from the target terminal 10, or may be a secondary counterpart terminal that receives visual data based on the specific viewpoint information from the primary counterpart terminal.

[0101] [Other Embodiments] The present invention has been described by the above disclosure, but the descriptions and drawings that form part of the above disclosure should not be understood to limit the present invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

[0102] In the above disclosure, the viewpoint information is transmitted from the counterpart terminal 20 to the target terminal 10 by terminal-to-terminal communication. However, the above disclosure is not limited to this. In a case where the address information of the target terminal 10 is not provided to the counterpart terminal 20, the viewpoint information may be transmitted indirectly from the counterpart terminal 20 to the target terminal 10 via the intermediary server 30.

[0103] Although not specifically mentioned in the above disclosure, a 3D model stored in the target terminal 10 may be provided to a predetermined counterpart terminal 20. The predetermined counterpart terminal 20 may be a counterpart terminal 20 authorized by the user of the target terminal 10 to use the 3D model. For example, when the target terminal 10 receives a request from the counterpart terminal 20 to view a 3D object, the target terminal 10 may transmit 3D model data (e.g., data managed by an extension such as “.obj,” “.ply,” or “.glb”) to the counterpart terminal 20. With this configuration, the counterpart terminal 20 can view the 3D model (3D object) in a virtual space without transmitting viewpoint information in the processing after receiving the 3D model data. Although the 3D model is transferred to a counterpart terminal 20 authorized to use the 3D model, unlimited leakage of the 3D model can be prevented.

[0104] Although not specifically mentioned in the above disclosure, when visual data for displaying a 3D object is not transmitted from the target terminal 10 to the counterpart terminal 20, the counterpart terminal 20 may display a sample of the 3D object. The data for displaying the sample may be transmitted from the target terminal 10 or from the intermediary server 30. The sample may be a mesh of the 3D object with a reduced resolution, or an image of the 3D object.

[0105] In the above disclosure, a case has been exemplified in which a sample of a 3D object is displayed in a virtual space as information about the 3D object when recruiting viewers (partner terminals 20) who are interested in information about the 3D object. However, the above disclosure is not limited to this. Information about the 3D object does not have to be displayed in the virtual space.

[0106] Although not specifically mentioned in the above disclosure, a request for viewing may include a case where a user actively makes a request, as well as a case where the intermediary server 30 (system) automatically makes a request by monitoring the viewer's viewpoint. For example, a request for viewing may be automatically executed when the distance between the display position of information about the 3D object and the viewer's viewpoint is equal to or less than a threshold. Alternatively, a request for viewing may be automatically executed when the viewer's viewpoint enters an area in the virtual space where the 3D object is located (e.g., a room, a world, etc.).

[0107] Although not specifically mentioned in the above disclosure, omitting transmission of visual data may mean that visual data displaying a 3D object at a standard frame rate or resolution is not transmitted. For example, even in a case where transmission of visual data is omitted, visual data displaying a 3D object at a frame rate lower than the standard frame rate or at a resolution lower than the standard resolution may be transmitted.

[0108] Although not specifically mentioned in the above disclosure, a program may be provided that causes a computer to execute each process performed by the intermediary server 30. The program may also be recorded on a computer-readable medium. Using the computer-readable medium, the program can be installed on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

[0109] Alternatively, a chip may be provided that is configured by a memory that stores programs for executing the processes performed by the intermediation server 30 and a processor that executes the programs stored in the memory.

[0110] [Note] A first feature is an image processing system comprising: a target terminal that stores a three-dimensional model used to generate visual data related to a target object that can be placed in a virtual space in which unspecified third parties can participate; and an intermediary device that mediates the target object in the virtual space, wherein the intermediary device comprises a device-side transmitting unit that transmits to the target terminal address information used for terminal-to-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space; and the target terminal comprises a receiving unit that receives viewpoint information including a line-of-sight direction for viewing the target object in the virtual space; a control unit that generates visual data related to the target object in the virtual space based on the viewpoint information; and a transmitting unit that transmits the visual data to the partner terminal via the terminal-to-terminal communication.

[0111] A second feature is the image processing system of the first feature, wherein the control unit generates, as the visual data, visual data that displays the target object in the virtual space as a two-dimensional image.

[0112] A third feature is the image processing system according to the first or second feature, wherein the transmission unit omits transmission of the visual data to a first counterpart terminal and executes transmission of the visual data to a second counterpart terminal.

[0113] A fourth feature is the image processing system according to the third feature, wherein the first counterpart terminal is the counterpart terminal that does not satisfy a specific condition, and the second counterpart terminal is the counterpart terminal that satisfies the specific condition.

[0114] A fifth feature is the image processing system according to the third or fourth feature, wherein the viewpoint information includes a viewpoint position from which the target object is viewed in the virtual space, the first counterpart terminal is the counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is farther than a threshold, and the second counterpart terminal is the counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is closer than the threshold.

[0115] A sixth feature is an image processing system in which, in at least one of the first to fifth features, the transmission unit transmits the visual data based on representative viewpoint information that represents two or more viewpoint information to two or more partner terminals that transmit two or more viewpoint information that satisfy a first condition.

[0116] A seventh feature is an image processing system in which, in at least one of the first to sixth features, the transmission unit transmits the visual data based on the specific viewpoint information to one or more specific partner terminals that transmit viewpoint information that satisfies a second condition for the specific viewpoint information.

[0117] An eighth feature is an image processing system in which, in at least one of the first to seventh features, the device-side transmission unit omits transmitting the address information corresponding to a first remote terminal to the target terminal, and transmits the address information corresponding to a second remote terminal to the target terminal.

[0118] A ninth feature is the image processing system according to the eighth feature, wherein the first counterpart terminal is the counterpart terminal that does not satisfy the mediation condition, and the second counterpart terminal is the counterpart terminal that satisfies the mediation condition.

[0119] A tenth feature is the image processing system according to the eighth or ninth feature, wherein the viewpoint information includes a viewpoint position from which the target object is viewed in the virtual space, the first remote terminal is the remote terminal that transmits the viewpoint position whose distance to the target object in the virtual space is farther than a threshold, and the second remote terminal is the remote terminal that transmits the viewpoint position whose distance to the target object in the virtual space is closer than the threshold.

[0120] An eleventh feature is an image processing method used in an image processing system including a target terminal that stores a three-dimensional model used to generate visual data related to a target object that can be placed in a virtual space in which unspecified third parties can participate, and an intermediary device that mediates the target object in the virtual space, the image processing method including the steps of: the intermediary device transmitting, to the target terminal, address information used for terminal-to-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space; the target terminal receiving viewpoint information including a line-of-sight direction for viewing the target object in the virtual space; the target terminal generating visual data related to the target object in the virtual space based on the viewpoint information; and the target terminal transmitting the visual data to the partner terminal via the terminal-to-terminal communication.

[0121] 10... Target terminal, 11... Communication unit, 12... Storage unit, 13... Control unit, 20... Partner terminal, 30... Intermediary server, 31... Communication unit, 32... Storage unit, 33... Control unit, 100... Image processing system, 110... Network, 200... 3D object, 210... Billboard, 400... Virtual space, 410... Torso

Claims

1. An image processing system comprising: a target terminal that stores a three-dimensional model used to generate visual data regarding a target object that may be placed in a virtual space in which unspecified third parties can participate; and an intermediary device that mediates the target object in the virtual space, wherein the intermediary device has a device-side transmitting unit that transmits to the target terminal address information used for terminal-to-terminal communication between the target terminal and a partner terminal that requests viewing of the target object in the virtual space, and the target terminal has: a receiving unit that receives viewpoint information including a line of sight direction for viewing the target object in the virtual space; a control unit that generates visual data regarding the target object in the virtual space based on the viewpoint information; and a transmitting unit that transmits the visual data to the partner terminal via the terminal-to-terminal communication.

2. The image processing system according to claim 1, wherein the control unit generates, as the visual data, visual data that displays the target object in the virtual space as a two-dimensional image.

3. The image processing system according to claim 1, wherein the transmission unit omits transmission of the visual data to a first remote terminal and executes transmission of the visual data to a second remote terminal.

4. The image processing system according to claim 3, wherein the first counterpart terminal is the counterpart terminal that does not satisfy a specific condition, and the second counterpart terminal is the counterpart terminal that satisfies the specific condition.

5. The image processing system of claim 3, wherein the viewpoint information includes a viewpoint position from which the target object is viewed in the virtual space, the first counterpart terminal is the counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is greater than a threshold value, and the second counterpart terminal is the counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is greater than the threshold value.

6. The image processing system according to claim 1, wherein the transmission unit transmits the visual data based on representative viewpoint information representative of the two or more viewpoint information to two or more partner terminals transmitting the two or more viewpoint information satisfying a first condition.

7. The image processing system according to claim 1, wherein the transmission unit transmits the visual data based on the specific viewpoint information to one or more specific partner terminals that transmit viewpoint information that satisfies a second condition for the specific viewpoint information.

8. The image processing system of claim 1, wherein the device-side transmission unit omits transmitting the address information corresponding to a first remote terminal to the target terminal, and transmits the address information corresponding to a second remote terminal to the target terminal.

9. The image processing system according to claim 8, wherein the first counterpart terminal is the counterpart terminal that does not satisfy the mediation conditions, and the second counterpart terminal is the counterpart terminal that satisfies the mediation conditions.

10. The image processing system of claim 8, wherein the viewpoint information includes a viewpoint position from which the target object is viewed in the virtual space, the first counterpart terminal is the counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is farther than a threshold value, and the second counterpart terminal is the counterpart terminal that transmits the viewpoint position whose distance to the target object in the virtual space is closer than the threshold value.

11. An image processing method for use in an image processing system comprising a target terminal storing a three-dimensional model used to generate visual data regarding a target object that may be placed in a virtual space in which unspecified third parties can participate, and an intermediary device that mediates the target object in the virtual space, the image processing method comprising the steps of: the intermediary device transmitting to the target terminal address information used for terminal-to-terminal communication between the target terminal and a partner terminal requesting viewing of the target object in the virtual space; the target terminal receiving viewpoint information including a line of sight direction for viewing the target object in the virtual space; the target terminal generating visual data regarding the target object in the virtual space based on the viewpoint information; and the target terminal transmitting the visual data to the partner terminal via the terminal-to-terminal communication.

Images