Image processing device, image processing method, and image display system

Abstract

An image processing device according to an embodiment of the present technology is provided with a division processing unit and a transmission unit. The division processing unit divides 3D model data to be displayed on a tiling display into first partial model data and second partial model data corresponding to a first display and a second display, respectively, of the tiling display on the basis of the positional relationship between the first display and the second display and a user's field of view of the tiling display. The transmission unit transmits the first partial model data and the second partial model data to a first display processing device and a second display processing device, respectively, that are configured to control the first display and the second display, respectively.

Description

Image processing device, image processing method, and image display system

[0001] The present technology relates to an image processing device, an image processing method, and an image display system.

[0002] Patent Literature 1 discloses a tiling display in which multiple unit panels are linked together to operate as a single display. In this tiling display, pixels are arranged so that the distance between adjacent pixels across unit panels is equal to or greater than the distance between adjacent pixels within the same unit panel. This makes it possible to display images in such a way that the seams are not noticeable when viewed.

[0003] Japanese Patent Application Laid-Open No. 2021-103209

[0004] In such a tiling display, there is a demand for a technique that can reduce the processing load of rendering.

[0005] In view of the above circumstances, an object of the present technology is to provide an image processing device, an image processing method, and an image display system that are capable of reducing the processing load of rendering.

[0006] To achieve the above object, an image processing device according to one aspect of the present technology includes a segmentation processing unit and a transmission unit. The segmentation processing unit segments 3D model data displayed on the tiling display into first partial model data and second partial model data corresponding to the first display and the second display, respectively, based on a positional relationship between a first display and a second display of the tiling display and a field of view of a user of the tiling display. The transmission unit transmits the first partial model data and the second partial model data to a first display processing device and a second display processing device configured to control the first display and the second display, respectively.

[0007] An image processing method according to one embodiment of the present technology includes dividing 3D model data displayed on a tiling display into first partial model data and second partial model data corresponding to the first display and the second display, respectively, based on a positional relationship between a first display and a second display of the tiling display and a field of view of a user of the tiling display, and transmitting the first partial model data and the second partial model data to a first display processing device and a second display processing device configured to control the first display and the second display, respectively.

[0008] According to an embodiment of the present technology, there is provided an image display system including the first display, the second display, the first display processing device, the second display processing device, and the image processing device, wherein the first display processing device and the second display processing device perform culling processing and rendering processing on the received first partial model data and the second partial model data, and control display of the first partial model data and the second partial model data after the culling processing and the rendering processing on the first display and the second display.

[0009] FIG. 1 is a schematic diagram showing a state in which a 3D model is displayed on a tiling display. FIG. 1 is a schematic diagram showing an example of the configuration of an image display system. FIG. 2 is a schematic diagram showing a 3D model, model placement information, and display position information. FIG. 3 is a schematic diagram showing animation data. FIG. 4 is a schematic diagram showing viewing range information. FIG. 5 is a flowchart showing an example of a process for calculating a visibility volume. FIG. 6 is a schematic diagram showing a process for projecting vertices onto a display. FIG. 7 is a flowchart showing an example of a process for splitting bones. FIG. 8 is a schematic diagram showing bone information. FIG. 9 is a flowchart showing an example of a process for removing vertices. FIG. 10 is a schematic diagram showing a division process and a removal process. FIG. 11 is a flowchart showing an example of a process for setting boundary information. FIG. 12 is a schematic diagram showing boundary information. FIG. 13 is a schematic diagram showing an optimized 3D model. FIG. 14 is a schematic diagram showing a culling process, a rendering process, and a display control process. FIG. 15 is a schematic diagram showing a culling process, a rendering process, and a display control process. FIG. 16 is a schematic diagram showing an image display system of a comparative example. FIG. 17 is a block diagram showing an example of the hardware configuration of a computer that can realize a parent PC and a child PC.

[0010] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0011] [Image Display System] Fig. 1 is a schematic diagram showing a state in which a 3D model is displayed on a tiling display. As shown in Fig. 1, in an image display system 100 according to this embodiment, a 3D model 1 is displayed on multiple displays. The 3D model 1 is an image that can be viewed stereoscopically, and a user can view the 3D model 1 in three dimensions by, for example, wearing dedicated glasses while viewing the display. Alternatively, a naked-eye stereoscopic display may be used as the display, allowing the user to view the 3D model in three dimensions without wearing dedicated equipment.

[0012] In this example, a human model is displayed as the 3D model 1, but the type of 3D model 1 is not limited, and an object such as a ball or an apple may be displayed. The 3D model 1 corresponds to an embodiment of 3D model data related to the present technology.

[0013] In this example, four displays 15, 25, 35, and 45 are arranged vertically, and these display one 3D model 1 in cooperation with one another. Specifically, the top display 15 displays the head and hands of a person. The second display 25 displays the upper half of the person's body. The remaining displays 35 and 45 also display parts of the person. A display that displays one image using multiple unit panels like this is called a tiling display, and is used as a means for displaying large images without using a huge panel.

[0014] In this embodiment, one PC (Personal Computer) is provided for each display. That is, image display control for each display is not performed by a single common PC, but rather display control is performed individually by individual PCs. Because the volume of image data to be rendered in a tiling display is extremely large, a configuration is often adopted in which a dedicated PC is provided for each display, as in this example, and the rendering process is distributed among the displays.

[0015] PC1 (slave PC 10) is provided for the top display 15. Slave PC 10 performs culling and rendering processes on the 3D model 1 and controls the display of the processed image on display 15. Similarly, PCs 2, 3, and 4 (slave PCs 20, 30, and 40) are provided for displays 25, 35, and 45, respectively.

[0016] Fig. 2 is a schematic diagram showing an example configuration of an image display system 100. The image display system 100 has a parent PC 50, multiple child PCs, and multiple displays. In the example of Fig. 1, the image display system 100 has four child PCs 10 to 40, but the specific number of child PCs is not limited, and may be any number depending on the number of displays. Fig. 2 shows multiple child PCs including child PCs 10 and 20.

[0017] As with the slave PCs, the number of displays is not limited. While displays 15 to 45 are arranged vertically in FIG. 1, they may be arranged horizontally or in a grid pattern. The shape of the displays is not limited to a rectangular shape. Figure 2 shows multiple displays, including displays 15 and 25.

[0018] The parent PC 50 has a display unit 51, an operation unit 52, a communication unit 53, a storage unit 54, and a controller 56. These blocks are interconnected via a bus 55. Instead of the bus 55, the blocks may be connected using a communication network or a non-standardized proprietary communication method.

[0019] The display unit 51 is a display device using, for example, a liquid crystal display, an electroluminescence display, or the like, and displays various images showing the status of the parent PC 50, a GUI (Graphical User Interface) for setting, and the like.

[0020] The operation unit 52 is, for example, a keyboard, a pointing device, a touch panel, or other operation device. When the operation unit 52 includes a touch panel, the touch panel can be integrated with the display unit 51.

[0021] The communication unit 53 is a communication module for communicating with other devices via a network such as a local area network (LAN) or a wide area network (WAN). A wireless LAN module such as Wi-Fi or a communication module for short-range wireless communication such as Bluetooth (registered trademark) may be provided. Alternatively, a communication device such as a modem or a router may be used.

[0022] The storage unit 54 is a storage device such as a non-volatile memory, and may be, for example, a hard disk drive (HDD) or a solid state drive (SSD). Any other computer-readable, non-transitory storage medium may also be used. The storage unit 54 stores a control program for controlling the overall operation of the parent PC 50. The method for installing the control program on the parent PC 50 is not limited. For example, the program may be installed via various recording media, or via the Internet, for example.

[0023] The controller 56 includes hardware necessary for configuring a computer, such as a processor such as a CPU, a GPU, or a DSP, memories such as a ROM or a RAM, and a storage device such as an HDD. For example, the CPU loads a program according to the present technology, which is pre-recorded in a ROM or the like, into a RAM and executes the program, thereby executing the image processing method according to the present technology. As the controller 56, for example, a programmable logic device (PLD) such as a field programmable gate array (FPGA), or other devices such as an application specific integrated circuit (ASIC), may be used.

[0024] In the present embodiment, the CPU of the controller 56 executes a program (e.g., an application program) according to the present technology to realize functional blocks including a visibility volume calculation unit 57, a division and removal unit 58, and a boundary information setting unit 59. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

[0025] The visibility volume calculation unit 57 calculates the visibility volume. The visibility volume is a parameter that indicates to what extent each part of the 3D model 1 is displayed. For simplicity, the visibility volume and the visibility volume calculation unit 57 may be abbreviated as VV and the VV calculation unit 57. The division and removal unit 58 generates an optimized 3D model. The boundary information setting unit 59 generates boundary information. Specific contents and generation methods of the VV, the optimized 3D model, and the boundary information will be described later. The parent PC 50 corresponds to an embodiment of an image processing device according to the present technology.

[0026] The child PC 10 has a controller 11. Although not shown, the child PC 10 may also have mechanisms such as a display unit, similar to the parent PC 50. In this embodiment, a CPU of the controller 11 executes a program according to the present technology, thereby realizing a culling processing unit 12, a rendering processing unit 13, and a display control unit 14 as functional blocks.

[0027] The culling processing unit 12 performs culling processing on the optimized 3D model. The rendering processing unit 13 performs rendering processing on the optimized 3D model. The display control unit 14 controls the display of the optimized 3D model that has been subjected to the culling processing and rendering processing on the display 15. The specific processing contents of these functional blocks will be described later.

[0028] The slave PC 20 has the same configuration as the slave PC 10. That is, the slave PC 20 has a controller 21, which implements a culling processing unit 22, a rendering processing unit 23, and a display control unit 24. The other slave PCs also have similar configurations. The slave PC 10 corresponds to an embodiment of a first display processing device according to the present technology. The slave PC 20 corresponds to an embodiment of a second display processing device according to the present technology.

[0029] The display 15 is, for example, a display panel such as a liquid crystal display or an EL (Electro-Luminescence) display. The display 25 and the other displays have a similar configuration. The display 15 corresponds to an embodiment of a first display according to the present technology. The display 25 corresponds to an embodiment of a second display according to the present technology. The displays 15, 25, and the other displays configure a tiling display according to the present technology.

[0030] If audio is output along with the video display, a speaker may be provided. Furthermore, since the image display system 100 displays the 3D model 1, it is necessary to change the display position of the image according to the user's viewing position. A camera or depth sensor may be provided to acquire the user's viewing position.

[0031] The following describes specific processing performed by the parent PC 50. In this embodiment, pre-processing (offline processing) is performed by the parent PC 50. That is, predetermined processing is performed in advance before the user uses the image display system 100 (before an image is displayed on each display).

[0032] Specifically, an optimized 3D model is generated by the parent PC 50. The optimized 3D model is a 3D model 1 optimized for each display, and a different optimized 3D model is generated for each display. That is, the parent PC 50 executes a process for generating an optimized 3D model for display 15, a process for generating an optimized 3D model for display 25, and so on. The process for generating an optimized 3D model for display 15 will be described below, but the process for generating an optimized 3D model for other displays such as display 25 is similar.

[0033] [Calculation of Visibility Volume] The following describes the processing performed by the VV calculation unit 57 (method of calculating VV). The VV calculation unit 57 receives the 3D model 1, model placement information M, display position information D, animation data A, and viewing range information W as input, and outputs VV. First, the contents of this information and parameters will be described.

[0034] 3 is a schematic diagram showing a 3D model 1, model placement information M, and display position information D. The 3D model 1 is image data displayed by the image display system 100. The model placement information M is information about the placement of the 3D model 1, and M pos , M rot , M scale It consists of three types of parameters:

[0035] M pos is a parameter representing the position of the 3D model 1, and M posx , M posy , M posz In the world space (the space in which the image display system 100 exists), a three-dimensional coordinate system consisting of x-axis, y-axis, and z-axis is defined in advance, and the x-coordinate of the 3D model 1 in this coordinate system is M posx Similarly, the y coordinate is M posy , z coordinate is M posz In this example, the coordinates of the tip of the foot of the 3D model 1 are expressed as M pos The present invention is not limited to this, and the position M of the 3D model 1 may be determined based on any arbitrary position such as the center of the head or body. pos may be defined.

[0036] M rot is a parameter that indicates how much the 3D model 1 is rotated, and M rotx , M roty , M rotz The rotation of the 3D model 1 in the x direction, y direction, and z direction is represented by M rotx , M roty , M rotz It is expressed as M scale is a parameter representing the scale of the 3D model 1, and M scalex , M scaley, M scalez The scales of the 3D model 1 in the x, y, and z directions are M scalex , M scaley , M scalez It is expressed as M scalex , M scaley , M scalez For example, each of the values ​​ranges from 0 to 1.

[0037] The display position information D is information about the position of the display 15. size , D pos , D rot , D scale It consists of four types of parameters. size is a parameter representing the size of the display 15, and D sizex , D sizey In this example, the display 15 is rectangular, and the horizontal and vertical lengths of the display 15 are D sizex , D sizey It is expressed as follows. pos , D rot , D scale Regarding M pos , M rot , M scale These parameters represent the same content as the coordinates, rotation, and scale of the display 15. pos may be defined based on any position of the display 15, such as the center, a side, or a corner.

[0038] FIG. 4 is a schematic diagram showing animation data A. Animation data A j,t is a parameter that represents time-series information of the three-dimensional position of the joint of the bone of the 3D model 1. In this example, the joint of the 3D model 1 is J min ≦j≦J max That is, the 3D model 1 is expressed as j = J min , J min +1, ..., J max -1, J max The total of J max -J min +1 joints. Similarly, when time is T min ≦t≦Tmax It is expressed as:

[0039] Animation Data A j,t depends on joint j and time t, and A posx , A posy , A posz The x-coordinate, y-coordinate, and z-coordinate of joint j at time t are respectively A posx , A posy , A posz For example, if the position of joint j=2 at time t=3 is (10, 20, 30), then A 2,3 = (10, 20, 30).

[0040] 5 is a schematic diagram showing viewing range information W. The viewing range information W is the range of the user's head position that can be tracked by the image display system 100. In this embodiment, for example, the user's head is photographed with a camera to acquire the head position, but the range of positions that can be acquired by the camera is limited due to the camera's angle of view. Similarly, when a depth sensor or the like is used, the range of positions that can be acquired is also limited. This range of positions that can be acquired (trackable range) becomes the viewing range information W.

[0041] Viewing range information W is W x , W y , W z In the example shown in FIG. 5, the trackable range is a rectangular parallelepiped range, and this rectangular parallelepiped range is the viewing range information W. In this case, W x , W y , W z Is W minx ≦W x ≦W maxx , W miny ≦W y ≦W maxy , W minz ≦W z ≦W maxz Of course, the viewing range information W is not limited to a rectangular parallelepiped range, and can be a range of various shapes depending on the position range that the image display system 100 can track.

[0042] Also, vertices are defined for the 3D model 1. For example, the top of the head, elbows, knees, and tips of the arms and legs of the 3D model 1 are defined as vertices. Of course, the specific positions of the vertices are not limited. Each vertex has an index v. min ≦v≦V max Furthermore, vertex information is defined for each vertex. The vertex information is a parameter that represents the three-dimensional position of each vertex at time t, and is expressed as {V v,t (V posx ,V posy ,V posz )|V min ≦v≦V max ∩T min ≦t≦T max}.

[0043] VV is a single-value (scalar) parameter specified for each vertex, and {VisibilityVolume v |V min ≦v≦V max} Note that VV does not depend on time t.

[0044] 6 is a flowchart showing an example of a Visibility Volume calculation process. Before the process of step 101 is executed, the VV calculation unit 57 first acquires the 3D model 1, model placement information M, display position information D, animation data A, and viewing range information W. This information is stored in advance in, for example, the storage unit 54. Thereafter, the processes of steps 101 to 109 are all executed by the VV calculation unit 57.

[0045] VV is initialized to 0 (step 101). VV at all vertices is initialized to 0 by the VV calculation unit 57. As shown in FIG. 6, the subsequent processing of steps 103 to 108 is performed for each vertex while the vertex index v is incremented by 1. Similarly, the processing of steps 104 to 107 is performed for each coordinate within the viewing range. Similarly, the processing of steps 105 and 106 is performed for each time t. In other words, the processing of steps 105 and 106 is performed for all times, all coordinates within the viewing range, and all vertices.

[0046] The vertex index is set to v=0 (step 102). In this example, V min = 0, i.e., v is first set to the minimum value. x ,W y ,W z ) = (W minx ,W miny ,W minz ) (Step 103). min (Step 104). First, the processes of steps 105 and 106 are executed for these vertex indices, coordinates, and times.

[0047] 7 is a schematic diagram showing the process of projecting vertices onto the display. In the process of step 105, the process of projecting vertices onto the display is executed. Specifically, when v=0 and t=T min The vertex information V of the vertex 0,Tmin , viewing position (W minx ,W miny ,W minz ), display position information D size , D pos , D rot , D scale Based on this, the time t=T min The viewing position is (W minx ,W miny ,W minz ), the projected position (two-dimensional coordinates) of the vertex at v=0 onto the display 15 is calculated.

[0048] In the process of step 106, it is determined whether the projection position is within the display 15. The determination is made based on the display position information D and the calculated projection position. FIG. 7 shows a schematic diagram of the state in which the projection position is within the display 15. In this case, VV, which is VV at vertex v=0, ０ On the other hand, if the projection position is not within the display 15, VV ０ Do not change the value of

[0049] 7 is merely a schematic diagram for the purpose of explanation, and does not actually project the 3D model 1 onto the display 15 or perform processing to determine whether or not the vertices are projected using a camera or the like. The projection processing is an internal processing that is realized only by numerical calculations by the parent PC 50.

[0050] It is determined whether or not the processing of the animation data A has been completed (step 107). max If the time t has not yet reached (No in step 107), t is incremented by 1 and the processes in steps 105 and 106 are executed again. min +1 、 Viewing position (W minx ,W miny ,W minz ) at vertex V 0,Tmin+1 Hereafter, when t is T max The projection process is repeated until

[0051] t is T max If the result of step 107 is "Yes", it is determined whether the processing within the viewing range has been completed (step 108). If the processing has not been completed at all viewing positions within the viewing range (step 108, "No"), the next viewing position, for example (W minx ,W miny ,W minz +1), the processing of steps 104 to 107 is executed. That is, the projection processing is executed at all viewing positions within the viewing range.

[0052] When the process is completed for all viewing positions (Yes in step 108), the projection process will have been performed (number of viewing positions) x (number of times). ０ The number to which 1 is added is the VV at that time. ０ The value is

[0053] Below, a processing example will be explained using specific values. Note that the values ​​used in the explanation are merely examples, and the values ​​in the actual processing are not limited to these. (Example) Range of time t: 1≦t≦10 Range of viewing position: 11≦W x ≦20, 21≦Wy ≦40, 11≦W z ≦50

[0054] In this case, the number of times of processing for time t is 10, and the number of times of processing for the viewing position is 8000 (W x： 10 times, W y： 20 times, W z： 40 times). That is, the total number of processes (number of projections) is the product of these, 80,000 times. For example, if the projection position was within the display 15 20,000 times, VV ０ Since the process of adding 1 to is performed 20,000 times, VV ０ The value is 20000.

[0055] It is determined whether all vertices have been processed (step 109). max If v does not reach V (No in step 109), 1 is added to v, and the processing in steps 103 to 108 is executed again for the next vertex. max If it has reached (Yes in step 109), the process ends.

[0056] In this way, VV at each vertex v ｖ In this example, the processing is performed in order from the smallest time t or vertex v, but the specific processing order is not limited. Similarly, the processing order for the viewing positions is not limited, as long as the processing is performed without omission for all possible values ​​of these.

[0057] 8 is a flowchart showing an example of bone division processing. Before the processing of step 201 is executed, the division removal unit 58 first acquires the 3D model 1 and the VV calculated by the VV calculation unit 57. Thereafter, the processing of steps 201 to 210 is all executed by the division removal unit 58.

[0058] FIG. 9 is a schematic diagram showing bone information. Bones are defined for the 3D model 1. For example, the head bones and leg bones of the 3D model 1 are defined as bones. Of course, the specific positions of the bones are not limited. Each bone has an index b.min ≦b≦B max In addition, bone information is defined for each bone. Bone information is a parameter that indicates the name of each bone, and is b |B min ≦b≦B max}.

[0059] Also, for each vertex v and each bone b, weight information {VW v,b |V min ≦v≦V max ,B min ≦b≦B max} is defined. Weight information is a parameter that indicates how much a vertex follows when a bone is moved, and takes a value between 0 and 1. The higher the weight information value, the more the vertex is affected by the movement of the bone, and if the weight information value is 0, the vertex is not affected by the movement of the bone. For example, if the vertex is the top of the head and the bone is a head bone, the vertex is relatively easily affected by the movement of the bone, so the weight information will have a relatively high value.

[0060] The following internal variables are defined for processing: the number of division levels N, an array th[n] of N VV thresholds, an array flag[n] of N binary values ​​(0 or 1), and an array Cluster[b] indicating the cluster number for each bone. The number of division levels N is the number of bone divisions in the process. The threshold array th[n] is the threshold used for division and is a fixed value determined in advance of the process. In this embodiment, the value of th[0] is set to 0. The value of th[n] is set so that it monotonically increases with n. That is, for example, th[2] does not exceed th[3]. When the number of division levels is N, flag[n] is defined as an array of N parameters from flag[0] to flag[N-1], and th[n] is defined as an array of N parameters from th[0] to th[N-1]. Bones are grouped into clusters, and the cluster to which each bone belongs is represented by Cluster[b].

[0061] The bone index is set to b=0 (step 201). In this example, B min= 0, i.e., b is first set to the minimum value. The binary value array flag[0] to flag[N-1] are all initialized to 1 (step 202). The vertex index is set to v = 0 (step 203). That is, the subsequent processing is first executed under b = 0 and v = 0.

[0062] It is determined whether the weight information is greater than 0 (step 204). That is, the weight information VW between the vertex v=0 and the bone b=0 is 0,0 A determination is made as to VW 0,0 If it is >0 (Yes in step 104), then VV at vertex v=0 ０ It is determined whether VV is greater than the threshold value th[0] (step 205). ０ If th[0] is true, 0 is assigned to flag[0]. ０ If VV>th[1], a process of substituting 0 for flag[1], if VV>th[2], a process of substituting 0 for flag[2], ... are executed in order for n=0 to N-1 (steps 205 to 207). Hereinafter, the processes of steps 205 to 207 may be referred to as threshold determination processes.

[0063] If it is determined in step 204 that the weight information is 0, the threshold determination process is not performed, and 1 is added to v, and the determination in step 204 is performed again. That is, the same process is performed for the next vertex v=1. After each threshold determination process, it is determined whether the process has been performed for all vertices (step 208), and whether v is V max That is, the threshold determination process is performed only on vertices whose weight information for bone b=0 is not 0, and no process is performed on vertices whose weight information is 0.

[0064] v is V maxWhen this is reached (Yes in step 208), the values ​​of the binary value array flag[0] to flag[N-1] for bone b=0 have been determined. Below, a processing example will be explained using specific values. (Example) Number of division levels: N=4 Threshold: th[0]=0, th[1]=1.5, th[2]=2.5, th[3]=3.5 Bone: b=0 Vertex: v=0 to 3 VV: VV ０ = 2, VV １ = 3, VV ２ = 1, VV ３ = 4 Weight information: VW 0,0 = 0.2, VW 1,0 = 0.5, VW 2,0 = 0.8, VW 3,0 =0.0

[0065] First, the process is performed for b=0 and v=0. First, it is determined whether the weight information is greater than 0. In this example, VW 0,0 = 0.2, the following threshold determination process is then executed: ０ = 2, - Since th[0] is greater than 0, 0 is assigned to flag[0] (1 → 0). - Since th[1] is greater than 1.5, 0 is assigned to flag[1] (1 → 0). - Since th[2] is not greater than 2.5, 0 is not assigned to flag[2] (remains 1). - Since th[3] is not greater than 3.5, 0 is not assigned to flag[3] (remains 1). Therefore, the values ​​of each flag[n] after processing for v=0 are flag[0] = 0, flag[1] = 0, flag[2] = 1, flag[3] = 1.

[0066] Next, for b = 0 and v = 1, in this example, VW 1,0 = 0.5, the threshold determination process is similarly performed. １= 3, - Since th[0] is greater than 0, 0 is assigned to flag[0] (remains 0). - Since th[1] is greater than 1.5, 0 is assigned to flag[1] (remains 0). - Since th[2] is greater than 2.5, 0 is assigned to flag[2] (1 → 0). - Since th[3] is not greater than 3.5, 0 is not assigned to flag[3] (remains 1). Therefore, the values ​​of each flag[n] after processing for v=1 are flag[0] = 0, flag[1] = 0, flag[2] = 0, flag[3] = 1.

[0067] Next, for b = 0 and v = 2, in this example, VW 2,0 = 0.8, the threshold determination process is similarly performed. ２ = 1, - Since th[0] is greater than 0, 0 is assigned to flag[0] (remains 0). - Since th[1] is not greater than 1.5, 0 is not assigned to flag[1] (remains 0). - Since th[2] is not greater than 2.5, 0 is not assigned to flag[2] (remains 0). - Since th[3] is not greater than 3.5, 0 is not assigned to flag[3] (remains 1). Therefore, the values ​​of each flag[n] after processing for v=2 are flag[0] = 0, flag[1] = 0, flag[2] = 0, flag[3] = 1 (no change from the values ​​after processing for v=1).

[0068] Next, for b = 0 and v = 3, VW 3,0 = 0.0, the threshold determination process is not executed. 3,0 If V is not 0 and threshold determination processing is performed, V ３ Since th[3] = 4 is greater than th[3] = 3.5, 0 is assigned to flag[3], and the values ​​of each flag[n] after processing are flag[0] = 0, flag[1] = 0, flag[2] = 0, flag[3] = 0.

[0069] In this way, threshold determination processing is performed for all vertices whose weight information is not 0 (follows) for a certain bone. ｖIf there is at least one vertex where th[n] is greater than th[n], the value of flag[n] will be 0. On the other hand, if there is no such vertex, the value of flag[n] will be 0.

[0070] Furthermore, since the value of each th[n] is set to monotonically increase, if a certain flag[n] is 1, then flag[m] with a larger argument will never be 0. In other words, for example, flag[0] = 0, flag[1] = 1, flag[2] = 0, and flag[3] = 0 will never occur.

[0071] The cluster of the bone is determined (step 209). Specifically, when the value of flag[n] becomes 1 for the first time in flag[m], the cluster of that bone is determined to be Cluster[b]=m. For example, in the above example, flag[0]=0, flag[1]=0, flag[2]=0, flag[3]=1, and the value of flag[n] becomes 1 for the first time in flag[3], so the cluster of bone b=0 is Cluster[0]=3. Also, for example, the VV of all vertices that follow the bone ｖ If is 0, flag[0]=1, and therefore Cluster[b]=0.

[0072] It is determined whether all bones have been processed (step 210). max If b has not reached B (No in step 210), the process of steps 202 to 210 is repeated for each bone while adding 1 to b. max If it reaches (Yes in step 210), the process ends. In this way, each bone b is grouped into one of the clusters Cluster[b]=0 to N-1.

[0073] 10 is a flowchart showing an example of a process for removing a vertex. The processes in steps 301 to 308 are all executed by the division and removal unit 58. A binary parameter flag2 is also defined as an internal variable for the process.

[0074] The vertex index is set to v=0 (step 301). The parameter flag2 is initialized to 1 (step 302). The bone index is set to b=0 (step 303). The subsequent processing in steps 304 to 306 is all performed under the assumption that v=0 and b=0.

[0075] It is determined whether the weight information is greater than 0 (step 304). 0,0 If Cluster[b]=0 is greater than 0 (Yes in step 304), it is determined whether the cluster of bone b=0 is a target for 3D model removal (step 305). Specifically, Cluster[b]=0 is a cluster for 3D model removal, and if Cluster[b]=0, bone b=0 is determined to be a target for removal. On the other hand, if Cluster[b]=0 is not 0, bone b=0 is determined not to be a target for removal. In this case, 0 is assigned to flag2.

[0076] On the other hand, VW 0,0 If b=0 (No in step 304), the determination as to whether bone b=0 is to be removed is not performed, and the process of step 304 is similarly performed for the next bone b=1.

[0077] After each processing, it is determined whether or not the processing has been performed on all bones (step 306). max If it is not reached (No in step 306), the process is performed on the next bone. That is, the process is performed only on bones whose weight information for vertex v=0 is not 0, and no process is performed on bones whose weight information is 0.

[0078] b is B maxWhen this is reached (Yes in step 306), the value of the binary value flag2 for vertex v=0 has been determined to be either 0 or 1. If all bones related to vertex v=0 are to be removed, the vertex is removed (step 307). Specifically, if flag2=1 remains, the vertex is removed. That is, if all bones with non-zero weight information for a vertex are to be removed (Cluster[b]=0), the vertex is removed. On the other hand, if there is even one bone with non-zero weight information that is not to be removed, the vertex is not removed. Furthermore, whether or not a bone with non-zero weight information is to be removed does not affect whether or not the vertex is removed.

[0079] It is determined whether all vertices have been processed (step 308). max If v does not reach V (No in step 308), the threshold determination process is performed on the unprocessed vertices. max (Yes in step 308), the process ends. In this way, the process of removing or not removing a vertex is completed for each vertex.

[0080] Below, we will explain a processing example using specific values. (Example) Bone: b = 0 to 2 Bone cluster: Cluster[0] = 0, Cluster[1] = 0, Cluster[2] = 2 Vertex: v = 0 to 2 Weight information VW v,b : VW 0,0 = 0.0, VW 0,1 = 0.0, VW 0,2 = 0.5 VW 1,0 = 0.5, VW 1,1 = 0.8, VW 1,2 = 0.0 VW 2,0 = 0.0, VW 2,1 = 0.5, VW 2,2 = 0.8

[0081] First, processing is performed for v=0. When b=0, VW 0,0 = 0.0, so no removal decision is made. When b = 1, VW0,1 = 0.0, so no removal decision is made. When b = 2, VW 0,2 = 0.5, so a removal decision is made. Since Cluster[2] = 2 for the cluster of bone b = 2, 0 is assigned to flag2 (1 → 0). Therefore, flag2 = 0, and vertex v = 0 is not removed.

[0082] Next, processing is performed for v=1. When b=0, VW 1,0 = 0.5, so a removal decision is made. Since the cluster of bone b = 0 has Cluster[0] = 0, 0 is not assigned to flag2 (it remains 1). When b = 1, VW 1,1 = 0.8, so removal is determined. Since the cluster of bone b = 1 has Cluster[1] = 0, flag2 is not set to 0 (it remains 1). When b = 2, VW 1,2 = 0.0, so no removal decision is made. Therefore, flag2 remains at 1, and vertex v=1 is removed.

[0083] Next, processing is performed for v=2. When b=0, VW 2,0 = 0.0, so no removal decision is made. When b = 1, VW 2,1 = 0.5, so a removal decision is made. Since the cluster of bone b = 1 has Cluster[1] = 0, 0 is not assigned to flag2 (it remains 1). When b = 2, VW 2,2 = 0.8, so a removal decision is made. Since Cluster[2] = 2 for the cluster of bone b = 2, 0 is assigned to flag2 (1 → 0). Therefore, flag2 = 0, and vertex v = 2 is not removed.

[0084] In this way, in this example, only vertex v=1 is removed. If there is no bone with weight information other than 0 (that follows) for a certain vertex, the removal determination is not performed even once, and flag2=1 remains. Therefore, the vertex is removed.

[0085] FIG. 11 is a schematic diagram showing the division process and the removal process. As an example, FIG. 11 shows a conceptual diagram of the process for the second display 15 from the top. In this example, the number of divisions N is set to 3, and each bone included in the 3D model 1 is assigned to one of clusters 0 to 3. As shown in FIG. 11, for example, each bone located in the upper body belongs to cluster 3. The cluster with the largest argument, such as cluster 3, becomes the main division area (the area with the highest display frequency). Furthermore, the upper part of the lower body belongs to cluster 2, and the center part belongs to cluster 1. The lower part of the lower body belongs to cluster 0 (shaded area, to be removed).

[0086] The 3D model 1 is divided into an upper body part 80 (corresponding to cluster 3), an upper lower body part 81 (corresponding to cluster 2), a middle lower body part 82 (corresponding to cluster 1), and a lower lower body part 83 (corresponding to cluster 0). Then, the lower lower body part 83 is removed, and an optimized 3D model 70 is generated from the remaining three parts 80 to 82.

[0087] [Setting of boundary information] Fig. 12 is a flowchart showing an example of the process of setting boundary information. All of the processes in steps 401 to 408 are executed by the boundary information setting unit 59. Fig. 13 is a schematic diagram showing boundary information. Boundary information is information that depends on cluster n and represents three-dimensional coordinates. Boundary information consists of a minimum value and a maximum value, and the minimum value is B nmin (B nminx ,B nminy ,B nminz ) and the maximum value is B nmax (B nmaxx ,B nmaxy ,B nmaxz )

[0088] First, cluster 0, vertex v=0 (meaning the first vertex belonging to cluster 0), time t=T min In step 404, the maximum value B 0max and minimum value B 0min At time t=T min The coordinates of the vertex v=0 in are substituted as they are.

[0089] Next, cluster 0, vertex v=0, time t=T min In step 404, the current maximum value B 0max and minimum value B 0min and time t = T min The coordinates of the vertex v=0 at +1 are compared with the maximum value B of the x-coordinate, and the maximum and minimum values ​​are updated. nmaxx If the maximum value is greater than the x-coordinate of the vertex, the maximum value is updated to the x-coordinate of the vertex. The same process is carried out for the y-coordinate and z-coordinate, and the minimum value.

[0090] Similarly, the maximum and minimum values ​​are updated for all subsequent times t. When all processing is completed (Yes in step 405), the maximum and minimum values ​​represent the range in which vertex v=0 moves at all times.

[0091] Similar processing is performed for all other vertices. When all processing is completed (Yes in step 406), the maximum and minimum values ​​represent the range in which all vertices belonging to cluster 0 can move at any time. In other words, the information reflects the maximum range of movement in animation. In FIG. 13, this range is schematically shown by a gray rectangular parallelepiped. The maximum and minimum values ​​are set as boundary information for cluster 0 (step 407).

[0092] Similar processing is performed for all other clusters, and boundary information is set for each cluster. When boundary information has been set for all clusters (step 408), the processing ends.

[0093] In this way, the division and removal unit 58 generates an optimized 3D model 70, and the boundary information setting unit 59 generates boundary information. In the optimized 3D model 70, a group of vertices of a lower body part 83 that cannot be displayed on the display 15 has been removed. Furthermore, division into three parts 80 to 82 is performed based on the results of the projection process onto the display 15. In this way, the optimized 3D model 70 becomes a 3D model 1 optimized for the display 15.

[0094] The optimized 3D model 70 and boundary information are transmitted to the slave PC 10, for example, by the communication unit 53. The received optimized 3D model 70 and boundary information are used in the culling process and rendering process in the slave PC 10. The generated optimized 3D model 70 and boundary information may be stored in the storage unit 54.

[0095] In this embodiment, the segmentation processing unit segments the 3D model 1 into a plurality of first unit model data that are different from each other, and generates first partial model data consisting of at least one of the plurality of first unit model data. Here, the VV calculation unit 57, segmentation removal unit 58, and boundary information setting unit 59 shown in FIG. 2 correspond to an embodiment of the segmentation processing unit according to the present technology. Also, the upper body part 80, the upper lower body part 81, the middle lower body part 82, and the lower lower body part 83 shown in FIG. 11 each correspond to the first unit model data according to the present technology. Also, the optimized 3D model 70 corresponds to the first partial model data according to the present technology.

[0096] The 3D model 1 is divided into four parts 80 to 83. These four parts are different from one another; that is, they do not have overlapping parts. The optimized 3D model 70 is made up of three parts 80 to 82, excluding the lower body part 83, out of the four parts 80 to 83. Note that the first partial model data may also be made up of a single piece of first unit model data.

[0097] Furthermore, in this embodiment, when generating the first partial model data, the segmentation processing unit excludes from the first partial model data the first hidden model data, which is the first unit model data that is not displayed on the display 25. Here, the lower body part 83 corresponds to the first hidden model data according to the present technology. In other words, the lower body part 83, which cannot be displayed on the display 25, is excluded from the optimized 3D model 70.

[0098] In this embodiment, the division processing unit sets multiple different first display frequency ranges, calculates the first display frequency for each portion of the 3D model data, and generates multiple first unit model data for each first display frequency range. Here, the range of values ​​separated by the threshold value th[n] used in the processing of steps 201 to 210 corresponds to the first display frequency range. In other words, the ranges such as th[0] (=0), th[0] to th[1], th[1] to th[2], ..., th[N-2] to th[N-1], th[N-1] and so on correspond to the first display frequency ranges. These ranges are different from each other.

[0099] Each bone corresponds to a portion of the 3D model data according to the present technology. The maximum value of VV of all vertices that follow a bone (whose weight information is not 0) corresponds to the first display frequency of the bone. When the first display frequency is high, the display frequency of each vertex also becomes relatively high, and the display frequency of the bone that follows all of them also becomes relatively high. Therefore, the first display frequency can be considered a parameter that represents the display frequency of a bone. In the processing of steps 201 to 210, the first display frequency of the bone (the maximum value of VV of all the following vertices) is calculated. Furthermore, the bones are divided into clusters according to the first display frequency, and parts 80 to 83 are generated for each cluster. That is, multiple first unit model data are generated for each range of the first display frequency.

[0100] In addition, in this embodiment, the division processing unit calculates a first display frequency, which is the display frequency of the 3D model 1 by the display 15, based on the position of the display 15 and the user's field of view of the display 15, and performs division into multiple first unit model data based on the calculated first display frequency.

[0101] The display frequency of each bone can also be said to be the display frequency of the 3D model 1 itself, and therefore the display frequency of the 3D model 1 is calculated. Furthermore, the process of generating a plurality of first unit model data for each range of the first display frequency is included in the process of executing division into a plurality of first unit model data based on the first display frequency. Furthermore, the display position information D corresponds to the position of the display 15, and the viewing range information W corresponds to the user's field of view. Since VV is calculated using these, the first display frequency is calculated based on the position of the display 15 and the user's field of view of the display 15.

[0102] The specific method for calculating the first display frequency based on the position of the display 15 and the user's field of view is not limited. That is, the first display frequency may be calculated using an algorithm or method different from those used in steps 101 to 109 and 201 to 210. Furthermore, the method for dividing the first display frequency into a plurality of first unit model data items is not limited, and a method other than the method of defining the range of the first display frequency using a threshold value may be used.

[0103] Furthermore, in this embodiment, the division processing unit performs division into a plurality of first unit model data based on the position of the display 15 and the user's field of view of the display 15. In fact, the processes of calculating VV using the display position information D and the viewing range information W, calculating the first display frequency based on VV, and generating a plurality of first unit model data for each range of the first display frequency are included in the process of performing division into a plurality of first unit model data based on the position of the display 15 and the user's field of view. Of course, the specific method of such processing is not limited, and division may be performed by a method that does not use the first display frequency.

[0104] In this embodiment, the segmentation processing unit excludes, from the optimized 3D model 70, first hidden model data, which is a portion of the 3D model 1 that is not displayed on the display 25. Here, the lower body part 83 corresponds to the first hidden model data. Note that removal of such hidden portions may be performed by a method that does not include segmentation into unit model data.

[0105] [Display Control] Fig. 14 is a schematic diagram showing an optimized 3D model. By the processing described above, an optimized 3D model and boundary information to be used on display 15 are generated and transmitted to the slave PC 10 corresponding to display 15. Furthermore, the same processing is performed for each display, and the optimized 3D model and boundary information to be used on displays 25, 35, and 45 are also transmitted to the corresponding slave PCs 20, 30, and 40, respectively.

[0106] 14 is a schematic diagram illustrating an example of the optimized 3D models 91 to 94 transmitted to each of the slave PCs 10 to 40. As described above, the parts that make up each of the optimized 3D models 91 to 94 are different, and the division methods are also different.

[0107] In this embodiment, the division processing units (VV calculation unit 57, division removal unit 58, boundary information setting unit 59) divide the 3D model 1 displayed on the tiling display into first partial model data and second partial model data corresponding to the first display and the second display, respectively, based on the positional relationship between the first display and the second display of the tiling display and the field of view of the user of the tiling display.

[0108] Here, for example, the display 15 corresponds to the first display, the display 25 corresponds to the second display, the optimized 3D model 91 corresponds to the first partial model data, and the optimized 3D model 92 corresponds to the second partial model data.

[0109] Furthermore, display position information D of displays 15 and 25 corresponds to the positional relationship between the first and second displays of the tiling display, and viewing range information W corresponds to the field of view of the user of the tiling display. In this embodiment, processing such as calculation of VV is performed based on the display position information D and viewing range information W of displays 15 and 25, and 3D model 1 is divided into optimized 3D models 91 and 92. Note that the division process into optimized 3D models 91 and 92 may be performed without calculating VV or dividing the unit model data.

[0110] In this embodiment, the communication unit 53 transmits the optimized 3D models 91 and 92 to the slave PCs 10 and 20, respectively.

[0111] In this embodiment, the first partial model data and the second partial model data include overlapping model data that is part of the 3D model data and is identical. For example, the head of 3D model 1 is included in both optimized 3D models 91 and 92, and therefore corresponds to overlapping model data according to the present technology. Furthermore, the toe tips are included in both optimized 3D models 93 and 94, and therefore also correspond to overlapping model data. Note that because each of the optimized 3D models 91 to 94 is divided differently, the overlapping model data may be part of the unit model data (each part). For example, the optimized 3D model 92 has one part in the upper body, but only the head is overlapping model data with the optimized 3D model 91.

[0112] In this example, the display 15 corresponds to the first display and the display 25 corresponds to the second display, but this is not limiting, and any two different displays can correspond to the first and second displays, such as displays 35 and 45. The same applies to the first and second partial model data, the first and second display processing devices, the first and second hidden model data, the first and second unit model data, and the first and second display frequencies.

[0113] 15 is a schematic diagram showing the culling process, rendering process, and display control process. These processes will be described below. Note that the process methods described below are merely examples, and each process may be performed using any known method.

[0114] First, the optimized 3D models 91 to 94, boundary information, model placement information M, display position information D, animation data A, and viewing position information are acquired by each of the slave PCs 10 to 40. For example, communication is performed between the communication unit of each of the slave PCs 10 to 40 and the communication unit 53 of the parent PC 50, and each piece of information is acquired.

[0115] 15 shows an example of a display in which the user moves, but the apparent image seen by the user does not change. In this case, in order to prevent the apparent image from changing, it is necessary to change the display in accordance with the user's movement. In this example, １ ~t ５ The user's viewing position moves downward in the Y direction, and the position of the 3D model 1 displayed on the display moves upward accordingly. The display position of the 3D model 1 is calculated based on the model placement information M, the display position information D, the animation data A, and the viewing position information.

[0116] The leftmost time t １ In this case, the 3D model 1 is not displayed on the top display 15. The optimized 3D model 91 consists of five parts: the head, shoulders, upper arms, chest, lower arms, and hands, and all of these parts are culled by the culling processing unit 12. In Fig. 15, the culled parts are shown with polka dots.

[0117] The head, upper body, arms, etc. of the 3D model 1 are displayed on the second display 25 from the top. The optimized 3D model 92 consists of the upper body, upper thighs, and lower thighs, of which the upper thighs and lower thighs are culled by the culling processing unit 22. In this way, culling is performed on parts that are not displayed in any part on the display 25. On the other hand, culling is not performed on parts that are displayed even in part, and rendering is performed. In other words, the rendering processing unit 23 renders the upper body parts. In Figure 15, the rendered parts are shown in black. Culling and rendering are performed based on boundary information.

[0118] Similarly, culling and rendering are performed on the optimized 3D models 93 and 94. Furthermore, the respective display control units control the display of the optimized 3D models 91 to 94 after culling and rendering on the displays 15 to 45. For example, on the display 15, the display control unit 14 controls the display so that the optimized 3D model 91 is not displayed. Furthermore, on the display 25, the display control unit 24 controls the display of the upper body parts of the optimized 3D model 92. Similar display control is performed on the displays 35 and 45.

[0119] As a result, the four displays 15 to 45 work together to display one 3D model 1. Note that display control may use techniques such as pixel shader (a type of function that processes light sources and shadows, and processes images pixel by pixel) or raster (a method of representing an image as a collection of pixels arranged in a grid).

[0120] Thereafter, as the time passes, the display on each of the displays 15 to 45 changes, and therefore the parts to be culled, the parts to be rendered, and the contents of display control also change in each of the slave PCs 10 to 40. For example, at time t ２ At time t ３ In this example, three parts are rendered: the head, the lower arms, and the hands.

[0121] 16 is a schematic diagram showing the culling process, rendering process, and display control process. In this example, the user does not move, but an animation of a 3D model approaching from a distance is displayed. In this case, the display is also realized by the same process as in FIG. 15.

[0122] As described above, in the image display system 100 according to this embodiment, the 3D model 1 is divided into optimized 3D models 91 to 94 based on the relative positions of the displays 15 to 45 and the user's field of view of the tiling display, and these are transmitted to the slave PCs 10 to 40, respectively. This reduces the processing load of rendering.

[0123] Because the volume of video data to be 3D rendered in a tiling display is extremely large, a dedicated PC is often installed for each display, and the rendering process is distributed among the displays. However, when displaying high-resolution 3D models, the processing load on a specific PC can become high, which can result in the frame rate of the entire system being kept low.

[0124] A known method for reducing the load during rendering is to cull areas that are not displayed on the display. Culling is a process that determines which areas are not displayed and reduces the data to be rendered, which makes the final rendering more efficient.

[0125] Rendering generally involves culling on an object-by-object basis. Culling reduces the amount of data to be rendered using boundary information preset for each object, and this process is performed using the Compute Shader resources of the CPU or GPU. While this makes rendering on an object-by-object basis more efficient, partial areas within an object are not culled, resulting in redundant rendering load for hidden areas within the object.

[0126] In response to this, an improvement method has been proposed in which objects are divided into smaller area units in the preprocessing stage and culled for each small area. In this method, the culling process involves reducing the amount of data to be rendered in small area units using the GPU's Task Shader. While this method makes rendering more efficient by culling for each small area, it has the problem of artifacts occurring for dynamic objects.

[0127] A method for improving artifacts in dynamic objects has been proposed that updates boundary settings for culling processing over time. This method follows the basic culling method described above (culling for each small region), but by calculating boundary information between each animation keyframe, it suppresses the occurrence of artifacts when small regions are deformed by animation.

[0128] On the other hand, in all of these methods, the input data acquired by each display for rendering processing includes not only the data to be displayed on that display, but also data that is not to be displayed on that display, i.e., data that should be displayed on another display.

[0129] 17 is a schematic diagram showing an image display system 200 of the comparative example. In the image display system 200 of the comparative example, the same 3D model and the same boundary information are transmitted to each slave PC. In other words, unlike the image display system 100 of the present technology, optimized 3D models 91 to 94 are not transmitted for each display, and a VV calculation unit 57 for generating the optimized 3D models 91 to 94 is not configured.

[0130] In this type of image display system 200, data that is not displayed on the display (unnecessary portions) is also transmitted to each child PC, resulting in an increased processing load before data reduction, including the culling process itself, and if the load on the PC reaches its processing performance cap, the PC could become a bottleneck in achieving a frame rate for the entire tiling display.

[0131] To address this issue, it is possible to ensure the frame rate by lowering the resolution of the displayed content, but this is not a desirable solution from the perspective of providing immersive content, and further improvements in processing are required.

[0132] In the image display system 100 of the present technology, optimized 3D models 91 to 94 are generated, with portions that cannot be displayed on the displays 15 to 45 removed, and are transmitted to the slave PCs 10 to 40. As a result, culling and rendering of unnecessary portions are not performed in the slave PCs 10 to 40, and the processing load is reduced.

[0133] Furthermore, data that may be displayed in the boundary area between adjacent displays (overlapping model data) is stored in duplicate on the adjacent displays. This makes it possible to efficiently perform rendering even when the display position of the 3D model 1 transitions across the boundary between displays, stabilizing the playback frame rate of the entire system.

[0134] Furthermore, smoothed spatiotemporal information based on time-series data of the animation of the 3D model 1 and information on the user's viewing range is used as an index for dividing and reducing the 3D model 1. Specifically, the division and reduction of the 3D model 1 is performed by calculating VV, determining bone clusters, removing vertices, setting boundary information, etc. In this way, the movement of the 3D model 1 and the viewing environment are taken into consideration, and the division and reduction of the 3D model 1 is performed in a form optimized for each display.

[0135] 18 is a block diagram showing an example of the hardware configuration of a computer 500 that can realize the parent PC 50 and the child PCs 10 to 40. The computer 500 includes a CPU 501, a ROM 502, a RAM 503, an input / output interface 505, and a bus 504 that interconnects these components. A display unit 506, an input unit 507, a storage unit 508, a communication unit 509, a drive unit 510, and the like are connected to the input / output interface 505.

[0136] The display unit 506 is a display device using, for example, a liquid crystal display (LCD), an electroluminescence (EL), or the like. The input unit 507 is, for example, a keyboard, a pointing device, a touch panel, or other operating device. If the input unit 507 includes a touch panel, the touch panel may be integrated with the display unit 506. The storage unit 508 is a non-volatile storage device, such as a HDD, a flash memory, or other solid-state memory. The drive unit 510 is a device capable of driving a removable storage medium 511, such as an optical storage medium or a magnetic recording tape. The communication unit 509 is a modem, a router, or other communication equipment that can be connected to a LAN, a WAN, or the like and is used to communicate with other devices. The communication unit 509 may be wired or wireless. The communication unit 509 is often used separately from the computer 500.

[0137] Image processing by the computer 500 having the above-described hardware configuration is realized by cooperation between software stored in the storage unit 508 or the ROM 502, etc., and the hardware resources of the computer 500. Specifically, the image processing method according to the present technology is realized by loading a program constituting the software stored in the ROM 502, etc., into the RAM 503 and executing it.

[0138] The program is installed in the computer 500 via, for example, a removable recording medium 511. Alternatively, the program may be installed in the computer 500 via a global network or the like. Alternatively, any non-transitory storage medium that can be read by the computer 500 may be used.

[0139] The image processing method according to the present technology may be executed by cooperation of multiple computers connected to each other in a manner that allows communication via a network, etc., thereby constructing an image processing device and an image display system according to the present technology. That is, the image processing method according to the present technology can be executed not only in a computer system constituted by a single computer, but also in a computer system in which multiple computers operate in conjunction with each other.

[0140] In this disclosure, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0141] The image display system, the parent PC, the child PC, the display, the processing flow, and the like described with reference to the drawings are merely exemplary embodiments, and can be modified as desired without departing from the spirit of the present technology. In other words, any other configurations, algorithms, and the like for implementing the present technology may be adopted.

[0142] It is also possible to combine at least two of the features of the present technology described above. That is, the various features described in each embodiment may be arbitrarily combined without distinguishing between the embodiments. Furthermore, the various effects described above are merely examples and are not intended to be limiting, and other effects may also be achieved.

[0143] The present technology may also be configured as follows: (1) An image processing device comprising: a division processing unit that divides 3D model data displayed on a tiling display into first partial model data and second partial model data corresponding to the first display and the second display, respectively, based on a positional relationship between a first display and a second display of the tiling display and a field of view of a user of the tiling display, and a transmission unit that transmits the first partial model data and the second partial model data to a first display processing device and a second display processing device configured to control the first display and the second display, respectively. (2) The image processing device according to (1), wherein, in the division, the division processing unit excludes, from the first partial model data, first hidden model data that is a portion of the 3D model data that is not displayed on the first display, and excludes, from the second partial model data, second hidden model data that is a portion of the 3D model data that is not displayed on the second display. (3) The image processing device according to (1) or (2), wherein the first partial model data and the second partial model data include overlapping model data that are part of the 3D model data and are identical. (4) The image processing device according to any one of (1) to (3), wherein the division processing unit divides the 3D model data into a plurality of first unit model data that are different from each other, and generates the first partial model data consisting of at least one of the plurality of first unit model data, and divides the 3D model data into a plurality of second unit model data that are different from each other, and generates the second partial model data consisting of at least one of the plurality of second unit model data.(5) The image processing device according to (4), wherein the segmentation processing unit, in generating the first partial model data, excludes first hidden model data, which is the first unit model data not displayed on the first display, from the first partial model data, and, in generating the second partial model data, excludes second hidden model data, which is the second unit model data not displayed on the second display, from the second partial model data. (6) The image processing device according to (4) or 5, wherein the segmentation processing unit performs the segmentation into the plurality of first unit model data based on a position of the first display and a user's field of view of the first display, and performs the segmentation into the plurality of second unit model data based on a position of the second display and a user's field of view of the second display. (7) The image processing device according to (6), wherein the segmentation processing unit calculates a first display frequency that is a display frequency of the 3D model data on the first display based on a position of the first display and a field of view of a user of the first display, and performs segmentation into the plurality of first unit model data based on the calculated first display frequency, calculates a second display frequency that is a display frequency of the 3D model data on the second display based on a position of the second display and a field of view of a user of the second display, and performs segmentation into the plurality of second unit model data based on the calculated second display frequency. (8) The image processing device according to (7), wherein the segmentation processing unit sets a plurality of ranges of the first display frequency that are different from each other, calculates the first display frequency for each portion of the 3D model data, generates the plurality of first unit model data for each range of the first display frequency, sets a plurality of ranges of the second display frequency that are different from each other, calculates the second display frequency for each portion of the 3D model data, and generates the plurality of second unit model data for each range of the second display frequency.(9) An image processing method comprising: dividing 3D model data displayed on a tiling display into first partial model data and second partial model data corresponding to the first display and the second display, respectively, based on a positional relationship between a first display and a second display of the tiling display and a field of view of a user of the tiling display; and transmitting the first partial model data and the second partial model data to a first display processing device and a second display processing device configured to control the first display and the second display, respectively. (10) An image display system comprising: an image processing device having: a first display and a second display of a tiling display; a first display processing device and a second display processing device configured to control the first display and the second display, respectively; a division processing device that divides 3D model data displayed on the tiling display into first partial model data and second partial model data corresponding to the first display and the second display, respectively, based on a positional relationship between the first display and the second display and a field of view of a user of the tiling display; and a transmission device that transmits the first partial model data and the second partial model data to the first display processing device and the second display processing device, respectively; wherein the first display processing device and the second display processing device perform culling processing and rendering processing on the received first partial model data and second partial model data, and control display of the first partial model data and the second partial model data after the culling processing and the rendering processing on the first display and the second display.

[0144] DESCRIPTION OF SYMBOLS 1...3D model 10, 20, 30, 40...child PC 12, 22...culling processing unit 13, 23...rendering processing unit 14, 24...display control unit 15, 25, 35, 45...display 50...parent PC 57...visibility volume calculation unit 58...division and removal unit 59...boundary information setting unit 80...upper body part 81...upper lower body part 82...middle lower body part 83...lower lower body part 91-94...optimized 3D model 100...image display system

Claims

1. An image processing device comprising: a division processing unit that divides 3D model data displayed on a tiling display into first partial model data and second partial model data corresponding to the first display and the second display, respectively, based on a positional relationship between a first display and a second display of the tiling display and a field of view of a user of the tiling display; and a transmission unit that transmits the first partial model data and the second partial model data to a first display processing device and a second display processing device configured to control the first display and the second display, respectively.

2. The image processing device according to claim 1, wherein the division processing unit, in the division, excludes first hidden model data which is a portion of the 3D model data that is not displayed on the first display from the first partial model data, and excludes second hidden model data which is a portion of the 3D model data that is not displayed on the second display from the second partial model data.

3. The image processing device according to claim 1, wherein the first partial model data and the second partial model data include overlapping model data that is part of the 3D model data and is identical to the 3D model data.

4. The image processing device according to claim 1, wherein the division processing unit divides the 3D model data into a plurality of first unit model data that are different from each other, and generates the first partial model data consisting of at least one of the plurality of first unit model data, and divides the 3D model data into a plurality of second unit model data that are different from each other, and generates the second partial model data consisting of at least one of the plurality of second unit model data.

5. The image processing device according to claim 4, wherein the division processing unit, in generating the first partial model data, excludes first hidden model data, which is the first unit model data not displayed on the first display, from the first partial model data, and, in generating the second partial model data, excludes second hidden model data, which is the second unit model data not displayed on the second display, from the second partial model data.

6. The image processing device according to claim 4, wherein the segmentation processing unit performs segmentation into the plurality of first unit model data based on the position of the first display and the user's field of view of the first display, and performs segmentation into the plurality of second unit model data based on the position of the second display and the user's field of view of the second display.

7. The image processing device according to claim 6, wherein the segmentation processing unit calculates a first display frequency, which is a display frequency of the 3D model data by the first display, based on the position of the first display and the field of view of the user of the first display, and performs the segmentation into the plurality of first unit model data based on the calculated first display frequency; calculates a second display frequency, which is a display frequency of the 3D model data by the second display, based on the position of the second display and the field of view of the user of the second display, and performs the segmentation into the plurality of second unit model data based on the calculated second display frequency.

8. The image processing device according to claim 7, wherein the segmentation processing unit sets a plurality of different ranges of the first display frequency, calculates the first display frequency for each portion of the 3D model data, generates the plurality of first unit model data for each range of the first display frequency, sets a plurality of different ranges of the second display frequency, calculates the second display frequency for each portion of the 3D model data, and generates the plurality of second unit model data for each range of the second display frequency.

9. An image processing method comprising: dividing 3D model data displayed on a tiling display into first partial model data and second partial model data corresponding to the first display and the second display, respectively, based on a positional relationship between a first display and a second display of the tiling display and a field of view of a user of the tiling display; and transmitting the first partial model data and the second partial model data to a first display processing device and a second display processing device configured to control the first display and the second display, respectively.

10. An image display system comprising: a first display and a second display of a tiling display; a first display processing device and a second display processing device configured to control the first display and the second display, respectively; a division processing device that divides 3D model data displayed on the tiling display into first partial model data and second partial model data corresponding to the first display and the second display, respectively, based on a positional relationship between the first display and the second display and a field of view of a user of the tiling display; and a transmission device that transmits the first partial model data and the second partial model data to the first display processing device and the second display processing device, respectively; wherein the first display processing device and the second display processing device perform a culling process and a rendering process on the received first partial model data and the second partial model data, and control the display of the first partial model data and the second partial model data after the culling process and the rendering process on the first display and the second display.

Images