Image processing device, image processing method, and image processing program

Abstract

A controller according to the present invention: determines, for a virtual viewpoint image viewed from a prescribed virtual viewpoint of a virtual space, the virtual viewpoint image being generated on the basis of first objects (321, 322) arranged in the virtual space and a second object (310) arranged in the virtual space excluding the first objects, whether one first object (321) among the first objects is arranged at a position where the field of view from the prescribed virtual viewpoint is obstructed; determines the relevance of the other first object (322) among the first objects to the one first object; groups the one first object and the other first object determined to have relevance to the one first object as an object group (380); and moves the first objects in the object group to positions in the virtual viewpoint image at which the field of view from the prescribed virtual viewpoint is not obstructed.

Description

Image processing device, image processing method, and image processing program

[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing program.

[0002] Japanese Patent Laid-Open No. 2006-129999 proposes an image processing device, an image processing method, and a program that reduce the possibility that a subject in a virtual viewpoint image and a virtual object area will overlap.

[0003] Japanese Patent Application Laid-Open No. 2019-67427

[0004] When the technology of Patent Literature 1 is applied to a case where multiple virtual object areas exist in a mixed reality (MR) or virtual reality (VR) space, the placement of each virtual object area relative to the subject is determined individually. Even if multiple comment objects exist in a virtual viewpoint image and the comment objects are related to each other, the relationship between the comment objects is not reflected in the placement of each comment object.

[0005] The present disclosure aims to provide an image processing device, an image processing method, and an image processing program that can appropriately arrange multiple objects to be arranged in a virtual viewpoint image, taking into account the relationships between the objects.

[0006] In the image processing device, image processing method, and image processing program according to the embodiments, in a virtual viewpoint image viewed from a predetermined virtual viewpoint in the virtual space, which is generated based on a first object placed in a virtual space and a second object placed in the virtual space excluding the first object, it is determined whether one of the first objects is placed in a position that obstructs the view from the predetermined virtual viewpoint, the relevance of other first objects among the first objects to the one first object is determined, the one first object and the other first objects determined to have a relevance to the one first object are grouped as an object group, and the first objects of the object group are moved to positions in the virtual viewpoint image from the predetermined virtual viewpoint so as not to obstruct the view.

[0007] According to the above configuration, it is possible to appropriately arrange a plurality of first objects to be arranged in the virtual viewpoint image, taking into consideration the relevance between the first objects.

[0008] FIG. 1 is a diagram illustrating an outline of the configuration of an image processing device according to an embodiment. FIG. 2 is a diagram illustrating an example of a comment written by a user in a virtual viewpoint image displayed on the display of FIG. 1. FIG. 3 is a diagram illustrating the relationship between the user's line of sight with respect to a comment written by another user and placed in the virtual viewpoint image and the placement of a subject object. FIG. 4 is a diagram illustrating an example of the placement of comments written by another user in the virtual viewpoint image. FIG. 5 is a diagram illustrating an example of the placement of comments written by another user in the virtual viewpoint image. FIG. 6 is a diagram illustrating an example of the placement of comments written by another user in the virtual viewpoint image. FIG. 7 is a diagram illustrating an example of the placement of comments written by another user in the virtual viewpoint image. FIG. 8 is a diagram illustrating an example of moving comments written by another user in the virtual viewpoint image. FIG. 9 is a diagram illustrating an example of the placement of comments written by another user in the virtual viewpoint image. FIG. 10 is a flowchart illustrating an example of the procedure of an image processing method according to an embodiment executed by each unit of the controller of FIG. 1. FIG. 11 is a flowchart illustrating an example of the procedure of an image processing method according to an embodiment executed by each unit of the controller of FIG. 1. FIG. 12 is a flowchart illustrating an example of the detailed procedure of the movement processing of FIG. 11.

[0009] Hereinafter, embodiments will be described with reference to the drawings. The same or equivalent parts or components are designated by the same reference numerals throughout the drawings.

[0010] The following embodiments are examples of devices and the like that embody the technical concept of the present invention. The technical concept of the present invention does not limit the materials, shapes, structures, arrangements, etc. of each component to those described below.

[0011] The image processing device 10 according to the embodiment shown in FIG. 1 includes a controller 20 and a display 30. The controller 20 generates a virtual viewpoint image viewed from a virtual viewpoint in the virtual space based on a plurality of first objects and a subject object, which is a second object, arranged in the virtual space. The plurality of first objects are objects related to the second objects, and include comment objects and predetermined objects other than comment objects. A comment object refers to a comment written by a user, including, for example, characters, symbols, numbers, tables, and figures, arranged in the virtual space. A predetermined object refers to an object other than a comment written by a user arranged in the virtual space. A subject object refers to an object arranged in the virtual space other than the plurality of first objects. For example, a user can write a first object in the virtual space as an annotation regarding a subject object in the virtual space shared by the user through a virtual viewpoint image. In this embodiment, an example is described in which the plurality of first objects are a plurality of comment objects. However, this is not limited thereto, and each first object may be a comment object or a predetermined object. Furthermore, the terms "comment" and "comment object" used in this embodiment may be appropriately interpreted as the terms "object" and "predetermined object." A virtual viewpoint is a predetermined viewpoint in a virtual space. A virtual viewpoint image is an image of a virtual space viewed from a virtual viewpoint. The predetermined viewpoint does not need to be fixed to a single location, and may be different for different users sharing the virtual space. The virtual viewpoint image can be used, for example, in the fields of mixed reality (MR) or virtual reality (VR). The virtual viewpoint of a virtual viewpoint image is set to correspond to the viewpoint of a user viewing the virtual viewpoint image.

[0012] The display 30 displays the virtual viewpoint image generated by the controller 20, with the user viewing the virtual viewpoint image as the viewing target. The display 30 may be, for example, a small display placed in front of both eyes of a pair of goggles or sunglasses worn on the head of each of multiple users sharing the virtual space, or multiple flat displays arranged to surround each user. The small display of the goggles or sunglasses displays a virtual viewpoint image in a line of sight corresponding to the orientation of the user's head. Each flat display displays a virtual viewpoint image in a line of sight corresponding to the orientation of the user's head when the user directs their gaze toward the flat display. In the following description, it is assumed that the display 30 is a small display of the goggles. Although the virtual space is shared by multiple users, the number of users viewing the virtual space simultaneously may be multiple or one. After a user places a comment in the virtual space, another user may view the virtual space with the comment placed therein.

[0013] The controller 20 may be provided for each pair of goggles worn by each user, or may be a single device capable of communicating with each user's goggles. The controller 20 has, for example, a general-purpose microcontroller. The microcontroller includes a CPU (Central Processing Unit) having an input / output unit and an arithmetic unit, and a memory. The memory includes a ROM (Read Only Memory) and a RAM (Random Access Memory). The microcontroller can virtually construct multiple information processing circuits by, for example, having the CPU execute a program stored in the memory.

[0014] The multiple information processing circuits can constitute each of the units 210 to 280 (excluding 240) and 261 to 264 of the controller 20, which will be described later. In this embodiment, an example is shown in which the multiple information processing circuits built in the microcontroller are realized by software. Of course, it is also possible to configure the information processing circuits by providing dedicated hardware for executing the information processing of each of the units 210 to 280 (excluding 240) and 261 to 264, which will be described later. Alternatively, the multiple information processing circuits may be configured by individual hardware. The dedicated hardware includes devices such as application specific integrated circuits (ASICs) and conventional circuit components arranged to perform the functions of each of the units 210 to 280 (excluding 240) and 261 to 264.

[0015] The comment information acquisition unit 210 acquires comment information. The comment information is information necessary for processing comments written by users in a virtual space as 3D objects. Comments include, for example, characters, symbols, numbers, tables, and figures. As described above, comments may be predetermined objects. Comments to be written in a virtual space can be input using, for example, an input device (not shown) such as a pen tablet, or can be input by operating input keys (not shown) displayed in a virtual viewpoint image displayed on the display 30. The comment information includes a 3D object model of a comment object, information on the position in the virtual space where the comment is written, and information on the orientation of the comment. As described above, a comment object refers to a comment written by a user placed in a virtual space.

[0016] The view flag generation unit 220 generates a view flag. The view flag is one-bit data indicating whether the orientation of a 3D object model of a comment object placed in virtual space is to be rotated in virtual space in accordance with the position of the virtual viewpoint (High) or fixed regardless of the position of the virtual viewpoint (Low). For example, the view flag generation unit 220 can generate a view flag of "rotate, orientation rotated state (High)" or "fix, orientation fixed state (Low)" for the 3D object model of a comment written in virtual space by a user.

[0017] The view flag generation unit 220 may determine whether or not the user is a predetermined user, and if the user is a predetermined user, generate the view flag as High (e.g., 1), and if the user is not a predetermined user, generate the view flag as Low (e.g., 0). A predetermined user refers to a user whose line-of-sight vector from the virtual viewpoint to the comment object does not match the direction of the comment in the comment information. The view flag generation unit 220 may determine whether or not the comment object should be rotated, and if the comment object should be rotated, generate the view flag as High, and if the comment object should not be rotated, generate the view flag as Low. The view flag generation unit 220 may determine that the comment object should not be rotated if the comment is a figure.

[0018] The view flag combining unit 230 generates comment model information. The comment model information is data obtained by combining the comment information acquired by the comment information acquiring unit 210 and the view flag generated by the view flag generating unit 220.

[0019] The model information storage unit 240 readably stores basic model information and comment model information generated by the view flag composition unit 230. The model information storage unit 240 can be configured with data storage such as a hard disk drive (HDD) or a solid state drive (SSD). The basic model information is data of a virtual space shared by users of each virtual viewpoint in the virtual space. The basic model information includes a 3D object model of a comment object of a comment written in the virtual space by a user, as well as a 3D object model of a subject object placed in the virtual space. The subject object refers to an object placed in the virtual space, excluding the comment object. For example, a user can write a comment in the virtual space as an annotation regarding a subject object in the virtual space shared by the user through a virtual viewpoint image.

[0020] The comment model information update unit 250 updates the comment model information in the model information storage unit 240. The comment model information update unit 250 adds the comment model information generated by the view flag merging unit 230 to the comment model information read from the model information storage unit 240, and stores the comment model information after the addition in the model information storage unit 240. The comment model information update unit 250 essentially updates the comment model information in the model information storage unit 240 every time the view flag merging unit 230 generates comment model information.

[0021] The head tracking information receiving unit 270 acquires head tracking information. The head tracking information is information that indicates the head position of the user wearing the goggles in the virtual space. The head tracking information can be acquired, for example, by a gyro sensor built into the goggles or a camera that captures an image of the user.

[0022] The eye tracking information receiving unit 290 acquires eye tracking information. The eye tracking information is information indicating the position of the user's viewpoint in the virtual space. The eye tracking information may be, for example, the center position of both eyes of the user in the virtual space. The eye tracking information can be acquired, for example, by a pupil sensor built into the goggles or a camera that captures an image of the user. The pupil sensor can be, for example, a sensor that detects the pupil positions of both eyes of the user wearing the goggles.

[0023] The rendering image generation unit 260 calculates a gaze vector VEYE representing the gaze direction from the user's viewpoint in the virtual space for each 3D object model. The rendering image generation unit 260 can calculate the gaze vector VEYE for each 3D object model from the head tracking information acquired by the head tracking information receiving unit 270 and the eye tracking information acquired by the eye tracking information receiving unit 290.

[0024] The rendering image generation unit 260 places a 3D object model in the user's line of sight in the virtual space for each comment object included in the comment model information. If the view flag indicates that the orientation of the 3D object model should be rotated in accordance with the position of the virtual viewpoint, the rendering image generation unit 260 rotates the 3D object model so that it faces forward as viewed from the viewpoint based on the line of sight vector VEYE. If the view of the 3D object model of a text object written by the user is obstructed by a subject object placed in the virtual space based on the basic model information, the rendering image generation unit 260 moves the 3D object model within the virtual space. To realize the process of moving the 3D object model of the text object within the virtual space, the rendering image generation unit 260 includes a field of view determination unit 261, a relevance determination unit 262, a grouping unit 263, and a movement unit 264.

[0025] The visibility determination unit 261 determines whether each comment input in a virtual viewpoint image from one virtual viewpoint is positioned at a position that obstructs the visibility in a virtual viewpoint image viewed from another virtual viewpoint different from the one virtual viewpoint. The one virtual viewpoint is the virtual viewpoint of a certain user wearing goggles, and the other virtual viewpoint is the virtual viewpoint of another user wearing goggles.

[0026] For example, suppose that a user writes first text 321 and second text 322 in the virtual space as comments about a subject object 310 in a virtual viewpoint image displayed on the display 30 of the user's goggles, as shown in FIG. 2 . Another user views the first text 321 and the second text 322 written in the virtual space by the user in a position corresponding to the virtual viewpoint of the other user in the virtual viewpoint image 300 displayed on the display 30 of the user's goggles. If the virtual viewpoint of the other user is in a different position from the virtual viewpoint of the first user, the positions of the subject object 310, the first text 321, and the second text 322 in the virtual viewpoint image 300 viewed by the other user will be different from those in the virtual viewpoint image 300 of the first user. This change in position may include a change in the relative positional relationship between the subject object 310 and the first text 321 and the second text 322 in the virtual viewpoint image 300.

[0027] As shown in FIG. 3, in the virtual viewpoint image 300 viewed by the user on the display 30 of the goggles 31, the first text 321 and the second text 322 may be positioned in each of the areas 330 to 360 around the subject object 310.

[0028] For example, first text 321 and second text 322, which are comment objects that are partially or completely located in area 330 behind subject object 310, are displayed as being partially or completely hidden behind subject object 310, as shown in Fig. 4. "Behind subject object 310" means, for example, when considering a virtual axis along the direction of the user's line of sight from the user's virtual viewpoint toward subject object 310 and the comment object, the distance on this axis is greater for the comment object than for the subject object 310. "The comment object is hidden behind subject object 310" means, for example, when considering a virtual axis that includes subject object 310 and points from the user's virtual viewpoint toward subject object 310, that the comment object is located inside a virtual line passing through the boundary of a predetermined area and that is perpendicular to this axis.

[0029] Here, the predetermined area perpendicular to the virtual axis is the portion onto which the subject object 310 is projected when considering a plane perpendicular to the virtual axis at a position where the distance on the virtual axis from the user's virtual viewpoint to the subject object 310 is minimum. Also, the virtual line is a virtual line extending radially from the user's virtual viewpoint. The visibility determination unit 261 may determine whether the view is obstructed by determining whether a part or all of the comment object is hidden behind the subject object 310.

[0030] Furthermore, even if a comment object is located inside a virtual line passing through the boundary of a predetermined area, the visibility determination unit 261 may determine that the visibility of the comment object is ensured to a predetermined degree and determine that the comment object is not hidden behind the subject object 310 if the entire subject object 310 is smaller than a predetermined size or if the subject object 310 has a predetermined transparency. Furthermore, the visibility determination unit 261 may determine that the comment object is partially hidden behind the subject object 310 or that the comment object is not hidden behind the subject object 310 if a virtual line located outside the comment object contacts a portion of the subject object 310 (e.g., the stem of an apple shown in FIG. 4 ) and this portion of the subject object 310 is smaller than a predetermined size. The predetermined size may be an absolute value or a value set taking into account the size of the comment object. As described above, each first object is not limited to a comment object and may be a predetermined object. The method of determining whether a predetermined object is hidden behind the subject object 310 is the same as that described above.

[0031] For example, the first text 321 and the second text 322 arranged in regions 340 and 350 above and below the subject object 310, which are within a predetermined distance from the virtual viewpoint, are displayed in their entirety without being obstructed by the subject object 310, as shown in FIG. 5 . The first text 321 and the second text 322 arranged in regions 340 and 350 are arranged within a vicinity region 370 of the subject object 310. The vicinity region 370 is an area that is within a predetermined distance from the subject object 310. For example, the first text 321 and the second text 322 arranged in a region 360 above the subject object 310, which is away from the virtual viewpoint by more than a predetermined distance, are also displayed in their entirety without being obstructed by the subject object 310, as shown in FIG. 6 . The first text 321 and the second text 322 arranged in region 360 are arranged outside the vicinity region 370 of the subject object 310. 4 and 6 that are hidden by the subject object 310 or that are positioned outside the vicinity region 370 of the subject object 310 are less visible as annotations of the subject object 310 than in Fig. 5. In other words, the first text 321 and the second text 322 in Fig. 4 and 6 are positioned in positions that are obscured from view.

[0032] The visibility determination unit 261 determines whether or not a situation occurs in which the first text 321 and the second text 322 in the virtual viewpoint image 300 viewed by another user are positioned in positions that obstruct the view, as shown in Fig. 4 or 6. The first text 321 and the second text 322 may be written in the virtual space by different users.

[0033] In the case of Fig. 4, the visibility determination unit 261 determines that the visibility is obstructed because the first text 321 and the second text 322 are obstructed by the subject object 310 in the vicinity region 370. In the case of Fig. 6, the visibility determination unit 261 determines that the visibility is obstructed because the first text 321 and the second text 322 are located outside the vicinity region 370. The visibility determination unit 261 may determine whether the visibility of the first text 321 and the second text 322 is obstructed by the subject object 310 in the vicinity region 370 or whether the first text 321 and the second text 322 are located outside the vicinity region 370.

[0034] The visibility determination unit 261 may determine whether the visibility of the first text 321 is obstructed from a predetermined viewpoint as follows: The visibility determination unit 261 sets a virtual area surrounding the first text 321 as a frame area. The visibility determination unit 261 sets virtual lines radiating from the predetermined viewpoint as virtual lines. When a virtual line reaches a subject object or a comment object from the predetermined viewpoint, the point at which the virtual line reaches the subject object or the comment object is set as the end point. The visibility determination unit 261 determines whether the number of virtual lines reaching the frame area exceeds a first threshold. If the number of virtual lines reaching the frame area exceeds the first threshold, the visibility determination unit 261 determines that the visibility is not obstructed, and if the number of virtual lines reaching the frame area is equal to or less than the first threshold, the visibility determination unit 261 may determine whether a proportion of virtual lines reaching the frame area exceeds a second threshold. The proportion of virtual lines reaching the frame area is the number of virtual lines divided by the area of ​​the frame area. The visibility determination unit 261 may change the first threshold value and the second threshold value based on the distance from a predetermined viewpoint to the frame area. When the distance from the predetermined viewpoint to the frame area is d1, the visibility determination unit 261 may change the first threshold value and the second threshold value to be smaller than when the distance from the predetermined viewpoint to the frame area is d2 (d2<d1). The visibility determination unit 261 may change the first threshold value and the second threshold value based on the size of the frame area. The visibility determination unit 261 may calculate the size of the frame area as the area of ​​the frame area. When the size of the frame area is s1, the visibility determination unit 261 may change the first threshold value and the second threshold value to be larger than when the size of the frame area is s2 (s2<s1).

[0035] The visibility determination unit 261 may determine that the visibility is obstructed if the distance from the specified viewpoint to the first text 321 is greater than a third threshold, and may determine that the visibility is not obstructed if the distance from the specified viewpoint to the first text 321 is equal to or less than the third threshold.

[0036] The relevance determination unit 262 determines the relevance of one or more comment objects to one comment object among the comment objects of multiple comments posted by each user in the virtual space. For example, assume that the first text 321 is an annotation regarding the subject object 310, and the second text 322 is an annotation supplementing the content of the annotation in the first text 321. In this case, displaying the comment object in the second text 322 in a position close to the comment object in the first text 321 in the virtual viewpoint space 300 viewed by any user may be important for understanding the meaning of the second text 322. For example, consider a case in which the first text 321 and the second text 322 in FIG. 2 are posted by different users in the virtual space. In this case, as shown in FIG. 7, the arrangement of the comment objects in the first text 321 and the second text 322 may have a positional relationship different from that shown in FIG. 2. The relevance determination unit 262 determines, for example, whether the comment objects in the first text 321 and the second text 322 are relevant to each other.

[0037] The relevance determination unit 262 may calculate the difference in position between one comment object and another comment object as a distance, and determine that the comments are related if the difference is smaller than a predetermined distance. The relevance determination unit 262 may calculate the difference in position between one comment object and another comment object as a distance, and calculate the degree of relevance between the comment objects based on the magnitude of the distance as a relevance degree. The relevance determination unit 262 may compare a first distance between the comment objects of the first comment 321 and the second comment 322 with a second distance between the comment objects of the first comment 321 and the third comment (not shown). In this case, the relevance determination unit 262 may calculate the relevance degree between comment objects with a shorter distance between them to be higher than the relevance degree between comment objects with a longer distance between them. The relevance determination unit 262 may determine whether multiple comment objects are related by, for example, analyzing the content of the comments in each comment object. The relevance determination unit 262 may determine whether or not there is a relationship between multiple comment objects based on whether or not a comment object has information indicating that it is related to other comment objects.

[0038] The relevance determination unit 262 may determine the relevance between comment objects by analyzing the content of each comment, for example, by evaluating whether the comments contain words with similar meanings. For this evaluation, the relevance determination unit 262 may evaluate the words of each comment for each semantic classification, score them, or convert them into vectors, and evaluate the distance between words in an evaluation space (not shown) in which the scored words are arranged, or the dot product of the word vectors. Distance evaluation can use known evaluation methods, such as Mahalanobis distance, Euclidean distance, or Manhattan distance. Distance evaluation can use, for example, a threshold value that serves as a criterion for determining whether the content of comments between multiple comment objects is related.

[0039] The relevance determination unit 262 may use, for example, a flag as information indicating the relevance between a comment object and another comment object. The relevance determination unit 262 may set the flag indicating the relevance, for example, by a user performing a specific operation when inputting a comment, or may logically determine and set the relevance based on the input manner of multiple comments into the virtual space. The relevance determination unit 262 may determine, for example, whether multiple comment objects are a common object as the logic for setting the flag indicating the relevance. The relevance determination unit 262 may determine, for example, whether comments of multiple comment objects were input from a common input device as the logic for setting the flag indicating the relevance. The relevance determination unit 262 may determine, for example, that a comment input using a pen tablet and a comment input by operating input keys displayed in the virtual viewpoint image on the display 30 were input from different input devices.

[0040] The relevance determination unit 262 may determine whether multiple comment objects are a common object by, for example, determining whether the comments of the multiple comment objects were input before and after the completion of an input operation. For example, if a text box opened for inputting a comment is left open and the next comment is subsequently input, the relevance determination unit 262 can determine that the comment objects of the previous and next comments are a common object. For example, if a text box that was once closed is reopened and the next comment is input, the relevance determination unit 262 can also determine that the comment objects of the previous and next comments are a common object. For example, if a text box is closed after inputting a comment, and then a new text box is opened and the next comment is input, the relevance determination unit 262 can determine that the comment objects of the two comments are not related to each other and are separate objects. For example, the relevance determination unit 262 may determine whether multiple comment objects are a common object by, for example, determining whether the two comments were input with a "non-input period" of at least a predetermined time between them.

[0041] The grouping unit 263 groups a comment object and one or more other comment objects determined to be related to the comment object as a comment group. As described above, each first object is not limited to a comment object and may be a predetermined object. When multiple first objects include at least one predetermined object, the "comment group" is referred to as an "object group" as described above in relation to the above-mentioned replacement. That is, the grouping unit 263 groups a predetermined object and one or more other predetermined objects determined to be related to the predetermined object as an object group. Furthermore, the grouping unit 263 groups a comment object and one or more predetermined objects determined to be related to the comment object as an object group. Furthermore, the grouping unit 263 groups a predetermined object and one or more comment objects determined to be related to the predetermined object as an object group. The target of one comment object may be limited to a comment determined by the visibility determination unit 261 as being placed in a position that obstructs visibility, such as the first text 321 and the second text 322 shown in Fig. 4 or 6. Reference numeral 380 in Fig. 4 or 6 indicates a group of comments of both the comment objects of the first text 321 and the second text 322 grouped by the grouping unit 263. The grouping unit 263 may group comment objects whose relevance calculated by the relevance determination unit 262 is greater than a predetermined value.

[0042] When generating a virtual viewpoint image from another virtual viewpoint, the moving unit 264 moves the comment objects of the grouped comment group 380 to positions in the virtual viewpoint image 300 from the other virtual viewpoint where the view is not obstructed. For example, a position where the view is not obstructed is a position within the areas 340 and 350 in the example of FIG. 3 , or a position within the nearby area 370 in the example of FIG. 5 . The moving unit 264 may arrange the moved comment objects based on the relevance. The moving unit 264 may arrange a comment object with a high relevance closer to a related comment object than a comment object with a low relevance.

[0043] The moving unit 264 may calculate a position where the view is not obstructed from a predetermined viewpoint as follows, and move the comment object of the comment group 380 to the calculated position. The moving unit 264 sets a virtual area surrounding the comment object of the comment group 380 as a frame area. The moving unit 264 sets virtual lines radiating from the predetermined viewpoint as virtual lines. When the virtual line reaches the subject object or the comment object from the predetermined viewpoint, the point where the virtual line reaches is set as the end point. The moving unit 264 assumes any position in the virtual space as a destination point Q, and moves the frame area to the position of point Q. The moving unit 264 may position point Q at the center of the frame area. The moving unit 264 determines whether the number of virtual lines reaching the frame area exceeds a first threshold. If the number of virtual lines reaching the frame area exceeds the first threshold, the moving unit 264 calculates a position where the view is not obstructed as the position of point Q. If the number of virtual lines reaching the frame area is equal to or less than the first threshold, the moving unit 264 assumes a new point other than point Q as point Q. The moving unit 264 repeats the process, starting with determining whether the number of virtual lines reaching the frame area exceeds the first threshold. The moving unit 264 may determine whether the proportion of virtual lines reaching the frame area exceeds a second threshold. Here, the proportion of virtual lines reaching the frame area is the number of virtual lines divided by the area of ​​the frame area. The moving unit 264 may change the first threshold or the second threshold based on the distance from a predetermined viewpoint to the frame area. When the distance from the predetermined viewpoint to the frame area is d1, the moving unit 264 may change the first threshold or the second threshold to a smaller value than when the distance from the predetermined viewpoint to the frame area is d2 (d2<d1). The visibility determination unit 261 may change the first threshold or the second threshold based on the size of the frame area. The moving unit 264 may calculate the size of the frame area as the area of ​​the frame area. When the size of the frame region is s1, the movement unit 264 may change the first threshold value and the second threshold value to be larger than when the size of the frame region is s2 (s2<s1).

[0044] The movement unit 264 may calculate, as a position where the field of view is not obstructed, any position in the virtual space where the distance from a predetermined viewpoint to the frame area is equal to or less than a third threshold value.

[0045] If the distance from the predetermined viewpoint to point Q is L, distance L may be calculated as the square root of the sum of the squares of the differences between the coordinate values ​​of two points P1 (x1, y1, z1) and P2 (x2, y2, z2). It can be expressed as:

[0046] The moving unit 264 moves the comment objects of the comment group 380 in the virtual viewpoint image 300 from another virtual viewpoint in a direction away from the subject object 310. The movement direction of the comment objects of the comment group 380 can be, for example, a direction along the coordinate axis with the larger coordinate value of the position of one comment object in a two-dimensional plane coordinate system based on the subject object 310. The coordinate value of the position of one comment object can be the coordinate value of a predetermined position that is appropriately set, such as the coordinate value of one comment object located closest to the subject object 310 or the coordinate value located at the center of one comment object. FIG. 8 shows a two-dimensional plane coordinate system based on the subject object 310 in the virtual viewpoint image 300. The two-dimensional plane coordinate system used here is not a world coordinate system, but a coordinate system on an XY plane in which the line of sight direction is the Z axis, the horizontal direction of the space is the X axis, and the vertical direction of the space is the Y axis. In the example of FIG. 8 , the coordinate value of the comment objects of the comment group 380 on the X-axis, which is the horizontal axis in a two-dimensional plane coordinate system, is greater than the coordinate value on the Y-axis, which is the vertical axis. In this case, the moving unit 264 moves the comment objects of the comment group 380 along the X-axis direction in a direction away from the subject object 310. After the movement, the comment objects of the comment group 381 are positioned in a position that does not obstruct the field of view. For example, as shown in FIG. 9 , when a leader line 390 corresponding to the comment group 380 is written in the virtual space and displayed in the virtual viewpoint image 300, the moving unit 264 may determine the movement direction of the comment objects of the comment group 380 based on the leader line 390. The leader line 390 is written in the virtual space with the intention of associating at least one of the first text 321 and the second text 322 of the comment group 380 with the subject object 310. The movement unit 264, for example, places the end point of the lead line 390 on the subject object 310 side at the origin of the two-dimensional plane coordinate system in Fig. 8 and develops the lead line 390 in the two-dimensional plane coordinate system. Based on the coordinate value of the end point of the lead line 390 on the comment group 380 side, the movement unit 264 can determine the movement direction of the comment object of the comment group 380 to be along the X-axis direction.The moving unit 264 may redraw the leader line 390 for the comment object of the moved comment group 381. In other words, the moving unit 264 may move the end point of the leader line 390, where the leader line 390 meets the comment group 380 before the movement, to the end point where the leader line 390 meets the comment object of the moved comment group 381. The moving unit 264 may move in a direction along a coordinate axis having a larger absolute value of the coordinate value of the position of one comment object in a two-dimensional plane coordinate system based on the subject object 310.

[0047] The rendering image transmission unit 280 transmits the data of the rendering image generated by the rendering image generation unit 260 to the display 30 which displays the rendering image as a virtual viewpoint image 300 .

[0048] An example of the procedure of the image processing method according to the embodiment executed by the image processing device will be described below with reference to the flowcharts of FIGS.

[0049] 10, the comment information acquisition unit 210 acquires comment information (step S101). The comment information includes a 3D object model of a comment object and information about the position and orientation in the virtual space where the comment is written.

[0050] The view flag generating unit 220 generates a view flag indicating whether the orientation of the comment object is fixed or rotated (step S102). For example, the flag is a 1-bit signal, and if it is High, it indicates a rotated orientation, and if it is Low, it indicates a fixed orientation.

[0051] The view flag combining unit 230 combines the comment information acquired by the comment information acquiring unit 210 with the view flag generated by the view flag generating unit 220. Through this combination, the view flag combining unit 230 generates comment model information (step S103).

[0052] The comment model information update unit 250 reads out the basic model information and comment model information stored from the model information storage unit 240 (step S104). The comment model information update unit 250 adds the comment model information generated by the view flag synthesis unit 230 to the comment model information read out from the model information storage unit 240 (step S105). The comment model information update unit 250 stores the comment model information including the added information in the model information storage unit 240, and updates the comment model information in the model information storage unit 240 (step S106).

[0053] The head tracking information receiving unit 270 acquires head tracking information (step S107). The eye tracking information receiving unit 290 acquires eye tracking information (step S108). The rendering image generating unit 260 calculates a gaze vector VEYE, which represents the gaze direction with the position of the user's eyeball in the virtual space as the viewpoint, from the head tracking information and the eye tracking information (step S109). The rendering image generating unit 260 assigns 1 to the value of variable i (step S110).

[0054] As shown in FIG. 11 , the rendering image generation unit 260 extracts the i-th comment model information related to the i-th comment object from the N comment model information related to the N comment objects (step S111). It determines whether the view flag of the i-th comment model information is High (step S112). If the view flag is Low (No in step S112), the rendering image generation unit 260 retains the 3D object model of the i-th comment object in the same orientation as when it was written (step S113). If the view flag is High (Yes in step S112), the rendering image generation unit 260 rotates the orientation of the 3D object model of the i-th comment object (step S114). This rotation causes the normal vector pointing toward the front of the text object and the line of sight vector VEYE to become parallel and opposite in direction. This allows the text object to appear directly in front when looking in the direction of the line of sight vector VEYE from the head tracking position acquired by the head tracking information receiving unit 270. The rendering image generating unit 260 places the 3D object model of the i-th text object in the virtual space of the basic model information read from the model information storage unit 240 based on the position information in the virtual space (step S115).

[0055] As a visibility determination step, the visibility determination unit 261 determines whether the visibility of the i-th comment object is obstructed when the virtual space is viewed in the direction of the line-of-sight vector VEYE (step S116). If the visibility of the i-th comment object is not obstructed (No in step S116), the current position of the comment object is maintained, and the process proceeds to step S118, which will be described later.

[0056] If the view of the i-th comment object in the virtual space is obstructed (Yes in step S116), the rendering image generation unit 260 moves the i-th comment object or a comment object in the comment group 380 to a position where the view is not obstructed (step S117).

[0057] The moving process of step S117 will be described with reference to FIG. 12 . As a moving step, the relevance determination unit 262 determines whether or not there are other comment objects that are related to the i-th comment object (step S1171). If there are no other comment objects that are related (No in step S1171), the process proceeds to step S1173, which will be described later. If there are other comment objects that are related (Yes in step S1171), the grouping unit 263 performs a grouping step (step S1172). In step S1172, the grouping unit 263 groups the i-th comment object and the other comment objects into one comment group 380. Thereafter, the process proceeds to step S1173. In step S1173, as a moving step, the moving unit 264 moves the i-th comment object to a position where the view is not obstructed. If there are other comment objects grouped with the i-th comment object, the moving unit 264 moves all of the comment objects in the comment group 380 to a position where the view is not obstructed. Thereafter, the moving process of step S117 ends, and the process proceeds to step S118.

[0058] As shown in FIG. 11 , in step S118, the rendering image generation unit 260 determines whether the variable i is smaller than N. If the variable i is smaller than N (Yes in step S118), the rendering image generation unit 260 adds 1 to the value of the variable i (step S119) and returns to step S111. If the variable i is not smaller than N (No in step S118), the rendering image generation unit 260 generates a rendering image (step S120). This rendering image is a 2D image of the spatial region visible from the head tracking position, representing the entire model of the virtual space after the placement and movement of N comment objects. This rendering image can be generated based on head tracking information acquired by the head tracking information receiving unit 270. The rendering image transmission unit 280 transmits the rendering image generated by the rendering image generation unit 260 to the display 30 (step S121). This completes the series of processes. The processes in steps S116 and S117 in FIG. 11 (steps S1171 to S1173 in FIG. 12) can constitute the steps of the image processing program of this embodiment.

[0059] When the grouping unit 263 groups the i-th comment object and other comment objects into one comment group 380 (step S1172), the other comment objects need to be treated the same as the i-th comment object. Therefore, the rendering image generation unit 260 may store, as the i-th overlap number, the ordinal number of the other comment objects grouped in the same comment group 380 as the i-th comment object. In this case, the rendering image generation unit 260 may include a ROM or a RAM. After adding 1 to the value of the variable i (step S119), if the variable i is equal to the stored overlap number, the rendering image generation unit 260 may determine whether the variable i is smaller than N (step S118) without returning to step S111.

[0060] In this embodiment, it is determined that comment objects such as the first comment 321 and the second comment 322 are positioned in a position that obstructs the view of the comment object in the virtual viewpoint image 300 displayed on the display 30. If there is a comment object related to the comment object positioned in the obstructed view, the grouping unit 263 groups the obstructed view comment object and the related comment objects into one text group 380. The moving unit 264 collectively moves the text objects of the grouped text group 380 to a position where the view of the comment object is not obstructed. Therefore, when one comment object is moved, other comment objects related to that comment object are also moved to nearby locations. This makes it possible to appropriately position multiple comment objects positioned in the virtual viewpoint image 300 while taking into account the relevance between the comment objects. Furthermore, it is possible to prevent multiple comment objects of the related comment group 380 from being positioned outside the vicinity region 370, far away from the subject object 310, due to movement. Alternatively, it is possible to prevent a plurality of comment objects in the comment group 380 from being moved to a position where they are hidden by the subject object 310, thereby preventing the view of the comment objects from being obstructed.

[0061] In this embodiment, the 3D object model of a text object that appears upside down in the virtual viewpoint image 300 is rotated based on the view flag by the processing of the rendering image generation unit 260 in steps S112 and S115 of Fig. 11. This processing of rotating the 3D object model based on the view flag may be omitted. When the processing of rotating the 3D object model is omitted, at least the processing of steps S102 and S103 of Fig. 10 and steps S112, S115, and S116 of Fig. 11 can be deleted.

[0062] An image processing device according to an embodiment includes a controller. In a virtual viewpoint image viewed from a predetermined virtual viewpoint in the virtual space, the virtual viewpoint image is generated based on a first object placed in a virtual space and second objects placed in the virtual space excluding the first object. The controller determines whether one of the first objects is located in a position that obstructs a view from the predetermined virtual viewpoint, determines a relationship between the one first object and other first objects among the first objects, groups the one first object and the other first objects determined to have a relationship with the one first object as an object group, and moves the first object of the object group to a position in the virtual viewpoint image viewed from the predetermined virtual viewpoint that does not obstruct the view. Each of the first objects may be a comment object or a predetermined object. The second object is a subject object.

[0063] As described above, the multiple first objects may be multiple comment objects. Therefore, the image processing device according to the present invention may employ the following configuration. That is, the image processing device according to the embodiment generates a virtual viewpoint image viewed from a virtual viewpoint in the virtual space based on comment objects in which comments are placed in a virtual space and subject objects, which are objects placed in the virtual space excluding the comment objects. The image processing device includes a visibility determination unit, a relevance determination unit, a grouping unit, and a moving unit. The visibility determination unit determines whether the comment objects are placed in a position that obstructs a view in the virtual viewpoint image from a predetermined virtual viewpoint. The relevance determination unit determines the relevance of other comment objects to one of the comment objects. The grouping unit groups the one comment object and the other comment objects determined to be relevant to the one comment object as a comment group. The moving unit moves the comment objects of the comment group to positions in the virtual viewpoint image from the predetermined virtual viewpoint that do not obstruct a view.

[0064] The visibility determination unit may determine whether the one comment object is blocked from view by the subject object in an area within a predetermined distance from the subject object, or whether the one comment object is located outside the area.

[0065] The moving unit may move the comment objects of the comment group along a coordinate axis on which the coordinate value of one of the comment objects is larger in a two-dimensional plane coordinate system based on the subject object.

[0066] An image processing method according to an embodiment generates a virtual viewpoint image viewed from a virtual viewpoint in a virtual space based on comment objects, each of which is a comment placed in a virtual space, and subject objects, each of which is an object placed in the virtual space excluding the comment object. The image processing method determines whether the comment object is placed in a position that obstructs a view in the virtual viewpoint image from a predetermined virtual viewpoint, determines the relevance of one of the comment objects to other comment objects among the comment objects, groups the one comment object and the other comment objects determined to have a relevance to the one comment object as a comment group, and moves the comment objects of the comment group to positions in the virtual viewpoint image from the predetermined virtual viewpoint that do not obstruct a view.

[0067] The image processing program according to the embodiment causes a controller having an arithmetic unit and an input / output unit to execute the following steps: a visibility determination step for determining whether a comment object is placed in a position that obstructs visibility in a virtual viewpoint image viewed from a predetermined virtual viewpoint in a virtual space, the virtual viewpoint image being generated based on a comment object in which a comment is placed in a virtual space and a subject object which is an object placed in the virtual space excluding the comment object; a relevance determination step for determining the relevance of one of the comment objects to other comment objects; a grouping step for grouping the one comment object and the other comment objects determined to have a relevance to the one comment object as a comment group; and a movement step for moving the comment objects of the comment group to positions in the virtual viewpoint image from the predetermined virtual viewpoint that do not obstruct visibility.

[0068] Although the embodiments of the present invention have been described above, the embodiments can be modified or varied based on the above disclosure. All components of the above embodiments and all features described in the claims may be individually extracted and combined as long as they are not mutually inconsistent.

[0069] The entire contents of Japanese Patent Application No. 2023-161773 (filing date: September 25, 2023) are incorporated herein by reference.

[0070] The present invention provides a technology that can be used for information sharing among multiple users who share a virtual space such as mixed reality or virtual reality.

Claims

1. An image processing device having a controller, wherein the controller: determines whether a first object among the first objects is located in a position that obstructs a view from a predetermined virtual viewpoint in a virtual space, the virtual viewpoint image being generated based on a first object located in a virtual space and a second object located in the virtual space excluding the first object; determines a relationship of other first objects among the first objects to the one first object; groups the one first object and the other first objects determined to have a relationship to the one first object as an object group; and moves the first object of the object group to a position in the virtual viewpoint image from the predetermined virtual viewpoint where the view is not obstructed.

2. The image processing device according to claim 1, wherein the position where the view is obstructed is a position where the one first object is hidden behind the second object as viewed from the specified virtual viewpoint, or a position outside an area that is less than a specified distance from the second object, and the position where the view is not obstructed is a position where the first object of the group of objects is not hidden behind the second object as viewed from the specified virtual viewpoint.

3. The image processing device according to claim 1 or 2, wherein the position where the view is obstructed is a position where a part or all of the one first object is hidden behind the second object, a position where the number of first virtual lines extending radially from the specified viewpoint reaching a first virtual area surrounding the one first object is equal to or less than a first threshold value, or a position where the ratio of the number of the first virtual lines to the area of ​​the first virtual area is equal to or less than a second threshold value.

4. The image processing device according to claim 1 or 2, wherein the position where the view is not obstructed is a position within an area where the second object is at a distance less than the specified distance, or a position where the distance from the specified viewpoint to the first object of the group of objects is less than a third threshold value.

5. The image processing device according to claim 1 or 2, wherein the position where the view is not obstructed is a position where the number of second virtual lines extending radially from the specified viewpoint reaching a second virtual area surrounding the first object of the group of objects exceeds a fourth threshold value, or a position where the ratio of the number of the second virtual lines to the area of ​​the second virtual area exceeds a fifth threshold value.

6. The image processing device of claim 1 or 2, wherein when the controller moves the first object of the object group to a position where the view is not obstructed, the controller moves the first object of the object group along the coordinate axis having the larger coordinate value of the position of the one first object in a two-dimensional plane coordinate system based on the second object.

7. An image processing method comprising: determining whether one of the first objects is located in a position that obstructs a view from a predetermined virtual viewpoint in a virtual space, the virtual viewpoint image being generated based on a first object located in a virtual space and a second object located in the virtual space excluding the first object; determining a relationship of other first objects among the first objects to the one first object; grouping the one first object and the other first objects determined to have a relationship to the one first object as an object group; and moving the first object of the object group to a position in the virtual viewpoint image from the predetermined virtual viewpoint where the view is not obstructed.

8. An image processing program that causes a controller having a calculation unit and an input / output unit to execute the following: determining whether or not one of the first objects is located at a position that obstructs a view from a predetermined virtual viewpoint in a virtual space, the virtual viewpoint image being generated based on a first object located in a virtual space and a second object located in the virtual space excluding the first object; determining a relationship of other first objects among the first objects to the one first object; grouping the one first object and the other first objects determined to have a relationship to the one first object as an object group; and moving the first objects of the object group to a position in the virtual viewpoint image from the predetermined virtual viewpoint where the view is not obstructed.

Images