Display method and display system

Abstract

To easily determine the placement of information.SOLUTION: A display method displays registration information associated with a parent object in a three-dimensional space together with the parent object in the subjective viewpoint of an observer. The display method comprises: energy map generation processing for generating one or more energy maps that are data regarding a two-dimensional plane indicating a region suitable for placement of the registration information at a viewpoint position of each of one or more observers; reference position determination processing for setting a reference position for displaying the registration information at a second position having lower energy than at a first position in the energy maps; and drawing processing for drawing the registration information to be drawn in a region including the parent object and the reference position in the subjective viewpoint of the observer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display method and a display system. [Background technology]

[0002] A configuration for simultaneously viewing an object and information in a three-dimensional space is widely known. Patent Document 1 discloses an information processing device that includes a control unit that controls the display of a character string represented by text information related to an object included in a content image displayed in a display area, and the control unit determines an arrangement area in the display area where the character string is to be arranged based on the position of the object relative to the display area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 181488 Summary of the Invention [Problem to be solved by the invention]

[0004] In the invention described in Patent Document 1, there is room for improvement in the method for determining the display position of information. [Means for solving the problem]

[0005] A display method according to a first aspect of the present invention is a display method for displaying registration information associated with a parent object in three-dimensional space together with the parent object from an observer's subjective viewpoint, and includes an energy map generation process for creating one or more energy maps, which are data on two-dimensional planes indicating areas suitable for arranging the registration information, at the viewpoint positions of one or more observers, respectively; a reference position determination process for setting a reference position for displaying the registration information at a second position in the energy map that has lower energy than a first position; and a drawing process for drawing the registration information drawn in an area including the parent object and the reference position from the observer's subjective viewpoint. A display system according to a second aspect of the present invention is a display system that displays registration information associated with a parent object in three-dimensional space together with the parent object from an observer's subjective viewpoint, and includes: an energy map generation unit that creates one or more energy maps, which are data on two-dimensional planes that indicate areas suitable for arranging the registration information, at the viewpoint positions of one or more of the observers; a reference position determination unit that sets a reference position in the energy map to display the registration information at a second position that has lower energy than a first position; and an image generation unit that draws the registration information drawn in an area including the parent object and the reference position from the observer's subjective viewpoint. [Effects of the Invention]

[0006] According to the present invention, the arrangement of information can be determined easily. [Brief explanation of the drawings]

[0007] [Figure 1] Overall configuration of the information presentation system [Figure 2] FIG. 10 is a diagram showing an example of an object table. [Figure 3] FIG. 10 is a diagram showing an example of an information table and a user table. [Figure 4] Hardware configuration diagram of a computing device representing a server and a client [Figure 5] A diagram explaining the situational setting that is the premise for explaining energy images [Figure 6] Diagram explaining energy images [Figure 7] Flowchart showing the processing of the information placement unit [Figure 8] Flowchart showing energy image generation processing [Figure 9] Flowchart showing reference position determination processing [Figure 10] Flowchart showing individual vector calculation processing [Figure 11] Flowchart showing the evaluation energy calculation process [Figure 12] Overall configuration diagram of information presentation system in Modification 1 [Figure 13] Overall configuration diagram of information presentation system in Modification 2 [Figure 14] Overall configuration diagram of an information presentation system according to a second embodiment [Figure 15] Diagram showing the worker [Figure 16] Diagram showing the worker's head [Figure 17] 10 is a flowchart showing an outline of processing performed by an information presentation system according to a second embodiment. [Figure 18] FIG. 10 is a diagram showing an example of a work record table. [Figure 19] FIG. 1 shows an example of an image table and an audio table. [Figure 20] FIG. 1 shows an example of a procedure table and a working posture table. [Figure 21] Detailed configuration diagram of the recording section and post-analysis section [Figure 22] Flowchart showing risk calculation processing by a risk judgment program [Figure 23] FIG. 10 is a diagram showing the display of registration information in the browsing phase of the second embodiment. [Figure 24] A diagram showing an example of a display template [Figure 25] A diagram showing the replay explanation video seen from behind the worker [Figure 26] A diagram showing the replay explanation video from the worker's side perspective [Figure 27] 10 is a flowchart showing an outline of processing performed by an information presentation system according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] -First embodiment- Hereinafter, a first embodiment of the display method will be described with reference to FIGS.

[0009] In this embodiment, a method of presenting information when viewing a virtual space from a first-person perspective will be described. However, as will be shown in a modified example described later, this method is also effective in cases where a computer image is superimposed on a video of the real world, a so-called augmented reality. In this embodiment, the registration information presented to the user is displayed in an area called an information presentation window. However, this name is for convenience, and the window may be transparent to display only the registration information. For convenience of explanation, the information presentation window will be described as rectangular, but it may also be polygonal or elliptical.

[0010] (Overall composition) FIG. 1 is a diagram illustrating the overall configuration of an information presentation system 1. The information presentation system 1 includes a server 10 and one or more clients 20. The server 10 and each client 20 are connected via a wireless or wired network. The distance between the server 10 and the clients 20 is not important; for example, they may be in the same room or thousands of kilometers apart. The distance between the clients 20 is also not important. Although FIG. 1 illustrates three clients 202 and only one user U, there is no limit to the number of clients 20 and users U included in the information presentation system 1. The information presentation system 1 includes the same number of clients 20 and users U. In the following, the users U are also referred to as "observers."

[0011] Each client 20 is used by an individual user U who uses an input device 961 and a video display device 962. The user U freely moves around and views the virtual space (hereinafter also referred to as a "specific space") generated by the server 10. That is, the server 10 generates an image of the virtual space from the viewpoint of each user U, i.e., the first-person viewpoint of the user U, and transmits it to each client 20. The client 20 transmits movement commands and viewpoint operation commands from the user U to the server 10. The server 10 updates the position and line of sight of each user based on the movement commands and viewpoint operation commands transmitted from the client 20, and generates an image of the virtual space corresponding to the updated viewpoint and transmits it to each client 20.

[0012] The server 10 includes a table update unit 11, an information placement unit 12, an image generation unit 13, and a storage device 14. The storage device 14 stores an object table 1000, an information table 1100, and a user table 1200. All of the data stored in the storage device 14 is pre-recorded, except for a viewpoint position 1204 and a line-of-sight direction 1205 in the user table 1200. The table update unit 11 updates the user table 1200 based on movement commands and line-of-sight operation commands transmitted from the client 20.

[0013] The information placement unit 12 places the registration information stored in the information table 1100 in the virtual space. The information placement unit 12 includes an energy map generation unit 12a and a reference position determination unit 12b. Detailed operations of the information placement unit 12 will be described later. The image generation unit 13 generates an image of the virtual space based on the viewpoint position and line of sight direction of each user U and transmits it to each client 20.

[0014] The storage device 14 is a non-volatile information storage device such as a hard disk drive. The object table 1000 stores data such as the position, size, and appearance of objects existing in the virtual space. The data stored in the object table 1000 is read by the image generation unit 13. The information table 1100 stores various pieces of information in the virtual space as registration information. The data stored in the information table 1100 is read by the image generation unit 13 when it is placed in the virtual space by the information placement unit 12. In other words, if the data stored in the object table 1000 is within the field of view of each user U, it is used for rendering processing by the image generation unit 13.

[0015] The user table 1200 stores data such as an avatar representing the appearance of each user U in the virtual space, a viewpoint position, and a viewpoint direction. The viewpoint position and viewpoint direction in the user table 1200 are updated as appropriate by the table update unit 11. Note that a predetermined fixed value is used as the viewing angle of the user U, so the field of view of the user U can be calculated from the viewpoint position and viewpoint direction. Specific examples of the object table 1000, the information table 1100, and the user table 1200 will be described later.

[0016] The client 20 includes a command transmission unit 21 and a video output unit 22. The command transmission unit 21 transmits movement commands and gaze operation commands to the server 10 based on the output of an input device 961 operated by the user U. The video output unit 22 receives a video of the virtual space from the server 10 and outputs the received video to the input device 961.

[0017] The user U operates an input device 961 and a video display device 962. The input device 961 transmits operation commands for the user U's movement and line of sight in the virtual space to the client 20. The input device 961 is a combination of a mouse and keyboard, a joystick, or an external camera and computing unit. When a mouse and keyboard are used, when the user U presses a predetermined key on the keyboard, a command to move forward, backward, left, or right is output to the client 20, and when the user U operates the mouse, a command to move the line of sight is output to the client 20. When a joystick is used, the movement command and line of sight movement are output to the client 20 depending on the tilt direction of two analog sticks provided on the joystick.

[0018] When an external camera and computing unit are used, the camera observing the movements of user U recognizes the movements of user U and the movements of the user U's head, and outputs movement commands and gaze movements to client 20. In this case, the camera may be attached integrally to user U, or may be fixed in a position where it can capture an image of user U. The video display device 962 is, for example, a stationary liquid crystal display or a head-mounted display. The functions of the computing unit described here may also be realized by client 20.

[0019] (table) FIG. 2 is a diagram showing an example of an object table 1000. The object table 1000 has multiple records, and each record has fields for an element model ID 1001, 3D model data 1002, item name 1003, bounding box 1004, center position 1005, importance 1006, related keyword tag 1007, and saved information ID 1008. The element model ID 1001 stores an element model ID that is an identifier unique to the object. The 3D model data 1002 stores a URI indicating the location where the 3D model of the item is saved. The item name 1003 stores the name of the object. In other words, while there may be multiple records with the same item name 1003 in the object table 1000, there are no records with the same element model ID 1001.

[0020] Bounding box 1004 stores two three-dimensional coordinates that are diagonal corners of a rectangular parallelepiped region that contains the object. Center position 1005 stores three-dimensional coordinates that are the center position of the object. Importance 1006 stores the importance of the object. In this embodiment, importance 1006 is expressed as an integer value, with a larger value indicating greater importance, but importance 1006 may also be a ranking result such as "A," "B," or "C." Related keyword tag 1007 stores keywords related to the object. Related saved information ID 1007 stores a saved information ID that is an identifier of saved information related to the object.

[0021] 3 is a diagram showing an example of the information table 1100 and the user table 1200. The information table 1100 has a plurality of records, and each record has fields for a saved information ID 1101, importance 1102, template ID 1103, keyword list 1104, description 1105, image 1106, related saved information 1107, and parent model ID 1108. The saved information ID 1101 stores an identifier of registered information. The importance 1102 stores the importance of registered information.

[0022] The template ID 1103 stores the identifier of the template used to display the registered information. The keyword list 1104 stores key sites related to the registered information. The description 1105 stores a sentence explaining the registered information. The image 1106 stores a URI indicating the location where an image related to the registered information is stored. The related saved information 1107 stores the saved information ID of other registered information related to that registered information. The parent model ID 1108 stores the element model ID 1001 of the object with which the registered information is associated.

[0023] The user table 1200 has multiple records, and each record has fields for a user ID 1201, a user name 1202, an avatar model 1203, a viewpoint position 1204, a gaze direction 1205, and a depth preference 1206. The user ID 1201 stores an identifier for identifying a user. The user name 1202 stores the user's name. The avatar model 1203 stores a URI indicating a location where a 3D model representing the user's appearance is saved. The viewpoint position 1204 stores three-dimensional coordinates indicating the user's most recent viewpoint position. The gaze direction 1205 stores data indicating the user U's most recent gaze direction, such as the yaw angle, roll angle, and pitch angle based on a predetermined posture. The depth preference 1206 stores the distance at which each user U prefers information to be displayed.

[0024] (Hardware configuration) 4 is a hardware configuration diagram of a computing device 40 representing the server 10 and the client 20. The hardware configurations of the server 10 and the client 20 are not necessarily identical, but the server 10 and the client 20 have in common the fact that they include the hardware of the computing device 40 described below. The computing device 40 includes a CPU 41, which is a central processing unit, a ROM 42, which is a read-only storage device, a RAM 43, which is a read / write storage device, an input / output interface 44, which is a contact point with the user interface, and a communication device 45. The CPU 41 performs various calculations by expanding programs stored in the ROM 42 into the RAM 43 and executing them.

[0025] The arithmetic device 40 may be realized by a field programmable gate array (FPGA), which is a rewritable logic circuit, or an application specific integrated circuit (ASIC), which is an integrated circuit for a specific application, instead of the combination of the CPU 41, the ROM 42, and the RAM 43. Furthermore, the arithmetic device 40 may be realized by a combination of different configurations, for example, a combination of the CPU 41, the ROM 42, the RAM 43, and the FPGA, instead of the combination of the CPU 41, the ROM 42, and the RAM 43.

[0026] The input / output interface 44 is a general-purpose communication port and video signal output device, and various peripheral devices can be connected to it. For example, a mouse, keyboard, joystick, camera, display, etc. can be connected to the input / output interface 44. The communication device 45 is a communication module compatible with IEEE802.3 or IEEE802.11, for example. The table update unit 11 and information placement unit 12 of the server 10 are realized by a program executed by the CPU 41. The video generation unit 13 is realized by a program executed by the CPU 41 and the communication device 45. The command transmission unit 21 of the client 20 is realized by a program executed by the CPU 41, the input / output interface 44, and the communication device 45. Similarly, the video output unit 22 is realized by a program executed by the CPU 41, the input / output interface 44, and the communication device 45.

[0027] (energy image) The energy image will be described with reference to FIGS. 5 and 6. Specifically, FIG. 5 is a diagram for explaining a situation setting that is a premise for explaining the energy image, and FIG. 6 is a diagram for explaining the energy image. In FIG. 5, an object 901 is placed in the line of sight of a user u in a virtual space, and an object 902 is placed to the left of the object 901. An information presentation window 904 presenting information about the object 901 is placed to the right of the object 901. Note that in FIG. 5, the dashed line indicates the field of view of the user u in the virtual space. In this case, the video generation unit 13 generated the scene image 905 shown in the lower part of FIG. 5 from the subjective viewpoint of the user u in the virtual space. The energy image will be described with reference to the following FIG. 6, assuming a situation in which the information placement unit 12 places another information presentation window related to the information presentation window 904 immediately after the scene image 905 is generated.

[0028] The energy image 80 is an image used in calculations to determine the position at which the next information presentation window will be displayed from the viewpoint of each user U. The energy image 80 is a collection of data for each grid-like region. However, the energy image is merely referred to as an "image" for convenience, and does not have to be an image file itself, such as a bitmap or JPEG. The energy image 80 can also be called an "energy map." In this embodiment, each pixel of the energy image has a value, for example, from -100 to +100. The numerical value of each pixel is called the "energy amount." The energy image indicates that the smaller the pixel numerical value, i.e., the lower the energy amount, the more suitable it is for arranging the information presentation window. For example, a position with an energy amount of -10 is more suitable for arranging the information presentation window than a position with an energy amount of zero.

[0029] Energy image 80 is created based on object map 81, target map 82, and adjustment map 83. In FIG. 6, areas without color indicate an initial value of zero, positive values ​​are indicated by diagonal hatching, and negative values ​​are indicated by point cloud hatching. Note that point cloud hatching is also called "point filling" or "dot hatching." The number of pixels in energy image 80, object map 81, target map 82, and adjustment map 83 is the same.

[0030] The object map 81 is a map in which a large positive value is set in the area where an object that exists in the field of view and has an importance higher than a predetermined threshold is present. Note that the objects referred to here also include existing information presentation windows. In the object map 81 shown in FIG. 6, a large positive value is set in the position where each object was located in the scene image 905. Since the object map 81 defines a positive energy amount, a newly placed information presentation window is placed so as to avoid the area set in the object map 81.

[0031] The object map 82 is a map in which a negative value is set for an object to which a newly placed information presentation window is related. For example, a negative value is set for a perfect circle or ellipse that is centered at the center of the related object and encompasses the object. Note that the star in the object map 82 shown in FIG. 6 indicates the center of the object 901. The object map 82 allows the new information presentation window to be placed near the related object.

[0032] The adjustment map 83 is set when a new information presentation window is related to another existing information presentation window. In the adjustment map 83, negative values ​​are set to the top, bottom, left, and right of the related existing information presentation window 904. No special value is set for the position of the information presentation window 904, and it remains at the initial value of zero. The adjustment map 83 makes it easier for the new information presentation window to be placed near the other existing information presentation window, specifically, above, below, left, or right.

[0033] The negative values ​​set in the adjustment map 83 do not have to be the same. For example, the right side of the existing information window may be set to the largest negative value, the left side to the next largest negative value, the top to the third largest negative value, and the bottom to the smallest negative value. With this skewed distribution, the new information window is most likely to be placed at the bottom of the existing information window.

[0034] These three maps, specifically, the object map 81, the target object map 82, and the adjustment map 83, are overlaid and blurred to form the energy image 80. Note that if the new information presentation window is not related to any other existing information presentation window, there is no adjustment map 83, and therefore the object map 81 and the target object map 82 are overlaid and blurred to form the energy image 80. However, due to limitations of the drawing, the blurring in the energy image 80 cannot be shown in Figure 6. Note that overlaying maps means adding the values ​​of corresponding pixels. The blurring can be performed using, for example, Gaussian blurring.

[0035] An energy graph 911 shown at the bottom of the energy image 80 illustrates the amount of energy at the position of the dashed dotted line 910 in the energy image 80. Reference numerals 912 to 915 in the energy graph 911 indicate the respective regions in the horizontal direction in the drawing. The region corresponding to region 912 in the energy image 80 is white, i.e., zero. However, since the adjacent region 913 is set to a large positive value in the object map 81, a blurring process results in a smooth increase from region 912 to region 913. Region 914 has a large positive value set in the object map 81, a negative value set in the target object map 82, and a negative value set in the adjustment map 83, and therefore has a smaller amount of energy than region 913.

[0036] (flowchart) 7 is a flowchart showing the processing of the information placement unit 12. First, in step S301, the information placement unit 12 identifies the current scene, i.e., the location and related users. For example, a specific range in the virtual space is designated as the current location from the outside, and the information placement unit 12 identifies users whose viewpoint positions 1204 in the user table 1200 are within that range. In the following step S302, the information placement unit 12 lists the registered information to be displayed.

[0037] There is no upper limit to the number of pieces of registration information to be displayed, but for example, the information arrangement unit 12 may list the 10 most important pieces of registration information of objects located at the location identified in step S301. The information arrangement unit 12 may list all pieces of registration information whose importance 1102 is higher than a predetermined threshold and whose parent object is within a predetermined distance from the current location of the user U. In the following step S303, the information arrangement unit 12 selects one piece of unprocessed registration information from the registration information listed in step S302 as information to be processed. In the following step S304, the information arrangement unit 12 lists the users identified in step S301.

[0038] In the following step S305, the information arrangement unit 12 selects one of the unprocessed users from the users listed in step S304 as a user to be processed, and proceeds to step S306. In step S306, the energy map generation unit 12a of the information arrangement unit 12 performs energy image generation processing, and proceeds to step S307. Details of the energy image generation processing will be described later with reference to FIG. 8. In step S307, the information arrangement unit 12 determines whether all users listed in step S304 have been processed. If the information arrangement unit 12 determines that all users have been processed, it proceeds to step S308, and if it determines that there is an unprocessed user, it returns to step S305.

[0039] In step S308, the reference position determination unit 12b of the information placement unit 12 executes a reference position determination process, and the process proceeds to step S309. In step S309, the information placement unit 12 places an information presentation window for each user U based on the reference position determined in step S803. However, the information placement unit 12 does not need to position the information presentation windows in the same position and in the same orientation for all users U; it is sufficient that the information presentation window having a spatial extent is placed so as to include the reference position. In other words, the normal to the information presentation window may be parallel to or coincident with the line of sight of each user U, and the position may be adjusted by referring to the energy image for each user U.

[0040] In the following step S310, the information arrangement unit 12 determines whether all of the registered information listed in step S302 has been processed. If the information arrangement unit 12 determines that all of the registered information listed has been processed, it ends the processing shown in Fig. 7, but if it determines that there is unprocessed registered information, it returns to step S303.

[0041] FIG. 8 is a flowchart showing the energy image generation process by the information arrangement unit 12, and also shows details of step S306 in FIG. 7. This flowchart is executed in a state where the target user and the target information have been identified. In other words, the process shown in FIG. 8 is executed a number of times equal to the product of the number of target users and the number of target information. This flowchart is executed by the energy map generation unit 12a of the information arrangement unit 12. First, in step S321, the information arrangement unit 12 identifies the viewpoint position and line of sight direction of the target user by referring to the user table 1200. In the following step S322, the information arrangement unit 12 calculates a projection matrix indicating the line of sight of the target user in the virtual space, based on the viewpoint position and line of sight direction identified in step S321.

[0042] In the following step S323, the information arrangement unit 12 generates an object map 81 for avoiding objects with high importance. Specifically, the information arrangement unit 12 applies a projection matrix to the bounding box 1004 of an object that exists at the location identified in step S301 and has importance 1006 higher than a predetermined threshold, and generates an object map 81 in which a large positive value is set in that area. Note that the value set for the projection position of each object may be set to a larger value as the importance of each object increases. In the following step S324, the information arrangement unit 12 generates an object map 82 for approximating the parent object of the information to be processed. Specifically, the information arrangement unit 12 applies a projection matrix to the center position 1005 of an object that has the value of the parent model ID 1108 of the information to be processed as the element model ID 1001, and generates an object map 82 in which a negative value is set in a perfect circle or ellipse area centered on the projected position.

[0043] In the next step S325, the information arrangement unit 12 generates an adjustment map 83 for adjusting the position with respect to the existing parent window. Specifically, the information arrangement unit 12 identifies the parent window by referring to the related saved information ID 1107 in the information to be processed, and sets negative values ​​to the top, bottom, left, and right of the display position. In the next step S326, the information arrangement unit 12 overlays the three maps generated in steps S323 to S325 and further applies a blurring process to generate an energy image 80, and then ends the process shown in FIG. 8. Hereinafter, the energy image 80 generated in this step will be called Euw. The subscript "u" is the user identifier, and the subscript "w" is the identifier of the information presentation window. Note that a specific example of steps S323 to S326 is as shown in FIG. 6.

[0044] FIG. 9 is a flowchart showing the reference position determination process performed by the information placement unit 12, and also shows details of step S308 in FIG. 7. This flowchart is executed by the reference position determination unit 12b of the information placement unit 12. First, in step S330, if the information to be processed is already displayed, the information placement unit 12 records the three-dimensional coordinates that are the current reference position as the previous display position p_old. If the information to be processed is not yet displayed, no processing is performed. In the following step S331, the information placement unit 12 sets the tentative reference position to an initial position. This initial position is the reference position if the information to be processed is already displayed, and if the information to be processed is not yet displayed, it is set to the origin (0,0,0) or the center position 1005 of the parent object of the presentation information to be processed. In the following step S332, the information placement unit 12 lists the users identified in step S301.

[0045] In the following step S333, the information arrangement unit 12 selects one of the unprocessed users from the users listed in step S333 as the user to be processed, and proceeds to step S334. In step S334, the information arrangement unit 12 executes an individual vector calculation process, which will be described later. In the individual vector calculation process, a three-dimensional vector that moves the optimal temporary reference position for each user is calculated. In the following step S335, the information arrangement unit 12 determines whether all of the registered information listed in step S332 has been processed. If the information arrangement unit 12 determines that all of the listed registered information has been processed, the process proceeds to step S336, and if the information arrangement unit 12 determines that there is unprocessed registered information, the process returns to step S333.

[0046] In step S336, the information arrangement unit 12 sums up the individual vectors of all users U. In the above-mentioned step S334, a three-dimensional vector is calculated as an individual vector for each user to be processed, so in this step, these three-dimensional vectors are simply added together. In the following step S337, the information arrangement unit 12 moves the tentative reference position according to the summation result in step S336. For example, if the tentative reference position immediately before executing step S337 was (100, 50, 10) and the summation result in step S336 was (0.2, -0.3, 0.1), the new tentative reference position will be (100.2, 49.7, 10.1).

[0047] In the next step S338, the information placement unit 12 calculates the evaluation energy En at the new tentative reference position. However, if this is the first time step S338 is executed and the information to be processed has already been displayed, the information placement unit 12 also calculates the evaluation energy E0 at the previously displayed position p_old. The method for calculating the evaluation energy En will be described later. In the next step S339, the information placement unit 12 determines whether the amount of change in the evaluation energy En, i.e., the absolute value of the difference between the evaluation energy at the previous tentative reference position and the evaluation energy at the current tentative reference position, is equal to or less than a threshold. If the information placement unit 12 determines that the amount of change in the evaluation energy En is equal to or less than the predetermined threshold, it determines that the calculation has ended and ends the processing shown in FIG. 9; if it determines that the amount of movement of the tentative reference position is greater than the predetermined threshold, it returns to step S332.

[0048] FIG. 10 is a flowchart showing the individual vector calculation process, and also shows details of step S334 in FIG. 9. First, in step S341, the information arrangement unit 12 calculates a tentative reference position in the energy image of the user to be processed. The projection matrix described above can be used for this calculation. In the following step S342, the information arrangement unit 12 determines whether the calculated tentative reference position exists in the energy image 80. If the information arrangement unit 12 determines that the tentative reference position exists in the energy image 80, the process proceeds to step S343, and if the information arrangement unit 12 determines that the tentative reference position does not exist in the energy image 80, the process proceeds to step S350.

[0049] In step S343, the information arrangement unit 12 reads the amount of energy at the tentative reference position in the energy image 80 and records it as a current value Ec. In the following step S344, the information arrangement unit 12 moves the tentative reference position in the energy image 80 by small distances in multiple directions and reads the amount of energy at the position after the movement. These multiple directions include, for example, a total of eight directions in the energy image 80, including up, down, left, right, and diagonal directions intermediate between these. In the following step S345, the information arrangement unit 12 determines the direction resulting in the smallest amount of energy among the amounts of energy read in step S344 as the optimal movement direction and calculates the reduction rate dE. The reduction rate dE is calculated using the following equation 1, where Em is the minimum value of the amount of energy read in step S344.

[0050] dE = (Ec-Em) / Ec (Equation 1)

[0051] In the following step S346, the information placement unit 12 calculates the distance d_tgt from the tentative reference position to the parent object of the information to be processed. Specifically, the distance d_tgt is the shortest distance between the viewpoint position 1204 and the bounding box 1004 of the parent object of the information to be processed. In the following step S347, the information placement unit 12 calculates the evaluation distance Ed using the following equations 2 to 4.

[0052] Ed = E_0 * (d_pref - d) (Formula 2)

[0053] Ed += E_1 if d < d_min (Formula 3)

[0054] Ed -= E_2 if d > (d_tgt - D_tgt_near) (Formula 4)

[0055] In Equations 2 to 4, E_0 to E_2 are predetermined constants, d is the distance between the viewpoint position 1204 and the temporary reference position to which the initial value was set in step S331 and which was moved in step S337, d_pref is the value of the depth preference 1206 in the user table 1200, and d_min is a predetermined threshold for limiting the short distance. Furthermore, D_tgt_near in Equation 4 is the distance to another object that is closer to the viewpoint position 1204 than the parent object of the information to be processed. Equations 2 to 4 can be explained as follows. That is, the approximate value of the evaluation distance Ed is determined by Equation 2, and if d is smaller than d_min, Equation 3 is corrected, and if d is larger than the difference between d_tgt and D_tgt_near, Equation 4 is corrected.

[0056] In the following step S348, the information arrangement unit 12 calculates a vector Vz pointing from the viewpoint position of the processing target user in the user table 1200 to the parent object of the processing target information. In the following step S349, the information arrangement unit 12 calculates an individual vector. This individual vector Vbest is calculated using the following equation 2, and the processing shown in Fig. 10 ends. Note that the single "x" in equation 5 is written as an operation symbol indicating multiplication.

[0057] Vbest = dE x Vxy + Ed x Vz (Formula 5)

[0058] In step S350, the information arrangement unit 12 sets the individual vector to zero, and ends the processing shown in FIG.

[0059] 11 is a flowchart showing the evaluation energy calculation process, and also shows details of step S338 in FIG. 9. First, in step S361, the information arrangement unit 12 initializes a variable UE to zero. In the following step S362, the information arrangement unit 12 lists the users identified in step S301. In the following step S363, the information arrangement unit 12 selects one of the unprocessed users from the users listed in step S362 as a user to be processed, and proceeds to step S364. In step S364, the information arrangement unit 12 identifies a tentative reference position on the energy image of the user to be processed.

[0060] In the following step S365, the information arrangement unit 12 adds the energy of the tentative reference position to the variable EU. That is, the energy at the tentative reference position of each user is added to the variable EU. In the following step S366, the information arrangement unit 12 determines whether all users have been processed. If it is determined that all users have been processed, the process proceeds to step S367, and if it is determined that there is an unprocessed user, the process returns to step S363.

[0061] In step S367, the information placement unit 12 determines whether or not a previously displayed position exists. The determination in this step also determines whether or not the information to be processed has already been displayed. If the information placement unit 12 determines that a previously displayed position exists, the process proceeds to step S368, and if it determines that a previously displayed position does not exist, the process proceeds to step S369. In step S368, the information placement unit 12 sets the sum of the variable EU repeatedly added in step S365 and the function E shown in Equation 6 as the evaluation energy En, and ends the process shown in FIG. 11.

[0062] E(p_new, p_old) =(p_new - p_old)^2 x E_constant (Equation 6)

[0063] In Equation 6, p_new is the tentative reference position, and p_old is the previously displayed position. That is, in Equation 6, the square of the distance in three-dimensional space between the tentative reference position and the previously displayed position is multiplied by a predetermined constant. In step S369, the information arrangement unit 12 sets the variable EU, which was repeatedly added in step S365, itself as the evaluation energy En, and then ends the processing shown in FIG.

[0064] According to the first embodiment described above, the following advantageous effects can be obtained. (1) The display method executed by the information presentation system 1 displays registration information associated with an object in a three-dimensional space and registered in the registration information table 1100 together with the object from the subjective viewpoint of the observer. This display method includes an energy map generation process (FIG. 8) that creates one or more energy images, which are data on a two-dimensional plane that indicate an area suitable for arranging the registration information, at the respective viewpoint positions of one or more users U; a reference position determination process (FIG. 9) that sets a reference position for displaying the registration information at a second position in the energy image that has lower energy than the first position; and a drawing process executed by the image generation unit 13 that draws the registration information drawn in an area including the parent object and the reference position from the subjective viewpoint of the observer. This allows for easy determination of the arrangement of the registration information.

[0065] (2) In the energy map generation process, the amount of energy is set high by assigning a positive amount of energy to the area of ​​the parent object as shown in the object map 81, and the amount of energy is set low by assigning a negative amount of energy to the periphery of the parent object as shown in the target object map 82. (3) In the energy map generation process, the amount of energy is set high by assigning a positive amount of energy to the area of ​​other registered information that is already displayed as shown in the object map 81. (4) In the energy map generation process, the amount of energy is set low by assigning a negative amount of energy to the area near the area where other registered information related to the registered information is already displayed as shown in the adjustment map 83. (5) In the energy map generation process, the energy is set high for the area of ​​an object other than the parent object that has a set importance greater than a predetermined threshold. Therefore, the arrangement of registered information can be easily determined by setting high energy to areas where it is not desired to display registered information superimposed and low energy to areas where it is desired to display registered information.

[0066] (6) In the reference position determination process, the reference position is determined so that the sum of the energy amounts at the reference position of each energy map is minimized. Specifically, in steps S343 to S345 in Fig. 10, a desired direction for moving the tentative reference position is determined for each energy map, and this vector amount is added up in S336 in Fig. 9 to move the tentative reference position.

[0067] (7) In the reference position determination process, a tentative reference position having three-dimensional coordinates is set in advance (S331), a position on each energy map corresponding to the tentative reference position is identified (S341), and a movement vector of the tentative reference position is set so that the total energy amount at the position corresponding to the tentative reference position is reduced in each energy map (S345). Furthermore, in the reference position determination process, the tentative reference position is updated by adding up the movement vectors calculated in each energy map for the tentative reference position (S336, S337), and when the change in the total energy amount at the position corresponding to the tentative reference position before and after updating the tentative reference position becomes equal to or less than a threshold (S339: YES), the tentative reference position is used as the reference position, and repeated calculations are performed.

[0068] (8) In the reference position determination process, the reference position is determined taking into consideration the evaluation distance Ed, which is based on the difference between the distance d from the viewpoint position to the reference position and the distance d_pref preferred by the viewer, and the vector Vz pointing from the viewpoint position to the parent object. Therefore, the distance direction from the reference position can also be adjusted for each user U.

[0069] (9) If the registered information is already displayed based on the previous reference position (S367: YES), the reference position is determined in the reference position determination process so that the sum of the energy amounts at the reference positions of the respective energy maps and the value based on the distance between the previous reference position and the reference position is small (S368). This prevents the display position of the existing information presentation window from changing significantly.

[0070] (Variation 1) In the first embodiment described above, the server 10 renders the video for each client 20. However, the server 10 may transmit the reference position of the information presentation window to each client 20 without performing the rendering, and each client 20 may perform the rendering.

[0071] 12 is an overall configuration diagram of an information presentation system 1A in Modification 1. In the function-saving server 10A, the image generation unit 13 is removed from the functions of the server 10 in the first embodiment. According to Modification 1, the amount of communication from the function-saving server 10A to the client 20 can be reduced.

[0072] (Variation 2) FIG. 13 is a diagram illustrating the overall configuration of an information presentation system 1B according to Modification 2. The information presentation system 1B is configured with a plurality of second extended clients 20B, and each of the second extended clients 20B has not only the functions of the client 20 according to the first embodiment but also the functions of the server 10. However, a command transmission unit 21 transmits location information of each user to another second extended client 20B, not to the server 10. The other configurations and processes are the same as those of the first embodiment, and therefore will not be described here. According to Modification 2, the server 10 that aggregates data is not required.

[0073] (Variation 3) In the individual vector calculation process, movement in the depth direction relative to the energy image does not need to be taken into consideration. In this case, steps S346 to S348 in Fig. 10 are omitted, and Vbest is calculated as the product of dE and Vxy in step S349.

[0074] (Variation 4) In step S347, the information arrangement unit 12 calculates the evaluation distance Ed using equations 2 to 4. However, the information arrangement unit 12 may calculate the evaluation distance Ed using only equation 2 without using equations 3 and 4.

[0075] (Variation 5) In the first embodiment described above, an example of viewing a virtual space has been described. However, the configuration described in the first embodiment may be used to view a real space and superimpose a computer image on a real-world image, which may be applied to so-called augmented reality. However, the input device 961 in this modification is a device that detects the movement of the user U in the real world. Specifically, the input device 961 is an accelerometer that detects the movement of the user U and a motion sensor attached to the body of the user U that detects the movement of the joints. However, a fixed camera that captures the user U from a distance may also be used as the input device 961. Furthermore, in this modification, the image generation unit 13 does not need to draw objects; it only needs to draw registration information. Furthermore, in this modification, the image display device 962 is a see-through display that allows the user to see what is behind it.

[0076] (Variation 6) In the first embodiment described above, only one energy image 80 was generated for each user U at the same time. However, multiple energy images 80 of user U at the same time may be created. That is, the information arrangement unit 12 may estimate the position and gaze direction of user U after a predetermined time, for example, 5 seconds, based on the time-series change in the viewpoint position 1204 in the user table 1200, and further create an energy image 80 based on the viewpoint position of user U at the estimated destination. This energy image 80 is also used in the reference position determination process (S308), and the reference position is determined taking into consideration the future position of user U.

[0077] This modification provides the following advantages. (10) In the energy map generation process, the movement of the user U is estimated, and an energy map is generated based on the viewpoint position of the observer at the estimated destination. Therefore, even if the user U moves, the reference position does not change significantly, and the user U can comfortably refer to the registered information.

[0078] --Second embodiment-- A second embodiment of the information presentation system will be described with reference to Figs. 14 to 26. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that data is also recorded.

[0079] 14 is a diagram showing the overall configuration of an information presentation system 1C according to the second embodiment. In addition to the configuration of the information presentation system 1 according to the first embodiment, the system includes a fixed camera 971, a microphone 972, earphones 973, and a wearable camera 974. The server 10C also includes a recording unit 16, a post-analysis unit 17, and a context evaluation unit 18, and the storage device 14 further stores an operation record table 1300, an image table 1400, a voice table 1500, and a procedure table 1600.

[0080] The worker W wearing the microphone 972, earphones 973, and wearable camera 974 may be the same as or different from the user U. For ease of explanation, the person who is in the specific space when data is recorded will be referred to as the worker W. Although only one worker W is shown in FIG. 14 for convenience of drawing, there may be two or more workers W. The recording unit 16, the post-analysis unit 17, and the context evaluation unit 18, like the table update unit 11, are realized by the CPU 41 expanding a program stored in the ROM 42 into the RAM 43 and executing it.

[0081] The object table 1000 and the procedure table 1600 are stored in advance in the storage device 14. The registration information table 1100 is created by the recording unit 16 and the post-analysis unit 17. The work record table 1300, the image table 1400, the voice table 1500, and the work posture table 1700 are created by the recording unit 16. The viewpoint position 1204 and the line of sight direction 1205 of the user table 1200 are updated by the table update unit 11, and other information of the user table 1200 is created in advance.

[0082] The worker W performs work in a specific space, which is a space where the objects listed in the object table 1000 exist. The specific space is, for example, the interior of a building, a bus, a train, a car, a ship, etc. The wearable camera 974 is, for example, an action camera attached to smart glasses or a helmet worn by the worker W. The wearable camera 974 may further have a function for calculating depth information, which is distance in the depth direction. The depth information may be realized by a combination of a laser rangefinder and a camera, or may be realized using images captured by multiple cameras installed at a predetermined baseline distance apart, or may be realized using multiple images captured by a single camera at different times.

[0083] The fixed camera 971 is one or more cameras. An image captured by the fixed camera 971 is hereinafter referred to as a "fixed camera image." The fixed camera 971 is placed in a space where the objects listed on the object table 1000 exist. The installation position and installation angle of the fixed camera 971 are known. Audio data recorded by the microphone 972 and images captured by the wearable camera 974 (hereinafter referred to as "subjective images") are sent to the server 10C. The server 10C can output audio to the earphone 973. The microphone 972, the earphone 973, and the wearable camera 974 are linked in advance to the worker W who uses them. For example, when the server 10C receives audio data from the microphone 972, it can determine which worker W the audio belongs to. The fixed camera image and the subjective image may be either a still image or a video.

[0084] FIG. 15 is a diagram showing the state of worker W, and corresponds to the registration phase described below. In FIG. 15, worker W is on a train, working by touching a work object with his or her hands. Although not shown in FIG. 15 due to drawing limitations, worker W has a microphone 972, earphones 973, and a wearable camera 974 fixed to his or her head. Fixed cameras 971 are placed on the left and right upper parts of worker W, and are capturing images of worker W.

[0085] Fig. 16 is a diagram showing the head of worker W. As described above, worker W has a microphone 972, earphones 973, and a wearable camera 974 fixed to his head. In Fig. 16, wearable camera 974 captures an image from the subjective viewpoint of worker W, that is, an image of worker W reaching out and touching a work object.

[0086] FIG. 17 is a flowchart showing an outline of the processing of the information presentation system 1C in the second embodiment. In the second embodiment, first, the recording unit 16 records the work of the worker W (S401) and registers the work log in the database, i.e., the storage device 14 (S402). The recording of the work and the registration in the storage device 14 are repeated until the work by the worker W is completed. When the work by the worker W is completed (S403: YES), items requiring attention are detected and evaluation KPIs are assigned (S404), and the results are written to the storage device 14 (S405). Hereinafter, S401 to S405 are referred to as the registration phase. When the registration phase is completed, the process of narrowing down the display information (S406) and displaying the items requiring attention (S407) is repeated.

[0087] 18 is a diagram showing an example of a work record table 1300. The work record table 1300 has multiple records, and each record has fields for a work class 1301, a work name 1302, a work execution registration ID 1303, an object class 1304, an object 3D model 1305, an individual object ID 1306, an individual object installation position 1307, an object placement angle 1308, a worker 1309, a start time 1310, an end time 1311, an interruption time list 1312, an image list 1313, work voice data 1314, and a work voice command 1315.

[0088] The work class 1301 stores a work class number that indicates the content of the work. The work name 1302 stores the name of the work. The work execution registration ID 1303 stores an identifier for the work record. In other words, when the same type of work is performed, the same values ​​are stored in the work class 1301 and work name 1302, but a different value is set for the work execution registration ID 1303 for each work.

[0089] The object class 1304 stores the class of the object of the work. The object 3D model 1305 stores the name of the 3D model of the object to be worked on or the file name. The object individual ID 1306 stores the individual ID of the object to be worked on. The object individual installation position 307 stores the installation position of the object to be worked on. The object placement angle 1308 stores the placement angle of the object to be worked on. The worker 1309 stores the name or full name of the worker. The start time 1310 stores the time when the work started. The end time 1311 stores the time when the work ended. The interruption time list 1312 stores a list of times when the work was interrupted. The image list 1313 stores a list of image IDs of images taken during work. The work audio data 1314 stores the audio ID of audio data recorded during work. The work audio command 1315 stores the identifier of the work audio command.

[0090] 19 is a diagram showing examples of an image table 1400 and an audio table 1500. The image table 1400 has multiple records, and each record has fields for an image ID 1401, an image URL 1402, an image type 1403, a photographing device ID 1404, a photographing user ID 1405, a photographing time 1406, and a work class 1407. The image ID 1401 stores an image identifier. The image ID 1401 included in the image list 1313 field in the work record table 1300 allows the work execution registration ID 1303 to be associated with the image ID 1401.

[0091] Image URL 1402 stores the URL where the image is saved. Image type 1403 stores the type of image. Photographic device ID 1404 stores the identifier of the device used to photograph the image. Photographic user ID 1405 stores the ID of the user who photographed the image. Photographic time 1406 stores the time the image was photographed. Work class 1407 stores a work class number that indicates the content of the work. Work class 1407 is the same type of information as work class 1301 in work record table 1300.

[0092] The voice table 1500 has multiple records, and each record has fields for a voice ID 1501, a voice URL 1502, a voice type 1503, a recording device ID 1504, a recording user ID 1505, a recording time 1506, and a work voice command 1507. The voice ID 1501 stores an identifier for the voice data. The voice ID 1501 included in the work voice data 1314 field in the work record table 1300 allows the work execution registration ID 1303 to be associated with the voice ID 1501. The voice URL 1502 stores the URL where the voice data is saved.

[0093] The voice type 1503 stores the type of voice data. The recording device ID 1504 stores the identifier of the device used to record the voice data. The recording user ID 1505 stores the ID of the user who recorded the voice data. The recording time 1506 stores the time when the voice data was recorded. Note that the recording time 1506 may store any of the time when recording started, the time when recording ended, or the time when recording started and the time when recording ended. The work voice command 1507 stores the identifier of the command for the work voice.

[0094] FIG. 20 shows examples of a procedure table 1600 and a work posture table 1700. The procedure table 1600 has multiple records, and each record has fields for a work class 1601, a work name 1602, a start condition 1603, a target location 1604, and a CAD model change 1605. The work class number indicating the content of the work is stored in the work class 1601. The work name 1602 stores the name of the work. The start condition 1603 stores the work class that must be completed before the work can begin. If there is no work class that must be completed beforehand, "none" is stored. The target location 1604 stores **, which indicates the target location of the work. The CAD model change 1605 stores data before and after the CAD model that changes due to the work.

[0095] The work posture table 1700 has multiple records, and each record has fields for a user ID 1701, acquisition time 1702, a task ID 1703, a position 1704, a base angle 1705, an estimated motion task 1706, first joint information 1707, and second joint information 1708. The user ID 1701 stores the identifier of the user who is the worker. The acquisition time 1702 stores the time when the posture information was acquired. The task ID 1703 stores the task execution registration ID 1303 of the work record table 1300, which is the task identifier.

[0096] The position 1704 stores the position of the user. The basic angle 1705 stores a basic angle. The estimated movement task 1706 stores the name of the task that is estimated to be performed by the user. The first joint information 1707 and the second joint information 1708 store angle information of each joint. That is, the task posture table 1700 stores the position, posture, etc. at one or more times when a certain worker W performs one task.

[0097] (Registration phase) FIG. 21 is a detailed configuration diagram of the recording unit 16 and the post-analysis unit 17. The recording unit 16 includes a hand posture extraction program 161, a target part estimation program 162, a task content estimation program 163, a user position / posture estimation program 164, and a fixed camera position / angle estimation program 165. The hand posture extraction program 161 extracts the hand posture from the subjective image. The target part estimation program 162 estimates the target part from the subjective image. The task content estimation program 163 estimates the task content from the estimated object, hand posture, and depth distance in the subjective image. The user position / posture estimation program 164 estimates the position and posture of the worker W during the task from the fixed camera image and records it in the task posture table 1700. The fixed camera position / angle estimation program 165 estimates the shooting position and angle of the subjective image from the position of the fixed camera 971 and the position of the target part.

[0098] Furthermore, the recording unit 16 detects the start and end of work using the subjectively photographed images and records the times in the work record table 1300. At this time, the recording unit 16 may output a voice from the earphones 973 of the worker W as follows. That is, when the start of work is detected, the recording unit 16 outputs "Work start confirmed with work target: ddd and start time: ddd," and when the end of work is detected, the recording unit 16 outputs "Work end confirmed with work target: ddd and start time: ddd." The worker W may speak into the microphone 972, saying "Work target ddd, start work on ddd" or "Finish work," and the recording unit 16 may recognize the content of the utterance using voice recognition and record the time of the utterance in the work record table 1300.

[0099] The post-analysis unit 17 includes a KPI generation program 171, a risk determination program 172, and an abnormal work determination program 173. The KPI generation program 171 generates KPIs (Key Performance Indicators) from sensing data registered in a work group. The risk determination program 172 uses the KPIs to determine whether or not a registered risk exists. The abnormal work determination program 173 uses the KPIs to determine abnormal values ​​in work.

[0100] The KPI generation program 171 of the post-analysis unit 17 creates KPIs using the recorded data. For example, any of the following eight can be used as KPIs: (1) Time required for the task; (2) Task time for each task if the task is divided into multiple steps; (3) Similarity with other cases based on the color and shape of the object; (4) ID of the object if one has been set; (5) Number of failures in processing the object if a failure has been confirmed; (6) Error in the task position of the object; (7) If torque fluctuations during the task can be calculated using another sensor, values ​​such as the maximum value, minimum value, and rate of change; (8) Values ​​obtained by analyzing the sound spectrum during the task. However, the method of calculating KPIs is not limited to these.

[0101] Among these calculation methods, image processing, deep learning, etc. can be used to calculate KPIs from images. Also known is a method (Explainable) that colors image elements that affect the KPI. The post-analysis unit 17 is not limited to the above explanations and can calculate KPIs using various known methods.

[0102] The risk assessment program 172 of the post-analysis unit 17 uses this KPI to calculate the potential risk using one of the following two methods, and enters the value of the potential risk in the importance 1102 column of the registration information table 1100. The first method is irregularity detection using clustering. In this case, identical or similar tasks are first grouped, and if the KPI to be evaluated within the same group is significantly different from other cases, it is determined that there is a risk that the task results contain problems. The magnitude of the risk can be evaluated by the size of the deviation of the outlier.

[0103] The second method is risk detection using a supervised model. In this case, if the KPI being evaluated is included in a class similar to that of problematic cases, it is determined that there is a risk that the work results contain problems. Specifically, identical or similar work is grouped, and cases containing problem elements are picked out from the work results. The problematic cases are then grouped according to the problematic case group tags. Furthermore, a classifier is created to distinguish between cases that are included in the problematic cases and cases that are not. Any classification method, such as a support vector machine, random forest, or deep learning, can be used for this classifier. This classifier can also calculate an index including the accuracy of the classification. When a new sample is obtained, the classifier is used to determine whether it belongs to the problematic case group.

[0104] FIG. 22 is a flowchart showing the risk calculation process by the risk judgment program 172. The process shown in FIG. 22 is executed in a state where the work result for which the risk is to be judged (hereinafter referred to as the "judgment target result") has been identified. The risk judgment program 172 first determines whether to use the first method or the second method in step S381. If the first method is to be used, the risk judgment program 172 proceeds to step S382, and if the second method is to be used, the risk judgment program 172 proceeds to step S387. In step S382, the risk judgment program 172 calls up data classified into a group that is the same as or similar to the judgment target result.

[0105] In the following step S383, the risk assessment program 172 determines whether the KPI of the assessment target result deviates from the KPI of the group, i.e., the KPI of the other data called up in step S382. If the risk assessment program 172 determines that the KPI of the assessment target result deviates from the KPI of the group, it proceeds to step S384, and if it determines that the KPI of the assessment target result does not deviate from the KPI of the group, it proceeds to step S386. In step S384, the risk assessment program 172 determines the magnitude of the risk as the magnitude of the deviation of the outlier. Specifically, it determines the magnitude of the risk as the product of the fixed value assigned to that KPI and the standard deviation of the KPI.

[0106] In the next step S385, the risk judgment program 172 adds a new record to the registration information table 1100 and ends the processing shown in Fig. 22. The magnitude of the risk calculated in step S384 is stored in the importance 1102 column of this new record. In step S386, the risk judgment program 172 determines that the judgment target result is no risk, and ends the processing shown in Fig. 22 without adding any data to the registration information table 1100.

[0107] In step S387, the risk judgment program 172 classifies the judgment target results using the discriminator and calls up the data of the group classified as having similar judgment target results. In the following step S388, the risk judgment program 172 determines whether the called group is a problematic group. If the group is a problematic group, the risk judgment program 172 proceeds to step S389, and if the group is not a problematic group, the risk judgment program 172 proceeds to step S386. In step S389, the risk judgment program 172 uses the index of the judgment certainty output by the discriminator as the magnitude of risk. In the following step S390, the risk judgment program 172 adds a new record to the registration information table 1100 and ends the processing shown in FIG. 22. The magnitude of risk calculated in step S389 is stored in the importance 1102 column of this new record.

[0108] Among the information registered in the registration information table 1100, when machine learning is used, the content of the explanation associated with the test data is used for the explanation 1105. Furthermore, when a determination is made based on the presence of an abnormal value in a specific KPI through clustering, the name of the KPI and the fluctuation range of the KPI are recorded as a character string. For example, this character string may be "required time: 100 seconds or more." When the machine learning system has a function for automatically generating warning messages, the post-analysis unit 17 saves the generated warning message in the explanation 1105. When the related risk explanation spans multiple windows, the post-analysis unit 17 sets a parent-child relationship in the display window and enters the related saved information ID in the related saved information ID 1107.

[0109] (Browse phase) In the viewing phase, for example, the user U, who is the viewer, refers to the registered information table 1100 or the like to display the areas that require inspection. This operation can be used for quality assurance activities after development or inspection activities during maintenance. However, since a large amount of information can be registered in the registered information table 1100, selection is required. The user U may explicitly select the registered information to display, or the context evaluation unit 18 may evaluate the context and the image generation unit 13 may display the registered information without the user U making an explicit selection. Instead of immediately displaying the registered information, the presence of the registered information may be notified to the user U with a marker, and the registered information may be displayed when the user U selects the marker.

[0110] FIG. 23 is a diagram showing the display of registered information in the browsing phase of the second embodiment. In FIG. 23, a 3D CAD model of an object existing in a specific space and registered information are displayed. This registered information can be considered the analysis result performed in the registration phase. Registered information with a high level of importance may be emphasized. Specifically, it may be displayed in a conspicuous color such as red, or the shape of the marker may be changed, or the size of the marker may be increased. The registered information may be displayed using a display template, which will be described next.

[0111] FIG. 24 is a diagram illustrating an example of a display template. The display template includes a saved information ID display field 981, a potential risk display field 982, a description display field 983, a related keyword display field 984, an image display area 985, and an option display field 986. The image generator 13 reads necessary information from the registered information table 1100 and applies it to the display template. For example, in the case of a risk estimated by clustering, the description display field 983 displays, "The KPI {value} is in the {value} state. This state is deviated by {value}σ from the general case." In addition, in the case of a risk estimated by machine learning, the following is displayed: "The KPI {value} is in the {value} state. This may be classified as the problem case {value} state. (Confidence {value})"

[0112] The window displaying each piece of registered information may further include buttons for referencing other information. Specifically, buttons for checking related images, displaying a separate graph of KPIs, and displaying a comparison with similar cases may also be included. Furthermore, buttons for displaying images of the same object taken at different times in the image display area 985 may also be included.

[0113] The context evaluation unit 18 may increase the importance 1102 of registered information having a parent object near the current location of the user U or registered information related to the content of the user U's utterance above the threshold value listed in step S302 of the information drawing process. For example, the context evaluation unit 18 acquires the user U's voice, searches data converted into text using a machine learning technique, and if there is content matching a keyword registered in the keyword list 1104, increases the importance 1102 by a value corresponding to the weight associated with each keyword. Alternatively, the user U may associate the content of the utterance with keywords in advance, and perform machine learning on the conversation data using the associated correspondence as correct answer data to create a machine learning model that appropriately interprets the user U's utterance.

[0114] When the user moves within the virtual space and approaches one of the markers, a summary subtitle of the marker is displayed. The subtitle may be, for example, the class name of the problem case or the type of KPI that indicates an outlier. Furthermore, if the user explicitly selects one of these using a pointing device or eye-gaze input, or if the degree of match between the conversation context and the keyword list 1104 exceeds a certain value, a summary screen is displayed. For example, the above-mentioned display template can be used for this summary screen. The summary screen can be displayed in a window that fits in a relatively small area within the 3D space, and can be processed, such as read aloud, if specified by the user. The display location of the summary screen uses the method shown in the first embodiment.

[0115] Furthermore, if the user U explicitly selects one of the summary screens using a pointing device or eye-gaze input, or if the conversation context match exceeds a certain value, a related window will open. This related window is created by preparing a template screen like the one shown in Figure 24 as an HTML template, and filling in the necessary parts with the character information and numerical information registered in 1100 to create a warning window. Buttons are placed within the window to call up detailed displays of similar cases and related information.

[0116] Fig. 25 is a diagram showing the replay explanation video as seen from a viewpoint behind the worker W, and Fig. 26 is a diagram showing the replay explanation video as seen from a viewpoint from the side of the worker W. If the user U selects to present detailed information, a reproduction model of the work display is displayed in the virtual space, as shown in Figs. 25 and 26.

[0117] The image generator 13 displays a 3D model that reproduces the posture of the worker W using the data recorded in the working posture table 1700. The image generator 13 may use a 3D model that reflects joint information in this 3D displayed worker model, or may use point cloud data based on a depth map of a fixed camera, i.e., point group data. The subjective image captured by the worker W is displayed, as shown in FIG. 25, clearly indicating the position from which the image was captured. This display shows a semi-transparent square with the shooting position as the vertex, and presents the subjectively captured image at the bottom, thereby expressing the position and shooting direction of the target.

[0118] However, if this rectangular shape overlaps with other objects and becomes difficult to see, or if the image size is too small or too large to see, a window displaying the information is placed at a low energy position using the method described in the first embodiment. As shown in Fig. 25, the vertices of the rectangular shape, which are pseudo-shooting positions, are set at positions different from the actual shooting positions, and leader lines are drawn from the pseudo-shooting positions to refer to the true shooting positions.

[0119] According to the second embodiment described above, the following advantageous effects can be obtained. (11) The display method executed by the information presentation system 1C includes a selection process executed by the context evaluation unit 18 to select registered information to be displayed based on the content of the user U's utterance and the distance from the user U to the parent object. Therefore, necessary information can be presented from a vast amount of registered information without burdening the user U.

[0120] (Modification of the second embodiment) The importance 1102 recorded in the registration information table 1100 may be a fixed value, and a mechanism may be provided for dynamically calculating the relevance with the context indicating the user U's current intention to use the information, and the display granularity of the information display may be adjusted according to the importance, relevance, and distance from the information subject. The display granularity of the information may also be determined taking into account the size of the display window that occupies the layout position when each display granularity form is adopted, and the apparent size of the display window for the user U.

[0121] -Third embodiment- A third embodiment of the information presentation system will be described with reference to FIG. 27. In the following description, the same components as those in the second embodiment are given the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the second embodiment. This embodiment differs from the first embodiment mainly in that data is also recorded for third parties who visit a specific space. In this embodiment, this third party is referred to as a visitor V to distinguish him or her from the worker W and user U described above.

[0122] The configuration of the information presentation system in this embodiment is the same as that in the second embodiment. When a visitor V who visits a specific space discovers something, such as a malfunction or a scratch, in a certain location (hereinafter referred to as a "location of interest"), the visitor V adds data to the registration information table 1100 stored in the storage device 14. For example, the visitor V starts a recording app on his or her smartphone, takes a photo of the location of interest, inputs a comment by voice or text, and registers information about the location of the photo in the registration information table 1100. Three methods for identifying and recording locations of interest will be described.

[0123] In the first method, a 3D model and an explanatory diagram are provided to the visitor V, and the visitor V himself / herself manually inputs points of interest into the registration information table 1100. In other words, the information presentation system does not perform a process of identifying points of interest.

[0124] In the second method, a fixed camera is present that captures images of the interior of a specific space, and visitor V is identified in the captured images using the method used to identify worker W in the second embodiment. A recording app instructs visitor V to move their smartphone while the fixed camera is capturing images, and visitor V moves the smartphone. The recording app acquires the output of the smartphone's built-in acceleration sensor, identifies the timing at which the smartphone was moved, and outputs the output to a device that processes the fixed camera's video. The device identifies the person in the video who is moving the smartphone at the timing when the smartphone's acceleration changes as visitor V, and identifies a point of interest from the position and orientation of the smartphone held by visitor V.

[0125] In the third method, a marker whose location can be uniquely identified from the video is first set within a specific space, such as a character string indicating a seat number, an AR marker, or a two-dimensional code. The location of this marker is recorded in advance in a recording app. Visitor V uses the recording app to take a photo of the marker near the location in question, then moves the camera from there to the problem location and takes another photo. The recording app tracks changes in the camera parameters from the changes in the captured video and estimates the location where the photo was taken.

[0126] FIG. 27 is a flowchart showing an outline of the processing of the information presentation system in the third embodiment. First, the visitor V starts the recording application (S411), records and photographs points of interest (S412), and registers the photographing location (S413). These data are then recorded in the storage device 14. Any of the above three specific methods may be used to identify and record points of interest. The visitor V can register any number of points of interest, and once recording is complete (S415: YES), the next processing is performed in the same manner as in the second embodiment. That is, items requiring attention are detected and assigned evaluation value KPIs (S416), and the database is updated (S417), and the processing shown in FIG. 27 ends.

[0127] According to the third embodiment described above, a visitor V who visits a specific space can add new data to the registration information table 1100.

[0128] In each of the above-described embodiments and modifications, the functional block configurations are merely examples. Some functional configurations shown as separate functional blocks may be configured as an integrated unit, or a configuration shown in a single functional block diagram may be divided into two or more functions. Furthermore, some of the functions of each functional block may be provided by other functional blocks.

[0129] In each of the above-described embodiments and modifications, the program may be stored in the storage device 14. Alternatively, the arithmetic device 40 may be provided with an input / output interface (not shown), and the program may be loaded from another device as needed via the input / output interface and a medium available to the arithmetic device 40. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, i.e., a wired, wireless, or optical network, or a carrier wave or digital signal propagating through the network. Alternatively, some or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

[0130] The above-described embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0131] 1, 1A, 1B, 1C... Information presentation system 10... Server 11 ... Table update section 12 … Information location department 12a ... Energy map generation section 12b... Reference position determining section 13...Video generation section 20 … Client U...User V … Visitor

Claims

1. A display method for displaying registered information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, comprising: an energy map generation process for generating one or more energy maps, which are data relating to a two-dimensional plane indicating an area suitable for arranging the registration information, at each viewpoint position of the one or more observers; a reference position determination process for setting a reference position for displaying the registration information at a second position in the energy map where the amount of energy is lower than that of a first position; a rendering process of rendering the registration information rendered in an area including the parent object and the reference position from a subjective viewpoint of the observer, In the energy map generation process, a positive amount of energy is assigned to the region of the parent object, and a negative amount of energy is assigned to the periphery of the parent object.

2. A display method for displaying registered information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, comprising: an energy map generation process for generating one or more energy maps, which are data relating to a two-dimensional plane indicating an area suitable for arranging the registration information, at each viewpoint position of the one or more observers; a reference position determination process for setting a reference position for displaying the registration information at a second position in the energy map where the amount of energy is lower than that of a first position; a rendering process of rendering the registration information rendered in an area including the parent object and the reference position from a subjective viewpoint of the observer, In the energy map generation process, a negative amount of energy is assigned to an area near an area where other registered information related to the registered information is already displayed.

3. A display method for displaying registered information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, comprising: an energy map generation process for generating one or more energy maps, which are data relating to a two-dimensional plane indicating an area suitable for arranging the registration information, at each viewpoint position of the one or more observers; a reference position determination process for setting a reference position for displaying the registration information at a second position in the energy map where the amount of energy is lower than that of a first position; a rendering process of rendering the registration information rendered in an area including the parent object and the reference position from a subjective viewpoint of the observer, In the reference position determination process, the reference positions are determined so that a total energy amount at the reference positions of the respective energy maps is small; The reference position determination process includes: A temporary reference position having three-dimensional coordinates is set in advance; identifying positions on the energy map corresponding to the tentative reference positions; a movement vector of the tentative reference position is set so that a total energy amount at a position corresponding to the tentative reference position is reduced in each of the energy maps; updating the tentative reference position by adding up the movement vectors calculated in each of the energy maps to the tentative reference position; A display method in which repeated calculations are performed with the tentative reference position set as the reference position when a change in the total amount of energy at a position corresponding to the tentative reference position before and after updating the tentative reference position becomes equal to or less than a threshold.

4. A display method for displaying registered information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, comprising: an energy map generation process for generating one or more energy maps, which are data relating to a two-dimensional plane indicating an area suitable for arranging the registration information, at each viewpoint position of the one or more observers; a reference position determination process for setting a reference position for displaying the registration information at a second position in the energy map where the amount of energy is lower than that of a first position; a rendering process of rendering the registration information rendered in an area including the parent object and the reference position from a subjective viewpoint of the observer, In the reference position determination process, the reference positions are determined so that a total energy amount at the reference positions of the respective energy maps is small; A display method in which the reference position determination process further determines the reference position by taking into account a distance based on the difference between the distance from the viewpoint position to the reference position and the distance preferred by the viewer, and a vector from the viewpoint position toward the parent object.

5. A display method for displaying registered information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, comprising: an energy map generation process for generating one or more energy maps, which are data relating to a two-dimensional plane indicating an area suitable for arranging the registration information, at each viewpoint position of the one or more observers; a reference position determination process for setting a reference position for displaying the registration information at a second position in the energy map where the amount of energy is lower than that of a first position; a rendering process of rendering the registration information rendered in an area including the parent object and the reference position from a subjective viewpoint of the observer, In the reference position determination process, the reference positions are determined so that a total energy amount at the reference positions of the respective energy maps is small; a display method in which, when the registered information is already displayed based on a previous reference position which is the reference position, in the reference position determination process, the reference position is determined so that the sum of the total energy amount at the reference position of each of the energy maps and a value based on the distance between the previous reference position and the reference position is small.

6. A display method for displaying registered information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, comprising: an energy map generation process for generating one or more energy maps, which are data relating to a two-dimensional plane indicating an area suitable for arranging the registration information, at each viewpoint position of the one or more observers; a reference position determination process for setting a reference position for displaying the registration information at a second position in the energy map where the amount of energy is lower than that of a first position; a rendering process of rendering the registration information rendered in an area including the parent object and the reference position from a subjective viewpoint of the observer, In the reference position determination process, the reference positions are determined so that a total energy amount at the reference positions of the respective energy maps is small; The energy map generation process further includes estimating movement of the observer, and further generating the energy map based on the viewpoint position of the observer at the estimated movement destination.

7. A display system that displays registration information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, an energy map generating unit that generates one or more energy maps, which are data relating to a two-dimensional plane that indicates an area suitable for arranging the registration information, at each of the viewpoint positions of the one or more observers; a reference position determination unit that sets a reference position for displaying the registration information at a second position in the energy map where the energy amount is lower than that of a first position; an image generating unit that draws the registration information drawn in an area including the parent object and the reference position from a subjective viewpoint of the observer, The energy map generator assigns a positive amount of energy to a region of the parent object and a negative amount of energy to a periphery of the parent object.

8. A display system that displays registration information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, an energy map generating unit that generates one or more energy maps, which are data relating to a two-dimensional plane that indicates an area suitable for arranging the registration information, at each of the viewpoint positions of the one or more observers; a reference position determination unit that sets a reference position for displaying the registration information at a second position in the energy map where the energy amount is lower than that of a first position; an image generating unit that draws the registration information drawn in an area including the parent object and the reference position from a subjective viewpoint of the observer, A display system, wherein the energy map generation unit assigns a negative amount of energy to an area near an area where other registration information related to the registration information is already displayed.

9. A display system that displays registration information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, an energy map generating unit that generates one or more energy maps, which are data relating to a two-dimensional plane that indicates an area suitable for arranging the registration information, at each of the viewpoint positions of the one or more observers; a reference position determination unit that sets a reference position for displaying the registration information at a second position in the energy map where the energy amount is lower than that of a first position; an image generating unit that draws the registration information drawn in an area including the parent object and the reference position from a subjective viewpoint of the observer, the reference position determination unit determines the reference positions so that a total energy amount at the reference positions of each of the energy maps is small; The reference position determination unit A temporary reference position having three-dimensional coordinates is set in advance; identifying positions on the energy map corresponding to the tentative reference positions; a movement vector of the tentative reference position is set so that a total energy amount at a position corresponding to the tentative reference position is reduced in each of the energy maps; updating the tentative reference position by adding up the movement vectors calculated in each of the energy maps to the tentative reference position; A display system that performs repeated calculations, setting the tentative reference position as the reference position when the change in the total amount of energy at the position corresponding to the tentative reference position before and after updating the tentative reference position becomes equal to or less than a threshold.

10. A display system that displays registration information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, an energy map generating unit that generates one or more energy maps, which are data relating to a two-dimensional plane that indicates an area suitable for arranging the registration information, at each of the viewpoint positions of the one or more observers; a reference position determination unit that sets a reference position for displaying the registration information at a second position in the energy map where the energy amount is lower than that of a first position; an image generating unit that draws the registration information drawn in an area including the parent object and the reference position from a subjective viewpoint of the observer, the reference position determination unit determines the reference positions so that a total energy amount at the reference positions of each of the energy maps is small; The reference position determination unit further determines the reference position by taking into consideration a distance based on the difference between the distance from the viewpoint position to the reference position and a distance preferred by the viewer, and a vector from the viewpoint position toward the parent object.

11. A display system that displays registration information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of a viewer, an energy map generating unit that generates one or more energy maps, which are data relating to a two-dimensional plane that indicates an area suitable for arranging the registration information, at each of the viewpoint positions of the one or more observers; a reference position determination unit that sets a reference position for displaying the registration information at a second position in the energy map where the energy amount is lower than that of a first position; an image generating unit that draws the registration information drawn in an area including the parent object and the reference position from a subjective viewpoint of the observer, the reference position determination unit determines the reference positions so that a total energy amount at the reference positions of each of the energy maps is small; When the registered information is already displayed based on the previous reference position, which is the reference position, the reference position determination unit determines the reference position so that the sum of the total energy amount at the reference position of each of the energy maps and a value based on the distance between the previous reference position and the reference position is small.

Images