Display control method and display control program

Abstract

The present invention improves work skill of a user.　In the present invention, a computer causes a display device to display an image, including a point of interest to which a user should pay attention, and executes display changing processing for changing the display of the image so as to emphasize the point of interest when detecting that the user has not noticed the point of interest. In the display changing processing, for example, a display state of the point of interest is changed so as to emphasize the point of interest, and / or an emphasis image for emphasizing the point of interest is displayed in the vicinity of the point of interest. Also, in the display changing processing, the computer may increase a degree of emphasis of the point of interest in accordance with increase in the number of times of detecting the user not noticing the point of interest.

Description

Display control method and display control program

[0001] The present invention relates to a display control method and a display control program.

[0002] In recent years, virtual reality (VR) technology has been applied to various fields. For example, an image showing a three-dimensional virtual space that recreates a work site is displayed on a head-mounted display (HMD) worn by a user. By depicting the situation in the virtual space from the user's viewpoint based on the orientation and position of the user's head, the user can experience as if they were actually in the work space. Such technology is being used in training and education to acquire and improve work skills.

[0003] Regarding training and education using images, the following proposals have been made. For example, a driving training device has been proposed that displays, in association with a distance display, an identifier indicating the position of a simulated vehicle at the time when the simulated vehicle becomes aware of an obstacle that may obstruct the simulated vehicle's path. Another proposal has been made of an information processing system that references operation information indicating a user's operation to generate and display an image in which objects are arranged in a virtual space representing a chemical laboratory or a chemical plant.

[0004] JP 2015-079223 A JP 2020-91475 A

[0005] However, when using VR technology to train or educate users on work at a workplace, there is a problem in that it is difficult to efficiently improve the users' work skills by simply giving them a passive experience, such as by displaying images of the workplace or the work being done.

[0006] In one aspect, the present invention aims to provide a display control method and a display control program that can improve a user's work skills.

[0007] In one proposal, a display control method is provided in which a computer displays an image including a feature that a user should pay attention to on a display device, and if it detects that the user has not noticed the feature, it performs a display change process that changes the display of the image to emphasize the feature.

[0008] In addition, one proposal provides a display control program that causes a computer to execute processing similar to the above-described display control method.

[0009] In one aspect, the user's work skills can be improved. The above and other objects, features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings illustrating preferred embodiments of the present invention.

[0010] FIG. 1 is a diagram illustrating a configuration example and a processing example of a display control system according to a first embodiment. FIG. 2 is a diagram illustrating a configuration example of a work training system according to a second embodiment. FIG. 3 is a diagram illustrating a configuration example of processing functions provided in an information processing device. FIG. 4 is a diagram illustrating a first display change processing example. FIG. 5 is a diagram illustrating a display change processing example. FIG. 6 is a diagram illustrating a display change processing example. FIG. 7 is a diagram illustrating a screen display example when the fourth display change processing example is used. FIG. 8 is a diagram illustrating a fifth display change processing example. FIG. 9 is a flowchart illustrating a display change processing procedure when the fifth display change processing example is used.

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The present invention will be described below with reference to the accompanying drawings. [First Embodiment] Fig. 1 is a diagram showing an example of the configuration and processing of a display control system according to a first embodiment. The display control system shown in Fig. 1 includes an information processing device 1 and a display device 2.

[0012] The information processing device 1 is, for example, a computer device such as a personal computer or a server device. The information processing device 1 generates images for training or educating a user on a predetermined task, and displays the generated images on the display device 2 so that the user can view them. The display device 2 may be, for example, an HMD worn on the user's head, or a general display placed on a desk or the like.

[0013] The processing of the information processing device 1 will be described below. The information processing device 1 displays an image 4 including a point of interest 3 to which the user should pay attention on the display device 2. The point of interest 3 is a target area for training or education, such as the user finding it among the displayed content of the image 4, carefully observing it, or taking some kind of action (touching it with their hands, repairing it, etc.). In order to improve the user's skills in such tasks, the information processing device 1 changes the displayed content of the image 4 depending on the occurrence of the user's behavior of "not noticing the point of interest 3."

[0014] 1 , the fact that the user has not noticed the attention point 3 is detected based on the detection result of the user's line of sight toward the display device 2. Specifically, when the user's gaze point passes through the attention point 3, it is determined that the user has not noticed the attention point 3.

[0015] However, this detection method is just one example, and for example, if the point of gaze does not overlap with the point of interest 3 for a certain period of time or more after the point of interest 3 is displayed, it may be determined that the user did not notice the point of interest 3. Furthermore, if the user performs an explicit input operation when they notice the point of interest 3, it may be determined that the user did not notice the point of interest 3 if no input operation indicating that they noticed the point of interest 3 is performed for a certain period of time or more after the point of interest 3 is displayed.

[0016] In the example of Fig. 1, it is assumed that in image 4, point of gaze 5 exists to the left of point of interest 3. From this state, it is assumed that point of gaze 5 passes through point of interest 3 and moves to the right of point of interest 3, as shown in image 4a. At this time, the information processing device 1 detects that the user did not notice point of interest 3.

[0017] Then, the information processing device 1 executes a display change process to change the display of the image 4a so as to emphasize the attention point 3. In the example of Fig. 1, the attention point 3 is linear, so the attention point 3 is displayed thicker (the width of the attention point 3 is enlarged) as shown in the image 4b, thereby emphasizing the attention point 3.

[0018] The display change process may be a process of changing the display state of the attention point 3 itself, as in the example of image 4b. As another such example, the attention point 3 may be enlarged. As another example, another image that highlights the attention point 3, such as an arrow, may be displayed near the attention point 3. As yet another example, the attention point 3 may be moved to a position that is easily noticeable or easy to find for the user.

[0019] In the above process, when it is determined that the user did not notice the attention point 3, the information processing device 1 performs a process of emphasizing the attention point 3, rather than giving the user the correct answer, to make it easier for the user to notice the attention point 3. This makes it possible to encourage the user to notice the attention point 3 by themselves, and enables active learning. For example, the user can acquire a task skill without losing the sense of performing the task of finding the attention point 3. As a result, the user's task skill can be improved.

[0020] Second Embodiment Fig. 2 is a diagram showing an example of the configuration of a work training system according to a second embodiment. The work training system includes an information processing device 100 and an HMD unit 200. Note that the information processing device 100 is an example of the information processing device 1 in Fig. 1, and the HMD unit 200 is an example of the display device 2 in Fig. 1.

[0021] This work training system allows a user wearing an HMD unit 200 to view an image showing a work site and receive work training or education. The information processing device 100 stores three-dimensional data showing a three-dimensional virtual space that recreates the work site, draws an image showing the virtual space based on the three-dimensional data, and displays the image on the HMD unit 200.

[0022] The HMD unit 200 is worn on the user's head. The HMD unit 200 includes a display 201, a direction / position sensor 202, and a gaze sensor 203. The display 201 is a non-transparent HMD that displays an image based on image data transmitted from the information processing device 100. The direction / position sensor 202 is a sensor for detecting the direction (direction pointing forward) and position of the user's head, and includes, for example, an acceleration sensor and an angular velocity sensor (gyro). The gaze sensor 203 is a sensor for detecting the direction of the user's gaze. The gaze sensor 203 is, for example, a near-infrared camera. In this case, the near-infrared camera captures images of the light reflection points on the cornea and the eyeball, and the direction and position of the gaze are estimated from the fluctuations of these images.

[0023] Next, the hardware configuration of the information processing device 100 will be described with reference to Fig. 2. The information processing device 100 is realized as, for example, a computer as shown in Fig. 2. The information processing device 100 includes a processor 101, a random access memory (RAM) 102, a hard disk drive (HDD) 103, a graphics processing unit (GPU) 104, an input interface (I / F) 105, a reader 106, and a communication interface (I / F) 107.

[0024] The processor 101 (processor circuit) performs overall control of the entire information processing device 100. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 101 may also be a combination of two or more elements of a CPU, an MPU, a DSP, an ASIC, or a PLD.

[0025] The RAM 102 is used as a main storage device of the information processing device 100. The RAM 102 temporarily stores at least a part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 also stores various data necessary for processing by the processor 101.

[0026] The HDD 103 is used as an auxiliary storage device for the information processing device 100. The HDD 103 stores an OS program, application programs, and various data. Note that other types of non-volatile storage devices, such as a solid state drive (SSD), can also be used as the auxiliary storage device.

[0027] A display 111 is connected to the GPU 104. The GPU 104 displays an image on the display 111 in accordance with an instruction from the processor 101. The display 111 may be a liquid crystal display, an organic EL (ElectroLuminescence) display, or the like.

[0028] An input device 112 is connected to the input interface 105. The input interface 105 transmits signals output from the input device 112 to the processor 101. Examples of the input device 112 include a keyboard and a pointing device. Examples of the pointing device include a mouse, a touch panel, a tablet, a touch pad, and a trackball.

[0029] A portable recording medium 113 is detachably attached to the reading device 106. The reading device 106 reads data recorded on the portable recording medium 113 and transmits the data to the processor 101. The portable recording medium 113 may be an optical disk, a semiconductor memory, or the like.

[0030] The communication interface 107 transmits and receives data to and from other devices. In the present embodiment, for example, the communication interface 107 performs wireless communication, and is capable of wirelessly communicating with the HMD unit 200.

[0031] The above hardware configuration can realize the processing functions of the information processing device 100. Fig. 3 is a diagram showing an example of the configuration of the processing functions of the information processing device. The information processing device 100 includes a storage unit 120, a direction / position detection unit 131, a gaze detection unit 132, and a display control unit 133.

[0032] The storage unit 120 is a storage area allocated in a storage device included in the information processing device 100, such as the RAM 102 or the HDD 103. The storage unit 120 stores three-dimensional data 121. The three-dimensional data 121 is data representing the three-dimensional shape of a virtual space that is a virtualized work site that is the subject of work training or education. The three-dimensional data 121 includes, for example, point cloud data representing the three-dimensional coordinates of multiple feature points set in the virtual space, and texture image data that is applied to each polygon based on the point cloud data. The virtual space may be a virtualized space of a real work site, or a space representing a fictitious work site created for training or education. The three-dimensional data 121 also includes data regarding points of interest set in the virtual space.

[0033] The processing of the direction / position detection unit 131, gaze detection unit 132, and display control unit 133 is realized, for example, by the processor 101 executing a predetermined program. The direction / position detection unit 131 detects the direction and position of the user's head based on the detection result of the direction / position sensor 202. The gaze detection unit 132 detects the direction of the user's gaze based on the detection result of the gaze sensor 203.

[0034] The display control unit 133 renders an image showing the virtual space of the work site based on the three-dimensional data 121, the direction and position of the user's head, and the direction of the line of sight, and displays the image on the display 201 of the HMD unit 200. In this image display, the direction and position of the user's head are associated with coordinates in the virtual space, and the situation of the virtual space as seen from the user's viewpoint is reproduced on the image based on these direction and position. For example, three-dimensional objects in the virtual space are placed in the virtual space with a posture and position corresponding to the direction and position of the user's head. Furthermore, if the direction or position of the user's head changes, the posture and position of the objects in the virtual space also change accordingly. This allows the user to experience as if they were actually present at the work site and visually observing the environment while moving around the work site.

[0035] Furthermore, the display control unit 133 detects the user's gaze point in the drawn image based on the detection result of the user's gaze direction. As will be described later, the user's gaze point is used to detect whether the user has not noticed a point of interest set in the virtual space. Note that the user's gaze direction may be detected as the forward direction of the user's head without using the gaze sensor 203. In this case, the user's gaze point is the center of the image drawn by the display control unit 133.

[0036] The processing of the display control unit 133 allows the user to grasp the situation at the work site in detail without actually going to the actual work site. This allows the user to receive training or education on work at the work site in a location other than the work site, such as an office. Furthermore, it is also possible to prepare three-dimensional data 121 showing an appropriate work site for training or education, and have the user receive training or education on that work site.

[0037] The information processing device 100 may, for example, detect the user's hand movements to allow the user to simulate a worksite experience. Recently, however, training and education services utilizing virtual spaces have been increasing. For example, in the medical field, errors in medical procedures are unacceptable, yet opportunities to work on cases are limited. To address this issue, virtual spaces allow for repeated training and failures on various cases, allowing for efficient skill improvement. Furthermore, with the increasing labor shortage and the spread of remote work, it is expected that training and education services in virtual spaces, which are not restricted by time or location, will continue to increase.

[0038] On the other hand, few current services can provide training or education that takes into account each user's individual characteristics, such as strengths and weaknesses, personality, hobbies, and preferences, as well as their work situation. Therefore, if there were services or technologies that could provide training or education that is suited to each user based on their individual characteristics and work situation while taking advantage of the characteristics of virtual space, it would be possible to efficiently improve users' skills.

[0039] There are also several findings regarding skill acquisition. For example, cognitive apprenticeship theory is an educational method that focuses on the apprenticeship-based vocational training that has long been conducted between experts and learners, and theorizes the learning process from a cognitive perspective. According to cognitive apprenticeship theory, the process by which a learner acquires a skill involves the stages of "modeling," "coaching," and "scaffolding and fading." Modeling is the stage in which an expert demonstrates an example to the learner, and the learner observes and visually grasps it. Coaching is the stage in which an expert assigns tasks appropriate to the learner's level and provides hints and advice to guide them to follow the example. Scaffolding and fading is the stage in which an expert asks the learner to complete moderate tasks, supports them, and then removes support as the learner becomes more capable.

[0040] Furthermore, the following reference 1 states that a sense of agency, or the feeling that the user is performing the action themselves, is necessary for skill acquisition. Reference 1: Ito et al., "Basic Research on Physical Skill Transfer Using Integrated Bodies," Proceedings of the 25th Virtual Reality Society of Japan Conference, September 2020, 3C3-7. Based on these findings, it is believed that learning while thinking and moving the user's body, rather than passive learning, is important for skill acquisition and improvement. For this reason, it is difficult to achieve sufficient learning effects by simply displaying the results of work in a virtual space using animations, for example.

[0041] In this embodiment, the information processing device 100 assists the user in the task by changing the display state of the virtual space based on the user's task status in the virtual space so that the user does not lose the sense that they are performing the task themselves, allowing the user to efficiently acquire skills. Specifically, the information processing device 100 sets a focus point in the virtual space that the user should pay attention to, and has the user perform the task of finding the focus point. If the information processing device 100 detects from the user's gaze movement that the user has not noticed the focus point, it executes a predetermined display change process to highlight the focus point. This makes it easier for the user to notice the focus point. This process does not instruct the user about the focus point at an early stage, but only provides a hint if the user has not noticed it, allowing the user to acquire skills without losing the sense that they are performing the task themselves.

[0042] Furthermore, the more times the information processing device 100 determines that the user has not noticed the attention point, the higher the degree of emphasis of the attention point in the display change process. This gradually provides hints when the user has not noticed the attention point, making the above-mentioned effect more pronounced.

[0043] As an example of a user's work, the following will be given: inspecting cracks in a bridge created in a virtual space. A specific crack location is set as a point of interest, and the user is asked to find the crack in that point of interest.

[0044] 4 is a diagram showing a first example of display change processing. An image 210 shown in FIG. 4 is an example of a depiction of the surface of a bridge pier. There are many cracks on the surface of the pier, and important cracks that an operator must find are set as focus points 211.

[0045] Here, a description will be given of the detection conditions for detecting that the user has not noticed the target part. As the detection condition, for example, any of the following detection conditions C1 to C3 can be adopted.

[0046] Detection condition C1: The user's gaze point passes through the target location. Detection condition C2: The target location is not visually recognized for a predetermined time after it is displayed on the screen.

[0047] Detection condition C3: The point of interest leaves the angle of view (changes from being present on the screen to being absent). Methods for detecting the point of interest include, for example, a method of detecting based on the detection results of the gaze sensor 203 (the intersection of the gaze and the screen is the point of interest), and a method of detecting the point of interest without using the gaze sensor 203 by regarding the direction of the gaze as being in front of the head (the center of the image is the point of interest). Of the above detection conditions C1 to C3, any of the above methods for detecting the point of interest can be applied to detection conditions C1 and C2. Note that "visual recognition" in detection condition C2 is determined, for example, when the point of interest remains at a part of the point of interest for a predetermined period of time. On the other hand, the latter of the above methods for detecting the point of interest is applied to detection condition C3.

[0048] In the following description, it is assumed that detection condition C1 is used. In image 210 in FIG. 4 , the user's gaze point 212 is located to the left of the attention location 211. From this state, it is assumed that the gaze point 212 crosses the attention location 211 and moves to the right, as shown in image 210a. In this case, it is detected that the user did not notice the attention location 211 at the time when the gaze point 212 crosses the attention location 211 and passes to the right (first detection). Furthermore, it is assumed that the gaze point 212 crosses the attention location 211 and moves to the left, as shown in image 210b. In this case, it is detected that the user did not notice the attention location 211 at the time when the gaze point 212 crosses the attention location 211 and passes to the left (second detection).

[0049] Note that the above detection condition C1 does not necessarily mean that the point of gaze passes through the location of interest in a straight line. For example, when a first coordinate immediately before the point of gaze reaches the location of interest is compared with a second coordinate immediately after the point of gaze leaves the location of interest, if at least one of the X coordinate or Y coordinate value is different between the first coordinate and the second coordinate, it may be determined that the point of gaze has "passed through." Alternatively, regardless of the first and second coordinates, it may be determined that the point of gaze has "passed through" if it once reaches the location of interest and then leaves the location of interest within a certain time.

[0050] Next, a description will be given of a display change process that is executed when it is detected that the user has not noticed the attention point 211. In a first example of the display change process, the display state of the attention point 211 itself is changed to emphasize the attention point 211. Specifically, a display change (highlighting) is performed such that the width of the line included in the attention point 211 becomes thicker. In the example of FIG. 4, since the attention point 211 is a linear crack, the width of the crack is enlarged. This emphasizes the attention point 211 (i.e., makes it more noticeable), making it easier for the user to find the attention point 211.

[0051] Furthermore, the display is changed so that the width of the crack becomes wider as the number of times the user fails to notice the attention area 211 increases. In the example of Figure 4, the width of the crack is enlarged in image 210a where the first detection was performed, and is further enlarged in image 210b where the second detection was performed. This further emphasizes the attention area 211, making it easier for the user to find the attention area 211.

[0052] 5 is a flowchart showing the display processing procedure when the first display change processing example is used. [Step S11] A point of interest in the virtual space comes into the user's field of view in the virtual space (a predetermined angle of view range centered in front of the user's head). At this time, the display control unit 133 of the information processing device 100 draws an image including the point of interest based on the three-dimensional data 121 and displays it on the display 201 of the HMD unit 200. The display control unit 133 changes the drawing area in the virtual space according to the direction and position of the user's head, and detects and tracks the position of the point of gaze in the drawn image.

[0053] [Step S12] The display control unit 133 determines whether the user has noticed the point of interest. For example, if the user's gaze point remains on a part of the point of interest for a predetermined period of time, it is determined that the user has noticed the point of interest. If it is determined that the user has noticed the point of interest, the processing of FIG. 5 ends. If not, the processing proceeds to step S13.

[0054] [Step S13] The display control unit 133 determines whether the user did not notice the attention spot. Here, if the user's gaze point passes through the attention spot, it is determined that the user did not notice the attention spot. If it is determined that the user did not notice the attention spot, the process proceeds to step S14; otherwise, the process proceeds to step S12.

[0055] The determination processes of steps S12 and S13 described above are repeatedly executed at regular intervals (for example, at a frame cycle). Note that in step S13, for example, it may be determined that the gaze point has "passed" if it once reaches the target location and then leaves the target location within a certain time T. In this case, in step S12, it may be determined that the gaze point has "noticed" if it remains at the target location for longer than the certain time T.

[0056] [Step S14] The display control unit 133 executes a display change process to highlight the point of interest. Here, the width of the crack designated as the point of interest is enlarged (first example of display change process).

[0057] [Step S15] The display control unit 133 determines whether the user has noticed the attention area using the same criteria as in step S12. If it is determined that the user has noticed the attention area, the process in FIG. 5 ends. If not, the process proceeds to step S16.

[0058] [Step S16] The display control unit 133 determines whether the user did not notice the attention area using the same criteria as in step S13. If it is determined that the user did not notice the attention area, the process proceeds to step S17; otherwise, the process proceeds to step S15.

[0059] [Step S17] The display control unit 133 increases the degree of emphasis for the point of interest and executes the same type of display change processing as in step S14. Here, the width of the crack designated as the point of interest is further enlarged. After this, the processing proceeds to step S15.

[0060] The above-described processing of steps S15 to S17 is repeatedly executed at regular intervals (for example, at a frame cycle). Note that in the processing of FIG. 5, a display change processing is executed in which the degree of emphasis of the attention area is increased each time the number of times it is determined that the attention area has not been noticed (i.e., the number of times that it has been detected that the attention area has not been noticed) increases by one. However, the interval of the number of times detection is performed to increase the degree of emphasis is not limited to one, and may be a predetermined value of two or more. Also, an upper limit may be set on the number of times the degree of emphasis is increased.

[0061] Next, a display change process different from the first display change process example will be described. Fig. 6 is a diagram showing a second display change process example. In the second display change process example, an image other than the focus point is displayed near the focus point to highlight the focus point. Here, an arrow image is used as an example of such an image.

[0062] 6 is a drawing of the surface of a bridge pier, similar to that of FIG. 4, and some of the many cracks that exist on the surface of the pier are set as points of interest 221. In addition, in image 220, the user's gaze point 222 is located to the left of point of interest 221.

[0063] From this state, it is assumed that the point of gaze 222 crosses the attention location 221 and moves to the right, as shown in image 220a. In this case, the display control unit 133 detects that the user did not notice the attention location 221 at the time the point of gaze 222 crosses the attention location 221 and passes to the right. Then, the display control unit 133 displays an arrow 223 pointing to the attention location 221 near the attention location 221 (second display change processing example). This emphasizes the attention location 221, making it easier for the user to find the attention location 221.

[0064] Thereafter, as shown in image 220b, the point of gaze 222 crosses the attention location 221 and moves to the left. In this case, the display control unit 133 detects that the user did not notice the attention location 221 at the time when the point of gaze 222 crosses the attention location 221 and passes to the left. Then, the display control unit 133 changes the display state of the arrow 223 so as to increase the degree of emphasis. In the example of FIG. 6, the size of the arrow 223 is enlarged compared to the time of image 220a. This further emphasizes the attention location 221, making it even easier for the user to find the attention location 221.

[0065] In the display process procedure when the second example of the display change process is used, for example, in step S14 of Fig. 5, an arrow 223 is displayed near the point of interest 221. In addition, in step S17, the size of the arrow 223 is enlarged.

[0066] 7 is a diagram showing a third example of display change processing. In the third example of display change processing, similar to the first example of display change processing, the display state of the attention point 211 itself is changed, thereby emphasizing the attention point 211. The difference from the first example of display change processing is that the size of the attention point 211 is enlarged.

[0067] 7 depicts the surface of a bridge pier, with a crack on the surface of the pier set as a point of interest 231. In addition, in image 230, a user's gaze point 232 is located to the right of point of interest 231.

[0068] From this state, suppose that the point of gaze 232 crosses the attention location 231 and moves to the left, as shown in image 230a. In this case, the display control unit 133 detects that the user did not notice the attention location 231 at the time when the point of gaze 232 crosses the attention location 231 and passes to the left. Then, the display control unit 133 enlarges the size of the attention location 231 (third example of display change processing). This emphasizes the attention location 231, making it easier for the user to find the attention location 231.

[0069] Thereafter, it is assumed that the point of gaze 232 crosses the attention location 231 and moves to the right, as shown in image 230b. In this case, the display control unit 133 detects that the user did not notice the attention location 231 at the time when the point of gaze 232 crosses the attention location 231 and passes to the right. Then, the display control unit 133 further enlarges the size of the attention location 231. This further emphasizes the attention location 231, making it even easier for the user to find the attention location 231.

[0070] In the display process procedure when the third example of the display change process is used, for example, the size of the attention point 231 is enlarged in step S14 of Fig. 5. Furthermore, the size of the attention point 231 is further enlarged in step S17.

[0071] 8 is a diagram showing a fourth example of the display change process. In the fourth example of the display change process, the display position of the attention point is changed so as to create a head redirection effect, thereby emphasizing the attention point.

[0072] Fig. 8 shows a downward view from above the user's head 243, and direction Dh indicates the front direction of the head 243. In the state at the top of Fig. 8, in the drawn image, the attention point 241 is displayed to the left of the front of the user's head 243. In this state, it is assumed that the user's gaze point 242 moves from the left side of the attention point 241 to the right, passing through the attention point 241, and it is detected that the user did not notice the attention point 241.

[0073] In this case, as a fourth example of display change processing, as shown in the lower part of Fig. 8 , the area of ​​the virtual space that is drawn on the image is changed so that the position of the attention point 241 approaches the gaze point 242. In the example in the lower part of Fig. 8 , the image display is changed so that the drawing area in the virtual space is rotated and moved to the right by an angle θ. In actual processing, for example, the horizontal angle of the head 243 based on the detection results of the direction / position sensor 202 is corrected so that it faces leftward by an angle θ from the actually measured angle, and the image is drawn based on the corrected angle. This processing guides the user's line of sight in the direction of the attention point 241, making it easier for the user to find the attention point 241.

[0074] 9 is a diagram showing an example of a screen display when the fourth display change processing example is used. An image 240 shown in FIG. 9 depicts the surface of a bridge pier, and the location of a crack on the surface of the pier is set as a focus point 241. In addition, in image 240, a user's gaze point 242 is located to the left of focus point 241.

[0075] From this state, assume that the point of gaze 242 crosses the attention location 241 and moves to the right, as shown in image 240a. In this case, the display control unit 133 detects that the user did not notice the attention location 241 when the point of gaze 242 crosses the attention location 241 and passes to the right (first detection). The display control unit 133 then changes the image display in the virtual space so that the attention location 241 moves a distance D1 to the right (in the direction of arrow 244) (fourth example of display change processing). This emphasizes the attention location 241, making it easier for the user to find the attention location 241.

[0076] Thereafter, as shown in image 240b, the gaze point 242 crosses the attention location 241 and moves to the left. In this case, the display control unit 133 detects that the user did not notice the attention location 241 when the gaze point 242 crosses the attention location 241 and passes to the left (second detection). The display control unit 133 then changes the image display in the virtual space so that the attention location 241 moves a distance D2 to the left (in the direction of arrow 245). At this time, the degree of emphasis is increased by making the moving distance D2 longer than the distance D1 at the time of the first detection. This makes it even easier for the user to find the attention location 241.

[0077] In the process of Figure 9, the direction of image movement in response to the detection that the user has not noticed the attention point 241 is limited to the horizontal direction, but the image may also be moved vertically so that the attention point 241 approaches the gaze point 242.

[0078] 5, the image display is changed so that the attention point 241 approaches the gaze point 242. Also, in step S17, the image display is changed so that the attention point 241 approaches the gaze point 242. In step S17, the amount of image movement is greater than in the most recently executed process of step S14 or step S17.

[0079] Next, an example of a task that can be the subject of training or education will be described, in which the most appropriate search result is selected from search results for similar cases. In this task, medical images of past cases similar to an original medical image (e.g., a CT image or an MRI image, such as a computed tomography (CT) or magnetic resonance imaging (MRI) image) that serves as a search key are searched for. In this task, the HMD unit 200 is not used, and images are displayed on a display 111 connected to the information processing device 100. The information processing device 100 is used as a terminal device for searching for similar cases, and the user operates the information processing device 100 to perform operations for searching for similar cases and visually checks the images displayed on the display 111 to perform the task.

[0080] Fig. 10 is a diagram showing a fifth example of the display change process. An image 250 shown in Fig. 10 displays an image area 251 including an original medical image that serves as a search key. In the example of Fig. 10, the image area 251 displays not only the medical image but also feature data (histogram) related to the medical image.

[0081] A user operation requests a search for similar case images that are similar to the medical image in the image area 251. The search process in response to this request is executed, for example, by an external server connected to the information processing device 100, and the search results are transmitted to the information processing device 100. As the search results, similar case images that are similar to the original medical image are transmitted in descending order of similarity. In addition to the similar case images, feature data related to the similar case images is also transmitted.

[0082] 10 displays similar case image regions 252a to 252f, each containing similar case images and feature data. In image 250, similar case image regions containing similar case images with high similarity are displayed from the upper left to the upper right, and then from the lower left to the lower right. That is, in image 250, of similar case image regions 252a to 252f, similar case image region 252a contains similar case image with highest similarity, and similar case image region 252f contains similar case image with lowest similarity.

[0083] However, the similar case image that actually depicts the lesion most similar to the lesion in the original medical image is considered to be included in similar case image region 252f, which is circled in a thick line. That is, of similar case image regions 252a to 252f, similar case image region 252f is the correct answer, and the task that is the subject of training or education is for the user to find and select the correct similar case image region 252f. In this case, similar case image region 252f corresponds to the area of ​​interest.

[0084] 10, it is assumed that the correct answer is selected by a user operation from among similar case image areas 252a to 252f. As an example, it is assumed that if the correct answer is not selected within a predetermined time after image 250 containing the search results is displayed, it is detected that the user did not notice the part of interest.

[0085] 10 , it is assumed that the correct similar case image region 252f was not selected after a predetermined time had elapsed since the display of the image 250, and that the user did not notice the region of interest. In this case, the display control unit 133 of the information processing device 100 moves the position of the similar case image region 252f to a higher position (a position with a higher degree of similarity) to make it easier for the user to find the similar case image region 252f. In the example of FIG. 10 , the position of the correct similar case image region 252f is moved from the bottom right to the bottom left, as in image 250a.

[0086] After that, suppose that a predetermined time has passed and it is detected that the user did not notice the region of interest. In this case, the display control unit 133 of the information processing device 100 moves the position of the similar case image region 252f to a higher position, thereby increasing the degree of emphasis of the similar case image region 252f. In the example of FIG. 10 , as shown in image 250b, the position of the correct similar case image region 252f is moved from the bottom left to the center of the top. This makes it easier for the user to find the similar case image region 252f.

[0087] 11 is a flowchart showing the procedure of the display process when the fifth example of the display change process is used. [Step S21] The display control unit 133 displays similar case image areas on the display 111 in descending order of similarity of the similar case images included in the areas.

[0088] [Step S22] The display control unit 133 determines whether a correct similar case image region has been selected by the user. If the correct region has been selected, the process in FIG. 11 ends. If the correct region has not been selected, the process proceeds to step S23.

[0089] [Step S23] The display control unit 133 determines whether a certain time has elapsed since the most recent execution of step S21 or step S24. If the certain time has not elapsed, the process proceeds to step S22. If the certain time has elapsed, the process proceeds to step S24.

[0090] [Step S24] The display control unit 133 moves the position of the correct similar case image region to a higher position (a position with a higher similarity) (fifth example of display change processing). After this, the processing proceeds to step S22. Note that each time the processing of step S24 is repeated, the position of the correct similar case image region is moved to a higher position, and the degree of emphasis of this region is increased.

[0091] In addition, in the above-mentioned fifth display change example, as another example, the position of the correct similar case image area 252f may be gradually moved closer to a position that is more easily noticeable to the user, such as the center of the entire display area of ​​the similar case image areas 252a to 252f.

[0092] In the fifth example of the display change process, a case has been described in which display change process is performed to highlight a focus area (a region of the correct similar case image) in an image displayed on the display 111. However, as another example, the above-described display change process may be performed on an image displayed on the display 201 of the HMD unit 200. For example, a display terminal device displaying an image such as that shown in FIG. 10 may be placed in a virtual space, and a user may search for search results by viewing the screen of the display terminal device. In this case, display change process such as that shown in FIG. 10 is performed on the screen of the display terminal device.

[0093] The processing functions of the devices (e.g., information processing devices 1, 100) shown in each of the above embodiments can be realized by a computer. In this case, a program describing the processing content of the functions to be possessed by each device is provided, and the processing functions are realized on the computer by executing the program. The program describing the processing content can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical discs, and semiconductor memories. Examples of magnetic storage devices include hard disk drives (HDDs) and magnetic tapes. Examples of optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0094] When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and sold. Alternatively, the program can be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.

[0095] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing in accordance with the program. Note that the computer can also read the program directly from a portable recording medium and execute processing in accordance with that program. The computer can also execute processing in accordance with the program received each time a program is transferred from a server computer connected via a network.

[0096] The foregoing merely illustrates the principles of the present invention. Further, since numerous modifications and changes will be apparent to those skilled in the art, the present invention is not limited to the exact construction and application shown and described above, and all corresponding modifications and equivalents are deemed to be within the scope of the present invention as defined by the appended claims and their equivalents.

[0097] REFERENCE SIGNS LIST 1 Information processing device 2 Display device 3 Attention point 4, 4a, 4b Image 5 Point of interest

Claims

1. A display control method, in which a computer causes a display device to display an image including a point of interest that the user should pay attention to, and when it is detected that the user has not noticed the point of interest, executes a display change process for changing the display of the image so as to emphasize the point of interest.

2. The display control method according to claim 1, wherein in the display change process, the degree of emphasis of the point of interest is increased as the number of times it is detected that the user has not noticed the point of interest increases.

3. The display control method according to claim 1, wherein in the display change process, the display state of the point of interest is changed so as to emphasize the point of interest.

4. The display control method according to claim 1, wherein in the display change process, an emphasis image for emphasizing the point of interest is displayed in the vicinity of the point of interest.

5. The display control method according to claim 1, wherein in the display change process, the display position of the point of interest in the image is changed according to a predetermined rule.

6. The image is drawn based on data indicating a three-dimensional space, and shows a region of the three-dimensional space as seen from the viewpoint of the user. In the display change process, the drawing region in the three-dimensional space is changed so that the display position of the point of interest in the image approaches the position of the user's fixation point in the image. The display control method according to claim 1.

7. The display control method according to claim 1, wherein the detection that the user has not noticed the point of interest is performed based on the positional relationship between the user's fixation point and the point of interest in the image.

8. A display control program for causing a computer to execute a process of causing a display device to display an image including a point of interest that the user should pay attention to, and when it is detected that the user has not noticed the point of interest, executing a display change process for changing the display of the image so as to emphasize the point of interest.

Images