Display device, display method, and program

Abstract

To properly adjust display mode according to users.SOLUTION: A display device includes: a display unit 22 for displaying an image; an output control unit 48A which causes the display unit 22 to display an image; a location information acquisition unit 50 which acquires location information of a user; and a visit frequency specifying unit 52 which specifies frequency of visit which is the number of times that the user visited the same place, on the basis of the location information of the user. The output control unit 48A causes the display unit 22 to display a sub image including information on an object of the same kind as an object located within a predetermined distance range from a place that the user visited, on the basis of the frequency of visit.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a display device, a display method, and a program.

Background Art

[0002] Recently, with the evolution of high-speed CPUs, high-definition screen display technologies, small and lightweight battery technologies, and the spread and broadband expansion of wireless network environments, information devices have evolved significantly. As such information devices, not only smartphones, which are typical examples, but also so-called wearable devices worn by users have become widespread. As wearable devices, there is an HMD (Head Mount Display) that provides visual information and is used for AR (Augumented Reality) and VR (Virtual Reality). For example, Patent Document 1 discloses a technique for correcting the color tone of an object based on the color tone of a marker arranged in the real space as a technique for considering the color tone of the object when superimposing the object on an image in the real space using AR technology.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, in a display device that displays an image such as an HMD, the display mode becomes uniform regardless of the user of the display device. Therefore, it is required to appropriately adjust the display mode according to the user of the display device.

[0005] In view of the above problems, an object of the present invention is to provide a display device, a display method, and a program capable of appropriately adjusting a display mode according to a user.

Means for Solving the Problem

[0006] A display device according to an aspect of the present invention includes a display unit that displays an image, an output control unit that causes the display unit to display an image, a position information acquisition unit that acquires position information of a user, and an access count specifying unit that specifies an access count, which is the number of times the user has visited the same place, based on the position information of the user. The output control unit causes the display unit to display a sub-image including information on an object of the same type as an object within a predetermined distance range from the place visited by the user, based on the access count.

[0007] A display method according to an aspect of the present invention is a display method for causing a display unit to display an image, including the steps of acquiring position information of the user, specifying an access count, which is the number of times the user has visited the same place, based on the position information of the user, and causing the display unit to display a sub-image including information on an object of the same type as an object within a predetermined distance range from the place visited by the user, based on the access count.

[0008] A program according to an aspect of the present invention is a program for causing a computer to execute a display method for causing a display unit to display an image, including the steps of acquiring position information of the user, specifying an access count, which is the number of times the user has visited the same place, based on the position information of the user, and causing the display unit to display a sub-image including information on an object of the same type as an object within a predetermined distance range from the place visited by the user, based on the access count.

Advantages of the Invention

[0009] According to the present invention, the display mode can be appropriately adjusted according to the user.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments described below.

[0012] (First Embodiment) FIG. 1 is a schematic diagram of a display device according to the first embodiment. The display device 10 according to the first embodiment is a display device that displays an image. As shown in FIG. 1, the display device 10 is a so-called wearable device that is worn on the body of the user U. In the example of the present embodiment, the display device 10 includes a device 10a worn on the eyes of the user U, a device 10b worn on the ears of the user U, and a device 10c worn on the arm of the user. The device 10a worn on the eyes of the user U includes a display unit 22 described later that outputs a visual stimulus (displays an image) to the user U. The device 10b worn on the ears of the user U includes an audio output unit that outputs an auditory stimulus (voice) to the user U, and the device 10c worn on the arm of the user U includes a tactile stimulus output unit that outputs a tactile stimulus to the user U. However, the configuration of FIG. 1 is an example, and the number of devices and the wearing position on the user U may be arbitrary. For example, the display device 10 may not include the device 10b or the device 10c. Also, for example, the display device 10 is not limited to a wearable device. For example, the display device 10 may be a device carried by the user U, such as a smartphone or a tablet terminal.

[0013] (Main Image) FIG. 2 is a diagram showing an example of an image displayed by the display device. As shown in FIG. 2, the display device 10 provides the main image PM to the user U through the display unit 22. As a result, the user U wearing the display device 10 can visually recognize the main image PM. In this embodiment, the main image PM is an image of a scenery that the user U would visually recognize when it is assumed that the user U is not wearing the display device 10, and it can also be said that it is an image of an actual object that enters the visual field range of the user U. The visual field range refers to the range that can be seen without moving the eyeball centered on the line of sight of the user U. In this embodiment, the display device 10 provides the main image PM to the user U by, for example, transmitting external light (peripheral visible light) from the display unit 22. That is, in this embodiment, it can be said that the user U directly visually recognizes the image of the actual scenery through the display unit 22. However, the display device 10 is not limited to directly allowing the user U to visually recognize the image of the actual scenery, and may provide the main image PM to the user U through the display unit 22 by displaying the image of the main image PM on the display unit 22. In this case, the user U will visually recognize the image of the scenery displayed on the display unit 22 as the main image PM. In this case, the display device 10 causes the display unit 22 to display, as the main image PM, the image that enters the visual field range of the user U captured by the imaging unit 28 described later. Note that in FIG. 2, a road and a building are included as the main image PM, but this is merely an example.

[0014] (Sub-image) As shown in FIG. 2, the display device 10 causes the display unit 22 to display the sub-image PS so as to be superimposed on the main image PM provided through the display unit 22. As a result, the user U will visually recognize the image in which the sub-image PS is superimposed on the main image PM. The sub-image PS is an image superimposed on the main image PM, and can be said to be an image other than the actual scenery that enters the visual field range of the user U. That is, it can be said that the display device 10 provides AR (Augumented Reality) to the user U by superimposing the sub-image PS on the main image PM which is an actual scenery.

[0015] In this embodiment, the sub-image PS is an image that contains information about the object included in the main image PM. That is to say, the sub-image PS can be said to be an image that contains information about the real object that has entered the user U's field of view (information about the object that the user U is currently visually recognizing). The sub-image PS is set for each object. When the object in the main image PM is included (when the user U is visually recognizing that object), the sub-image PS for that object is displayed. In the example of FIG. 2, the object that the user U is visually recognizing is a building, and the sub-image PS, which is information about that building, is being displayed. In FIG. 2, the sub-image PS is the letter "BU", but this is merely an example.

[0016] In this way, the display device 10 provides the main image PM and the sub-image PS. In addition to this, different content images of the main image PM and the sub-image PS may also be displayed on the display unit 22. The content image may be an image of any content, such as a movie or a TV program. Further, the display device 10 may be one that displays the sub-image PS without providing the main image PM.

[0017] (Configuration of the display device) FIG. 3 is a schematic block diagram of the display device according to the first embodiment. As shown in FIG. 3, the display device 10 includes an input unit 20, a display unit 22, a storage unit 24, a communication unit 26, an imaging unit 28, a GNSS receiver 30, and a control unit 32.

[0018] The input unit 20 is a device that receives the user's operations and may be, for example, a touch panel. The display unit 22 is a display that displays images. In this embodiment, the display unit 22 is a so-called HMD (Head Mount Display). In addition to the display unit 22, an audio output unit (speaker) that outputs audio and a tactile stimulus output unit that outputs a tactile stimulus to the user U may also be provided as output units. The tactile stimulus output unit outputs a tactile stimulus to the user by physically operating such as vibrating, but the type of tactile stimulus is not limited to vibration and may be any arbitrary one.

[0019] The storage unit 24 is a memory that stores various types of information such as the calculation content and programs of the control unit 32, and includes, for example, at least one of a main memory device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and an external storage device such as an HDD (Hard Disk Drive). The program for the control unit 32 stored by the storage unit 24 may be stored in a recording medium readable by the display device 10.

[0020] The communication unit 26 is a module that communicates with external devices and the like, and may include, for example, an antenna. The communication method by the communication unit 26 is wireless communication in this embodiment, but the communication method may be arbitrary.

[0021] The imaging unit 28 is a camera that images an object within the visual field range of the user U. The imaging unit 28 is provided at a position such that the imaging range overlaps with the visual field range of the user U. In the example of FIG. 1, the imaging unit 28 is provided in the device 10a and is attached so that the imaging direction is the direction in which the face of the user U is facing. Thereby, the imaging unit 28 can image an object within the visual field range of the user U. Note that the imaging unit 28 may be a video camera that images at a predetermined frame rate. The number of imaging units 28 is arbitrary and may be single or plural. When there are a plurality of imaging units 28, information on the direction in which the camera 20A is facing may also be acquired.

[0022] The GNSS receiver 30 is a device that detects the position information of the display device 10. The position information here is earth coordinates. In this embodiment, since the display device 10 is worn by the user U, the positions of the user U and the display device 10 are the same. Therefore, it can be said that the GNSS receiver 30 detects the position information of the user U. In this embodiment, the GNSS receiver 30 is a so-called GNSS (Global Navigation Satellite System) module that receives radio waves from satellites and detects the position information of the display device 10 (user U). Note that in the first embodiment, the GNSS receiver 30 is not an essential component.

[0023] The control unit 32 is an arithmetic unit, that is, a CPU (Central Processing Unit). The control unit 32 includes an image acquisition unit 40, an object identification unit 42, a viewing frequency identification unit 44, a display mode setting unit 46, and an output control unit 48. The control unit 32 reads a program (software) from the storage unit 24 and executes it, thereby realizing the image acquisition unit 40, the object identification unit 42, the viewing frequency identification unit 44, the display mode setting unit 46, and the output control unit 48, and executing their processes. Note that the control unit 32 may execute these processes by one CPU, or may include a plurality of CPUs and execute the processes by these plurality of CPUs. Further, at least a part of the image acquisition unit 40, the object identification unit 42, the viewing frequency identification unit 44, the display mode setting unit 46, and the output control unit 48 may be realized by hardware.

[0024] The image acquisition unit 40 controls the imaging unit 28 to cause the imaging unit 28 to capture an image, and acquires the image (image data) captured by the imaging unit 28. That is, the image acquisition unit 40 acquires an image (image data) with the imaging range being the visual field range of the user U captured by the imaging unit 28. The image acquisition unit 40 causes the imaging unit 28 to capture an image at predetermined intervals. Each time the imaging unit 28 captures an image, the image acquisition unit 40 acquires the image data captured by the imaging unit 28.

[0025] Based on the image captured by the imaging unit 28, the object identification unit 42 identifies the type of the object within the visual field range of the user U. The process of the object identification unit 42 will be described later.

[0026] Based on the identification result of the object identification unit 42, that is, based on the type of the object within the visual field range of the user U, the viewing frequency identification unit 44 identifies the viewing frequency, which is the number of times the user U has viewed the same type of object. The process of the viewing frequency identification unit 44 will be described later.

[0027] The display mode setting unit 46 determines the display mode of the sub-image PS based on the number of visual recognitions. The display mode of the sub-image PS refers to the way of displaying the sub-image PS and can also be called the display mode. The processing of the display mode setting unit 46 will be described later.

[0028] The output control unit 48 controls the display unit 22 to display an image on the display unit 22. The output control unit 48 causes the display unit 22 to display the sub-image PS so as to be superimposed on the main image PM. Further, the output control unit 48 causes the sub-image PS to be displayed based on the display mode of the sub-image PS determined by the display mode setting unit 46. Since the display mode of the sub-image PS is set based on the number of visual recognitions, it can be said that the output control unit 48 causes the sub-image PS to be displayed based on the number of visual recognitions. The specific processing content of the output control unit 48 will be described later.

[0029] The display device 10 has the configuration as described above. Hereinafter, the processing content by the display device 10 will be described.

[0030] (Setting process of display mode) First, as the processing content by the display device 10, the setting process of the display mode will be described. FIG. 4 is a flowchart for explaining the setting process of the display mode.

[0031] (Acquisition of image) As shown in FIG. 4, the control unit 32 acquires the image captured by the imaging unit 28 through the image acquisition unit 40 (step S10). The image acquisition unit 40 acquires an image (image data) that is captured by the imaging unit 28 with the visual field range of the user U as the imaging range. That is, it can be said that the image acquisition unit 40 acquires an image including the main image PM within the visual field range of the user U that is visually recognized through the display unit 22.

[0032] (Specification of type of object to be visually recognized) The control unit 32 extracts the object to be visually recognized and identifies the type of the object to be visually recognized based on the image acquired by the image acquisition unit 40 by the visually recognized object identification unit 42 (step S12). The object to be visually recognized refers to the object visually recognized by the user U through the display unit 22, and can be said to be an object existing within the visual field range of the user U. The visually recognized object identification unit 42 extracts the object shown in the image acquired by the image acquisition unit 40 as the object to be visually recognized and identifies the type of the object to be visually recognized. The type of the object refers to the category to which the object belongs when the object is classified according to a predetermined classification criterion. The classification criterion here may be arbitrary.

[0033] The visually recognized object identification unit 42 may extract the object to be visually recognized and identify the type in any method. For example, an AI (Artificial Intelligence) model may be used to identify the object. In this case, the AI model is stored in the storage unit 24 and is a model for extracting the object shown in the image from the image data and identifying the type. The AI model uses the image data and the information indicating the type of the object shown in the image as one data set, and is a trained AI model constructed by learning a plurality of data sets as teacher data. The visually recognized object identification unit 42 inputs the image data acquired by the image acquisition unit 40 into the trained AI model, obtains the information identifying the type of the object to be visually recognized shown in the image, and identifies the type of the object to be visually recognized. Note that any model may be used as the AI model. For example, a CNN (Conventional Neural Network: Convolutional Neural Network) model may be used.

[0034] Note that the visible object specifying unit 42 may extract all the objects shown in the image acquired by the image acquisition unit 40 (objects existing within the visual field range of the user U) as visible objects, but is not limited thereto, and may extract some of the objects included in the image acquired by the image acquisition unit 40 (objects existing within the visual field range of the user U) as visible objects. In this case, for example, the visible object specifying unit 42 may extract, as visible objects, the objects that overlap with the line of sight of the user U among the objects included in the image acquired by the image acquisition unit 40. The visible object specifying unit 42, for example, inputs the image data in a predetermined region (for example, the central region) of the image acquired by the image acquisition unit 40 into the AI model, obtains information specifying the types of visible objects shown in the predetermined region, and specifies the types of visible objects.

[0035] Also, for example, the visible object specifying unit 42 may extract, as visible objects, the objects that are within the visual field range continuously for a predetermined time or more. In this case, the visible object specifying unit 42 extracts, as visible objects, the objects that are within the visual field range continuously for a predetermined time or more from the images continuously captured in time series by the imaging unit 28, that is, the objects included in all of the images continuously for a predetermined time or more. In this case, for example, the visible object specifying unit 42 inputs the image data of the images continuously captured in time series into the AI model respectively, and obtains information on the types of the objects shown in each image. Then, when the same type of object is shown in all of the images continuously captured for a predetermined time or more, the visible object specifying unit 42 sets that object as a visible object.

[0036] Also, the visible object specifying unit 42 may also obtain information indicating the positions of the visible objects in the main image PS (the coordinates of the visible objects in the image acquired by the image acquisition unit 40).

[0037] (Specification of the number of viewing times) Next, based on the identification result of the object identification unit 42, the control unit 32 causes the visual recognition count determination unit 44 to determine the number of times the user U has visually recognized the same type of object (step S14). The visual recognition count determination unit 44 counts the number of times the object has been extracted as the visual recognition object by the object identification unit 42 for each type of visual recognition object, sets it as the visual recognition count, and stores it in the storage unit 24. That is, the visual recognition count determination unit 44 determines the number of times the same type of visual recognition object has been extracted by the object identification unit 42 as the visual recognition count. For example, when the object identification unit 42 extracts an object of the type "banana" five times at different timings, the visual recognition count determination unit 44 determines the visual recognition count of the object of the type "banana" as five times.

[0038] When a predetermined period has elapsed, the visual recognition count determination unit 44 may reset the visual recognition count to zero. That is, the visual recognition count determination unit 44 may set the number of times the same type of visual recognition object has been extracted by the object identification unit 42 within a predetermined period as the visual recognition count. The predetermined period may be set arbitrarily, for example, it may be one month or one year.

[0039] (Setting of display mode) Based on the visual recognition count, the display mode setting unit 46 sets the display mode of the sub-image PS. The sub-image PS is information about an object within the visual field range of the user U (the object included in the main image PS), and is displayed when the object is within the visual field range of the user U. The display mode of the sub-image PS refers to the method of displaying the sub-image PS as described above. The display mode setting unit 46 sets the display mode of the sub-image PS for each type of object.

[0040] More specifically, based on the visual recognition count, the display mode setting unit 46 sets the display mode of the sub-image PS for each type of object. That is, based on the visual recognition count, the display mode setting unit 46 sets the display mode of the sub-image PS for the type of object for which the visual recognition count has been counted. For example, based on the visual recognition count of an object of the type "banana", the display mode setting unit 46 sets the display mode of the sub-image PS for the object of the type "banana".

[0041] FIG. 5 and FIG. 6 are diagrams showing an example of displaying a sub-image in the normal mode and the enhanced mode. The display mode setting unit 46 sets the display mode of the sub-image PS to either the normal mode or the enhanced mode according to the number of times of visual recognition. The enhanced mode is a mode in which information about the object is displayed more prominently than in the normal mode. That is, the sub-image PS set to the enhanced mode is displayed more conspicuously than the sub-image PS set to the normal mode. For example, in the example of FIG. 5, when an object of the type of banana enters the visual field range of the user U, the case where the character "banana" is displayed as the sub-image PS is shown. In the example of FIG. 5, the character "banana" in the enhanced mode is larger than the character "banana" in the normal mode, and the character in the enhanced mode is displayed more prominently. Also, in the example of FIG. 6, the case where an arrow indicating an object of the type of banana is displayed as the sub-image PS is shown. In the example of FIG. 6, the arrow in the enhanced mode is larger than the arrow in the normal mode, and the arrow in the enhanced mode is displayed more prominently.

[0042] However, FIGS. 5 and 6 are just examples, and the sub-image PS may be any image as long as it shows information about the object. For example, the sub-image PS may be characters in any language that show information about the object, or may be an icon of any shape such as an exclamation mark. Also for example, the sub-image PS may be a sentence explaining the object. In this case, the enhanced mode may refer to a higher degree of detail in the explanation of the object than in the normal mode. Also for example, as the sub-image PS, information about the same type of visually recognized object extracted previously may be displayed, or comparison information that is information comparing the object currently being visually recognized by the user U with the same type of visually recognized object extracted previously may be displayed. In this case, for example, when the user U is currently visually recognizing a banana, information about the banana previously visually recognized by the user U may be displayed as the sub-image PS, or information comparing the currently visually recognized banana with the previously visually recognized banana may be displayed.

[0043] Also, in the examples of FIGS. 5 and 6, the sub-image PS in the emphasis mode was made larger than the sub-image PS in the normal mode, but it is not limited thereto, and any display method that emphasizes the sub-image PS in the emphasis mode more than the sub-image PS in the normal mode may be used. For example, the display position of the sub-image PS in the emphasis mode within the main image PS may be made more prominent (e.g., the center, etc.) than the display position of the sub-image PS in the normal mode. Also, the luminance of the sub-image PS in the emphasis mode may be made higher than the luminance of the sub-image PS in the normal mode. Also, the display frequency of the sub-image PS in the emphasis mode may be made higher than the display frequency of the sub-image PS in the normal mode. Here, the display frequency may refer to the display time of the sub-image PS per unit time, or may refer to the speed at which the sub-image PS blinks.

[0044] Hereinafter, returning to the description in FIG. 4, a method for setting the display mode based on the number of visual recognitions will be specifically described. As shown in FIG. 4, the control unit 32 determines whether the number of visual recognitions is equal to or more than a predetermined number by the display mode setting unit 46 (step S16). When the number of visual recognitions is equal to or more than the predetermined number (step S16; Yes), the display mode of the sub-image PS for the object of the same type as the object of visual recognition is set to the emphasis mode (step S18). On the other hand, when the number of visual recognitions is not equal to or more than the predetermined number (step S16; No), that is, when the number of visual recognitions is less than the predetermined number, the process proceeds to step S20 without setting the display mode of the sub-image PS for the object of the same type as the object of visual recognition to the emphasis mode. More specifically, the display mode setting unit 46 sets the display mode of the sub-image PS for the object of the same type as the object of visual recognition with a number of visual recognitions less than the predetermined number to the normal mode. For example, when the number of visual recognitions of the object of the type of banana is equal to or more than the predetermined number and the number of visual recognitions of the object of the type of apple is less than the predetermined number, the display mode setting unit 46 sets the display mode of the sub-image PS for the object of the type of banana to the emphasis mode and sets the display mode of the sub-image PS for the object of the type of apple to the normal mode.

[0045] After that, if the process ends (step S20; Yes), this process ends. If the process does not end (step S20; No), it returns to step S10 and repeats the process. At this time, if there is a visual recognition object whose visual recognition count was less than the predetermined number but the visual recognition count became equal to or more than the predetermined number in the subsequent process, the display mode of the sub-image PS for the object of the same type as the visual recognition object is switched from the normal mode to the emphasized mode.

[0046] As described above, the display mode setting unit 46 sets the sub-image PS when the visual recognition count is less than the predetermined number to the normal mode, and sets the sub-image PS when the visual recognition count is equal to or more than the predetermined number to the emphasized mode. However, the process of the display mode setting unit 46 is not limited to this, and the display mode of the sub-image PS may be set by any method based on the visual recognition count. For example, when the visual recognition count is less than the predetermined number, the display mode setting unit 46 may set it to the non-display mode in which the sub-image PS is not displayed, and when the visual recognition count becomes equal to or more than the predetermined number, it may be set to the mode in which the sub-image PS is displayed. Also, for example, the display mode setting unit 46 may set the display mode so that the degree of emphasis of the sub-image PS increases as the visual recognition count increases. For example, the display mode setting unit 46 may enlarge the sub-image PS as the visual recognition count increases.

[0047] For example, the display mode setting unit 46 may select whether to set to a non-display mode in which the sub-image PS is not displayed or a mode in which the sub-image PS is displayed for each type of object. In this case, for example, the display mode setting unit 46 acquires type information indicating the type of object for which the sub-image PS is to be displayed. Then, for objects of types other than those indicated by the type information, the display mode setting unit 46 sets to the non-display mode. On the other hand, for objects of the type indicated by the type information, the display mode setting unit 46 sets to the mode in which the sub-image PS is to be displayed. Note that the type information may be information indicating the type of object for which the sub-image PS is not to be displayed. In this case, for objects of the type indicated by the type information, the non-display mode is set. This type information may be set in advance, for example, set in advance by the user U. That is, for example, when the user U is interested in fruits and not interested in meats, the mode for fruits is set to the display mode and the mode for meats is set to the non-display mode. In this way, by setting the display mode for each type of object, the sub-image PS can be provided according to the preferences of the user U.

[0048] (Display process of sub-image) Next, the display process of the sub-image PS will be described. The output control unit 48 causes the display unit 22 to display the sub-image PS related to the visual object extracted by the visual object specifying unit 42 so as to be superimposed on the main image PM including the visual object. In other words, when a visual object exists within the visual field range of the user U, the output control unit 48 causes the display unit 22 to display the sub-image PS related to the visual object.

[0049] The output control unit 48 causes the display unit 22 to display the sub-image PS related to the visual object (visual object within the visual field range of the user U) extracted by the visual object specifying unit 42 based on the display mode set by the display mode setting unit 46, that is, based on the number of visual recognitions. The output control unit 48 reads out the display mode of the sub-image PS related to the type of the visual object extracted by the visual object specifying unit 42, and displays the sub-image PS for the extracted visual object in that display mode. This will be described in more detail below.

[0050] FIG. 7 is a flowchart for explaining the display process of the sub-image. As shown in FIG. 7, the processes up to S12 for extracting the object to be visually recognized and specifying the type of the object to be visually recognized are the same as the processes in the setting of the display mode shown in FIG. 4. That is, in the present embodiment, the control unit 32 performs the processes after step S14 (display mode setting process) shown in FIG. 4 and the processes after step S30 (display process of the sub-image PS) shown in FIG. 7 in parallel.

[0051] When the object to be visually recognized is extracted in step S12 and the type of the object to be visually recognized is specified, the control unit 32 determines whether the display mode of the sub-image PS for the type of the extracted object to be visually recognized is the emphasis mode by the output control unit 48 (step S30). When the display mode for the type of the extracted object to be visually recognized is the emphasis mode (step S30; Yes), the output control unit 48 causes the display unit 22 to display the sub-image PS for the object to be visually recognized in the emphasis mode (step S32). The output control unit 48 displays the sub-image PS in the emphasis mode so as to overlap the main image PM including the object to be visually recognized. On the other hand, when the display mode for the type of the extracted object to be visually recognized is not the emphasis mode (step S30; No), that is, when it is the normal mode, the output control unit 48 causes the display unit 22 to display the sub-image PS for the object to be visually recognized in the normal mode (step S34). The output control unit 48 displays the sub-image PS in the normal mode so as to overlap the main image PM including the object to be visually recognized. That is, it can be said that the output control unit 48 emphasizes and displays the sub-image (sub-image PS set in the emphasis mode) to be displayed when the number of visual recognitions is equal to or more than a predetermined number, rather than the sub-image PS (sub-image PS set in the normal mode) to be displayed when the number of visual recognitions is less than the predetermined number.

[0052] For example, when the type of the extracted object to be visually recognized is a banana, in the above example, since the sub-image PS for the banana is in the enhanced mode, the sub-image PS related to the banana is displayed in the enhanced mode. That is, when the user U has visually recognized a banana a predetermined number of times or more, since the banana is set to the enhanced mode, when the user U visually recognizes a banana this time, the sub-image PS related to the banana is displayed in the enhanced mode. On the other hand, when the type of the extracted object to be visually recognized is an apple, in the above example, since the sub-image PS for the apple is in the normal mode, the sub-image PS related to the apple is displayed in the normal mode.

[0053] (Effect) As described above, the display device 10 according to the present embodiment includes a display unit 22 that displays an image, an imaging unit 28, an object recognition unit 42, a recognition count determination unit 44, and an output control unit 48. The imaging unit 28 captures an object within the visual field range of the user U. The object recognition unit 42 identifies the type of the object to be visually recognized within the visual field range of the user U based on the image captured by the imaging unit 28. The recognition count determination unit 44 determines the recognition count, which is the number of times the user U has visually recognized the same type of object to be visually recognized, based on the identification result of the object recognition unit 42. The output control unit 48 causes the display unit 22 to display an image so as to be superimposed on the main image PM within the visual field range of the user U that is visually recognized through the display unit 22. The output control unit 48 causes the display unit 22 to display a sub-image PS including information related to the object of that type so as to be superimposed on the main image PM including the object of that type based on the recognition count.

[0054] In a display device, the display mode may be uniform regardless of the user of the display device. On the other hand, the display device 10 according to the present embodiment counts the number of times the user U has visually recognized the same type of object as the number of visual recognition times. Then, when the user U visually recognizes an object of the same type as the object, the display device 10 displays a sub-image PS related to the object based on the counted number of visual recognition times. According to the display device 10 according to the present embodiment, it is possible to adjust the display mode of the sub-image PS according to the user U's past actions, and it is possible to appropriately adjust the display mode according to the user U.

[0055] The output control unit 48 emphasizes and displays the sub-image PS to be displayed when the number of visual recognition times is equal to or more than a predetermined number, rather than the sub-image PS to be displayed when the number of visual recognition times is less than the predetermined number. According to the display device 10 according to the present embodiment, in order to emphasize and display the sub-image PS for the object that the user U has seen many times, it is possible to appropriately adjust the display mode according to the user U. For example, when the user U is interested in bananas, the number of times of visually recognizing bananas increases. In such a case, by emphasizing and displaying the sub-image PS when seeing bananas, it is possible to appropriately convey information about bananas to the user U.

[0056] (Second Embodiment) Next, the second embodiment will be described. In the second embodiment, it is different from the first embodiment in that the display mode of the sub-image PS is set based on the number of visits, which is the number of times of visiting the same place. The parts common in configuration between the second embodiment and the first embodiment will be omitted from the description.

[0057] (Configuration of Display Device) FIG. 8 is a schematic block diagram of a display device according to the second embodiment. As shown in FIG. 8, the control unit 32A of the display device 10A according to the second embodiment includes a position information acquisition unit 50, a visit number specifying unit 52, a display mode setting unit 46A, and an output control unit 48A.

[0058] The location information acquisition unit 50 acquires the location information of the user U (display device 10A). The location information acquisition unit 50 causes the GNSS receiver 30 to detect the location information of the user U, and acquires the detection result of the location information of the user U from the GNSS receiver 30. The location information acquisition unit 50 causes the GNSS receiver 30 to detect the location information of the user U at regular intervals. Each time the GNSS receiver 30 detects the location information of the user U, the location information acquisition unit 50 acquires the location information of the user U detected by the GNSS receiver 30.

[0059] The visit count specifying unit 52 specifies the visit count of the user U based on the location information of the user U. The visit count is the number of times the user U has visited the same location. The processing of the visit count specifying unit 52 will be described later.

[0060] The display mode setting unit 46A determines the display mode of the sub-image PS based on the visit count. The processing of the display mode setting unit 46A will be described later.

[0061] The output control unit 48A causes the sub-image PS to be displayed based on the display mode of the sub-image PS determined by the display mode setting unit 46A. Since the display mode of the sub-image PS is set based on the visit count, it can be said that the output control unit 48A causes the sub-image PS to be displayed based on the visit count. The specific processing content of the output control unit 48A will be described later.

[0062] (Setting process of display mode) FIG. 9 is a flowchart for explaining the setting process of the display mode.

[0063] (Acquisition of location information) As shown in FIG. 9, the control unit 32 acquires the location information of the user U by the location information acquisition unit 50 (step S40).

[0064] (Specification of visit count) The control unit 32 specifies, based on the position information of the user U, the number of visits, which is the number of times the user U has visited the same location, by the visit frequency specifying unit 52 (step S42). The visit frequency specifying unit 52 specifies the visited location, which is the location visited by the user U, from the position information of the user U, counts the number of times the user U has visited for each visited location, and stores it in the storage unit 24 as the visit frequency. The location in the present embodiment refers to a region occupying a predetermined area in the earth coordinate system. For example, the visit frequency specifying unit 52 divides the earth coordinate system into a plurality of regions (locations), and determines the region (location) whose range includes the position of the user U as the visited location. Then, the visit frequency specifying unit 52 counts the number of times regarded as the visited location for each region (location) and sets it as the visit frequency. For example, when the total number of times the position of the user U detected at different timings is within the region AR1 is 5 times, the visit frequency of the region AR1 is set to 5 times.

[0065] More specifically, the visit frequency specifying unit 52 may regard the region as the visited location when the user U is continuously located within the same region for a predetermined time or more. That is, the visit frequency specifying unit 52 may determine that the region has been visited when the user U has stayed in the same region for a predetermined time or more. In this case, the visit frequency specifying unit 52 determines the region as the visited location when each of the positions indicated by the position information of the user U continuously detected in time series is within the same region based on the position information of the user U continuously detected for a predetermined time or more.

[0066] However, the visit frequency specifying unit 52 is not limited to counting the visit frequency by dividing the earth coordinate system into a plurality of regions in this way. For example, the visit frequency specifying unit 52 may compare the positions of the user U detected at different timings, and when those positions are within a predetermined distance range, determine that the same location has been visited and count the visit frequency.

[0067] The visit frequency specifying unit 52 may reset the visit frequency to zero when a predetermined period has elapsed. That is, the visit frequency specifying unit 52 may use the number of times the user U has visited the same location within a predetermined period as the view count. The predetermined period may be arbitrarily set, for example, it may be one month or one year.

[0068] (Setting of display mode) The display mode setting unit 46A sets the display mode of the sub-image PS based on the visit frequency. The display mode setting unit 46A extracts an object within a predetermined distance range from the location where the visit frequency is specified as a proximity object. For example, the display mode setting unit 46A reads map data from the storage unit 24 and extracts, based on the map data, an object within a predetermined distance range from the location where the visit frequency is specified as a proximity object. Then, the display mode setting unit 46A sets the display mode of the sub-image PS related to the object of the same type as the proximity object based on the visit frequency. For example, if the location where the visit frequency is specified is the area AR1, the display mode setting unit 46A reads out the facilities within a predetermined distance range from the area AR1 from the map data and uses the read facilities as the proximity objects. Then, the display mode setting unit 46A sets the display mode of the sub-image PS related to the facilities of the same type as the proximity objects based on the visit frequency of the area AR1.

[0069] More specifically, as shown in FIG. 9, the control unit 32 determines whether the number of visits is equal to or greater than a predetermined number by the display mode setting unit 46A (step S44). When the number of visits is equal to or greater than the predetermined number (step S44; Yes), the display mode setting unit 46A extracts proximity objects within a predetermined distance range from the location where the number of visits has reached the predetermined number or more. Then, the display mode setting unit 46A sets the display mode of the sub-image PS for objects of the same type as the proximity objects to the emphasized mode (step S46). On the other hand, when the number of visits is less than the predetermined number (step S44; No), that is, when the number of visits is less than the predetermined number, the process proceeds to step S48 without setting the display mode to the emphasized mode. More specifically, initially, the display mode setting unit 46A sets the display mode of the sub-image PS for all types of objects to the normal mode. Then, for locations where the number of visits is less than the predetermined number, the display mode setting unit 46A does not extract proximity objects or set the display mode. However, when there is a location where the number of visits reaches the predetermined number or more, the proximity objects at that location are extracted, and the display mode for objects of the same type as the proximity objects is switched to the emphasized mode.

[0070] Thereafter, when the process ends (step S48; Yes), this process ends. When the process does not end (step S48; No), the process returns to step S10 and repeats.

[0071] (Sub-image display process) The output control unit 48A causes the display unit 22 to display the sub-image PS related to the visual recognition object (the visual recognition object within the visual field range of the user U) extracted by the visual recognition unit 42 based on the display mode set by the display mode setting unit 46A, that is, based on the number of visits. The output control unit 48 reads the display mode of the sub-image PS related to the type of the visual recognition object extracted by the visual recognition unit 42, and displays the sub-image PS for the extracted visual recognition object in that display mode. This will be described more specifically below.

[0072] FIG. 10 is a flowchart for explaining the display process of sub-images. As shown in FIG. 10, the control unit 32 acquires, via the image acquisition unit 40, the image captured by the imaging unit 28 (step S10A). Based on the image acquired by the image acquisition unit 40, the visual recognition object identification unit 42 extracts the visual recognition object and identifies the type of the visual recognition object (step S12A). The control unit 32 determines, via the output control unit 48A, whether the display mode of the sub-image PS for the type of the extracted visual recognition object is the highlighting mode (step S30A). If the display mode for the type of the extracted visual recognition object is the highlighting mode (step S30A; Yes), the output control unit 48 causes the display unit 22 to display the sub-image PS for that visual recognition object in the highlighting mode (step S32A). The output control unit 48A causes the sub-image PS to be displayed in the highlighting mode so as to overlap the main image PM that includes that visual recognition object. On the other hand, if the display mode for the type of the extracted visual recognition object is not the highlighting mode (step S30A; No), that is, if it is the normal mode, the output control unit 48A causes the display unit 22 to display the sub-image PS for that visual recognition object in the normal mode (step S34A). The output control unit 48A causes the sub-image PS to be displayed in the normal mode so as to overlap the main image PM that includes that visual recognition object. That is, it can be said that the output control unit 48A highlights and displays the sub-image (the sub-image PS set to the highlighting mode) to be displayed when the number of visits is equal to or more than a predetermined number, rather than the sub-image PS (the sub-image PS set to the normal mode) to be displayed when the number of visits is less than the predetermined number.

[0073] In this way, when the display mode for the object viewed by the user U is the emphasis mode, the output control unit 48A displays the sub-image PS for that object in the emphasis mode. Therefore, when the user U visits again a place where the number of visits has reached a predetermined number or more and visually recognizes the immediate object itself at that place, the sub-image PS for that immediate object is displayed in the emphasis mode. Also, when the user U visits a place other than the place where the number of visits has reached a predetermined number or more and visually recognizes an object that is different from the immediate object at the place where the number of visits has reached a predetermined number or more but is of the same type, the sub-image PS for the object visually recognized is also displayed in the emphasis mode. As a specific example, consider the case where the number of visits to the area AR1 has reached a predetermined number or more and the immediate object in the area AR1 is a pharmacy. In this case, when the user U visits the area AR1 again and visually recognizes the same pharmacy which is the immediate object, or when the user U visually recognizes another pharmacy at a place other than the area AR1, the sub-image PS related to the pharmacy is displayed in the emphasis mode.

[0074] FIG. 11 is a diagram showing an example of displaying a sub-image in a normal mode and an enhanced mode. In the example of FIG. 11, when an object of the type of pharmacy enters the visual field range of the user U, a case where the characters "pharmacy" are displayed as the sub-image PS is shown. In the example of FIG. 11, the characters "pharmacy" in the enhanced mode are larger than the characters "pharmacy" in the normal mode, and the characters in the enhanced mode are displayed in an emphasized manner. However, FIG. 11 is just an example, and the sub-image PS may be any image as long as it shows information about the object. For example, the sub-image PS may be characters in any language that show information about the object, or may be an icon of any shape such as an exclamation mark or an arrow. Also, for example, the sub-image PS may be a sentence that explains the object. In this case, the enhanced mode may refer to a higher degree of detail in the explanation of the object than in the normal mode. Also, for example, as the sub-image PS, information about the same type of visually recognized object extracted previously may be displayed, or comparison information that is information comparing the object currently visually recognized by the user U with the previously extracted adjacent object may be displayed. In this case, for example, when the user U is currently visually recognizing a pharmacy, information about the pharmacy that the user U visited previously may be displayed as the sub-image PS, or information comparing the currently visually recognized pharmacy with the previously visited pharmacy may be displayed.

[0075] Also, in the example of FIG. 11, the sub-image PS in the enhanced mode is made larger than the sub-image PS in the normal mode, but it is not limited to this, and any display method that emphasizes the sub-image PS in the enhanced mode more than the sub-image PS in the normal mode may be used. For example, the display position of the sub-image PS in the enhanced mode within the main image PS may be made more prominent (e.g., the center, etc.) than the display position of the sub-image PS in the normal mode. Also, the brightness of the sub-image PS in the enhanced mode may be made higher than the brightness of the sub-image PS in the normal mode. Also, the display frequency of the sub-image PS in the enhanced mode may be made higher than the display frequency of the sub-image PS in the normal mode. Here, the display frequency may refer to the display time of the sub-image PS per unit time, or may refer to the speed at which the sub-image PS blinks.

[0076] Also, in the present embodiment, when the number of visits is less than a predetermined number, the sub-image PS is in the normal mode, and when the number of visits is equal to or more than the predetermined number, the sub-image PS is in the enhanced mode. However, the method for setting the display mode in the second embodiment is not limited to this, and the display mode of the sub-image PS may be set by any method based on the number of visits. For example, when the number of visits is less than the predetermined number, the display mode setting unit 46A may set the sub-image PS to a non-display mode in which the sub-image PS is not displayed, and when the number of visits becomes equal to or more than the predetermined number, the sub-image PS may be set to a mode in which the sub-image PS is displayed. Also, for example, the display mode setting unit 46A may set the display mode such that the degree of enhancement of the sub-image PS increases as the number of visits increases. For example, the display mode setting unit 46 may enlarge the sub-image PS as the number of visits increases.

[0077] (Effect) As described above, the display device 10A according to the second embodiment includes a display unit 22 that displays an image, an output control unit 48A, a position information acquisition unit 50, a visit number specifying unit 52, and an output control unit 48A. The position information acquisition unit 50 acquires the position information of the user U. The visit number specifying unit 52 specifies the number of visits, which is the number of times the user U has visited the same location, based on the position information of the user U. The output control unit 48A causes the display unit 22 to display an image so as to overlap the main image PM within the visual field range of the user U that is viewed through the display unit 22. The output control unit 48A causes the display unit 22 to display a sub-image PS including information on an object (proximity object) of the same type as an object within a predetermined distance range from the location visited by the user U, based on the number of visits.

[0078] In a display device, the display mode may be uniform regardless of the user of the display device. In contrast, the display device 10A according to the present embodiment counts the number of times the user U has visited the same location as the visit count. Then, when the user U visually recognizes an object of the same type as the proximity object near that location, the display device 10A displays a sub-image PS related to that object based on the counted visit count. According to the display device 10 according to the present embodiment, it is possible to adjust the display mode of the sub-image PS according to the user U's past actions, and it is possible to appropriately adjust the display mode according to the user U.

[0079] Further, the output control unit 48A emphasizes and displays the sub-image PS to be displayed when the visit count is equal to or more than a predetermined number of times, rather than the sub-image PS to be displayed when the visit count is less than the predetermined number of times. According to the display device 10A according to the present embodiment, the sub-image PS for an object of the same type as the proximity object near a location with a high visit count is emphasized and displayed. Therefore, according to the display device 10A, it is possible to appropriately adjust the display mode according to the user U's actions.

[0080] Further, even when an object of the same type as the proximity object enters the user U's visual field range at a location different from the location where the visit count has become equal to or more than a predetermined number of times, the output control unit 48A emphasizes and displays the sub-image PS related to that object. Thereby, for example, even at a location where the user U visits for the first time, it becomes possible to make it easier for the user U to recognize an object of interest, and it becomes possible to appropriately adjust the display mode according to the user U's actions.

[0081] Further, the visit count specifying unit 52 determines that the user U has visited a location when the user U has been located at the same location for a predetermined time or more. Thereby, it becomes possible to adjust the display mode according to the user U's actions.

[0082] Note that the second embodiment and the first embodiment may be combined. That is, while displaying the sub-image PS based on the number of visual recognitions using the process described in the first embodiment, the sub-image PS may be displayed based on the number of visits using the process described in the second embodiment.

[0083] (Third Embodiment) Next, the third embodiment will be described. In the third embodiment, it is different from the first and second embodiments in that the display mode of the sub-image PS is set based on the number of times the user U has visually recognized the same type of object at the same location. The parts common to the first and second embodiments in the third embodiment will be omitted from the description.

[0084] (Configuration of the Display Device) FIG. 12 is a schematic block diagram of a display device according to the third embodiment. As shown in FIG. 12, the control unit 32B of the display device 10B according to the third embodiment includes an image acquisition unit 40, a visual recognition object identification unit 42, a visual recognition count identification unit 44, a position information acquisition unit 50, a visit count identification unit 2, a duplication count identification unit 60, a display mode setting unit 46B, and an output control unit 48B.

[0085] The duplication count identification unit 60 identifies the duplication count, which is the number of times the user U has visually recognized the same type of object at the same location. The processing details of the duplication count identification unit 60 will be described later.

[0086] (Display Mode Setting Process) The control unit 32B according to the third embodiment sets the display mode of the sub-image PS based on the number of visual recognitions in the same manner as in the first embodiment, and sets the display mode of the sub-image PS based on the number of visits in the same manner as in the second embodiment. However, the control unit 32B is different from the first and second embodiments in that it also sets the display mode of the sub-image PS based on the duplication count. Hereinafter, the display mode setting process based on the duplication count will be described.

[0087] FIG. 13 is a flowchart for explaining the setting process of the display mode. As shown in FIG. 13, the duplication number specifying unit 60 of the control unit 32B specifies the duplication number based on the type of the visual object specified by the visual object specifying unit 42 and the position information of the user U acquired by the position information acquisition unit 50 (step S50). Specifically, the duplication number specifying unit 60 uses the information on the type of the visual object specified by the visual object specifying unit 42 and the position information of the user U at the timing when the image used to specify the type of the visual object was captured as one data set, and acquires the data set for each timing when the image was captured. Then, for example, the duplication number specifying unit 60 classifies those data sets for each combination of the type of the visual object and the location where the user U is located, and counts the number of data sets that are the same type of object and where the user U is located as the duplication number. The duplication number specifying unit 60 counts the duplication number for each combination of the type of the visual object and the location where the user U is located. That is, for example, when the pharmacy in the area AR1 is visually recognized 5 times, the duplication number for the pharmacy in the area AR1 is 5 times.

[0088] The duplication number specifying unit 60 may reset the duplication number to zero when a predetermined period has elapsed. That is, the duplication number specifying unit 60 may use the number of times the user U has visually recognized the same object at the same location within a predetermined period as the duplication number. The predetermined period may be arbitrarily set, and may be, for example, one month or one year.

[0089] The display mode setting unit 46B sets the display mode of the sub-image PS based on the number of repetitions. The display mode setting unit 46B sets the display mode for the object of visual recognition visually recognized at the location where the number of repetitions is counted, based on the number of repetitions. Specifically, as shown in FIG. 13, the display mode setting unit 46B determines whether the number of repetitions is equal to or greater than a predetermined number (step S52). If it is equal to or greater than the predetermined number (step S52; Yes), the display mode of the sub-image PS for the object of visual recognition at the location where the number of repetitions has reached the predetermined number or more is set to the non-display mode (step S54). For example, if the number of repetitions, which is the number of times a pharmacy is visually recognized in the area AR1, is equal to or greater than the predetermined number, the display mode for the pharmacy in the area AR1 is set to the non-display mode.

[0090] On the other hand, if the number of repetitions is less than the predetermined number (step S52; No), that is, if the number of repetitions is less than the predetermined number, the process proceeds to step S56 without setting the display mode to the non-display mode. Thereafter, if the process ends (step S56; Yes), this process ends. If the process does not end (step S56; No), the process returns to step S50 and the process is repeated.

[0091] Note that the display mode setting unit 46B may stop setting the non-display mode and switch to the mode for displaying the sub-image PS after a predetermined period has elapsed since the non-display mode was set.

[0092] (Display process of sub-image) FIG. 14 is a diagram showing an example in the non-display mode. The output control unit 48B causes the display unit 22 to display a sub-image PS related to a visual object (a visual object within the visual field range of the user U) extracted by the visual object identification unit 42 based on the display mode set by the display mode setting unit 46B. For example, similar to the first embodiment and the second embodiment, when the display mode of the sub-image PS for the type of the extracted visual object is the emphasis mode, the output control unit 48B displays the sub-image PS in the emphasis mode, and when the display mode of the sub-image PS for the type of the extracted visual object is the normal mode, the output control unit 48B displays the sub-image PS in the normal mode. On the other hand, when the display mode of the sub-image PS for the extracted visual object is the non-display mode, the output control unit 48B does not display the sub-image PS for that visual object. In this case, the output control unit 48B acquires the position information of the user U when the extracted visual object is visually recognized, and when the position of the user U when the visual object is visually recognized is the same as the place where the number of overlaps is equal to or more than a predetermined number, and the type of the visual object matches the type of the adjacent object at the place where the number of overlaps of the type of the visual object is equal to or more than a predetermined number, it can be said that the output control unit 48B does not display the sub-image PS for the visual object. FIG. 14 shows an example in the non-display mode. As shown in FIG. 14, in the non-display mode, the sub-image PS is not displayed.

[0093] (Effect) As described above, the display device 10B according to the third embodiment includes a duplication count specifying unit 60. The duplication count specifying unit 60 specifies the duplication count, which is the number of times the user has visually recognized the same type of object at the same location, based on the specification result of the visual object specifying unit 42 and the position information of the user U. When the duplication count is equal to or greater than a predetermined count, the output control unit 48B stops displaying the sub-image PS for that object. Here, generally, the user U lives a life in which the action pattern is cycled once a week. Therefore, it is also assumed that the user U visually recognizes the same object or visits the same location without interest. In such a case, if the sub-image PS of that object is displayed, it may be troublesome for the user U. On the other hand, the display device 10B according to the third embodiment stops displaying the sub-image PS of that object when the duplication count is equal to or greater than a predetermined count. Therefore, according to the display device 10B according to the third embodiment, it is possible to suppress the user U from feeling troubled and appropriately adjust the display mode according to the action of the user U.

[0094] In the above description, the sub-image PS for the visual object at the location where the duplication count is equal to or greater than the predetermined count is in the non-display mode, and the sub-image PS for the same type of object at a location different from that location is not in the non-display mode. However, this is not the only case, and the sub-image PS for the same type of object at a different location may also be in the non-display mode. That is, the display device 10B may set the sub-image PS for the object of the same type as the neighboring object at the location where the duplication count is equal to or greater than the predetermined count to the non-display mode. For example, when the duplication count of the pharmacy in the region AR1 is equal to or greater than the predetermined count, the display mode of the sub-image PS for the pharmacy at a location other than the region AR1 may also be set to the non-display mode.

[0095] Also, in the above description, the display device 10B sets the display mode based on both the visual recognition count and the visit count, and also sets the display mode based on the duplication count, but it is not limited thereto. For example, the display device 10B may set the display mode based only on the duplication count without performing the display mode setting process based on the visual recognition count and the visit count.

[0096] (Fourth Embodiment) Next, the fourth embodiment will be described. In the fourth embodiment, it is different from the first embodiment in that the display mode of the sub-image PS is set based on the number of times the user U has looked away from the object. The parts that are common in configuration between the fourth embodiment and the first embodiment will not be described. Note that the fourth embodiment can also be combined with the second embodiment and the third embodiment.

[0097] (Configuration of Display Device) FIG. 15 is a schematic block diagram of a display device according to the fourth embodiment. As shown in FIG. 15, the control unit 32C of the display device 10C according to the fourth embodiment includes an image acquisition unit 40, a visual object identification unit 42C, a line-of-sight switching frequency identification unit 70, a display mode setting unit 46C, and an output control unit 48C.

[0098] The visual object identification unit 42C identifies the type of object that the user U has looked away from within a predetermined time based on the image captured by the imaging unit 28. The processing of the visual object identification unit 42C will be described later.

[0099] The line-of-sight switching frequency identification unit 70 identifies the line-of-sight switching frequency, which is the number of times the user U has looked away from the same type of object, based on the identification result of the visual object identification unit 42C. The processing of the line-of-sight switching frequency identification unit 70 will be described later.

[0100] (Display Mode Setting Process) The control unit 32C according to the fourth embodiment sets the display mode of the sub-image PS based on the line-of-sight switching frequency. Note that in the fourth embodiment as well, the display mode may be set based on the number of viewings in the same manner as in the first embodiment, or the display mode may be set based on the number of visits in the same manner as in the second embodiment, or the display mode may be set based on the number of duplicates in the same manner as in the third embodiment.

[0101] FIG. 16 is a flowchart for explaining the setting process of the display mode. As shown in FIG. 16, the control unit 32C acquires, through the image acquisition unit 40, the image captured by the imaging unit 28 (step S60), and the visual object identification unit 42C identifies the type of the object to be gaze-switched based on the image acquired by the image acquisition unit 40 (step S62). The object to be gaze-switched refers to an object that the user U has looked away from within a predetermined time.

[0102] The visual object identification unit 42C extracts the object to be gaze-switched from the image data of the images continuously captured in time series and identifies the type of the object to be gaze-switched. Specifically, the visual object identification unit 42C extracts, as the object to be gaze-switched, an object that has gone out of the visual field range within a predetermined time after once entering the visual field range. That is, the visual object identification unit 42C extracts, as the object to be gaze-switched, an object that appears in the image and no longer appears in the images captured until a predetermined time has elapsed from that image. However, the visual object identification unit 42C is not limited to extracting, as the object to be gaze-switched, an object that has gone out of the visual field range within a predetermined time after once entering the visual field range. For example, the visual object identification unit 42C may extract, as the object to be gaze-switched, an object that has stopped overlapping with the user U's gaze within a predetermined time after once overlapping with the gaze. In this case, for example, the visual object identification unit 42C extracts, as the object to be gaze-switched, an object that appears in a predetermined area (for example, the central area) of the image and no longer appears in the predetermined area (for example, the central area) in the images captured until a predetermined time has elapsed from that image. Note that the method of identifying the type of the object to be gaze-switched is arbitrary, and for example, it may be identified using an AI model as in the first embodiment.

[0103] The predetermined time here may be arbitrarily set, but for example, it is preferably 200 milliseconds or more and 900 milliseconds or less. In terms of neuroscience, it is said that the reaction time when a known type of stimulus is presented and a determined type of reaction is made (simple detection task) is about 150 milliseconds to 300 milliseconds. However, in the present embodiment, in order to appropriately detect the reaction of looking away after seeing an object and then determining that one does not want to see the object, in addition to the simple reaction, it is preferable to also take into account the discrimination reaction time for judgment. That is, in the present embodiment, by setting the predetermined time to 200 milliseconds or more and 900 milliseconds or less, it is possible to appropriately take into account the round-trip time (time of simple reaction) for the viewpoint to move between two objects and the discrimination reaction time for judgment, which is preferable.

[0104] Based on the identification result of the visual recognition object identification unit 42C, the gaze switching count identification unit 70 identifies the gaze switching count indicating the number of times the user U has looked away from the same type of object (step S64). The gaze switching count identification unit 70 counts the number of times extracted as the gaze switching object for each type of gaze switching object, sets it as the gaze switching count, and stores it in the storage unit 24. That is, the gaze switching count identification unit 70 identifies the number of times the same type of gaze switching object has been extracted by the visual recognition object identification unit 42C as the gaze switching count. For example, when the gaze switching object of the type of spider is extracted 5 times at different timings, the gaze switching count identification unit 70 identifies the gaze switching count of the spider as 5 times.

[0105] When a predetermined period has elapsed, the gaze switching count identification unit 70 may reset the gaze switching count and return it to zero. That is, the gaze switching count identification unit 70 may set the number of times the same type of gaze switching object has been extracted by the visual recognition object identification unit 42C within the predetermined period as the gaze switching count. The predetermined period may be arbitrarily set, for example, it may be one month or one year.

[0106] The display mode setting unit 46C sets the display mode of the sub-image PS based on the number of times of line-of-sight switching. The display mode setting unit 46C sets the display mode for an object of the same type as the object of line-of-sight switching for which the number of times of line-of-sight switching has been counted, based on the number of times of line-of-sight switching. Specifically, as shown in FIG. 16, the display mode setting unit 46C determines whether the number of times of line-of-sight switching is equal to or greater than a predetermined number (step S66). If it is equal to or greater than the predetermined number (step S66; Yes), the display mode of the sub-image PS for an object of the same type as the object of line-of-sight switching is set to the non-display mode (step S68). For example, when the number of times of line-of-sight switching for an object of the type "spider" is equal to or greater than the predetermined number, the display mode of the sub-image PS for the spider is set to the non-display mode.

[0107] On the other hand, if the number of times of line-of-sight switching is less than the predetermined number (step S66; No), that is, if the number of times of line-of-sight switching is less than the predetermined number, the process proceeds to step S70 without setting the display mode to the non-display mode. After that, if the process ends (step S70; Yes), this process ends. If the process does not end (step S70; No), the process returns to step S60 and the process is repeated.

[0108] Note that the display mode setting unit 46C may stop setting the non-display mode and switch to the mode for displaying the sub-image PS after a predetermined period has elapsed since the non-display mode was set.

[0109] (Display process of sub-image) Based on the display mode set by the display mode setting unit 46C, the output control unit 48C causes the display unit 22 to display a sub-image PS related to the visual object (visual object within the visual field range of the user U) extracted by the visual object identification unit 42C. For example, similar to the first embodiment, when the display mode of the sub-image PS for the type of the extracted visual object is the emphasis mode, the output control unit 48C causes the sub-image PS to be displayed in the emphasis mode, and when the display mode of the sub-image PS for the type of the extracted visual object is the normal mode, the output control unit 48C causes the sub-image PS to be displayed in the normal mode. On the other hand, when the display mode of the sub-image PS for the extracted visual object is the non-display mode, the output control unit 48C does not display the sub-image PS for that visual object.

[0110] (Effect) As described above, the display device 10 according to the present embodiment includes a display unit 22 that displays an image, an imaging unit 28, a visual object identification unit 42C, a line-of-sight switching frequency identification unit 70, and an output control unit 48C. The imaging unit 28 images an object within the visual field range of the user U. The visual object identification unit 42C identifies the type of the object (line-of-sight switching object) that the user U has looked away from within a predetermined time based on the image captured by the imaging unit 28. The line-of-sight switching frequency identification unit 70 identifies the line-of-sight switching frequency, which is the number of times the user U has looked away from the same type of object, based on the identification result of the visual object identification unit 42C. The output control unit 48C causes the display unit 22 to display an image so as to be superimposed on the main image PM within the visual field range of the user U that is viewed through the display unit 22. When the line-of-sight switching frequency is equal to or greater than a predetermined number of times, the output control unit 48C stops displaying the sub-image PS including information related to the object of that type.

[0111] Here, there may be objects that user U does not want to see. If a sub-image PS of an object that user U does not want to see is displayed, user U may become aware of the object that user U does not want to see, which may inhibit the mood of user U. On the other hand, the display device 10C according to the present embodiment stops displaying the sub-image PS of an object for which the number of times user U has averted their eyes is large. Therefore, according to the display device 10B according to the third embodiment, it is possible to suppress inhibiting the mood of user U and appropriately adjust the display mode according to the behavior of user U.

[0112] In addition, when the number of times of line-of-sight switching is less than a predetermined number of times, the output control unit 48C causes the display unit 22 to display the sub-image PS so as to overlap with the main image PM including the object of that type. The display device 10C according to the present embodiment can appropriately adjust the display mode according to the behavior of user U by displaying the sub-image PS for an object for which the number of times of averting the eyes is small.

[0113] Also, the predetermined time is preferably 200 milliseconds or more and 900 milliseconds or less. By setting the predetermined time within this range, it is possible to more appropriately adjust the display mode according to the behavior of user U by appropriately taking into account the simple reaction time and the discrimination reaction time.

[0114] As described above, the embodiments of the present invention have been described, but the embodiments are not limited by the contents of these embodiments. In addition, the above-described components include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those within a so-called equivalent range. Furthermore, the above-described components can be combined as appropriate, and the configurations of each embodiment can also be combined. Furthermore, various omissions, substitutions, or changes of the components can be made without departing from the gist of the above-described embodiments.

Explanation of Reference Numerals

[0115] 10 Display device 22 Display unit 28 Imaging unit 30 GNSS receiver 40 Image acquisition unit 42 Visual object identification unit 44 Visual recognition count determination unit 46 Display mode setting unit 48 Output control unit 50 Position information acquisition unit 52 Visit count determination unit 60 Duplication count determination unit 70 Gaze switching count determination unit PM Main image PS Sub-image

Claims

1. A display unit for displaying an image; An imaging unit for imaging an object within the user's field of view; An output control unit that causes the display unit to display the user's field of view as a main image by transmitting external light from the display unit or by the imaging unit imaging; A visible object identification unit that identifies the type of object within the user's field of view by obtaining information indicating the type of object from the image captured by the imaging unit; A position information acquisition unit for acquiring the position information of the user; A visit count identification unit that identifies the visit count, which is the number of times the user has visited the same location, based on the position information of the user; A display mode setting unit that sets the first display mode as the display mode for the information indicating the type of object acquired from the visible object identification unit, and sets the second display mode as the display mode if the visit count is equal to or greater than a first predetermined number of times; The output control unit superimposes, as a sub-image, an image consisting of information about the object including at least one of the type of object, an icon indicating the object, a text explaining the object, and information about the same type of object extracted previously, on the main image in the display mode and causes the display unit to display it; The output control unit emphasizes and displays the sub-image to be displayed when the visit count is equal to or greater than the first predetermined number of times more than the sub-image to be displayed when the visit count is less than the first predetermined number of times; If the information indicating the type of object and the information indicating the type of object acquired at a location where the visit count is equal to or greater than the first predetermined number of times are the same at a location different from the location where the visit count has become equal to or greater than the first predetermined number of times, the output control unit causes the display unit to display the sub-image in the second display mode; The visit count identification unit determines that the user has visited a location when the user has been located at the same location for a first predetermined time or more; The display mode setting unit is configured to set at least one of the following: not display the sub-image in the first display mode and display the sub-image in the second display mode; make the sub-image in the second display mode larger than the sub-image in the first display mode; make the brightness of the sub-image in the second display mode higher than the brightness of the sub-image in the first display mode; or make the display frequency of the sub-image in the second display mode higher than the display frequency of the sub-image in the first display mode. Display device. **Claim 2** The visual object specifying unit extracts, as a visual object, an object that appears continuously for a second predetermined time or longer among the objects appearing in the image captured by the imaging unit, and specifies the type of the visual object by acquiring information indicating the type of the visual object. The apparatus further includes a duplication count specifying unit that specifies a duplication count, which is the number of times the user has viewed the same type of object at the same location, based on the type of the visual object specified by the visual object specifying unit and the position information of the user. When the duplication count is equal to or greater than a second predetermined count, the output control unit stops displaying the sub-image for the object. The display device according to claim 1. **Claim 3** A display method for causing a display unit to display an image, the method comprising: imaging an object within a visual field of a user; causing the display unit to display, as a main image, the visual field of the user by transmitting external light through the display unit or by the imaging unit performing imaging; specifying the type of the object within the visual field of the user by acquiring information indicating the type of the object from the captured image; acquiring the position information of the user; specifying an access count, which is the number of times the user has visited the same location, based on the position information of the user; setting a first display mode as the display mode for the information indicating the type of the acquired object, and setting a second display mode as the display mode if the access count is equal to or greater than a first predetermined count; and causing the display unit to display, in the display mode, an image including at least one of the type of the object, an icon indicating the object, a text explaining the object, and information about the same type of object extracted previously, as a sub-image, by superimposing the sub-image on the main image. When the number of visits is less than the first predetermined number, the sub-image to be displayed is more than When the number of visits is equal to or more than the first predetermined number, emphasizing and displaying the sub-image to be displayed; If, at a location different from the location where the number of visits has reached the first predetermined number or more, the information indicating the type of the object is the same as the information indicating the type of the object obtained at the location where the number of visits has reached the first predetermined number or more, displaying the sub-image on the display unit in the second display mode; When the user is located at the same place for a predetermined time or more, determining that the user has visited that place; Setting to include at least one of: not displaying the sub-image in the first display mode but displaying it in the second display mode, making the sub-image in the second display mode larger than the sub-image in the first display mode, making the brightness of the sub-image in the second display mode higher than the brightness of the sub-image in the first display mode, or making the display frequency of the sub-image in the second display mode higher than the display frequency of the sub-image in the first display mode; including Display method.

4. A program for causing a computer to execute a display method for displaying an image on a display unit, imaging an object within the user's field of view; either transmitting external light from the display unit or causing the imaging unit to image, and displaying the user's field of view as a main image on the display unit; identifying the type of the object within the user's field of view by obtaining information indicating the type of the object from the captured image; obtaining the user's position information; identifying the number of visits, which is the number of times the user has visited the same place, based on the user's position information; For the obtained information indicating the type of the object, setting the first display mode as the display mode, and setting the second display mode as the display mode if the number of visits is equal to or more than the first predetermined number; A step of causing an image including at least one of the type of the object, an icon indicating the object, a sentence explaining the object, and information regarding the same type of object extracted previously, to be superimposed on the main image in the display mode as a sub-image and displayed on the display unit; When the number of visits is less than the first predetermined number, compared with the sub-image to be displayed; A step of prominently displaying the sub-image to be displayed when the number of visits is equal to or more than the first predetermined number; If the information indicating the type of the object and the information indicating the type of the object obtained at the location where the number of visits is equal to or more than the first predetermined number are the same at a location different from the location where the number of visits has become equal to or more than the first predetermined number, a step of causing the sub-image to be displayed on the display unit in the second display mode; A step of determining that the user has visited a location when the user has been located at the same location for a predetermined time or more; A step of setting to include at least one of: not displaying the sub-image in the first display mode and displaying the sub-image in the second display mode, making the sub-image larger in the second display mode than the sub-image in the first display mode, making the brightness of the sub-image higher in the second display mode than the brightness in the first display mode, or making the display frequency of the sub-image higher in the second display mode than the display frequency in the first display mode; To be executed by a computer; Program.

Images