Display device, display method, and program

Abstract

To appropriately adjust display modes according to users.SOLUTION: A display device 10 includes: a display unit 22 that displays an image; an output control unit 48 that causes the display unit 22 to display the image; an imaging unit 28 that takes an image of an object within a user's field of view; a visual recognition object identifying unit 42 that identifies a type of the object within the user's field of view based on the image taken by the imaging unit 28; and a visual recognition count identifying unit 44 that identifies a visual recognition count, which is the number of times the user has visually recognized objects of the same type, based on an identification result of the visual recognition object identifying unit 42. The output control unit 48 causes the display unit 22 to display a sub-image including information about objects of that type based on the visual recognition count.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a display device, a display method, and a program. [Background technology]

[0002] Recently, information devices have been evolving significantly with the advancement of high-speed CPUs, high-definition screen display technology, small and lightweight battery technology, the spread of wireless network environments, and broadband. As such information devices, not only smartphones, which are a representative example, but also so-called wearable devices worn by users have become widespread. Wearable devices include HMDs (Head Mount Displays) that provide visual information and are used for AR (Augumented Reality) and VR (Virtual Reality). For example, Patent Document 1 discloses a technology for correcting the color tone of an object based on the color tone of a marker placed in real space, as a technology for taking into account the color tone of an object when superimposing the object on an image in real space using AR technology. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-170316 A Summary of the Invention [Problem to be solved by the invention]

[0004] Here, in a display device that displays an image, such as an HMD, the display mode is uniform regardless of the user of the display device. Therefore, it is required to appropriately adjust the display mode according to the user of the display device.

[0005] In view of the above-mentioned problems, an object of the present invention is to provide a display device, a display method, and a program that are capable of appropriately adjusting a display mode to suit a user. [Means for solving the problem]

[0006] A display device according to one aspect of the present invention includes a display unit that displays an image, an output control unit that causes the display unit to display the image, an imaging unit that captures an image of an object within a user's field of view, a viewed object identification unit that identifies the type of object within the user's field of view based on the image captured by the imaging unit, and a viewing count identification unit that identifies the viewing count, which is the number of times the user has viewed the same type of object, based on the identification result of the viewed object identification unit, and the output control unit causes the display unit to display a sub-image including information about the type of object based on the viewing count.

[0007] A display method according to one aspect of the present invention is a display method for displaying an image on a display unit, and includes the steps of: acquiring an image of an object within a user's field of view; identifying the type of object within the user's field of view based on the image; identifying a number of viewings, which is the number of times the user has viewed the same type of object, based on the result of identifying the type of object within the user's field of view; and displaying a sub-image including information about that type of object on the display unit based on the number of viewings.

[0008] A program according to one aspect of the present invention is a program that causes a computer to execute a display method for displaying an image on a display unit, and causes the computer to execute the following steps: acquiring an image of an object within a user's field of view; identifying the type of object within the user's field of view based on the image; identifying a number of viewings, which is the number of times the user has viewed the same type of object, based on the result of identifying the type of object within the user's field of view; and displaying a sub-image including information about that type of object on the display unit based on the number of viewings. Effect of the Invention

[0009] According to the present invention, the display mode can be appropriately adjusted to suit the user. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of a display device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of an image displayed by the display device. [Diagram 3] FIG. 3 is a schematic block diagram of the display device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating the display mode setting process. [Diagram 5] FIG. 5 is a diagram showing an example in which sub-images are displayed in the normal mode and the emphasis mode. [Figure 6] FIG. 6 is a diagram showing an example in which sub-images are displayed in the normal mode and the emphasis mode. [Figure 7] FIG. 7 is a flowchart illustrating the sub-image display process. [Figure 8] FIG. 8 is a schematic block diagram of a display device according to the second embodiment. [Figure 9] FIG. 9 is a flowchart illustrating the display mode setting process. [Figure 10] FIG. 10 is a flowchart illustrating the sub-image display process. [Figure 11] FIG. 11 is a diagram showing an example in which sub-images are displayed in the normal mode and the emphasis mode. [Figure 12] FIG. 12 is a schematic block diagram of a display device according to the third embodiment. [Figure 13] FIG. 13 is a flowchart illustrating the display mode setting process. [Figure 14] FIG. 14 is a diagram showing an example of the non-display mode. [Figure 15] FIG. 15 is a schematic block diagram of a display device according to the fourth embodiment. [Figure 16] FIG. 16 is a flowchart illustrating the display mode setting process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiment described below.

[0012] (First embodiment) FIG. 1 is a schematic diagram of a display device according to a first embodiment. The display device 10 according to the first embodiment is a display device that displays an image. As shown in FIG. 1, the display device 10 is a so-called wearable device that is worn on the body of a user U. In the example of this embodiment, the display device 10 includes a device 10a worn on the eye of the user U, a device 10b worn on the ear of the user U, and a device 10c worn on the arm of the user. The device 10a worn on the eye of the user U includes a display unit 22 described later that outputs a visual stimulus to the user U (displays an image). The device 10b worn on the ear of the user U includes an audio output unit that outputs an auditory stimulus (audio) to the user U, and the device 10c worn on the arm of the user U includes a tactile stimulus output unit that outputs a tactile stimulus to the user U. However, the configuration of FIG. 1 is an example, and the number of devices and the positions at which they are worn on the user U may be arbitrary. For example, the display device 10 may not include the device 10b or the device 10c. Also, for example, the display device 10 is not limited to a wearable device. For example, the display device 10 may be a device carried by the user U, such as a smartphone or a tablet terminal.

[0013] (Main statue) FIG. 2 is a diagram showing an example of an image displayed by the display device. As shown in FIG. 2, the display device 10 provides the user U with a main image PM through the display unit 22. As a result, the user U wearing the display device 10 can view the main image PM. In this embodiment, the main image PM is an image of a landscape that the user U would view if the user U were not wearing the display device 10, and can also be said to be an image of an actual object that falls within the field of view of the user U. The field of view refers to a range that can be seen without moving the eyeballs, centered on the line of sight of the user U. In this embodiment, the display device 10 provides the user U with the main image PM by, for example, transmitting external light (ambient visible light) from the display unit 22. That is, in this embodiment, it can be said that the user U directly views an image of an actual landscape through the display unit 22. However, the display device 10 is not limited to allowing the user U to directly view an image of an actual landscape, and may provide the user U with the main image PM through the display unit 22 by displaying an image of the main image PM on the display unit 22. In this case, the user U visually recognizes the image of the scenery displayed on the display unit 22 as the main image PM. In this case, the display device 10 displays an image captured by the imaging unit 28, which will be described later, within the visual field of the user U as the main image PM on the display unit 22. Note that, although a road and a building are included as the main image PM in FIG. 2, this is merely an example.

[0014] (Sub image) As shown in Fig. 2, the display device 10 displays the sub-image PS on the display unit 22 so as to be superimposed on the main image PM provided through the display unit 22. This allows the user U to view an image in which the sub-image PS is superimposed on the main image PM. The sub-image PS is an image superimposed on the main image PM, and can be said to be an image other than an actual scene that falls within the visual field of the user U. In other words, the display device 10 can be said to provide the user U with AR (Augumented Reality) by superimposing the sub-image PS on the main image PM, which is an actual scene.

[0015] In this embodiment, the sub-image PS is an image that includes information about an object included in the main image PM. In other words, the sub-image PS can be said to be an image that includes information about an actual object that is within the visual field of the user U (information about an object currently being viewed by the user U). The sub-image PS is set for each object. When an object in the main image PM is included (when the user U is viewing the object), the sub-image PS for that object is displayed. In the example of FIG. 2, the object that the user U is viewing is a building, and the sub-image PS, which is information about the building, is displayed. In FIG. 2, the sub-image PS is the letters BU, but this is just an example.

[0016] In this way, the display device 10 provides the main image PM and the sub-image PS, but may also display a content image having a different content from the main image PM and the sub-image PS on the display unit 22. The content image may be an image of any content, such as a movie or a television program. The display device 10 may also display the sub-image PS without providing the main image PM.

[0017] (Display Device Configuration) Fig. 3 is a schematic block diagram of the display device according to the first embodiment. As shown in Fig. 3, the display device 10 includes an input unit 20, a display unit 22, a storage unit 24, a communication unit 26, an imaging unit 28, a GNSS receiver 30, and a control unit 32.

[0018] The input unit 20 is a device that accepts user operations and may be, for example, a touch panel. The display unit 22 is a display that displays images. In this embodiment, the display unit 22 is a so-called HMD (Head Mount Display). In addition to the display unit 22, an audio output unit (speaker) that outputs audio and a tactile stimulus output unit that outputs a tactile stimulus to the user U may be provided as output units. The tactile stimulus output unit outputs a tactile stimulus to the user by physically operating, such as by vibration, but the type of tactile stimulus is not limited to vibration and may be any type.

[0019] The storage unit 24 is a memory that stores various information such as the contents of calculations and programs of the control unit 32, and includes at least one of a main storage device such as a RAM (Random Access Memory), a ROM (Read Only Memory), and an external storage device such as a HDD (Hard Disk Drive). The program for the control unit 32 saved in the storage unit 24 may be stored in a recording medium readable by the display device 10.

[0020] The communication unit 26 is a module that communicates with an external device, etc., and may include, for example, an antenna, etc. In this embodiment, the communication method used by the communication unit 26 is wireless communication, but the communication method may be any other method.

[0021] The imaging unit 28 is a camera that captures an object within the field of view of the user U. The imaging unit 28 is provided at a position where the imaging range overlaps with the field of view of the user U. In the example of FIG. 1, the imaging unit 28 is provided in the device 10a and is attached so that the imaging direction is the direction in which the face of the user U is facing. This allows the imaging unit 28 to capture an object within the field of view of the user U. The imaging unit 28 may be a video camera that captures images at a predetermined frame rate. The number of imaging units 28 is arbitrary, and may be single or multiple. When there are multiple imaging units 28, information on the direction in which the camera 20A is facing may also be acquired.

[0022] The GNSS receiver 30 is a device that detects position information of the display device 10. The position information here is Earth coordinates. In this embodiment, the display device 10 is worn by the user U, so the user U and the display device 10 are in the same position. Therefore, it can be said that the GNSS receiver 30 detects the position information of the user U. In this embodiment, the GNSS receiver 30 is a so-called GNSS (Global Navigation Satellite System) module, and receives radio waves from a satellite to detect the position information of the display device 10 (user U). Note that in the first embodiment, the GNSS receiver 30 is not an essential component.

[0023] The control unit 32 is a calculation device, that is, a CPU (Central Processing Unit). The control unit 32 includes an image acquisition unit 40, a visual object identification unit 42, a visual count identification unit 44, a display mode setting unit 46, and an output control unit 48. The control unit 32 realizes the image acquisition unit 40, the visual object identification unit 42, the visual count identification unit 44, the display mode setting unit 46, and the output control unit 48 by reading and executing a program (software) from the storage unit 24, and executes the processes. The control unit 32 may execute these processes by one CPU, or may include multiple CPUs and execute the processes by the multiple CPUs. Also, at least a part of the image acquisition unit 40, the visual object identification unit 42, the visual count identification unit 44, the display mode setting unit 46, and the output control unit 48 may be realized by hardware.

[0024] The image acquisition section 40 controls the imaging section 28 to cause the imaging section 28 to capture an image, and acquires the image (image data) captured by the imaging section 28. That is, the image acquisition section 40 acquires an image (image data) captured by the imaging section 28, with the visual field range of the user U as the imaging range. The image acquisition section 40 causes the imaging section 28 to capture an image at predetermined intervals. The image acquisition section 40 acquires the image data captured by the imaging section 28 each time the imaging section 28 captures an image.

[0025] The viewed object specifying unit 42 specifies the type of object within the visual field of the user U based on the image captured by the imaging unit 28. The processing of the viewed object specifying unit 42 will be described later.

[0026] The viewing count specifying unit 44 specifies the viewing count, which is the number of times the user U viewed the same type of object, based on the identification result of the viewed object specifying unit 42, i.e., based on the type of object within the visual field of the user U. The processing of the viewing count specifying unit 44 will be described later.

[0027] The display mode setting unit 46 determines the display mode of the sub-image PS based on the viewing count. The display mode of the sub-image PS refers to the way in which the sub-image PS is displayed, and can also be called the display mode. The processing of the display mode setting unit 46 will be described later.

[0028] The output control unit 48 controls the display unit 22 to display an image on the display unit 22. The output control unit 48 causes the display unit 22 to display the sub-image PS so as to be superimposed on the main image PM. The output control unit 48 also causes the sub-image PS to be displayed based on the display mode of the sub-image PS determined by the display mode setting unit 46. Since the display mode of the sub-image PS is set based on the number of times it is viewed, it can be said that the output control unit 48 displays the sub-image PS based on the number of times it is viewed. Specific processing contents of the output control unit 48 will be described later.

[0029] The display device 10 has the above-described configuration. The processing performed by the display device 10 will now be described.

[0030] (Display mode setting process) First, a display mode setting process will be described as a process performed by the display device 10. Fig. 4 is a flowchart illustrating the display mode setting process.

[0031] (Image Acquisition) 4, the control unit 32 causes the image acquisition unit 40 to acquire an image captured by the imaging unit 28 (step S10). The image acquisition unit 40 acquires an image (image data) captured by the imaging unit 28 and having an imaging range within the visual field of the user U. That is, it can be said that the image acquisition unit 40 acquires an image including a main image PM within the visual field of the user U visually recognized through the display unit 22.

[0032] (Identifying the type of object seen) The control unit 32 extracts the viewed object based on the image acquired by the image acquisition unit 40 using the viewed object identification unit 42, and identifies the type of the viewed object (step S12). The viewed object refers to an object that the user U views through the display unit 22, and can be said to be an object that exists within the field of view of the user U. The viewed object identification unit 42 extracts an object shown in the image acquired by the image acquisition unit 40 as a viewed object, and identifies the type of the viewed object. The type of object refers to the category of the object when the object is classified according to a predetermined classification criterion. The classification criterion here may be arbitrary.

[0033] The visual object identification unit 42 may extract the visual object by any method and identify the type, but may also use an AI (Artificial Intelligence) model to identify the object. The AI ​​model in this case is stored in the storage unit 24, and is a model for extracting the object appearing in the image from the image data and identifying the type. The AI ​​model is a trained AI model constructed by learning the image data and information indicating the type of the object appearing in the image as one data set and using multiple data sets as teacher data. The visual object identification unit 42 inputs the image data acquired by the image acquisition unit 40 into the trained AI model, acquires information identifying the type of the visual object appearing in the image, and identifies the type of the visual object. Note that any model may be used as the AI ​​model, but may use, for example, a CNN (Conventional Neural Network) model.

[0034] The viewed object identifying unit 42 may extract all objects (objects present within the visual field of the user U) in the image acquired by the image acquiring unit 40 as viewed objects, but is not limited thereto. The viewed object identifying unit 42 may extract some of the objects (objects present within the visual field of the user U) included in the image acquired by the image acquiring unit 40 as viewed objects. In this case, for example, the viewed object identifying unit 42 may extract an object that overlaps the line of sight of the user U among the objects included in the image acquired by the image acquiring unit 40 as a viewed object. For example, the viewed object identifying unit 42 inputs image data in a predetermined area (for example, a central area) of the image acquired by the image acquiring unit 40 into an AI model, acquires information that identifies the type of the viewed object appearing in the predetermined area, and identifies the type of the viewed object.

[0035] For example, the visual object identification unit 42 may extract an object that is continuously within the visual field for a predetermined time or more as the visual object. In this case, the visual object identification unit 42 extracts an object that is continuously within the visual field for a predetermined time or more from the images captured continuously in time series by the imaging unit 28, that is, an object included in all of the images that are continuous for a predetermined time or more, as the visual object. In this case, for example, the visual object identification unit 42 inputs image data of the images captured continuously in time series to an AI model, respectively, to obtain information on the type of object captured in each image. Then, when an object of the same type is captured in all of the images captured continuously for a predetermined time or more, the visual object identification unit 42 identifies the object as the visual object.

[0036] The viewed object specifying section 42 may also acquire information indicating the position of the viewed object in the main image PS (the coordinates of the viewed object in the image acquired by the image acquiring section 40).

[0037] (Specifying the number of views) Next, the control unit 32 uses the viewing count specifying unit 44 to specify the viewing count indicating the number of times the user U viewed the same type of object based on the identification result of the viewed object specifying unit 42 (step S14). The viewing count specifying unit 44 counts the number of times the viewed object is extracted as a viewed object by the viewed object specifying unit 42 for each type of viewed object, sets it as the viewing count, and stores it in the storage unit 24. That is, the viewing count specifying unit 44 specifies the number of times the same type of viewed object is extracted by the viewed object specifying unit 42 as the viewing count. For example, if a viewed object of a type called banana is extracted five times at different times by the viewed object specifying unit 42, the viewing count specifying unit 44 specifies the viewing count of the object of a type called banana as five times.

[0038] The viewing count specifying unit 44 may reset the viewing count to zero after a predetermined period has elapsed. That is, the viewing count specifying unit 44 may set the number of times that the same type of viewed object is extracted by the viewed object specifying unit 42 within the predetermined period as the viewing count. The predetermined period may be set arbitrarily, and may be, for example, one month or one year.

[0039] (Display mode settings) The display mode setting unit 46 sets the display mode of the sub-image PS based on the number of times of viewing. The sub-image PS is information about an object (an object included in the main image PS) within the visual field of the user U, and is displayed when the object is within the visual field of the user U. The display mode of the sub-image PS refers to the manner in which the sub-image PS is displayed, as described above. The display mode setting unit 46 sets the display mode of the sub-image PS for each type of object.

[0040] More specifically, the display mode setting unit 46 sets the display mode of the sub-image PS for each type of object based on the number of times of viewing. That is, the display mode setting unit 46 sets the display mode of the sub-image PS for the type of object whose number of times of viewing has been counted based on the number of times of viewing. For example, the display mode setting unit 46 sets the display mode of the sub-image PS for the type of object called banana based on the number of times of viewing of the type of viewed object called banana.

[0041] 5 and 6 are diagrams showing an example of a case where a sub-image is displayed in a normal mode and an emphasis mode. The display mode setting unit 46 sets the display mode of the sub-image PS to either the normal mode or the emphasis mode according to the number of times of viewing. The emphasis mode is a mode in which information about the object is displayed more emphasized than in the normal mode. That is, the sub-image PS set to the emphasis mode is displayed more prominently than the sub-image PS set to the normal mode. For example, the example of FIG. 5 shows a case where the character "banana" is displayed as the sub-image PS when an object of a type called banana is within the visual field range of the user U. In the example of FIG. 5, the character "banana" in the emphasis mode is larger than the character "banana" in the normal mode, and the character in the emphasis mode is displayed emphasized. In addition, the example of FIG. 6 shows a case where an arrow pointing to an object of a type called banana is displayed as the sub-image PS. In the example of FIG. 6, the arrow in the emphasis mode is larger than the arrow in the normal mode, and the arrow in the emphasis mode is displayed emphasized.

[0042] However, FIG. 5 and FIG. 6 are only examples, and the sub-image PS may be any image that shows information about the object. For example, the sub-image PS may be characters in any language that show information about the object, or an icon of any shape such as an exclamation mark. For example, the sub-image PS may be a sentence that describes the object. In this case, the highlight mode may indicate that the description of the object is more detailed than the normal mode. For example, the sub-image PS may display information about the same type of visually recognized object that was previously extracted, or may display comparison information that is information comparing the object currently visually recognized by the user U with the same type of visually recognized object that was previously extracted. In this case, for example, if the user U is currently visualizing a banana, information about the banana previously visually recognized by the user U may be displayed as the sub-image PS, or information comparing the currently visually recognized banana with the previously visually recognized banana may be displayed.

[0043] 5 and 6, the sub-image PS in the emphasis mode is larger than the sub-image PS in the normal mode, but the present invention is not limited to this, and any display method for emphasizing the sub-image PS in the emphasis mode more than the sub-image PS in the normal mode may be used. For example, the display position of the sub-image PS in the emphasis mode in the main image PS may be a more prominent position (e.g., the center) than the display position of the sub-image PS in the normal mode. The luminance of the sub-image PS in the emphasis mode may be higher than the luminance of the sub-image PS in the normal mode. The display frequency of the sub-image PS in the emphasis mode may be higher than the display frequency of the sub-image PS in the normal mode. The display frequency here may refer to the display time of the sub-image PS per unit time, or may refer to the blinking speed of the sub-image PS.

[0044] Returning to the description in FIG. 4, the method of setting the display mode based on the viewing count will be specifically described below. As shown in FIG. 4, the control unit 32 uses the display mode setting unit 46 to determine whether the viewing count is a predetermined number or more (step S16), and if the viewing count is the predetermined number or more (step S16; Yes), sets the display mode of the sub-image PS for the object of the same type as the viewed object to the emphasis mode (step S18). On the other hand, if the viewing count is not the predetermined number or more (step S16; No), that is, if the viewing count is less than the predetermined number, the display mode of the sub-image PS for the object of the same type as the viewed object is not set to the emphasis mode, and proceeds to step S20. More specifically, the display mode setting unit 46 sets the display mode of the sub-image PS for the object of the same type as the viewed object whose viewing count is less than the predetermined number to the normal mode. For example, if the number of times a visual object of the type banana is viewed is equal to or greater than a predetermined number and the number of times a visual object of the type apple is viewed is less than a predetermined number, the display mode setting unit 46 sets the display mode of the sub-image PS for the object of the type banana to the emphasis mode and the display mode of the sub-image PS for the object of the type apple to the normal mode.

[0045] Thereafter, if the processing is to be completed (step S20; Yes), this processing is terminated, and if the processing is not to be completed (step S20; No), the process returns to step S10 and repeats the processing. At that time, if there is a visually recognized object whose visual recognition count was less than the predetermined number of times but whose visual recognition count becomes equal to or greater than the predetermined number of times in the subsequent processing, the display mode of the sub-image PS for the object of the same type as the visually recognized object is switched from the normal mode to the highlight mode.

[0046] In this way, the display mode setting unit 46 sets the sub-image PS in the normal mode when the number of times of viewing is less than a predetermined number, and sets the sub-image PS in the emphasis mode when the number of times of viewing is equal to or greater than the predetermined number. However, the processing of the display mode setting unit 46 is not limited to this, and the display mode of the sub-image PS may be set by any method based on the number of times of viewing. For example, the display mode setting unit 46 may set the display mode to a non-display mode in which the sub-image PS is not displayed when the number of times of viewing is less than a predetermined number, and may set the display mode to a mode in which the sub-image PS is displayed when the number of times of viewing is equal to or greater than the predetermined number. Also, for example, the display mode setting unit 46 may set the display mode such that the greater the number of times of viewing, the greater the degree of emphasis of the sub-image PS. For example, the display mode setting unit 46 may set the sub-image PS to a larger size as the number of times of viewing increases.

[0047] Also, for example, the display mode setting unit 46 may select whether to set a non-display mode in which the sub-image PS is not displayed or a mode in which the sub-image PS is displayed for each type of object. In this case, for example, the display mode setting unit 46 acquires type information indicating the type of object for which the sub-image PS is displayed. Then, the display mode setting unit 46 sets the non-display mode for objects of types other than those indicated by the type information. On the other hand, the display mode setting unit 46 sets the mode in which the sub-image PS is displayed for objects of types indicated by the type information. Note that the type information may be information indicating the type of object for which the sub-image PS is not displayed, and in this case, the non-display mode is set for objects of the type indicated by the type information. This type information may be set in advance, for example, by the user U. That is, for example, if the user U is interested in fruits and not interested in meat, the display mode is set for fruits and the non-display mode is set for meat. In this way, by setting the display mode for each type of object, the sub-image PS can be provided according to the preferences of the user U.

[0048] (Sub-image display processing) Next, the display process of the sub-image PS will be described. The output control unit 48 causes the display unit 22 to display the sub-image PS related to the visual target extracted by the visual target identification unit 42 so as to be superimposed on the main image PM including the visual target. In other words, when the visual target exists within the visual field of the user U, the output control unit 48 causes the display unit 22 to display the sub-image PS related to the visual target.

[0049] The output control unit 48 causes the display unit 22 to display the sub-image PS relating to the viewed object (viewed object within the visual field of the user U) extracted by the viewed object specifying unit 42 based on the display mode set by the display mode setting unit 46, i.e., based on the viewing count. The output control unit 48 reads out the display mode of the sub-image PS relating to the type of viewed object extracted by the viewed object specifying unit 42, and causes the sub-image PS relating to the extracted viewed object to be displayed in that display mode. This will be described in more detail below.

[0050] Fig. 7 is a flowchart for explaining the display processing of the sub-image. As shown in Fig. 7, the processing up to S12 for extracting the visual object and identifying the type of the visual object is the same as the processing in setting the display mode shown in Fig. 4. That is, in this embodiment, the control unit 32 performs the processing from step S14 shown in Fig. 4 (display mode setting processing) and the processing from step S30 shown in Fig. 7 (display processing of the sub-image PS) in parallel.

[0051] When the visual object is extracted and the type of the visual object is specified in step S12, the control unit 32 determines whether the display mode of the sub-image PS for the type of the extracted visual object is the highlight mode through the output control unit 48 (step S30). If the display mode for the type of the extracted visual object is the highlight mode (step S30; Yes), the output control unit 48 causes the display unit 22 to display the sub-image PS for the visual object in the highlight mode (step S32). The output control unit 48 causes the sub-image PS to be displayed in the highlight mode so as to be superimposed on the main image PM including the visual object. On the other hand, if the display mode for the type of the extracted visual object is not the highlight mode (step S30; No), that is, if it is the normal mode, the output control unit 48 causes the display unit 22 to display the sub-image PS for the visual object in the normal mode (step S34). The output control unit 48 causes the sub-image PS to be displayed in the normal mode so as to be superimposed on the main image PM including the visual object. In other words, the output control unit 48 emphasizes the sub-image (sub-image PS set to emphasis mode) to be displayed when the number of times viewed is equal to or greater than a predetermined number of times, compared to the sub-image PS (sub-image PS set to normal mode) to be displayed when the number of times viewed is less than a predetermined number of times.

[0052] For example, if the type of the extracted visual object is a banana, in the above example, the sub-image PS for bananas is in the highlighted mode, so the sub-image PS for bananas is displayed in the highlighted mode. That is, if the user U has previously viewed bananas a predetermined number of times, bananas are set in the highlighted mode, so when the user U views bananas this time, the sub-image PS for bananas is displayed in the highlighted mode. On the other hand, if the type of the extracted visual object is an apple, in the above example, the sub-image PS for apples is in the normal mode, so the sub-image PS for apples is displayed in the normal mode.

[0053] (effect) As described above, the display device 10 according to the present embodiment includes the display unit 22 that displays an image, the imaging unit 28, the viewed object identification unit 42, the viewing count identification unit 44, and the output control unit 48. The imaging unit 28 images an object within the visual field of the user U. The viewed object identification unit 42 identifies the type of viewed object within the visual field of the user U based on the image captured by the imaging unit 28. The viewing count identification unit 44 identifies the viewing count, which is the number of times the user U viewed the same type of viewed object, based on the identification result of the viewed object identification unit 42. The output control unit 48 causes the display unit 22 to display an image so as to be superimposed on the main image PM within the visual field of the user U viewed through the display unit 22. The output control unit 48 causes the display unit 22 to display a sub-image PS including information on the type of object so as to be superimposed on the main image PM including the type of object based on the viewing count.

[0054] In some display devices, the display mode may be uniform regardless of the user of the display device. In contrast, the display device 10 according to the present embodiment counts the number of times that the user U has viewed the same type of object as the number of viewings. Then, when the user U views an object of the same type as the object, the display device 10 displays a sub-image PS related to the object based on the counted number of viewings. The display device 10 according to the present embodiment makes it possible to adjust the display mode of the sub-image PS according to the past actions of the user U, and thus makes it possible to appropriately adjust the display mode according to the user U.

[0055] The output control unit 48 emphasizes the sub-image PS to be displayed when the number of times of viewing is equal to or greater than a predetermined number, compared to the sub-image PS to be displayed when the number of times of viewing is less than a predetermined number. According to the display device 10 of this embodiment, the sub-image PS for an object that the user U has viewed many times is emphasized, so that it is possible to appropriately adjust the display mode to suit the user U. For example, if the user U is interested in bananas, the number of times bananas are viewed will be large. In such a case, the sub-image PS when bananas are viewed is emphasized, so that information about bananas can be appropriately conveyed to the user U.

[0056] Second embodiment Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that the display mode of the sub-image PS is set based on the number of visits, which is the number of times the same place has been visited. Explanations of the configurations of the second embodiment and the first embodiment will be omitted.

[0057] (Display Device Configuration) Fig. 8 is a schematic block diagram of a display device according to the second embodiment. As shown in Fig. 8, a control unit 32A of a display device 10A according to the second embodiment includes a position information acquisition unit 50, a visit count determination unit 52, a display mode setting unit 46A, and an output control unit 48A.

[0058] The position information acquisition unit 50 acquires position information of the user U (the display device 10A). The position information acquisition unit 50 causes the GNSS receiver 30 to detect the position information of the user U, and acquires the detection result of the position information of the user U from the GNSS receiver 30. The position information acquisition unit 50 causes the GNSS receiver 30 to detect the position information of the user U at predetermined intervals. The position information acquisition unit 50 acquires the position information of the user U detected by the GNSS receiver 30 every time the GNSS receiver 30 detects the position information of the user U.

[0059] The visit count identifying unit 52 identifies the number of visits of the user U based on the location information of the user U. The visit count is the number of times the user U visits the same place. The process of the visit count identifying unit 52 will be described later.

[0060] The display mode setting unit 46A determines the display mode of the sub-image PS based on the number of visits. The process of the display mode setting unit 46A will be described later.

[0061] The output control unit 48A displays the sub-image PS based on the display mode of the sub-image PS determined by the display mode setting unit 46A. Since the display mode of the sub-image PS is set based on the number of visits, it can be said that the output control unit 48A displays the sub-image PS based on the number of visits. Specific processing contents of the output control unit 48A will be described later.

[0062] (Display mode setting process) FIG. 9 is a flowchart illustrating the display mode setting process.

[0063] (Location information acquisition) As shown in FIG. 9, the control unit 32 acquires the position information of the user U by the position information acquisition unit 50 (step S40).

[0064] (Specifying the number of visits) The control unit 32 uses the visit count specification unit 52 to specify the number of visits, which is the number of times the user U has visited the same place, based on the position information of the user U (step S42). The visit count specification unit 52 specifies the visited places, which are places visited by the user U, from the position information of the user U, counts the number of times the user U has visited each visited place, and stores the number of visits in the storage unit 24. In this embodiment, a place refers to an area that occupies a predetermined area in the Earth coordinate system. For example, the visit count specification unit 52 divides the Earth coordinate system into a plurality of areas (places) and determines an area (place) that includes the position of the user U within its range as a visited place. Then, the visit count specification unit 52 counts the number of times that the area (place) is determined to be a visited place, and sets the number of visits. For example, if the number of times that the position of the user U detected at different times is within the area AR1 is five times in total, the visit count specification unit 52 determines the number of visits to the area AR1 as five times.

[0065] More specifically, when the user U is located in the same area for a predetermined time or more consecutively, the visit count identifying unit 52 may determine that the user U has visited the area when the user U has stayed in the same area for a predetermined time or more consecutively. In this case, the visit count identifying unit 52 determines that the user U has visited the area when the positions indicated by the position information of the user U detected continuously for a predetermined time or more consecutively are within the same area based on the position information of the user U detected continuously in time series, as a visited place.

[0066] However, the visit count determination unit 52 is not limited to dividing the Earth coordinate system into a plurality of regions and counting the number of visits in this manner. For example, the visit count determination unit 52 may compare the positions of the user U detected at different times, and if those positions are within a predetermined distance range, determine that the user U visited the same place, and count the number of visits.

[0067] The visit count determination unit 52 may reset the visit count to zero after a predetermined period has elapsed. That is, the visit count determination unit 52 may determine the number of times the user U visited the same place within the predetermined period as the viewing count. The predetermined period may be set arbitrarily, and may be, for example, one month or one year.

[0068] (Display mode settings) The display mode setting unit 46A sets the display mode of the sub-image PS based on the number of visits. The display mode setting unit 46A extracts an object within a predetermined distance range from the place where the number of visits is specified as a nearby object. For example, the display mode setting unit 46A reads map data from the storage unit 24, and extracts an object within a predetermined distance range from the place where the number of visits is specified as a nearby object based on the map data. Then, the display mode setting unit 46A sets the display mode of the sub-image PS related to the same type of object as the nearby object based on the number of visits. For example, if the place where the number of visits is specified is the area AR1, the display mode setting unit 46A reads out equipment within a predetermined distance range from the area AR1 from the map data, and sets the read equipment as the nearby object. Then, the display mode setting unit 46A sets the display mode of the sub-image PS related to the same type of equipment as the equipment set as the nearby object based on the number of visits to the area AR1.

[0069] More specifically, as shown in FIG. 9, the control unit 32 determines whether the visit count is equal to or greater than a predetermined number by the display mode setting unit 46A (step S44). If the visit count is equal to or greater than the predetermined number (step S44; Yes), the display mode setting unit 46A extracts nearby objects within a predetermined distance range from the location where the visit count is equal to or greater than the predetermined number. Then, the display mode setting unit 46A sets the display mode of the sub-image PS for the object of the same type as the nearby object to the highlight mode (step S46). On the other hand, if the visit count is not equal to or greater than the predetermined number (step S44; No), that is, if the visit count is less than the predetermined number, the display mode is not set to the highlight mode and the process proceeds to step S48. More specifically, the display mode setting unit 46A initially sets the display mode of the sub-image PS for all types of objects to the normal mode. The display mode setting unit 46A does not extract nearby objects or set the display mode for locations that have been visited less than a predetermined number of times, but if there is a location that has been visited more than a predetermined number of times, it extracts nearby objects at that location and switches the display mode for objects of the same type as the nearby objects to an emphasis mode.

[0070] Thereafter, if the processing is to be completed (step S48; Yes), this processing is terminated, and if the processing is not to be completed (step S48; No), the processing returns to step S10 and the processing is repeated.

[0071] (Sub-image display processing) The output control unit 48A causes the display unit 22 to display the sub-image PS related to the viewed object (the viewed object within the visual field of the user U) extracted by the viewed object specifying unit 42 based on the display mode set by the display mode setting unit 46A, i.e., based on the number of visits. The output control unit 48 reads out the display mode of the sub-image PS related to the type of viewed object extracted by the viewed object specifying unit 42, and causes the sub-image PS related to the extracted viewed object to be displayed in that display mode. This will be described in more detail below.

[0072] FIG. 10 is a flowchart for explaining the display process of the sub-image. As shown in FIG. 10, the control unit 32 causes the image acquisition unit 40 to acquire an image captured by the imaging unit 28 (step S10A), and causes the visual object identification unit 42 to extract a visual object based on the image acquired by the image acquisition unit 40 and identify the type of the visual object (step S12A). The control unit 32 causes the output control unit 48A to determine whether the display mode of the sub-image PS for the type of the extracted visual object is the emphasis mode (step S30A). If the display mode for the type of the extracted visual object is the emphasis mode (step S30A; Yes), the output control unit 48 causes the display unit 22 to display the sub-image PS for the visual object in the emphasis mode (step S32A). The output control unit 48A causes the sub-image PS to be displayed in the emphasis mode so as to be superimposed on the main image PM including the visual object. On the other hand, if the display mode for the type of the extracted visual object is not the highlight mode (step S30A; No), i.e., if it is the normal mode, the output control unit 48A causes the display unit 22 to display the sub-image PS for that visual object in the normal mode (step S34A). The output control unit 48A causes the sub-image PS to be displayed in the normal mode so as to be superimposed on the main image PM including that visual object. That is, it can be said that the output control unit 48A displays the sub-image (sub-image PS set to the highlight mode) to be displayed when the number of visits is equal to or greater than the predetermined number, more emphatically than the sub-image PS (sub-image PS set to the normal mode) to be displayed when the number of visits is less than the predetermined number.

[0073] In this way, when the display mode for the object viewed by the user U is the highlight mode, the output control unit 48A displays the sub-image PS for the object in the highlight mode. Therefore, when the user U visits a place where the number of visits is equal to or more than the predetermined number of times again and views the nearby object itself at that place, the sub-image PS for the nearby object is displayed in the highlight mode. Also, when the user U visits a place other than the place where the number of visits is equal to or more than the predetermined number of times and views an object of the same type as the nearby object at the place where the number of visits is equal to or more than the predetermined number of times, the sub-image PS for the viewed object is displayed in the highlight mode. As a specific example, consider a case where the number of visits to the area AR1 is equal to or more than the predetermined number of times and the nearby object of the area AR1 is a pharmacy. In this case, when the user U visits the area AR1 again and views the same pharmacy as the nearby object, or when the user U views a different pharmacy at a place other than the area AR1, the sub-image PS for the pharmacy is displayed in the highlight mode.

[0074] FIG. 11 is a diagram showing an example of a case where a sub-image is displayed in normal mode and emphasis mode. In the example of FIG. 11, when an object of a type called a pharmacy is within the visual field range of the user U, the character "pharmacy" is displayed as the sub-image PS. In the example of FIG. 11, the character "pharmacy" in the emphasis mode is larger than the character "pharmacy" in the normal mode, and the character in the emphasis mode is displayed with emphasis. However, FIG. 11 is an example, and the sub-image PS may be any image that shows information about the object. For example, the sub-image PS may be characters in any language that show information about the object, or may be an icon of any shape such as an exclamation mark or an arrow. For example, the sub-image PS may be a sentence that describes the object. In this case, the emphasis mode may indicate that the description of the object is more detailed than the normal mode. For example, the sub-image PS may display information about the same type of visual object extracted previously, or may display comparison information that is information comparing the object currently visually recognized by the user U with a nearby object extracted previously. In this case, for example, when user U is currently viewing a pharmacy, information regarding a pharmacy that user U previously visited may be displayed as a sub-image PS, or information comparing the pharmacy currently being viewed with the pharmacy previously visited may be displayed.

[0075] In the example of FIG. 11, the sub-image PS in the emphasis mode is larger than the sub-image PS in the normal mode, but the present invention is not limited to this, and any display method for emphasizing the sub-image PS in the emphasis mode more than the sub-image PS in the normal mode may be used. For example, the display position of the sub-image PS in the emphasis mode in the main image PS may be a more prominent position (e.g., the center) than the display position of the sub-image PS in the normal mode. The luminance of the sub-image PS in the emphasis mode may be higher than the luminance of the sub-image PS in the normal mode. The display frequency of the sub-image PS in the emphasis mode may be higher than the display frequency of the sub-image PS in the normal mode. The display frequency here may refer to the display time of the sub-image PS per unit time, or may refer to the blinking speed of the sub-image PS.

[0076] In the present embodiment, the sub-image PS is in the normal mode when the number of visits is less than a predetermined number, and in the emphasized mode when the number of visits is equal to or greater than the predetermined number. However, the method of setting the display mode in the second embodiment is not limited to this, and the display mode of the sub-image PS may be set by any method based on the number of visits. For example, the display mode setting unit 46A may set the display mode to a non-display mode in which the sub-image PS is not displayed when the number of visits is less than a predetermined number, and may set the display mode to a mode in which the sub-image PS is displayed when the number of visits is equal to or greater than the predetermined number. For example, the display mode setting unit 46A may set the display mode such that the greater the number of visits, the greater the degree of emphasis of the sub-image PS. For example, the display mode setting unit 46 may set the sub-image PS to a larger size when the number of visits is greater.

[0077] (effect) As described above, the display device 10A according to the second embodiment includes the display unit 22 that displays an image, the output control unit 48A, the position information acquisition unit 50, the visit count specification unit 52, and the output control unit 48A. The position information acquisition unit 50 acquires the position information of the user U. The visit count specification unit 52 specifies the visit count, which is the number of times the user U has visited the same place, based on the position information of the user U. The output control unit 48A causes the display unit 22 to display an image so as to be superimposed on the main image PM within the visual field of the user U viewed through the display unit 22. The output control unit 48A causes the display unit 22 to display a sub-image PS including information on an object of the same type as an object (nearby object) within a predetermined distance range from the place visited by the user U, based on the visit count.

[0078] In some display devices, the display mode may be uniform regardless of the user of the display device. In contrast, the display device 10A according to the present embodiment counts the number of times that the user U has visited the same place as the number of visits. Then, when the user U visually recognizes an object of the same type as a nearby object near the place, the display device 10A displays a sub-image PS related to the object based on the counted number of visits. The display device 10 according to the present embodiment makes it possible to adjust the display mode of the sub-image PS according to the past actions of the user U, and thus makes it possible to appropriately adjust the display mode according to the user U.

[0079] Furthermore, the output control unit 48A emphasizes the sub-image PS to be displayed when the number of visits is equal to or greater than a predetermined number, compared to the sub-image PS to be displayed when the number of visits is less than a predetermined number. According to the display device 10A of this embodiment, the sub-image PS for an object of the same type as a nearby object near a location visited frequently is emphasized. Therefore, according to the display device 10A, it is possible to appropriately adjust the display mode in accordance with the behavior of the user U.

[0080] Furthermore, when an object of the same type as the nearby object enters the user U's field of view in a location other than the location where the user U has visited a predetermined number of times, the output control unit 48A also emphasizes and displays the sub-image PS relating to the object. This makes it possible to make the user U easily recognize an object that interests him / her even in a location that the user U has never visited before, and makes it possible to appropriately adjust the display mode in accordance with the user U's actions.

[0081] Furthermore, the visit count determination unit 52 determines that the user U has visited a location when the user U has been in the same location for a predetermined period of time or more. This makes it possible to adjust the display mode in accordance with the user U's actions.

[0082] The second embodiment and the first embodiment may be combined. That is, the sub-image PS may be displayed based on the number of views using the process described in the first embodiment, while the sub-image PS may be displayed based on the number of visits using the process described in the second embodiment.

[0083] Third embodiment Next, a third embodiment will be described. The third embodiment differs from the first and second embodiments in that the display mode of the sub-image PS is set based on the number of times the user U views the same type of object in the same place. Explanations of the configurations of the third embodiment common to the first and second embodiments will be omitted.

[0084] (Display Device Configuration) Fig. 12 is a schematic block diagram of a display device according to a third embodiment. As shown in Fig. 12, a control unit 32B of a display device 10B according to a third embodiment includes an image acquisition unit 40, a viewed object identification unit 42, a viewing count identification unit 44, a position information acquisition unit 50, a visit count identification unit 2, an overlap count identification unit 60, a display mode setting unit 46B, and an output control unit 48B.

[0085] The overlap count identifying unit 60 identifies the overlap count, which is the number of times the user U visually recognizes the same type of object in the same place. The processing content of the overlap count identifying unit 60 will be described later.

[0086] (Display mode setting process) The control unit 32B according to the third embodiment sets the display mode of the sub-image PS based on the number of views in a manner similar to that of the first embodiment, and sets the display mode of the sub-image PS based on the number of visits in a manner similar to that of the second embodiment. However, the control unit 32B differs from the first and second embodiments in that it sets the display mode of the sub-image PS based on the number of overlaps as well. The process of setting the display mode based on the number of overlaps will be described below.

[0087] FIG. 13 is a flowchart for explaining the display mode setting process. As shown in FIG. 13, the overlapping number identification unit 60 of the control unit 32B identifies the overlapping number based on the type of the visually recognized object identified by the visually recognized object identification unit 42 and the position information of the user U acquired by the position information acquisition unit 50 (step S50). Specifically, the overlapping number identification unit 60 sets the information on the type of the visually recognized object identified by the visually recognized object identification unit 42 and the position information of the user U at the timing when the image used to identify the type of the visually recognized object was captured as one data set, and acquires a data set for each timing when the image was captured. Then, for example, the overlapping number identification unit 60 classifies these data sets by combination of the type of the visually recognized object and the location where the user U is located, and counts the number of data sets that are the same type of object and the same location where the user U is located as the overlapping number. The overlapping number identification unit 60 counts the overlapping number for each combination of the type of the visually recognized object and the location where the user U is located. That is, for example, if a pharmacy is viewed five times in area AR1, the number of overlaps for the pharmacy in area AR1 is five.

[0088] The overlap count identification unit 60 may reset the overlap count to zero after a predetermined period of time has elapsed. That is, the overlap count identification unit 60 may determine the number of times that the user U views the same object in the same place within the predetermined period of time as the overlap count. The predetermined period of time may be set arbitrarily, and may be, for example, one month or one year.

[0089] The display mode setting unit 46B sets the display mode of the sub-image PS based on the overlapping number. The display mode setting unit 46B sets the display mode for the visually recognized object at the location where the overlapping number is counted based on the overlapping number. Specifically, as shown in FIG. 13, the display mode setting unit 46B determines whether the overlapping number is a predetermined number or more (step S52), and if it is the predetermined number or more (step S52; Yes), sets the display mode of the sub-image PS for the visually recognized object at the location where the overlapping number is the predetermined number or more to a non-display mode (step S54). For example, if the overlapping number, which is the number of times the pharmacy is viewed in the area AR1, is the predetermined number or more, the display mode for the pharmacy in the area AR1 is set to a non-display mode.

[0090] On the other hand, if the number of overlaps is not equal to or greater than the predetermined number (step S52; No), that is, if the number of overlaps is less than the predetermined number, the display mode is not set to the non-display mode and the process proceeds to step S56. If the process is to be ended (step S56; Yes), the process is ended, and if the process is not to be ended (step S56; No), the process returns to step S50 and is repeated.

[0091] Note that the display mode setting unit 46B may stop the non-display mode setting and switch to a mode in which the sub-image PS is displayed when a predetermined period of time has elapsed since the non-display mode was set.

[0092] (Sub-image display processing) FIG. 14 is a diagram showing an example of the non-display mode. Based on the display mode set by the display mode setting unit 46B, the output control unit 48B causes the display unit 22 to display the sub-image PS related to the visual object (visual object within the visual field of the user U) extracted by the visual object identification unit 42. For example, similar to the first and second embodiments, when the display mode of the sub-image PS related to the type of the extracted visual object is the highlight mode, the output control unit 48B causes the sub-image PS to be displayed in the highlight mode, and when the display mode of the sub-image PS related to the type of the extracted visual object is the normal mode, the output control unit 48B causes the sub-image PS to be displayed in the normal mode. On the other hand, when the display mode of the sub-image PS related to the extracted visual object is the non-display mode, the output control unit 48B does not cause the sub-image PS related to the visual object to be displayed. In this case, the output control unit 48B acquires the position information of the user U when the extracted visual target object is visually recognized, and when the position of the user U when the visual target object is visually recognized is the same as the location where the overlapping number is equal to or greater than the predetermined number, and the type of the visual target object matches the type of the nearby object at the location where the overlapping number is equal to or greater than the predetermined number, it can be said that the sub-image PS of the visual target object is not displayed. Fig. 14 shows an example in the non-display mode. As shown in Fig. 14, in the non-display mode, the sub-image PS is not displayed.

[0093] (effect) As described above, the display device 10B according to the third embodiment includes the overlapping number specification unit 60. The overlapping number specification unit 60 specifies the overlapping number, which is the number of times the user has viewed the same type of object at the same place, based on the specification result of the viewed object specification unit 42 and the position information of the user U. If the overlapping number is equal to or greater than a predetermined number, the output control unit 48B stops displaying the sub-image PS for the object. Here, generally, the user U lives a life in which a behavior pattern rotates for one cycle per week. Therefore, it is assumed that the user U may view the same object or visit the same place even though he or she is not interested. In such a case, if the sub-image PS of the object is displayed, it may be bothersome for the user U. In contrast, the display device 10B according to the third embodiment stops displaying the sub-image PS of the object when the overlapping number is equal to or greater than a predetermined number. Therefore, according to the display device 10B according to the third embodiment, it is possible to appropriately adjust the display mode according to the behavior of the user U while suppressing the user U from feeling bothered.

[0094] In the above description, the sub-image PS of the visual object at the location where the overlap count is equal to or greater than the predetermined number is set to the non-display mode, and the sub-image PS of the same type of object at a different location from the location is not set to the non-display mode. However, this is not limited to this, and the sub-image PS of the same type of object at a different location may also be set to the non-display mode. That is, the display device 10B may set the sub-image PS of the same type of object as the nearby object at the location where the overlap count is equal to or greater than the predetermined number to the non-display mode. For example, when the overlap count of the pharmacy in the area AR1 is equal to or greater than the predetermined number, the display mode of the sub-image PS of the pharmacy at a location other than the area AR1 may also be set to the non-display mode.

[0095] In the above description, the display device 10B sets the display mode based on the number of views and the number of visits, and also sets the display mode based on the number of overlaps, but this is not limited thereto. For example, the display device 10B may set the display mode based on the number of overlaps without setting the display mode based on the number of views and the number of visits.

[0096] (Fourth embodiment) Next, a fourth embodiment will be described. The fourth embodiment differs from the first embodiment in that the display mode of the sub-image PS is set based on the number of times the user U looks away from the object. Explanation of the configuration of the fourth embodiment common to the first embodiment will be omitted. Note that the fourth embodiment can be combined with the second and third embodiments.

[0097] (Display Device Configuration) Fig. 15 is a schematic block diagram of a display device according to the fourth embodiment. As shown in Fig. 15, a control unit 32C of a display device 10C according to the fourth embodiment includes an image acquisition unit 40, a visual object identification unit 42C, a line of sight switching count identification unit 70, a display mode setting unit 46C, and an output control unit 48C.

[0098] The viewed object specifying unit 42C specifies the type of object to which the user U averts his / her eyes within a predetermined time based on the image captured by the imaging unit 28. The process of the viewed object specifying unit 42C will be described later.

[0099] The gaze switching count specifying unit 70 specifies the gaze switching count, which is the number of times the user U averts his / her gaze from the same type of object, based on the specification result of the viewed object specifying unit 42C. The processing of the gaze switching count specifying unit 70 will be described later.

[0100] (Display mode setting process) The control unit 32C according to the fourth embodiment sets the display mode of the sub-image PS based on the number of times the line of sight is switched. Note that, in the fourth embodiment, the display mode may be set based on the number of times of viewing as in the first embodiment, the display mode may be set based on the number of times of visiting as in the second embodiment, or the display mode may be set based on the number of times of overlap as in the third embodiment.

[0101] Fig. 16 is a flowchart for explaining the display mode setting process. As shown in Fig. 16, the control unit 32C acquires an image captured by the imaging unit 28 through the image acquisition unit 40 (step S60), and the visual object identification unit 42C identifies the type of the line of sight switching object based on the image acquired by the image acquisition unit 40 (step S62). The line of sight switching object refers to an object to which the user U has averted his / her eyes within a predetermined time.

[0102] The visual object identification unit 42C extracts a line of sight switching object based on image data of images captured continuously in time series, and identifies the type of line of sight switching object. Specifically, the visual object identification unit 42C extracts an object that once entered the visual field and then went out of the visual field within a predetermined time as a line of sight switching object. That is, the visual object identification unit 42C extracts an object that appears in an image and no longer appears in an image captured before a predetermined time has elapsed since the image was captured as a line of sight switching object. However, the visual object identification unit 42C is not limited to extracting an object that once entered the visual field and then went out of the visual field within a predetermined time as a line of sight switching object. For example, the visual object identification unit 42C may extract an object that once overlapped the user U's line of sight and then no longer overlaps the line of sight within a predetermined time as a line of sight switching object. In this case, for example, the visual object specification unit 42C extracts, as a line of sight switching object, an object that is captured within a predetermined area (for example, a central area) of an image and that is no longer captured within the predetermined area (for example, a central area) in an image captured within a predetermined time period after the image. Note that any method for specifying the type of line of sight switching object may be used, but for example, an AI model may be used to specify the type, as in the first embodiment.

[0103] The predetermined time here may be set arbitrarily, but is preferably, for example, 200 milliseconds or more and 900 milliseconds or less. In brain science, it is said that the reaction time when a known type of stimulus is presented and a predetermined type of reaction is made to it (simple detection task) is about 150 milliseconds to 300 milliseconds. However, in this embodiment, in order to properly detect the reaction of looking at an object, deciding that you do not want to look at the object, and then looking away, it is preferable to take into account the discrimination reaction time for making a judgment in addition to the simple reaction. That is, in this embodiment, it is preferable to set the predetermined time to 200 milliseconds or more and 900 milliseconds or less, because it is possible to properly take into account the round-trip time (simple reaction time) for the gaze to move to two objects and the discrimination reaction time for making a judgment.

[0104] The gaze switching count identifying unit 70 identifies the gaze switching count indicating the number of times the user U averted his / her eyes from the same type of object based on the identification result of the viewed object identifying unit 42C (step S64). The gaze switching count identifying unit 70 counts the number of times each type of gaze switching object is extracted as a gaze switching object, sets it as the gaze switching count, and stores it in the storage unit 24. That is, the gaze switching count identifying unit 70 identifies the number of times the same type of gaze switching object is extracted as the gaze switching count. For example, if a type of gaze switching object called a spider is extracted five times at different times, the gaze switching count identifying unit 70 identifies the gaze switching count of the spider as five times.

[0105] The gaze switching count specifying unit 70 may reset the gaze switching count to zero after a predetermined period has elapsed. That is, the gaze switching count specifying unit 70 may determine the number of times that the same type of gaze switching object is extracted by the visual object specifying unit 42C within the predetermined period as the gaze switching count. The predetermined period may be set arbitrarily, and may be, for example, one month or one year.

[0106] The display mode setting unit 46C sets the display mode of the sub-image PS based on the number of times of gaze switching. The display mode setting unit 46C sets the display mode for the object of the same type as the gaze switching object for which the number of times of gaze switching has been counted, based on the number of times of gaze switching. Specifically, as shown in FIG. 16, the display mode setting unit 46C judges whether the number of times of gaze switching is a predetermined number or more (step S66), and if it is the predetermined number or more (step S66; Yes), sets the display mode of the sub-image PS for the object of the same type as the gaze switching object to the non-display mode (step S68). For example, if the number of times of gaze switching for an object of the type spider is the predetermined number or more, the display mode of the sub-image PS for the spider is set to the non-display mode.

[0107] On the other hand, if the number of times the line of sight has been changed is not equal to or greater than the predetermined number (step S66; No), that is, if the number of times the line of sight has been changed is less than the predetermined number, the display mode is not set to the non-display mode and the process proceeds to step S70. If the process is to be ended (step S70; Yes), the process is ended, and if the process is not to be ended (step S70; No), the process returns to step S60 and is repeated.

[0108] Note that the display mode setting unit 46C may stop the non-display mode setting and switch to a mode in which the sub-image PS is displayed when a predetermined period of time has elapsed since the non-display mode was set.

[0109] (Sub-image display processing) The output control unit 48C causes the display unit 22 to display the sub-image PS related to the visual object extracted by the visual object specifying unit 42C (the visual object within the visual field of the user U) based on the display mode set by the display mode setting unit 46C. For example, similar to the first embodiment, when the display mode of the sub-image PS related to the type of the extracted visual object is the highlight mode, the output control unit 48C causes the sub-image PS to be displayed in the highlight mode, and when the display mode of the sub-image PS related to the type of the extracted visual object is the normal mode, the output control unit 48C causes the sub-image PS to be displayed in the normal mode. On the other hand, when the display mode of the sub-image PS related to the extracted visual object is the non-display mode, the output control unit 48C does not cause the sub-image PS related to the visual object to be displayed.

[0110] (effect) As described above, the display device 10 according to the present embodiment includes the display unit 22 that displays an image, the imaging unit 28, the visual object specification unit 42C, the line of sight switching count specification unit 70, and the output control unit 48C. The imaging unit 28 captures an image of an object within the visual field of the user U. The visual object specification unit 42C specifies the type of object (line of sight switching object) to which the user U averts his / her eyes within a predetermined time based on the image captured by the imaging unit 28. The visual object specification unit 70 specifies the number of times of visual sight switching, which is the number of times the user U averts his / her eyes from the same type of object, based on the specification result of the visual object specification unit 42C. The output control unit 48C causes the display unit 22 to display an image so as to be superimposed on the main image PM within the visual field of the user U viewed through the display unit 22. When the number of times of visual sight switching reaches a predetermined number or more, the output control unit 48C stops displaying the sub-image PS including information on the type of object.

[0111] Here, there may be an object that the user U does not want to see. If a sub-image PS for the object that the user U does not want to see is displayed, the user U may become aware of the object that the user does not want to see, which may upset the user U's mood. In response to this, the display device 10C according to the present embodiment stops displaying the sub-image PS for an object that the user U has looked away from many times. Therefore, the display device 10B according to the third embodiment makes it possible to appropriately adjust the display mode according to the user U's behavior while suppressing upset of the user U's mood.

[0112] Furthermore, when the number of times the line of sight has been switched is less than a predetermined number of times, the output control unit 48C causes the display unit 22 to display the sub-image PS so as to be superimposed on the main image PM including that type of object. The display device 10C according to this embodiment can appropriately adjust the display mode in accordance with the behavior of the user U by displaying the sub-image PS for objects for which the number of times the user has looked away is small.

[0113] Moreover, the predetermined time is preferably 200 milliseconds or more and 900 milliseconds or less. By setting the predetermined time in this range, it becomes possible to more appropriately adjust the display mode in accordance with the behavior of the user U by appropriately taking into account the simple reaction time and the discrimination reaction time.

[0114] Although the embodiments of the present invention have been described above, the embodiments are not limited to the contents of these embodiments. The above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined, and the configurations of each embodiment can also be combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the above-described embodiments. [Explanation of symbols]

[0115] 10 Display device 22 Display section 28 Imaging unit 30 GNSS receiver 40 Image acquisition unit 42 Visual Identification Department 44 Number of views specified 46 Display mode setting section 48 Output control section 50 Location information acquisition section 52 Visit count specification part 60 Number of overlaps specified 70 Eye gaze switching count determination section PM Main Statue PS Sub Image

Claims

1. A display unit for displaying an image; An imaging unit that captures an image of an object within a user's visual field; an output control unit that causes the display unit to display the user's visual field range as a main image by transmitting external light through the display unit or by the imaging unit capturing an image; a visual object identification unit that identifies the type of object within the user's visual field by acquiring information indicating the type of object from the image captured by the imaging unit; a visual recognition count specifying unit that specifies a visual recognition count, which is the number of times that the visual recognition unit specifies that the information indicating the type of the object acquired at different times is the same; A location information acquisition unit that acquires location information of the user; a visit count determination unit that determines a visit count, which is the number of times the user has visited the same place, based on the location information of the user; and an overlap count specification unit that specifies an overlap count, which is the number of times the user visually recognizes the same type of object at the same place, based on the identification result of the visual object specification unit and the user's position information, the output control unit causes the display unit to display an image consisting of information about the object including at least one of a type of the object, an icon indicating the object, a sentence explaining the object, and information about the object of the same type that has been previously extracted, as a sub-image superimposed on the main image, based on the number of views and the number of visits; When the number of overlaps is equal to or greater than a predetermined number, the output control unit stops displaying the sub-image for the object. Display device.

2. The display device according to claim 1 , wherein the output control unit emphasizes the sub-image to be displayed when the number of views is equal to or greater than a predetermined number of times compared to the sub-image to be displayed when the number of views is less than a predetermined number of times.

3. A display method for displaying an image on a display unit, comprising: capturing an image of an object within a user's field of view; A step of displaying the visual field of the user as a main image on the display unit by transmitting external light through the display unit or by an imaging unit capturing an image; Identifying a type of object within the user's visual field by acquiring information indicating the type of object from the captured image; A step of determining a number of times of visual recognition, which is the number of times that information indicating the type of the object obtained at different times is determined to be the same; acquiring location information of the user; determining a visit count, which is the number of times the user has visited the same location, based on the location information of the user; A step of identifying a number of overlaps, which is the number of times the user visually recognizes the same type of object at the same location, based on the result of identifying the type of object and the user's location information; a step of displaying, on the display unit, an image consisting of information about the object including at least one of a type of the object, an icon indicating the object, a sentence explaining the object, and information about the object of the same type that has been previously extracted, as a sub-image superimposed on the main image, based on the number of views and the number of visits; If the number of overlaps is equal to or greater than a predetermined number, stopping the display of the sub-image for the object; Including, Display method.

4. A program for causing a computer to execute a display method for displaying an image on a display unit, capturing an image of an object within a user's field of view; A step of displaying the visual field of the user as a main image on the display unit by transmitting external light through the display unit or by an imaging unit capturing an image; Identifying a type of object within the user's visual field by acquiring information indicating the type of object from the captured image; A step of determining a number of times of visual recognition, which is the number of times that information indicating the type of the object obtained at different times is determined to be the same; acquiring location information of the user; determining a visit count, which is the number of times the user has visited the same location, based on the location information of the user; A step of identifying a number of overlaps, which is the number of times the user visually recognizes the same type of object at the same location, based on the result of identifying the type of object and the user's location information; a step of displaying, on the display unit, an image consisting of information about the object including at least one of a type of the object, an icon indicating the object, a sentence explaining the object, and information about the object of the same type that has been previously extracted, as a sub-image superimposed on the main image, based on the number of views and the number of visits; If the number of overlaps is equal to or greater than a predetermined number, stopping the display of the sub-image for the object; to cause a computer to execute program.

Images