Device, method and recording medium

Abstract

The present invention provides an apparatus, a method and a recording medium, wherein the apparatus comprises: a detection unit for detecting an object viewed by a subject; a storage unit for storing each of a plurality of conditions occurring in a scene in association with historical record data of an object viewed by the subject according to the occurrence of the condition; and an output unit for outputting information for identifying an object associated with the condition in the historical record data when any one of the plurality of conditions occurs.

Description

Technical Field

[0001] The present invention relates to an apparatus, a method and a recording medium. Background Art

[0002] Patent Document 1 states that “the electronic device may include a sight line detection unit that detects a user's sight line, and the control unit may obtain an area in a web page that the user is paying attention to based on the user's sight line information obtained from the sight line detection unit.”

[0003] Patent Document 1: Japanese Patent Publication No. 2015-191551 Summary of the invention

[0004] In order to solve the above-mentioned problems, a device is provided in a first embodiment of the present invention. The device may include a detection unit for detecting an object viewed by a subject. The device may include a storage unit, which associates the object viewed by the subject with each of the multiple conditions in a scene according to the occurrence of each of the multiple conditions and stores them as history data in a storage mode. The device may include an output unit, which outputs information for identifying the object associated with the one of the multiple conditions when any one of the multiple conditions associated with the object stored in the history data occurs in an output mode.

[0005] The detection unit may detect the object viewed by one subject person, and the output unit may output the information to other subject persons.

[0006] The storage unit can store the multiple objects in coordination with the detection order of the detection unit. The output unit can output the information of the multiple objects in sequence according to the detection order of the detection unit.

[0007] The device may further include a second detection unit for detecting a change in a condition in the scene. The second detection unit may detect that the condition in the scene has changed based on at least one of an operation of a target device and a change in a measurement value of a sensor crossing a threshold.

[0008] The output unit may output the information for identifying the target by at least one of displaying the information for identifying the target on a display device worn by the target person, brightening a display screen of the target device, and indicating the target device with sound.

[0009] The output unit may output information for identifying the position of the object.

[0010] In a second aspect of the present invention, a method is provided. The method may include a detection stage of detecting an object viewed by a subject. The method may include a storage stage of associating the object viewed by the subject with each of the multiple conditions in a scene according to the occurrence of each of the multiple conditions and storing the object as historical record data in a storage mode. The method may include an output stage of outputting information for identifying the object associated with the one of the multiple conditions stored in the historical record data when any one of the multiple conditions associated with the object occurs in an output mode.

[0011] In a third aspect of the present invention, a recording medium having a program recorded thereon is provided. The computer functions as a detection unit for detecting an object viewed by a subject person by executing the program. The computer functions as a storage unit for associating the object viewed by the subject person with each of the multiple conditions according to the occurrence of each of the multiple conditions in a scene in a storage mode and storing the object as history data by executing the program. The computer functions as an output unit for outputting information for identifying the object associated with one of the multiple conditions stored in the history data when any one of the multiple conditions associated with the object occurs in an output mode by executing the program.

[0012] In addition, the above summary of the invention does not list all the essential features of the present invention. In addition, sub-combinations of these feature groups may also constitute inventions. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A maintenance management system 1 according to this embodiment is shown.

[0014] Figure 2 The output device 17 is shown.

[0015] Figure 3 The operation of the output device 17 is shown.

[0016] Figure 4 Other operations of the output device 17 are shown.

[0017] Figure 5 Indicates the display screen.

[0018] Figure 6 An example of a computer 2200 is shown in which various aspects of the present invention may be implemented in whole or in part.

[0019] Description of Reference Numerals

[0020] 1 Maintenance management system, 11 Equipment, 12 Maintenance terminal, 15 Operation control device, 16 Interface device, 17 Output device, 18 Resource management device, 100 Control network, 171 Camera unit, 172 Detection unit, 173 Acquisition unit, 174 Second detection unit, 175 Input / output control unit, 176 Storage unit, 177 Output unit, 180 Display screen, 2200 Computer, 2201 DVD-ROM, 2210 Main controller, 2212 CPU, 2214 RAM, 2216 Graphics controller, 2218 Display device, 2220 Input / output controller, 2222 Communication interface, 2224 Hard disk drive, 2226 DVD-ROM drive, 2230 ROM, 2240 Input / output chip, 2242 Keyboard. DETAILED DESCRIPTION

[0021] The present invention will be described below by way of the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all combinations of features described in the embodiments are essential to the solution means of the invention.

[0022] Figure 1 The maintenance management system 1 of this embodiment is shown. The maintenance management system 1 performs maintenance management of a factory and includes a plurality of devices 11 , a maintenance terminal 12 , an operation control device 15 , an interface device 16 , an output device 17 , and a resource management device 18 .

[0023] Here, as factories, in addition to chemical and other industrial factories, there are factories that manage and control wellheads and their surroundings such as gas fields or oil fields, factories that manage and control power generation such as hydropower, thermal power, and nuclear power, factories that manage and control environmental power generation such as solar power or wind power, and factories that manage and control water supply and drainage or dams, etc. Some of the multiple devices 11 and the maintenance terminal 12 can be configured at the site where the process is executed in the factory. For example, there are pipelines that allow the measured fluid to flow, and flow meters that are installed in the pipeline and measure the flow rate of the fluid, etc. at the site. The operation control device 15, other parts of the multiple devices 11, the interface device 16, the output device 17, and the resource management device 18 can be configured at the management center of the factory.

[0024] The plurality of devices 11 are tools, machines or devices, for example, sensors that measure physical quantities such as pressure, temperature, pH, speed, flow rate, etc. in a factory process, actuators such as valves, flow control valves, on-off valves, pumps, fans, motors, heating devices, cooling devices, etc. that control any physical quantity, sound equipment such as microphones and speakers that collect abnormal sounds in the factory or issue alarms, position detection equipment that outputs position information of each device, pipes that allow fluid to flow, switches, cameras, PCs (Personal Computers), etc. that are arranged in a room such as a management center, etc., or other equipment. Each device 11 in the plurality of devices 11 may be of different types, or at least two or more devices 11 may be of the same type.

[0025] At least some of the plurality of devices 11 can be connected to the operation control device 15 via the control network 100 in a wired or wireless manner. The communication within the control network 100 can be digital communication or a mixed communication in which a digital signal is superimposed on an analog signal (4-20 mA signal, etc.), and can be at a speed of about 1000 bps to 10000 bps (1200 bps, 2400 bps as an example). The communication within the control network 100 can be performed, for example, by a wireless communication protocol of ISA (International Society of Automation), and can be performed, for example, by ISA100, HART (Highway Addressable Remote Transducer) (registered trademark), BRAIN (registered trademark), FOUNDATION Fieldbus, PROFIBUS, etc.

[0026] Each device 11 may have inherent identification information (also referred to as device inherent information). The device inherent information is information used to uniquely identify the device 11. In the present embodiment, as an example, it may be at least one of a serial number assigned to the device 11 through a communication protocol (as an example, HART), a serial number set by the manufacturer of the device 11, and a device ID assigned by a user.

[0027] The maintenance terminal 12 accesses the setting parameters of some of the multiple devices 11 to refer to, set, and change the setting parameter values. The maintenance terminal 12 may be a handheld terminal (HHT) carried by the subject person on site (for example, a smartphone or a tablet PC), or a fixed PC. When the maintenance terminal 12 is a handheld terminal, the maintenance terminal 12 may be detachably connected to the device 11.

[0028] The operation control device 15 communicates with some of the multiple devices 11 to control the process. For example, the operation control device 15 obtains a process value as measurement data from the device 11 as a sensor and drives the device 11 as an actuator. In addition, the operation control device 15 can supply the process value to the interface device 16 and receive the target value of the process value from the interface device 16. In addition, in this embodiment, as an example, it is explained that the maintenance management system 1 is equipped with one operation control device 15 to control multiple devices 11, but it is also possible to have multiple operation control devices 15, each of which performs distributed control on a part of the devices 11. The operation control device 15 can be an FCS (Field Control Station) as an example.

[0029] The interface device 16 displays various data in the factory on the display screen and provides an interface between the subject and the factory. The interface device 16 can control the process of the factory through the operation control device 15 according to the operation of the subject. For example, the interface device 16 can receive the process value from the operation control device 15 and supply the target value of the process value to the operation control device 15. In addition, the interface device 16 can change the setting parameter value of the device 11 through the operation control device 15. In addition, the interface device 16 can store at least a part of the device 11 in association with the setting parameter value of the device 11. As an example, the interface device 16 can be an HIS (Human Interface Station) and can be composed of a PC or the like.

[0030] The output device 17 is an example of a device that outputs various information to the subject person in various conditions of the factory. For example, the output device 17 can display the object that the subject person should watch in various conditions. The output device 17 can be composed of a wearable device, a PC, a camera, etc. as an example. In this embodiment, the output device 17 is connected to the factory equipment 11, the resource management device 18, the interface device 16, the operation control device 15, and the maintenance terminal 12 (hereinafter also referred to as an external device) so as to be communicable, but it is not necessary to be connected to the external device.

[0031] The resource management device 18 performs online monitoring and centralized management of the factory. For example, the resource management device 18 can manage data (for example, setting parameter values ​​or process values) of the equipment 11 acquired by the operation control device 15. The resource management device 18 can be composed of a PC or the like.

[0032] Figure 2The output device 17 is shown. The output device 17 stores the object viewed by the subject in the storage mode, and displays the object that the subject should view in the output mode. The output device 17 includes an imaging unit 171, a detection unit 172, an acquisition unit 173, a second detection unit 174, an input / output control unit 175, a storage unit 176, and an output unit 177.

[0033] The camera unit 171 captures the subject in order to detect the subject's line of sight. The camera unit 171 captures a portion of the subject (eyeballs, head, etc.). The camera unit 171 may be a glasses-type wearable device, a camera provided on the periphery of the display screen of the output device 17 or an external device, or a surveillance camera provided on the ceiling or wall of a management center, etc. The camera unit 171 may supply the captured data to the detection unit 172.

[0034] The detection unit 172 detects the object that the subject is looking at. The detection unit 172 can detect the line of sight by analyzing the image supplied from the camera unit 171. The detection unit 172 can detect the gaze position of both eyes from the image of the eyeball, and detect the straight line connecting the middle point of both eyes and the gaze position of the subject as the subject's line of sight. The detection unit 172 can also detect the direction of the subject's head based on the position of the subject's eyes, nose, mouth, etc., and detect the straight line of the detected direction as the subject's line of sight. The detection unit 172 can detect the area on the detected line of sight or the device 11 as the object. The detection unit 172 can detect the object as a coordinate range, in which case, there can be a table indicating the correspondence between the coordinate range and the device 11. The detection unit 172 can further detect the position of the subject based on GPS or images, and detect the object based on the position and line of sight of the subject. The detection unit 172 can supply data representing the object to the input / output control unit 175 and the second detection unit 174.

[0035] The acquisition unit 173 acquires input related to the status in the factory from the subject or the external device. The acquisition unit 173 can receive various data from the external device in the factory in a wired or wireless manner. The acquisition unit 173 may include at least one of a keyboard, a touch panel, a communication terminal, and a button. The acquisition unit 173 may supply the acquired content to the second detection unit 174.

[0036] Here, the conditions may include at least one of the subject's exit / entry relative to a building or room such as a management center, arrival at a predetermined time, the subject's performance of work in the factory (as an example, prescribed operations on external devices), and detection of an abnormality in the process value of equipment 11 in the factory.

[0037] The second detection unit 174 detects a change or occurrence of a condition in the scene based on the content acquired from the acquisition unit 173. The second detection unit 174 can detect that a condition in the scene has changed based on at least one of an operation on an external device (such as the interface device 16 as an example) as a target and a change in a measurement value of the external device (such as the sensor device 11 as an example) crossing a threshold. The second detection unit 174 can use the data from the detection unit 172 to detect the object viewed by the subject, and detect that a condition has changed based on the judgment that the subject has viewed a specific object. The second detection unit 174 can also further detect a change in the scene. The second detection unit 174 can store conditions (such as a threshold value or range as an example) for determining a change or occurrence of a condition for each condition. The second detection unit 174 can supply data indicating a detected change or occurrence of a condition to the input / output control unit 175.

[0038] Here, the scene can be a specified space in a factory. As an example, it can be the shooting range of the camera unit 171, a building, a room, a workplace in the factory, an installation area of ​​one or more devices 11, and at least one of the display screens of the output device 17 or an external device.

[0039] The input / output control unit 175 controls input / output of data stored in the storage unit 176. In the storage mode, the input / output control unit 175 can associate the status with the object viewed by the subject (for example, the coordinate range viewed by the subject, the device 11 viewed by the subject, etc.) and store it in the storage unit 176. The input / output control unit 175 can determine the stored object based on the data indicating the object supplied from the detection unit 172. The input / output control unit 175 can determine the current status based on the data supplied from the second detection unit 174.

[0040] In the output mode, the input / output control unit 175 can obtain information for identifying an object associated with the situation from the storage unit 176 and supply it to the output unit 177. The input / output control unit 175 can switch between the storage mode and the output mode according to the mode input by the subject via the acquisition unit 173.

[0041] Here, the information for identifying the object may include at least one of a coordinate range of the object, device-specific information of the object, an installation area of ​​the device 11 of the object, and information indicating any one window displayed on the display screen.

[0042] The storage unit 176 stores each of the multiple situations occurring in the scene in association with the history data of the object viewed by the subject according to the occurrence of the situation. The storage unit 176 can store the data supplied from the input / output control unit 175 as the history data in the storage mode. The storage unit 176 can store the multiple objects in accordance with the detection order of the detection unit 172.

[0043] When any one of a plurality of conditions occurs, the output unit 177 outputs information for identifying an object associated with the condition in the history data supplied from the input / output control unit 175. The output unit 177 may output information for identifying the position of an object. The output unit 177 may output information of a plurality of objects in sequence according to the detection order of the detection unit 172. The output unit 177 may output information for identifying an object by displaying the information for identifying an object on a display device (e.g., a head mounted display) worn by the subject, brightening the display screen of an external device such as the device 11 as the subject, and representing the external device such as the device 11 as the subject with sound. The output unit 177 may be connected to an external device in a wired or wireless manner to output information for identifying an object. In addition, the output unit 177 may include a display screen or a speaker, etc., for outputting the information.

[0044] Figure 3 The operation of the output device 17 is shown. The output device 17 supports the maintenance management of the factory by performing the processing of steps S11 to S17. Steps S11 to S17 show the operation in the storage mode of the output device 17. In addition, this operation can be started in accordance with the startup of the factory.

[0045] In step S11, the output device 17 determines whether the condition for starting storage is met. If a specified input is made from the subject or the external device via the acquisition unit 173, the output device 17 can start the storage action. The specified input may include at least one of an input of a specific condition, an input of a specific scene, an input of identification information of the subject, and an input for setting a storage mode. In addition, the output device 17 may also start the storage action when a predetermined time is reached or when the second detection unit 174 detects a change in the condition. When the condition for starting storage is met (step S11; yes), the processing can be transferred to step S13. When the condition for starting storage is not met (step S11; no), it is possible to wait until the condition is met.

[0046] In step S13, the detection unit 172 detects the object viewed by the subject person based on the data supplied from the camera unit 171. The detection unit 172 may pre-store a plurality of predetermined detection objects (as an example, coordinate ranges, device-specific information, etc.), and detect the object when the subject person views the object. The detection unit 172 may pre-store a plurality of predetermined detection objects for each scene. When the coordinate range represented by the data supplied from the camera unit 171 coincides with at least a portion of the predetermined detection object, the detection unit 172 may detect the predetermined detection object as the object viewed by the subject person. The detection unit 172 may detect as an object an area or device 11 that has been viewed by the subject person for more than a predetermined period (e.g., 1 second).

[0047] In step S15, the input / output control unit 175 associates the detected object with the current situation in the current scene and stores it in the storage unit 176 as history data. The input / output control unit 175 may further associate and store at least one of the identification information of the subject and the detection order of the object with the history data. When the input / output control unit 175 receives a selection to delete any one of the plurality of objects in the history data from the subject via the acquisition unit 173, the selected object may be deleted from the history data in the storage unit 176.

[0048] In step S17, the second detection unit 174 detects whether the condition has changed. The second detection unit 174 can receive various data from an external device of the factory via the acquisition unit 173, and detect the condition change based on the data. The second detection unit 174 can detect the condition change using different measurement values ​​and / or different thresholds according to the scene or condition. As an example, in the case where the temperature sensor in the factory indicates an abnormal value, the second detection unit 174 can detect that the condition has changed when the measured temperature from the temperature sensor becomes within the normal range. In addition, in the case where a signal indicating that the operation is completed is received from an external device such as the interface device 16 via the acquisition unit 173, the second detection unit 174 can detect that the condition has changed. In addition, when a predetermined moment or a predetermined period has passed, the second detection unit 174 can detect that the condition has changed.

[0049] If the second detection unit 174 detects a change in the condition (step S17; yes), the storage mode may be terminated. If the second detection unit 174 does not detect a change in the condition (step S17; no), the object may be detected and the storage mode may be continued until the condition is satisfied. In addition, if a new condition or scene is detected by the second detection unit 174, the output device 17 may start the storage mode for the new condition or scene from step S13.

[0050] Figure 4Indicates other actions of the output device 17. The output device 17 supports the maintenance management of the factory by performing the processing of steps S21 to S27. Steps S21 to S27 indicate actions in the output mode of the output device 17. In addition, this action can be started by starting the factory. Here, the subject in the processing of steps S21 to S27 may be different from the subject in the processing of steps S11 to S17 in the storage mode.

[0051] In step S21, the second detection unit 174 detects whether a condition has occurred. The second detection unit 174 can detect the occurrence of a condition in the same manner as step S17. As an example, when the measurement value of the temperature sensor in the factory exceeds a threshold value, the second detection unit 174 can detect that a specific condition has occurred. In addition, when a signal indicating the start of an operation is received from an external device such as the interface device 16 via the acquisition unit 173, the second detection unit 174 can detect that a condition has occurred. In addition, if an input indicating a condition that a prescribed operation has started is received from the subject via the acquisition unit 173, the second detection unit 174 can detect that a condition has occurred.

[0052] In step S23, the input / output control unit 175 specifies a plurality of objects corresponding to the conditions detected by the second detection unit 174, and acquires data indicating the objects and the detection order from the history data in the storage unit 176. The input / output control unit 175 supplies the acquired data to the output unit 177.

[0053] In step S25, the output unit 177 outputs information for identifying multiple objects in the order of detection. After outputting information representing one object, the output unit 177 may output information representing the next object in the order of detection when the second detection unit 174 receives an input indicating that the operation corresponding to the object is completed from the subject or an external device. As an example, the output unit 177 may cause the head-mounted display worn by the subject to display the name, setting area, etc. of the device 11 as the object. In addition, as an example, the output unit 177 may output control information for brightening the window of the object, enlarging the window of the object, or changing the color of the window of the object in the display screen of the device 11 as the object.

[0054] In step S27, the second detection unit 174 may detect a change in the state in the same manner as step S17. When the second detection unit 174 detects a change in the state (step S27; yes), the input / output control unit 175 may determine an object corresponding to the changed state. When the second detection unit 174 does not detect a change in the state (step S27; no), the process may return to step S25 and output information until the state changes. In addition, when the changed state detected by the second detection unit 174 does not meet the output condition (for example, when the state is within the normal range), the output device 17 may end the output mode.

[0055] According to the above-described output device 17 , it is possible to store operations performed by an expert and to efficiently transfer the same operations performed by the expert to a novice.

[0056] Figure 5 The display screen 180 is shown as an example of an object in the output mode. The display screen 180 may be a display screen of the output device 17 or a display screen of an external device in the factory. In this case, the camera unit 171 may be provided at the end of the display screen 180 to capture the eyes of the object viewing the display screen 180 in order to detect the line of sight.

[0057] The process values ​​of a portion of the devices 11 selected by the subject person among the devices 11 in the factory can be displayed on the display screen 180. For example, a selection area 1641 for selecting the installation area of ​​the device 11 in the factory and a data display area 1642 for displaying the process values ​​of the devices 11 belonging to the selected installation area can be provided on the display screen 180. When an abnormality occurs in the process value of any device 11 and the output device 17 determines that a situation has occurred, the output unit 177 can change the display method of the data display area 1642 of the process value.

[0058] In this figure, as an example, in the selection area 1641, the buildings and rooms in the factory are displayed as the options for the installation area of ​​the device 11, and "Room a" of "Building B" is selected, and the history of the process values ​​of "Device (1)" to "Device (4)" installed in the installation area is displayed in the data display areas 1642 (1) to 1642 (4). In addition, when an abnormality occurs in the process value of "Device (1)" and the data display area 1642 (1) is the object to be viewed, the background color of the data display area 1642 (1) is changed through the control of the output unit 177. In addition, since it is detected that the line of sight of the subject is in the area 1643, the output device 17 determines that the situation has not changed and continues to change the background color of the data display area 1642 (1).

[0059] In addition, in the present embodiment, the output device 17 executes the output mode and the storage mode for different subjects at different times, but the output mode and the storage mode may be executed in parallel for the same subject. As an example, when the output device 17 detects that the subject has viewed a new object that is not stored in the storage unit 176 in the output mode, the new object may be stored in the storage mode.

[0060] In addition, the camera unit 171 can be installed on the subject (e.g., head, shoulders, etc.) to capture the direction the subject is facing. For example, the camera unit 171 can capture a code (e.g., a barcode, a QR code (registered trademark)) installed on the device 11, the detection unit 172 recognizes the device 11 through the code, and the input / output control unit 175 stores the device 11 as an object in the storage unit 176.

[0061] In addition, the output device 17 may not include the camera unit 171. In this case, the detection unit 172 may receive data for detecting the line of sight of the subject from an external device (eg, a wearable device, a camera, etc.).

[0062] Furthermore, the output device 17 may be a part of the plant equipment 11 , the resource management device 18 , the interface device 16 , the operation control device 15 , or the maintenance terminal 12 .

[0063] In addition, various embodiments of the present invention may be described with reference to flow charts and block diagrams, where a module may represent (1) a stage of a process for performing an operation or (2) a portion of a device having the function of performing an operation. Specific stages and portions may be implemented by dedicated circuits, programmable circuits supplied together with computer-readable instructions stored on a computer-readable medium, and / or processors supplied together with computer-readable instructions stored on a computer-readable medium. Dedicated circuits may include digital and / or analog hardware circuits, and may also include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits, which include logical AND, logical OR, logical XOR, logical NAND, logical NOR and other logical operations, flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs) and other memory elements, etc.

[0064] Computer-readable media may include any tangible device capable of storing instructions executed by an appropriate device, and as a result, a computer-readable medium having instructions stored therein includes a product containing instructions that can be executed in order to make a means for performing the operations specified by the flowchart or block diagram. Examples of computer-readable media may include: electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable media may include: floppy (registered trademark) disks, magnetic disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), electrically erasable programmable read-only memories (EEPROM), static random access memories (SRAM), compact disk read-only memories (CD-ROM), digital versatile disks (DVD), Blu-ray (RTM) disks, memory sticks, integrated circuit cards, etc.

[0065] Computer readable instructions include any of source code and object code described by any combination of one or more programming languages ​​including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or object-oriented programming languages ​​such as Smalltalk (registered trademark), JAVA (registered trademark), C++, and existing procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0066] The computer readable instructions may be provided to a processor or programmable circuit of a general-purpose computer, a special-purpose computer or other programmable data processing device via a local or local area network (LAN), a wide area network (WAN) such as the Internet, and the computer readable instructions are executed to produce a means for performing the operations specified by the flowchart or block diagram. Examples of processors include: computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0067] Figure 6 The computer 2200 is an example of a computer that can implement various modes of the present invention in whole or in part. The computer 2200 can perform operations associated with the device of the embodiment of the present invention or functions of one or more parts of the device, or perform the operations or the one or more parts, and / or the computer 2200 can perform processes of the embodiment of the present invention or stages of the processes, through the program installed in the computer 2200. In order to make the computer 2200 perform specific operations associated with some or all of the modules in the flowcharts and block diagrams described in this specification, the CPU 2212 can execute such a program.

[0068] The computer 2200 of this embodiment includes a CPU 2212, a RAM 2214, a graphic controller 2216, and a display device 2218, which are connected to each other through a main controller 2210. The computer 2200 also includes an input / output unit such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the main controller 2210 via an input / output controller 2220. The computer also includes a conventional input / output unit such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0069] The CPU 2212 controls each unit by operating according to the program stored in the ROM 2230 and the RAM 2214. The graphic controller 2216 acquires image data generated by the CPU 2212 in a frame buffer or the like provided in the RAM 2214 or in itself, and displays the image data on the display device 2218.

[0070] The communication interface 2222 can communicate with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides the programs or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0071] The ROM 2230 stores therein a boot program or the like executed by the computer 2200 upon activation and / or a program that depends on the hardware of the computer 2200. The input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0072] The program is provided by a computer-readable medium such as DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium and installed in the hard disk drive 2224, RAM 2214 or ROM 2230, which is also an example of a computer-readable medium, and is executed by the CPU 2212. The information processing described in these programs is read to the computer 2200, thereby bringing about cooperation between the program and the various types of hardware resources described above. The device or method can be constituted by realizing the operation or processing of information with the use of the computer 2200.

[0073] For example, when communication is performed between the computer 2200 and an external device, the CPU 2212 can execute a communication program loaded in the RAM 2214, and instruct the communication interface 2222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 2212, the communication interface 2222 reads the transmission data stored in the transmission buffer processing area provided in the RAM 2214, the hard disk drive 2224, the DVD-ROM 2201, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes the reception data received from the network to the reception buffer processing area provided on the recording medium, etc.

[0074] Furthermore, the CPU 2212 can read all or a necessary part of a file or database stored in an external recording medium such as the hard disk drive 2224, the DVD-ROM drive 2226 (DVD-ROM 2201), or an IC card into the RAM 2214, and perform various types of processing on the data on the RAM 2214. Then, the CPU 2212 writes the processed data back to the external recording medium.

[0075] Various types of information such as various types of programs, data, tables, and databases can be stored in the recording medium and receive information processing. CPU2212 performs various types of processing described in various places of this disclosure on the data read from RAM2214 and writes the results back to RAM2214. The various types of processing include various types of operations specified by the instruction sequence of the program, information processing, conditional judgment, conditional branching, unconditional branching, information retrieval / replacement, etc. In addition, CPU2212 can retrieve information in files, databases, etc. in the recording medium. For example, in the case where multiple entries having attribute values ​​of the first attribute associated with the attribute values ​​of the second attribute are stored in the recording medium, CPU2212 can retrieve an entry consistent with the condition specifying the attribute value of the first attribute from the multiple entries, and read the attribute value of the second attribute stored in the entry, thereby obtaining the attribute value of the second attribute associated with the first attribute that meets the predetermined condition.

[0076] The program or software module described above may be stored in a computer-readable medium on or near the computer 2200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the program to the computer 2200 via the network.

[0077] The present invention has been described above using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. According to the description of the claims, the method of making such changes or improvements can also be included in the technical scope of the present invention.

[0078] The execution order of each process such as actions, processes, steps and stages in the apparatus, system, program and method shown in the claims, specifications and drawings is not specifically indicated as "earlier", "before", etc. In addition, it should be noted that as long as the output of the previous process is not used in the subsequent process, it can be implemented in any order. Even if the action flow in the claims, specifications and drawings is described using "first," "next," etc. for the sake of convenience, it does not mean that it must be implemented in this order.

Claims

1. A device, characterized in that include: A detection unit, detecting an area in the sight line of a subject in the scene or a device in the scene as an object viewed by the subject; A storage unit that associates an object viewed by a subject, i.e., an area in the subject's line of sight in the scene, or a device in the scene with each of the plurality of conditions according to occurrence of the plurality of conditions in the scene in a storage mode and stores the association as history data; as well as an output unit, which outputs information for identifying the object associated with any one of the multiple conditions in the historical record data, that is, an area in the line of sight of the subject in the scene or a device in the scene, when any one of the multiple conditions associated with the object in the historical record data occurs in the output mode, In the storage mode, The detection unit detects an object viewed by a subject, that is, an area in the subject's line of sight in the scene or a device in the scene, based on a specific situation occurring in the scene. The storage unit associates the object viewed by the one subject with the specific situation and stores the associated information as the history data. In the output mode, When the specific situation occurs, the output unit outputs information for identifying an object associated with the specific situation in the historical record data to other subjects, that is, outputs information for identifying an object viewed by one of the subjects in the storage mode, that is, an area within the line of sight of one of the subjects in the scene or a device in the scene.

2. The device according to claim 1, characterized in that The storage unit stores a plurality of objects in coordination with the detection order of the detection unit. The output unit sequentially outputs the information on the plurality of objects according to the detection order of the detection unit.

3. The device according to claim 1 or 2, characterized in that: Also includes a second detection unit that detects a change in a condition within the scene, The second detection unit detects the change in the situation in the scene based on at least one of an operation of the target device and a change in a measurement value of a sensor crossing a threshold value.

4. The device according to claim 1 or 2, characterized in that: The output unit outputs the information for identifying the object by displaying the information for identifying the object on a display device worn by the subject, brightening the display screen of the device serving as the object, and indicating the device serving as the object with sound.

5. The device according to claim 1 or 2, characterized in that: The output unit outputs information for identifying the position of the object.

6. A method, characterized in that include: In a detection stage, an area in the line of sight of a subject in the scene or a device in the scene is detected as an object viewed by the subject; A storage stage, in which, according to the occurrence of a plurality of conditions in the scene, an object viewed by the subject, i.e., an area in the subject's line of sight in the scene or a device in the scene is associated with each of the plurality of conditions and stored as history record data in a storage mode; as well as In the output stage, when any one of the multiple conditions associated with the object in the historical record data occurs in the output mode, information for identifying the object associated with the any one of the multiple conditions in the historical record data, that is, the area in the line of sight of the subject in the scene or the equipment in the scene is output, In the storage mode, The detection stage includes a stage of detecting an object viewed by a subject, that is, an area in the subject's line of sight in the scene or a device in the scene according to a specific situation occurring in the scene. The storage stage includes a stage of associating the object viewed by the one subject with the specific situation and storing the object as the history data. In the output mode, The output stage includes outputting information for identifying an object associated with the specific situation in the historical record data to other subjects when the specific situation occurs, that is, outputting information for identifying an object viewed by one of the subjects in the storage mode, that is, an area within the line of sight of one of the subjects in the scene or a device in the scene.

7. A recording medium having a program recorded thereon, characterized in that: The computer functions as a detection unit, a storage unit, and an output unit by executing the program. The detection unit detects an area in the sight line of the subject person in the scene or a device in the scene as an object viewed by the subject person, In the storage mode, the storage unit associates an object viewed by a subject, that is, an area in the subject's line of sight in the scene or a device in the scene with each of the multiple conditions according to the occurrence of the multiple conditions in the scene, and stores the associations as history data. The output unit outputs information for identifying the object associated with any one of the multiple conditions in the historical data, that is, an area in the line of sight of the subject in the scene or a device in the scene, when any one of the multiple conditions associated with the object in the historical data occurs in the output mode, In the storage mode, The detection unit detects an object viewed by a subject, that is, an area in the subject's line of sight in the scene or a device in the scene, based on a specific situation occurring in the scene. The storage unit associates the object viewed by the one subject with the specific situation and stores the associated information as the history data. In the output mode, When the specific situation occurs, the output unit outputs information for identifying an object associated with the specific situation in the historical record data to other subjects, that is, outputs information for identifying an object viewed by one of the subjects in the storage mode, that is, an area within the line of sight of one of the subjects in the scene or a device in the scene.

Images