Information Processing Method and System

Abstract

To provide an information processing method capable of identifying a location where a worker carefully observes based on a brain wave signal measured from a worker, and a system.SOLUTION: An information processing method for identifying a location that a worker carefully observes includes the steps of: acquiring images that represent at least a portion of the worker's field of view at a given time as image data; acquiring the direction of the worker's line of sight at a given time as a gaze data; acquiring brainwave signals of the worker during the time including the predetermined time as brainwave signal data; detecting a predetermined waveform from the acquired brainwave signal data; and determining the worker's visual position when a predetermined waveform occurs based on the image data and the gaze data.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to an information processing method and system.

Background Art

[0002] Conventionally, a device that detects a user's field of view and line of sight and displays information at a position visually recognized by the user is known. For example, Patent Document 1 describes an interface that overlays and displays information required by the user on a region of interest that the user is focusing on within the field of view.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The interface described in Patent Document 1 detects only the user's line of sight to identify the region of interest. Therefore, the interface described in Patent Document 1 cannot determine whether the user is attentively visually recognizing the region of interest.

[0005] An object of the present disclosure is to provide an information processing method and system that identify a position attentively visually recognized by an operator based on an electroencephalogram signal measured from the operator.

Means for Solving the Problems

[0006] An information processing method for identifying the position visually recognized by an operator of the present disclosure includes: a step of acquiring, as image data, an image representing at least a part of the operator's visual field at a predetermined time; a step of acquiring, as gaze data, the gaze direction of the operator at the predetermined time; a step of acquiring, as electroencephalogram signal data, the electroencephalogram signal of the operator in a time period including the predetermined time; a step of detecting a predetermined waveform from the acquired electroencephalogram signal data; and a step of determining the visual recognition position of the operator when the predetermined waveform occurs based on the image data and the gaze data. This is the gist of the present disclosure.

Effect of the Invention

[0007] According to the present disclosure, it is possible to provide an information processing method and system for identifying the position visually recognized by an operator based on the electroencephalogram signal measured from the operator.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings. However, the configurations described below are merely examples of the present disclosure, and the present disclosure is not limited to the following embodiments. Even if they are other than these embodiments, various changes can be made according to the design and the like as long as the technical idea according to the present disclosure is not deviated from.

[0010] (Embodiment 1) FIG. 1 is a block diagram of an example of a system 1 for executing the information processing method according to Embodiment 1 of the present disclosure. The information processing method according to the present embodiment relates to a method for efficiently transmitting a location that a specific operator consciously or unconsciously pays attention to during work to another operator. For example, the specific operator is an operator who is skilled in a specific task, and the other operator is an operator who is less skilled in the specific task than the specific operator (that is, an operator with less experience than the specific operator). The location to be noted is, for example, a position that attracts the attention of the specific operator during work. The information processing method according to the present embodiment acquires image data representing at least a part of the field of view of the specific operator, gaze data representing the gaze direction of the specific operator, and electroencephalogram signal data representing the electroencephalogram signal of the specific operator while the specific operator is working on site. Then, the information processing method determines in advance an electroencephalogram signal induced when the specific operator sees a predetermined event and sets it as a reference waveform. The predetermined event represents, for example, an event that attracts the attention of the specific operator. The method acquires the image data, the gaze data, and the electroencephalogram signal data again when the specific operator works at the work site again. If there is a waveform similar to the reference waveform in the acquired electroencephalogram signal data, the method determines the position that the specific operator was looking at when the waveform occurred based on the image data and the gaze data, and determines the position as a location representing the event that attracted the attention of the specific operator. By applying a video effect to the image data so as to emphasize the location, another operator different from the specific operator can efficiently understand the location that the specific operator feels should be paid attention to by looking at the image data with the video effect applied. In this way, the information processing method can efficiently transmit the location that the specific operator consciously or unconsciously pays attention to during work to another operator different from the specific operator.

[0011] As shown in FIG. 1, the system 1 includes a wearable device 10, a control device 20, and a display device 30. The wearable device 10 is attached to a specific operator. The wearable device 10 includes a device that can be worn on the head and eyes of a specific operator. The wearable device 10 includes an arithmetic circuit 11, a storage device 12, a communication circuit 13, a vision camera 14, a gaze specifying device 15, and an electroencephalogram measuring device 16.

[0012] The arithmetic circuit 11 executes processing in the wearable device 10. The arithmetic circuit 11 includes a general-purpose processor such as a CPU or MPU that realizes a predetermined function by executing a program. The arithmetic circuit 11 is configured to be communicable with the storage device 12, and by calling and executing an arithmetic program or the like stored in the storage device 12, it realizes various processes in the arithmetic circuit 11, such as a process of storing data acquired by the vision camera 14 or the like. The arithmetic circuit 11 is not limited to a mode in which hardware resources and software cooperate to realize a predetermined function, and may be a hardware circuit designed specifically to realize a predetermined function. That is, the arithmetic circuit 11 can be realized by various processors such as a GPU, FPGA, DSP, ASIC, etc., in addition to a CPU and MPU. Such an arithmetic circuit 11 can be composed of, for example, a signal processing circuit that is a semiconductor integrated circuit.

[0013] The storage device 12 is a storage medium that can store various information. The storage device 12 is realized, for example, by appropriately combining memories such as DRAM, SRAM, flash memory, HDD, SSD, and other storage devices. The storage device 12 stores a program for realizing various processes performed by the arithmetic circuit 11 as described above. Further, the storage device 12 stores each data acquired by the vision camera 14, the gaze specifying device 15, and the electroencephalogram measuring device 16 as will be described later.

[0014] The communication circuit 13 is an interface device for connecting to other devices or systems via a communication line by wire or wirelessly. The interface device can perform communication compliant with a wired communication standard such as, for example, USB (registered trademark) or Ethernet (registered trademark). Also, the interface device can perform communication compliant with a wireless communication standard such as, for example, Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile phone line, etc.

[0015] The vision camera 14 is an imaging device including an imaging element such as a CCD or a CMOS. In the present embodiment, the vision camera 14 is configured to image the direction in which a specific operator is facing. The vision camera 14 can acquire, as image data, an image (for example, a moving image) representing at least a part of the field of view of the specific operator. The acquired image data is stored in the storage device 12. The acquired image data may be stored in a storage device (not shown) that the vision camera 14 has and read out as necessary.

[0016] The gaze identification device 15 is an imaging device including an imaging element such as a CCD or a CMOS. The gaze identification device 15 is configured to image at least one eye of a specific operator. The gaze identification device 15 is a device called, for example, an eye tracker or an eye tracking module, and can identify or detect the gaze of a specific operator wearing it using infrared rays and acquire it as gaze data. The acquired gaze data is stored in the storage device 12. The acquired gaze data may be stored in a storage device (not shown) that the gaze identification device 15 has and read out as necessary.

[0017] The electroencephalogram measurement device 16 is a device capable of measuring an electroencephalogram signal, and is, for example, an electroencephalograph. In the present embodiment, the electroencephalogram measurement device 16 is attached to a specific operator and configured to measure the electroencephalogram signal of the specific operator. The electroencephalogram measurement device 16 has a plurality of electrodes. The plurality of electrodes include a measurement electrode and a reference electrode. The plurality of electrodes may include a ground electrode. The measurement electrode may be arranged at any of the sites defined by the international 10-20 method.

[0018] In the system 1 according to the present disclosure, the electroencephalogram measurement device 16 may be configured to measure an event-related potential among the electroencephalograms of a specific operator on whom electrodes are disposed. The measurement electrodes may be disposed, for example, on the top of the head of the specific operator. The reference electrode may be disposed, for example, on the left and right earlobes of the specific operator. The ground electrode may be disposed, for example, on the left and right earlobes of the specific operator. The potential measured by the electroencephalogram measurement device 16 is not limited to the event-related potential and may be other potentials.

[0019] The electroencephalogram measurement device 16 measures the potential difference between a plurality of electrodes as the electroencephalogram of a specific operator. The potential difference measured by the plurality of electrodes is stored in the storage device 12 as measured electroencephalogram data. The electroencephalogram measurement device 16 may amplify the measured potential difference by an amplifier and store it in the storage device 12 as measured electroencephalogram data. The measured potential difference may be stored in a storage device (not shown) included in the electroencephalogram measurement device 16 and read out as necessary.

[0020] The wearable device 10 transmits the data acquired by the visual field camera 14, the line-of-sight specifying device 15, and the electroencephalogram measurement device 16 to the control device 20 by wired or wireless communication via the communication circuit 13.

[0021] The control device 20 is, for example, a computer. The control device 20 includes an arithmetic circuit 21, a storage device 22, an input / output device 23, and a communication circuit 24. The control device 20 stores each data received from the wearable device 10 in the storage device 22 via the communication circuit 24.

[0022] The arithmetic circuit 21 may be configured in the same manner as the arithmetic circuit 11 and executes the processing in the control device 20. The arithmetic circuit 21 is configured to be communicable with the storage device 22, and realizes various processes in the arithmetic circuit 21, such as a process of creating a reference waveform and a process of specifying a point of attention, by calling and executing an arithmetic program and the like stored in the storage device 22.

[0023] The memory device 22 may be configured in the same manner as the memory device 12, and stores a program for realizing various processes performed by the arithmetic circuit 21 as described above.

[0024] The input / output device 23 has functions as an input device for inputting information from a user and an output device for outputting information to the user. The input / output device 23 includes one or more human-machine interface devices. The human-machine interface devices include, for example, input devices such as a keyboard, a pointing device (mouse, trackball, etc.), a touch pad, output devices such as a display, a speaker, and input / output devices such as a touch panel.

[0025] The communication circuit 24 is configured in the same manner as the communication circuit 13.

[0026] The display device 30 is, for example, a display, and can display an image or the like output from the control device 20.

[0027] (Information Processing Method) Next, with reference to FIGS. 1 to 8, an information processing method according to the present disclosure will be described. The information processing method according to the present disclosure can be divided into a plurality of stages. The plurality of stages include a creation stage, a point of attention identification stage, and a transmission stage. The creation stage indicates a stage of creating a reference waveform indicating a specific brain wave signal induced when a specific operator sees a predetermined event. For example, the predetermined event represents an event that attracts the attention of a specific operator. Hereinafter, a position having an event that attracts the attention of a specific operator is also appropriately referred to as a "point of attention". The point of attention identification stage indicates a stage of identifying a point of attention using the reference waveform. The transmission stage indicates a stage of transmitting the point of attention to another operator different from the specific operator. Note that the point of attention may be a place that the specific operator feels should be consciously or unconsciously attended to.

[0028] FIG. 2 is a flowchart showing part of the creation stage of the information processing method according to Embodiment 1. The information processing method according to the present disclosure relates to a method for efficiently communicating to other workers the position where an event that attracts the attention of a specific worker occurs. This method determines the position that the specific worker visually recognized attentively using the brain wave signal measured from the specific worker. In the creation stage, this method determines the waveform (i.e., the reference waveform) of the brain wave signal (hereinafter, appropriately referred to as the "specific brain wave signal") induced when the specific worker visually recognizes the attention point. FIG. 2 is a flowchart showing the stage of collecting data for determining the reference waveform in the creation stage.

[0029] First, the specific worker moves to the work site and wears the wearable device 10. Then, the specific worker turns on the switch of the wearable device 10. When the wearable device 10 starts up, the arithmetic circuit 11 of the wearable device 10 activates the vision camera 14, the gaze specifying device 15, and the electroencephalogram measuring device 16.

[0030] In step S11, the arithmetic circuit 11 acquires, as image data, an image representing at least a part of the field of view of the specific worker by the vision camera 14. The arithmetic circuit 11 stores the acquired image data in the storage device 12. In step S12, the arithmetic circuit 11 acquires the gaze data of the specific worker by the gaze specifying device 15 and specifies the gaze direction. The arithmetic circuit 11 stores the acquired gaze data in the storage device 12. In step S13, the arithmetic circuit 11 acquires electroencephalogram signal data representing the electroencephalogram signal of the specific worker by the electroencephalogram measuring device 16. The arithmetic circuit 11 stores the acquired electroencephalogram signal data in the storage device 12. When the specific worker starts up the wearable device 10, the specific worker starts working at the work site. The arithmetic circuit 11 of the wearable device 10 can acquire the image data at a predetermined time, the gaze data at the predetermined time, and the electroencephalogram signal data in a time period including the predetermined time and store them in the storage device 12.

[0031] Then, when the specific worker finishes working at the work site, the specific worker turns off the switch of the wearable device 10.

[0032] Next, the stage after data collection in the creation stage will be described. FIG. 3 is a flowchart showing other parts of the creation stage of the information processing method according to Embodiment 1.

[0033] First, in step S21, the arithmetic circuit 21 of the control device 20 acquires image data, gaze data, and electroencephalogram signal data from the storage device 12 of the wearable device 10 via the communication circuit 24. The arithmetic circuit 21 stores each acquired data in the storage device 22.

[0034] Next, in step S22, the arithmetic circuit 21 synchronizes each acquired data with respect to time. By synchronizing the image data, gaze data, and electroencephalogram signal data with respect to time, they can be associated with each other. The arithmetic circuit 21 can synchronize each data, for example, using the time information each data has. Therefore, the arithmetic circuit 21 can acquire and store in the storage device 22 the image data at a predetermined time, the gaze data at the predetermined time, and the electroencephalogram signal data in a time period including the predetermined time.

[0035] In step S23, the arithmetic circuit 21 sets a position that is determined to be a location where a specific operator should pay attention, that is, an attention point. The attention point may be set, for example, by a specific operator while referring to the image data stored in the storage device 22 of the control device 20 and operating the input / output device 23 (such as a mouse) of the control device 20. Also, the setting of the attention point does not necessarily need to be performed by the specific operator and may be performed by another operator different from the specific operator.

[0036] In step S24, the arithmetic circuit 21 specifies the visual recognition position of a specific operator in the image data based on the image data and the gaze data. Thereby, the arithmetic circuit 21 can specify which position the specific operator was looking at during the work (i.e., the visual recognition position) after step S13. For example, the arithmetic circuit 21 may create visual recognition position data in which the gaze data is superimposed and displayed on the image data by combining the image data and the gaze data.

[0037] In step S25, the arithmetic circuit 21 specifies the timing at which the specific operator visually recognized the attention point. Then, in step S26, the arithmetic circuit 21 obtains electroencephalogram signal data including a specific electroencephalogram signal representing the electroencephalogram signal of the specific operator when visually recognizing the attention point by extracting the electroencephalogram signal at the timing. That is, in step S26, the arithmetic circuit 21 obtains electroencephalogram signal data including a specific electroencephalogram signal induced when the specific operator sees a predetermined event. Here, the predetermined event represents an event that attracts the attention of the specific operator. The attention point corresponds to the position having the predetermined event. Generally, the electroencephalogram signal induced in the process of decision-making in the event-related potential is called P300, and a large potential change can occur about 300 ms to 500 ms after the occurrence of some trigger. The trigger corresponds to the timing when the specific operator sees a predetermined event (i.e., the timing when visually recognizing the attention point) in the method according to the present disclosure. Therefore, the arithmetic circuit 21 can obtain the event-related potential induced when the specific operator sees a predetermined event by using, for example, the electroencephalogram signal between 200 ms and 600 ms after the timing (i.e., 0 ms).

[0038] Naturally, the period of the obtained electroencephalogram signal is not limited to the above, and the arithmetic circuit 21 may use electroencephalogram signals of other periods, such as electroencephalogram signals between 0 ms and 1000 ms, electroencephalogram signals between 100 ms and 700 ms, or electroencephalogram signals between 300 ms and 500 ms. In step S25, the arithmetic circuit 21 can specify a plurality of timings at which the specific operator visually recognized the attention point. Therefore, the arithmetic circuit 21 can obtain a plurality of specific electroencephalogram signals in step S26.

[0039] In step S27, the arithmetic circuit 21 adds and averages the acquired plurality of specific electroencephalogram signals to create a reference waveform. Generally, since the component related to the event-related potential in the electroencephalogram signal has a small potential change, it may not be possible to obtain a desired waveform due to noise such as other components of the electroencephalogram signal. Therefore, the arithmetic circuit 21 acquires a plurality of electroencephalogram signals related to the event-related potential, and by adding and averaging the plurality of electroencephalogram signals, it is possible to reduce the influence of the component corresponding to the noise and obtain a desired electroencephalogram signal. For example, the arithmetic circuit 21 can create a more appropriate reference waveform by adding and averaging 10 or more specific electroencephalogram signals. The number of specific electroencephalogram signals to be added and averaged is not limited to 10 or more, and may be 13 or more, or may be 20 or more. Also, it may be 5 or more.

[0040] FIG. 4 is a graph showing an example of the waveform of the reference waveform created in the creation stage and the waveform of an electroencephalogram signal different from the reference waveform. In FIG. 4, waveform A indicates the reference waveform, and waveform B indicates the waveform of an electroencephalogram signal different from the reference waveform. The horizontal axis of FIG. 4 indicates time. The vertical axis of FIG. 4 indicates amplitude (that is, potential), with the upper side indicating a potential in the negative direction and the lower side indicating a potential in the positive direction. As shown in waveform A of FIG. 4, a large potential change occurs in the reference waveform about 300 to 500 ms after the input of the trigger.

[0041] By processing as described above, the arithmetic circuit 21 can create a reference waveform determined based on the specific electroencephalogram signal induced when a specific operator sees a predetermined event.

[0042] In the above-described process, the arithmetic circuit 21 sets the attention point after synchronizing the image data, the gaze data, and the electroencephalogram signal data. However, the timing for setting the attention point is not limited to this. For example, the attention point may be set after the arithmetic circuit 21 acquires the image data, the gaze data, and the electroencephalogram signal data. Further, the attention point may be set before the vision camera acquires the image data in S11. In this case, for example, image data related to the work site where a specific worker performs work is acquired in advance, and the attention point can be set by the specific worker or another worker operating the input / output device 23 while referring to the image data. When performing such a process, since the attention point can be grasped in advance, it is possible to pay attention to the specific worker visually recognizing the attention point a plurality of times after step S13. Therefore, the arithmetic circuit 21 can more reliably acquire a plurality of desired specific electroencephalogram signals from the electroencephalogram signal data.

[0043] Next, the attention point specifying stage will be described with reference to FIG. 5. FIG. 5 is a flowchart showing a part of the attention point specifying stage of the information processing method according to Embodiment 1. The flowchart shown in FIG. 5 shows the stage of collecting the image data, the gaze data, and the electroencephalogram signal data in the attention point specifying stage.

[0044] First, the specific worker moves to the work site and wears the wearable device 10. Then, the specific worker turns on the switch of the wearable device 10. When the wearable device 10 is activated, the arithmetic circuit 11 of the wearable device 10 activates the vision camera 14, the gaze specifying device 15, and the electroencephalogram measuring device 16.

[0045] In step S31, the arithmetic circuit 11 acquires, as image data, an image representing at least a part of the field of view of a specific operator by the vision camera 14. The arithmetic circuit 11 stores the acquired image data in the storage device 12. In step S32, the arithmetic circuit 11 acquires the gaze data of the specific operator by the gaze identification device 15 and identifies the gaze direction. The arithmetic circuit 11 stores the acquired gaze data in the storage device 12. In step S33, the arithmetic circuit 11 acquires electroencephalogram signal data representing the electroencephalogram signal of the specific operator by the electroencephalogram measuring device 16. The arithmetic circuit 11 stores the acquired electroencephalogram signal data in the storage device 12. When the specific operator activates the wearable device 10, the operator starts working at the work site. The arithmetic circuit 11 of the wearable device 10 can acquire the image data at a predetermined time, the gaze data at the predetermined time, and the electroencephalogram signal data in a time period including the predetermined time and store them in the storage device 12.

[0046] Then, when the specific operator finishes the work at the work site, the operator turns off the switch of the wearable device 10.

[0047] Next, the stage after data collection in the attention point identification stage will be described. FIG. 6 is a flowchart showing other parts of the attention point identification stage of the information processing method according to the first embodiment.

[0048] First, in step S41, the arithmetic circuit 21 of the control device 20 acquires the image data, the gaze data, and the electroencephalogram signal data from the storage device 12 of the wearable device 10 via the communication circuit 24. The arithmetic circuit 21 stores each acquired data in the storage device 22.

[0049] Next, in step S42, the arithmetic circuit 21 synchronizes the acquired data with respect to time. By synchronizing the image data, the gaze data, and the electroencephalogram signal data with respect to time, the arithmetic circuit 21 can associate them with each other. The arithmetic circuit 21 can synchronize each data, for example, using the time information included in each data. Therefore, the arithmetic circuit 21 can obtain and store in the storage device 22 the image data at a predetermined time, the gaze data at the predetermined time, and the electroencephalogram signal data in a time period including the predetermined time.

[0050] In step S43, the arithmetic circuit 21 detects, based on the reference waveform, an electroencephalogram signal of a predetermined waveform that is induced when a specific operator sees a predetermined event (that is, when visually recognizing a point of attention) in the acquired electroencephalogram signal data. More specifically, the arithmetic circuit 21 determines whether there is a waveform similar to the reference waveform in the acquired electroencephalogram signal data. The arithmetic circuit 21 detects a predetermined waveform from the electroencephalogram signal data when a part of the electroencephalogram signal included in the acquired electroencephalogram signal data is a waveform corresponding to the reference waveform. The predetermined waveform is a waveform corresponding to the reference waveform in the electroencephalogram signal included in the acquired electroencephalogram signal data. The arithmetic circuit 21 can determine whether there is a waveform similar to the reference waveform in the acquired electroencephalogram signal data, for example, by using a cross-correlation function (CCF) or a dynamic time warping method (DTW). Since CCF and DTW are well known, a specific description including the calculation method is omitted. Note that the determination of whether there is a waveform similar to the reference waveform is not limited to the above-described method, and the arithmetic circuit 21 can make the determination using any method.

[0051] When a brainwave signal induced when a specific operator sees a predetermined event is detected (S43: YES), that is, when it is determined that there is a waveform similar to the reference waveform, the arithmetic circuit 21 executes steps S44 and the like. When the brainwave signal is not detected (S43: NO), the arithmetic circuit 21 ends the process. The arithmetic circuit 21 obtains a period during which the potential difference regarding each waveform at a predetermined time is within a predetermined error between the reference waveform and the waveform indicating a part of the brainwave signal, and if the total of the periods is equal to or longer than a predetermined time, it may be determined that the reference waveform and the waveform indicating a part of the brainwave signal are similar. In addition, the arithmetic circuit 21 may determine that the reference waveform and the waveform indicating a part of the brainwave signal are similar when the potential regarding the waveform indicating a part of the brainwave signal is within a predetermined range. The predetermined range may be determined by, for example, the maximum value and the minimum value of the potential in the brainwave signal (that is, the specific brainwave signal) used when creating the reference waveform. However, the method for determining similarity by the arithmetic circuit 21 is not limited to this.

[0052] For example, the arithmetic circuit 21 obtains measurement points at which the potential difference regarding each waveform at a predetermined time is within a predetermined error between the reference waveform and the waveform indicating a part of the brainwave signal. Then, the arithmetic circuit 21 may determine that the reference waveform and the waveform indicating a part of the brainwave signal are similar when the total of the measurement points is equal to or more than a predetermined number. Alternatively, the arithmetic circuit 21 may calculate the DTW distance between the waveform in a predetermined time zone of the reference waveform and the waveform indicating a part of the brainwave signal, and determine that the reference waveform and the waveform indicating a part of the brainwave signal are similar when the calculated value is equal to or less than a predetermined value. The predetermined time zone of the reference waveform is, for example, as described above, the period between 300 ms and 500 ms after the timing (0 ms) when the specific operator sees the predetermined event.

[0053] In step S44, the arithmetic circuit 21 identifies the visual recognition position of a specific operator in the image data based on the image data and the line-of-sight data. Thereby, the arithmetic circuit 21 can identify which position the specific operator was looking at during the work (i.e., the visual recognition position) after step S33. For example, the arithmetic circuit 21 may create visual recognition position data by combining the image data and the line-of-sight data and superimposing the line-of-sight data on the image data for display.

[0054] In step S45, the arithmetic circuit 21 obtains the generation time of a waveform similar to the reference waveform. When the arithmetic circuit 21 determines that there are waveforms similar to the reference waveform in a plurality of time intervals in the electroencephalogram signal data, it identifies the time at which each waveform occurred.

[0055] In step S46, the arithmetic circuit 21 identifies the visual recognition position at the timing when a waveform similar to the reference waveform occurred. The arithmetic circuit 21 identifies, for example, the visual recognition position of the specific operator at the generation time identified in step S45 in the created visual recognition position data. Thereby, the arithmetic circuit 21 can identify the position corresponding to the attention point in the image data based on the electroencephalogram signal data. When there are a plurality of identified generation times, the arithmetic circuit 21 identifies the visual recognition position of the specific operator for each time.

[0056] In step S47, the arithmetic circuit 21 applies a video effect to the image data based on the position identified in step S46. Thereby, the arithmetic circuit 21 creates the applied image data. For example, the arithmetic circuit 21 applies a video effect so that the specific position can be discriminated in the image data. For example, the arithmetic circuit 21 may perform highlighting by any image processing such as applying a color to the specific position in the image data, indicating it with an arrow, surrounding the specific position with a frame, or blinking it so that it can be discriminated.

[0057] With the above, the arithmetic circuit 21 finishes the processing in the attention point identification stage. In the processing in the above-mentioned attention point identification stage, after collecting the image data, gaze data, and electroencephalogram signal data, each data is stored in the control device 20 and each process is performed to identify the attention point, but the processing is not limited to this. For example, at the timing when a specific operator performs an operation (that is, after S13), having a mobile terminal having the same functions as the control device 20, the arithmetic circuit of the mobile terminal may perform the processing described in steps S41 to S46. That is, the arithmetic circuit of the mobile terminal may identify the attention point while acquiring the image data, gaze data, and electroencephalogram signal data. By processing in this way, the arithmetic circuit of the mobile terminal can identify the attention point in parallel with the operation by the specific operator.

[0058] Next, while referring to FIG. 7, the transmission stage will be described. FIG. 7 is a flowchart showing the transmission stage of the information processing method according to Embodiment 1.

[0059] First, in step S51, the arithmetic circuit 21 reads out the assigned image data stored in the storage device 22. In step S52, the arithmetic circuit 21 outputs the assigned image data to the display device 30 via the communication circuit 24. The display device 30 displays the output assigned image data. Another operator different from the specific operator can grasp which position of the image (i.e., the work site) the specific operator is paying attention to by looking at the displayed assigned image data. Thereby, the other operator can learn the position that the specific operator is considered to pay attention to consciously or unconsciously during the operation.

[0060] With the above, the arithmetic circuit 21 finishes the processing in the transmission stage. In the processing in the above-mentioned transmission stage, the arithmetic circuit 21 displayed the assigned image data on the display device 30, but the processing is not limited to this.

[0061] For example, the transmission process may be executed by having a different operator from the specific operator wear the wearable device 40 shown in FIG. 8. FIG. 8 is a block diagram showing an example of a system 2 which is a modification of the system 1. The system 2 includes a wearable device 40 and a control device 20. The wearable device 40 includes, for example, a head-mounted display, a glasses-type device, and the like. The wearable device 40 has an arithmetic circuit 41, a storage device 42, a communication circuit 43, a vision camera 44, and an AR (augmented reality) display device 45.

[0062] The arithmetic circuit 41 may be configured in the same manner as the arithmetic circuit 21 and executes processing in the wearable device 40. The storage device 42 may be configured in the same manner as the storage device 22 and stores information regarding the above-mentioned points of attention in association with the image data. The communication circuit 43 may be configured in the same manner as the communication circuit 24. The arithmetic circuit 41 receives the points of attention specified by the control device 20 via the communication circuit 43 and stores them in the storage device 42. The vision camera 44 is an imaging device that may be configured in the same manner as the vision camera 14. The AR display device 45 is a display device that can display an AR image so as to overlay the image acquired by the vision camera 44. The AR display device 45 may be disposed in front of at least one eye of the other operator. For example, based on the image acquired by the vision camera 44, the arithmetic circuit 41 may identify the scene that the other operator is looking at and display an arbitrary AR image on the AR display device 45 so as to overlay the above-mentioned points of attention on the scene.

[0063] By displaying in this way, the arithmetic circuit 41 can display the AR image in an appropriate area according to the scene that the other operator who has moved to the work site is looking at, and can notify the other operator of the position that the specific operator felt should be paid attention to. The AR image may be created so as to emphasize the position to the other operator, similar to the image processing in step S47.

[0064] Further, the wearable device 40 may further include a line-of-sight identification device that identifies the line-of-sight direction of another operator. By providing the line-of-sight identification device, the arithmetic circuit 41 can more appropriately display the AR image on the AR display device 45 based on the line-of-sight direction of the other operator wearing the wearable device 40. For example, when the line of sight of the other operator is directed in a direction other than the attention point, the arithmetic circuit 41 may display an AR image in the area corresponding to the attention point on the AR display device 45. Further, when the line of sight of the other operator is directed at the attention point, the arithmetic circuit 41 may display an AR image in the area corresponding to the attention point on the AR display device 45.

[0065] (Embodiment 2) With reference to FIGS. 9 to 12, an information processing method according to Embodiment 2 will be described. When a specific operator (for example, an experienced operator) is working at a work site, the judgment ability of the operator may be reduced by the external environment. Further, the judgment ability may also be reduced depending on the state of the specific operator (for example, the physical condition of the specific operator). When the specific operator works in a state where the judgment ability is reduced, the work can be executed in a state where the judgment ability is reduced. Therefore, the reliability of the work is reduced. However, according to the information processing method according to Embodiment 2, the arithmetic circuit 21 of the control device 20 can determine whether the judgment ability of the specific operator is reduced depending on the external environment and the state of the specific operator. Therefore, according to the information processing method according to Embodiment 2, when the environment detection device 50 worn by the specific operator determines that the judgment ability of the specific operator is reduced, it can notify the specific operator to that effect. Note that the information processing method according to Embodiment 2 is not limited to a specific operator, and it is similarly possible to determine whether the judgment ability of another operator (for example, an inexperienced operator) different from the specific operator is reduced.

[0066] FIG. 9 shows a block diagram of an example of a system 3 for executing the information processing method according to Embodiment 2. The system 3 includes a wearable device 10B and a control device 20. The wearable device 10B further includes a temperature sensor 17 and a humidity sensor 18 as external environment sensors, and a body temperature sensor 19 as a vital sensor, in addition to the wearable device 10 shown in FIG. 1. The external environment sensor is not limited to a temperature sensor, and may be, for example, any sensor such as a temperature sensor, a humidity sensor, a vibration sensor, a sound sensor, a pressure sensor, or an odor sensor, or a combination of these sensors. The vital sensor is not limited to the body temperature sensor 19, and may be any sensor such as a body temperature sensor, a heart rate sensor, or a blood pressure sensor, or a combination of these sensors.

[0067] The information processing method according to Embodiment 2 first executes the creation stage of the information processing method according to Embodiment 1 to create a reference waveform of a specific operator. It is desirable that the reference waveform be created by an operation performed in an environment where the specific operator is not burdened as much as possible. Here, the burdened environment may include a low temperature, rain, vibration, strange odor, or noise.

[0068] Next, in the information processing method according to Embodiment 2, a specific operator performs an operation at a work site while wearing the wearable device 10B. FIG. 10 is a flowchart showing a part of the creation stage of the information processing method according to Embodiment 2. First, the specific operator moves to the work site and wears the wearable device 10B. Then, the specific operator turns on the switch of the wearable device 10B. When the wearable device 10B starts up, the arithmetic circuit 11 of the wearable device 10B activates the vision camera 14, the gaze identification device 15, the electroencephalogram measurement device 16, the temperature sensor 17, the humidity sensor 18, and the body temperature sensor 19.

[0069] In step S61, the arithmetic circuit 11 acquires, as image data, an image representing at least a part of the field of view of a specific operator by the vision camera 14. The arithmetic circuit 11 stores the acquired image data in the storage device 12. In step S62, the arithmetic circuit 11 acquires the gaze data of the specific operator by the gaze identification device 15 and identifies the gaze direction. The arithmetic circuit 11 stores the acquired gaze data in the storage device 12. In step S63, the arithmetic circuit 11 acquires, by the electroencephalogram measuring device 16, electroencephalogram signal data representing the electroencephalogram signal of the specific operator. The arithmetic circuit 11 stores the acquired electroencephalogram signal data in the storage device 12.

[0070] In step 64, the arithmetic circuit 11 acquires, as temperature data, the air temperature at the work site by the temperature sensor 17. The arithmetic circuit 11 stores the acquired temperature data in the storage device 12. In step 65, the arithmetic circuit 11 acquires the humidity at the work site as humidity data. The arithmetic circuit 11 stores the acquired humidity data in the storage device 12. In step 66, the arithmetic circuit 11 acquires the body temperature of the specific operator as body temperature data. The arithmetic circuit 11 stores the acquired body temperature data in the storage device 12. When the specific operator activates the wearable device 10B, the operation at the work site is started. The arithmetic circuit 11 of the wearable device 10B can acquire the image data at a predetermined time, the gaze data at a predetermined time, and the electroencephalogram signal data in a time period including the predetermined time and store them in the storage device 12. Further, the arithmetic circuit 11 can acquire the temperature data at a predetermined time, the humidity data at a predetermined time, and the body temperature data at a predetermined time and store them in the storage device 12.

[0071] Then, when the specific operator finishes the operation at the work site, the specific operator turns off the switch of the wearable device 10B.

[0072] Next, the stage after data collection in the creation stage of the method according to Embodiment 2 will be described. FIG. 11 is a flowchart showing other parts of the creation stage of the information processing method according to Embodiment 2.

[0073] First, in step S71, the arithmetic circuit 21 of the control device 20 acquires image data, gaze data, electroencephalogram signal data, temperature data, humidity data, and body temperature data from the storage device 12 of the wearable device 10B via the communication circuit 24. The arithmetic circuit 21 stores each acquired data in the storage device 22.

[0074] Next, in step S72, the arithmetic circuit 21 synchronizes at least the image data, gaze data, and electroencephalogram signal data among the acquired data with respect to time. The arithmetic circuit 21 may further synchronize other data with respect to time. By synchronizing the image data, gaze data, and electroencephalogram signal data with respect to time, the arithmetic circuit 21 can associate them with each other. The arithmetic circuit 21 can synchronize each data, for example, using the time information each data has. Therefore, the arithmetic circuit 11 can acquire and store in the storage device 12 the image data at a predetermined time, the gaze data at the predetermined time, and the electroencephalogram signal data in a time period including the predetermined time. Also, the arithmetic circuit 11 can acquire and store in the storage device 12 the temperature data at a predetermined time, the humidity data at the predetermined time, and the body temperature data at the predetermined time.

[0075] In step S73, the arithmetic circuit 21 sets a point of attention. The point of attention may be set, for example, by a specific operator while referring to the image data stored in the storage device 22 of the control device 20 and operating the input / output device 23 (such as a mouse) of the control device 20. Also, the setting of the point of attention does not necessarily need to be performed by a specific operator and may be performed by another operator different from the specific operator.

[0076] In step S74, the arithmetic circuit 21 specifies the visual recognition position of the specific operator in the image data based on the image data and the gaze data. Thereby, the arithmetic circuit 21 can specify which position the specific operator was looking at during the work (that is, the visual recognition position) after step S66. For example, the arithmetic circuit 21 may create visual recognition position data in which the gaze data is overlaid on the image data by combining the image data and the gaze data.

[0077] In step S75, the arithmetic circuit 21 identifies the timing at which the specific operator visually recognized the attention point. Then, in step S76, the arithmetic circuit 21 extracts the brain wave signal at that timing, thereby obtaining brain wave signal data representing the brain wave signal of the specific operator when visually recognizing the attention point. That is, the arithmetic circuit 21 obtains brain wave signal data including the brain wave signal induced when the specific operator sees a predetermined event.

[0078] In step S77, the arithmetic circuit 21 adds and averages the acquired plurality of brain wave signals to create a predetermined environmental reference waveform. The arithmetic circuit 21 stores the created predetermined environmental reference waveform in the storage device 22 in association with the external environmental information (that is, the temperature information and humidity information at the work site) and the body temperature information of the specific operator. The temperature information, humidity information, and body temperature information may be average values during the working hours, or may be maximum and minimum values. Also, the temperature information, humidity information, and body temperature information may use the values at the timing when the above-mentioned attention point was visually recognized.

[0079] In step S78, the arithmetic circuit 21 compares the previously created reference waveform with the predetermined environmental reference waveform. When the arithmetic circuit 21 determines that there is no difference between the reference waveform and the predetermined environmental reference waveform (S78: NO), in step S79, it stores in the storage device that the temperature information, humidity information, and body temperature information have no influence on the judgment ability. When the arithmetic circuit 21 determines that there is a difference between the reference waveform and the predetermined environmental reference waveform (S78: YES), in step S80, it stores in the storage device that the temperature information, humidity information, and body temperature information have an influence on the judgment ability.

[0080] For example, when the potential of the predetermined environmental reference waveform is within a predetermined range set in advance, the arithmetic circuit 21 may determine that there is no difference between the reference waveform and the predetermined environmental reference waveform. The predetermined range may be determined, for example, by the maximum value and the minimum value of the potential in the electroencephalogram signal (i.e., the specific electroencephalogram signal) used when creating the reference waveform. The method for determining the presence or absence of the difference between the reference waveform and the predetermined environmental reference waveform is not limited to this. For example, the arithmetic circuit 21 may make a determination based on the difference in the average value or the median value of the potential for each waveform, or may make a determination based on the value calculated using CCF or DTW between each waveform. The arithmetic circuit 21 may make a determination in the same manner as in step S43.

[0081] By processing as described above, the arithmetic circuit 21 can determine whether there is an influence on the judgment ability of a specific operator in a predetermined environment. By performing the above processing in various environments, the arithmetic circuit 21 can determine whether there is an influence on the judgment ability of a specific operator in various environments. For example, the arithmetic circuit 21 may store in the storage device as judgment ability information whether there is an influence on the judgment ability of a specific operator in a predetermined environment.

[0082] FIG. 12 shows a block diagram of an environment detection device 50 that can be worn by a specific operator for executing the information processing method according to Embodiment 2. The environment detection device 50 includes an arithmetic circuit 51, a storage device 52, a communication circuit 53, a temperature sensor 54, a humidity sensor 55, a body temperature sensor 56, and a notification device 57. The configuration of the environment detection device 50 is not limited to the above. For example, the environment detection device 50 can include any sensor that the wearable device 10B can include, such as a vibration sensor or a sound sensor.

[0083] The arithmetic circuit 51 can be configured in the same way as the arithmetic circuit 21 and executes the processing in the environment detection device 50. The storage device 52 can be configured in the same way as the storage device 22 and stores the judgment ability information. The temperature sensor 54 can detect the ambient temperature and transmits the temperature as temperature data to the arithmetic circuit 51. The humidity sensor 55 can detect the ambient humidity and transmits the humidity as humidity data to the arithmetic circuit 51. The body temperature sensor 56 can detect the body temperature of a specific worker and transmits the body temperature as body temperature data to the arithmetic circuit 51. The notification device 57 can notify a specific worker of predetermined information based on an instruction from the arithmetic circuit 51. The notification device 57 is, for example, a speaker that can notify a specific worker of sound.

[0084] Based on the judgment ability information stored in the storage device 52, the arithmetic circuit 51 determines whether each data obtained from the temperature sensor 54, the humidity sensor 55, and the body temperature sensor 56 affects the judgment ability of the specific worker. When it is determined that the judgment ability of the specific worker may decrease, the arithmetic circuit 51 notifies the specific worker using the notification device 57 that the judgment ability of the specific worker may decrease in the current environment. Therefore, the specific worker wearing the environment detection device 50 can grasp that there is a possibility that his or her judgment ability is decreased in the environment of the work site.

[0085] For example, when the judgment ability may decrease in a state where the humidity is high and the temperature and the body temperature of the specific worker are low, the arithmetic circuit 51 may notify the specific worker that the judgment ability may decrease when the data obtained from each of the sensors 54 to 56 is equal to or lower than a predetermined threshold. In this case, the predetermined threshold is the temperature in the temperature data and the body temperature data, and the humidity in the humidity data. Also, when it is determined based on the data obtained from each of the sensors 54 to 56 that the judgment ability of the specific worker does not decrease, the arithmetic circuit 41 may notify the specific worker using the notification device 57 that the judgment ability of the specific worker does not decrease in the current environment.

[0086] The arithmetic circuit 51 may determine whether it affects the judgment ability of a specific operator based on only any one of the temperature data, humidity data, and body temperature data. For example, when the judgment ability of a specific operator may decline when the temperature is low, the arithmetic circuit 51 may notify the specific operator using the notification device 57 that the judgment ability of the specific operator may decline in the current environment based on only the temperature data.

[0087] As described above, the environment detection device 50 can include any sensor that the wearable device 10B can include. Therefore, for example, when the wearable device 10B and the environment detection device 50 include a vibration sensor, the arithmetic circuit 51 can notify the specific operator using the notification device 57 that the judgment ability of the specific operator may decline based on the presence or absence of vibration.

[0088] (Modification example) The method according to the above-described embodiment is not limited to the case where a specific operator moves to a work site and acquires image data as a part of the field of view of the specific operator at the work site. For example, instead of a specific operator moving to a work site, an image corresponding to the work site is displayed on a computer display in a building, and the arithmetic circuit 21 may specify a position that the specific operator feels is a point of attention using the image. Further, the arithmetic circuit 21 may create a reference waveform of an electroencephalogram signal generated when the specific operator visually recognizes a point of attention using the image.

[0089] In the above-described embodiment, the specific operator wears the wearable device 10 having the vision camera 14, the gaze specifying device 15, and the electroencephalogram measuring device 16. However, the device worn by the specific operator is not limited to this. For example, instead of the wearable device 10, the specific operator may include the vision camera 14, the gaze specifying device 15, and the electroencephalogram measuring device 16. In this case, the control device 20 acquires image data, gaze data, and electroencephalogram signal data from the vision camera 14, the gaze specifying device 15, and the electroencephalogram measuring device 16, respectively.

[0090] In the above-described embodiment, the data acquired by the wearable device 10 is transmitted to the control device 20, and the arithmetic circuit 21 of the control device 20 performs processes such as creation of a reference waveform and identification of points of attention, but is not limited thereto. For example, the wearable device 10 may transmit each data to the server device via the communication circuit 24, and each process may be performed in the server device.

[0091] In the above-described embodiment, the arithmetic circuit 51 notified the worker of a decrease in the worker's judgment ability based on environmental information such as the temperature during the worker's operation or vital information such as body temperature, but the processing by the environment detection device 50 is not limited thereto. For example, the environment detection device 50 may further include an electroencephalogram measuring device, and the arithmetic circuit 51 may determine a decrease in the worker's judgment ability using an electroencephalogram signal in addition to the environmental information and vital information. Specifically, when the body temperature detected by the body temperature sensor 56 is a value at which the judgment ability of a specific worker may decrease and the value of the potential of the acquired electroencephalogram signal is outside a predetermined range, the arithmetic circuit 51 may notify the specific worker through the notification device 57 that the judgment ability may decrease. The predetermined range may be determined by the maximum value and the minimum value of the potential in the electroencephalogram signal (that is, the specific electroencephalogram signal) used when creating the reference waveform as described above, but is not limited thereto. Thereafter, when the body temperature detected by the body temperature sensor 56 changes to a value at which it is determined that the judgment ability does not decrease and the value of the potential of the acquired electroencephalogram signal is within the predetermined range, the arithmetic circuit 51 may notify the specific worker through the notification device 57 that the decrease in the judgment ability has been eliminated.

[0092] (Summary of Aspects) As is clear from the above description, the present disclosure includes the following aspects. Below, for the sole purpose of clarifying the correspondence with the embodiments, reference numerals are attached in parentheses.

[0093] (Aspect 1) The information processing method for identifying the position visually recognized by an operator with attention includes the steps of: acquiring, as image data, an image representing at least a part of the operator's field of view at a predetermined time; acquiring, as gaze data, the gaze direction of the operator at the predetermined time; acquiring, as electroencephalogram signal data, the electroencephalogram signal of the operator in a time period including the predetermined time; detecting a predetermined waveform from the acquired electroencephalogram signal data; and determining the visually recognized position of the operator when the predetermined waveform occurs based on the image data and the gaze data. Thereby, the information processing method can identify the position visually recognized by the operator with attention based on the electroencephalogram signal of the operator in a time period including the predetermined time measured from the operator.

[0094] (Aspect 2) In the information processing method of Aspect 1, the step of detecting the predetermined waveform may detect the predetermined waveform when a part of the electroencephalogram signal is a waveform corresponding to a reference waveform. Thereby, the information processing method can detect the predetermined waveform when the electroencephalogram signal includes a waveform corresponding to the reference waveform. Therefore, the information processing method can detect the predetermined waveform when a waveform corresponding to the reference waveform occurs in the electroencephalogram signal and determine the position visually recognized by the operator.

[0095] (Aspect 3) In the information processing method of Aspect 2, the reference waveform may be determined based on a specific electroencephalogram signal measured in advance, which is induced when the operator sees a predetermined event. Thereby, the information processing method can detect the predetermined waveform based on the specific electroencephalogram signal induced when the operator sees a predetermined event and determine the position visually recognized by the operator.

[0096] (Aspect 4) In the information processing method of Aspect 3, the predetermined event may represent an event that attracts the operator's attention. Thereby, the information processing method can detect the predetermined waveform based on the specific electroencephalogram signal induced when the operator visually recognizes a specific event and is attracted by attention, and determine the position visually recognized by the operator. Therefore, the information processing method can determine the visually recognized position when the operator visually recognizes an event that attracts the operator's attention.

[0097] (Aspect 5) Any of the information processing methods according to Aspect 1 to Aspect 4 may further include a step of transmitting the visual recognition position to another worker different from the worker. Thereby, the information processing method can convey the position visually recognized by a specific worker to another worker. Therefore, another worker can know the position visually recognized by a specific worker with attention, and the information processing method can efficiently transmit know-how from a specific worker to another worker.

[0098] (Aspect 6) Any of the information processing methods according to Aspect 1 to Aspect 5 may further include a step of imparting a video effect representing the visual recognition position on the image. Thereby, the information processing method can create an image representing the position visually recognized by the worker with attention. Therefore, by referring to the image by the user, the information processing method can easily and efficiently convey the visual recognition position to the user.

[0099] (Aspect 7) The system (1) includes an imaging device (14) that acquires, as image data, an image representing at least a part of the field of view of a worker at a predetermined time, a gaze direction specifying device (15) that acquires, as gaze data, the gaze direction of the worker at a predetermined time, an electroencephalogram measuring device (16) that acquires, as electroencephalogram signal data, the electroencephalogram signal of the worker in a time period including the predetermined time, a storage device (22) that stores the image data acquired from the imaging device (14), the gaze data acquired from the gaze direction specifying device (15), and the electroencephalogram signal data acquired from the electroencephalogram measuring device (16), and a control device (20) having an arithmetic circuit (21) that detects a predetermined waveform from the electroencephalogram signal data and determines the visual recognition position of the worker when the predetermined waveform occurs based on the image data and the gaze data. Thereby, the system (1) can specify the position visually recognized by the worker with attention based on the electroencephalogram signal of the worker in a time period including the predetermined time measured from the worker.

[0100] (Aspect 8) The system (1) of Aspect 7 may further include a display device (30) that displays image data and the imparted image data created based on the visual recognition position data representing the visual recognition position. Thereby, the system (1) can cause the display device (30) to display the specified visual recognition position, and can communicate the visual recognition position to any user.

[0101] The system described in the present disclosure is realized by the cooperation of hardware resources such as a processor and a memory, and software resources (computer programs).

Industrial Applicability

[0102] According to the present disclosure, an information processing method and system for specifying the position visually recognized by an operator based on the brain wave signal measured from the operator can be provided, and thus it can be suitably used in this type of industrial field.

Explanation of Signs

[0103] 1 System 10 Wearable Device 11 Arithmetic Circuit 12 Storage Device 13 Communication Circuit 14 Visual Field Camera 15 Line-of-Sight Specifying Device 16 Brain Wave Measuring Device 17 Temperature Sensor 18 Humidity Sensor 19 Body Temperature Sensor 20 Control Device 21 Arithmetic Circuit 22 Storage Device 23 Input / Output Device 24 Communication Circuit 30 Display Device

Claims

1. Obtaining, as image data, an image representing at least a part of an operator's field of view at a predetermined time; Obtaining, as line-of-sight data, the operator's line-of-sight direction at the predetermined time; Obtaining, as electroencephalogram signal data, the operator's electroencephalogram signal in a time period including the predetermined time; Detecting a predetermined waveform from the obtained electroencephalogram signal data; Determining the operator's visual recognition position when the predetermined waveform occurs based on the image data and the line-of-sight data; An information processing method for identifying a position visually recognized by an operator with attention, including the above steps.

2. The step of detecting the predetermined waveform is to detect the predetermined waveform when a part of the electroencephalogram signal is a waveform corresponding to a reference waveform. The information processing method according to claim 1.

3. The reference waveform is determined based on a pre-measured specific electroencephalogram signal induced when the operator sees a predetermined event. The information processing method according to claim 2.

4. The predetermined event represents an event that attracts the operator's attention. The information processing method according to claim 3.

5. The information processing method according to claim 1, further including the step of transmitting the visual recognition position to another operator different from the operator.

6. The information processing method according to any one of claims 1 to 5, further including the step of applying a video effect representing the visual recognition position on the image.

7. An imaging device for obtaining, as image data, an image representing at least a part of an operator's field of view at a predetermined time; A line-of-sight specifying device for obtaining, as line-of-sight data, the operator's line-of-sight direction at the predetermined time; An electroencephalogram measuring device for obtaining, as electroencephalogram signal data, the operator's electroencephalogram signal in a time period including the predetermined time; A storage device for storing the image data obtained from the imaging device, the line-of-sight data obtained from the line-of-sight specifying device, and the electroencephalogram signal data obtained from the electroencephalogram measuring device, and a control device having an arithmetic circuit for detecting a predetermined waveform from the electroencephalogram signal data and determining the operator's visual recognition position when the predetermined waveform occurs based on the image data and the line-of-sight data; A system comprising the above components.

8. The system according to claim 7, further comprising a display device for displaying the image data and the imparted image data created based on the visual recognition position data representing the visual recognition position.

Images