Behavioral history display device

Abstract

To provide a technology for easily recording an action history of a worker.SOLUTION: Voice recognition means 120 determines a word included in voice emitted by a worker and outputs word information in time series. Gaze point position detection means 130 detects a position of a gaze point of the worker, and outputs gaze point position information in time series. Fingertip position detection means 140 detects a position of a fingertip of the worker, and outputs fingertip position information in time series. Integration means 111 outputs, in time series, integration information obtained by integrating the word information, gaze point position information, and fingertip position information and stores them in storage means 150, on the basis of the time-series word information, the time-series gaze point position information, and the time-series fingertip position information. Image information generation means 112 generates, in time series, on the basis of the time-series integration information, image information including the gaze point position and the fingertip position of the worker, and a word series indicating voice emitted by the worker in a predetermined period including an arbitrary point of time, to be output on display means 160.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention records the behavioral history of a worker. display This relates to the technology. [Background technology]

[0002] In recent years, there has been an increasing need in industry to record the work know-how (called "tacit knowledge") accumulated by workers with skilled work skills (hereinafter referred to as "skilled workers") and pass it on to other workers. However, it is not easy to systematize tacit knowledge and realize a means to share skills within an organization. Therefore, various techniques related to skill transfer have been proposed. For example, a work support device disclosed in Patent Document 1 measures the amount of movement of a worker's line of sight. The measured amount is then compared with a predetermined reference value, and the worker's proficiency is determined based on the comparison result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 119127 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the work assistance device disclosed in Patent Document 1 is primarily intended for comparing proficiency levels. Furthermore, it uses only information about the worker's gaze point, without taking into account information such as language and gestures. Therefore, the work assistance device disclosed in Patent Document 1 has limited application purposes and is not suitable for recording and sharing a wide range of work skills. Therefore, after examining various related technologies, including the above-mentioned prior art, we developed a technology that digitally records information such as language and gestures in addition to gaze point information, and by integrating this information, enables the recording and sharing of a wider range of work skills. In recent years, multimodal information processing technology has been developed that integrates and processes multiple pieces of information acquired by sensors. By using this multimodal information processing technology, it is possible to integrate multiple pieces of input information (multimodal information) and generate information with new added value. When a worker is performing a task, the system can distinguish between audio information generated during the task, as well as gaze point position information and fingertip position information synchronized with the audio information, and selectively integrate and process this multimodal information to record the worker's behavioral history, including their tacit knowledge, and share this behavioral history information. The present invention makes it easy to record the worker's behavior history. Show The present invention aims to provide a technology that can [Means for solving the problem]

[0005] First Invention The behavior history display device displays integrated information that combines the words contained in the voice spoken by the worker with the positions of the worker's gaze point and fingertip. First Invention comprises image information generating means, storage means, and display means. The storage means includes: The time-series word information indicating words contained in the voice uttered by the worker, output from the voice recognition means, the time-series gaze point position information indicating the gaze point position at which the worker is gazing, output from the gaze point position detection means, and the time-series fingertip position information indicating the fingertip position of the worker, output from the fingertip position detection means, are integrated. Integrated information is recorded over time. The integrated information at time t(n) is composed of the words uttered by the worker at time t(n), the position of the worker's gaze point, and the position of the fingertip at time t(n). The voice recognition means may be any of various known voice recognition means. For example, a voice recognition means composed of a voice input means and a word discrimination means may be used. The voice input means inputs voice spoken by the worker and outputs voice information indicating the voice in time series. The word discrimination means discriminates words contained in the voice based on the voice information and outputs the word information in time series. When displaying a behavior history including a gaze point position on the display means, it is necessary to set the gaze point position on the display screen of the display means. For this reason, the system is configured to be able to determine the correspondence between the gaze point position detected by the gaze point position detection means and the coordinate system on the display screen of the display means. The storage means records a behavior history that integrates the voice uttered by the worker and the position of the worker's gaze point. The fingertip position detection means can use various known fingertip position detection methods. For example, a fingertip position detection means composed of an imaging means and an image processing means can be used. The imaging means captures an image of an imaging area and outputs time-series imaging information. The image processing means detects the fingertip position in the imaging area based on the imaging information and outputs fingertip position information indicating the fingertip position in time-series. When displaying the behavior history including the fingertip position on the display means, it is necessary to set the fingertip position on the display screen of the display means. For this reason, the fingertip position detection means is configured to be able to determine the correspondence between the fingertip position detected by the fingertip position detection means and the coordinate system on the display screen of the display means. the image information generating means generates time-series image information based on time-series integrated information obtained by integrating the time-series word information, the time-series gaze point position information, and the time-series fingertip position information, all of which are recorded in the storage means; The generated time-series image information and the equipment image in the worker's field of view are The data is output to a display means. The image information at any given time point includes the positions of the worker's gaze point and fingertip at that given time point, and a word sequence indicating the voice uttered by the worker within a predetermined period including that given time point. A "predetermined period including an arbitrary point in time" means, for example, if the arbitrary point in time is included in a speech period sandwiched between silent periods of a predetermined duration, the speech period. Alternatively, if the arbitrary point in time is included in a word utterance period from the start time to the end time of a word, the word utterance period. A "word sequence indicating the speech uttered by the worker within a predetermined period" means, for example, a combination of words included in the speech if the predetermined period is included in a speech period sandwiched between silent periods of a predetermined duration. Alternatively, if the predetermined period is a word utterance period, the word sequence means the word. First InventionWith this system, the worker's behavioral history can be easily and accurately confirmed by correlating the voice with the gaze point and fingertip position. Second Invention The behavior history display device displays integrated information that combines the words contained in the voice spoken by the worker with the positions of the worker's fingertips. Second Invention comprises image information generating means, storage means, and display means. The storage means includes: The time-series word information output from the voice recognition means, which indicates words contained in the voice uttered by the worker, and the time-series fingertip position information output from the fingertip position detection means, which indicates the fingertip positions of the worker, are integrated. Integrated information is recorded over time. The integrated information at time t(n) is composed of the words uttered by the worker at time t(n) and the position of the worker's fingertip at time t(n). The image information generating means generates an image based on time-series integrated information obtained by integrating the time-series word information and the time-series fingertip position information, which are stored in the storage means. The position of the worker's fingertip at any point in time and a word sequence indicating the voice uttered by the worker within a predetermined period including the point in time are included. Generate time-series image information, The generated time-series image information and the equipment image in the worker's field of view are The data is output to a display means. The image information at any given time point includes the position of the worker's fingertip at that given time point and a word sequence indicating the voice uttered by the worker within a predetermined period including that given time point. Second Invention With this system, the worker's behavior history can be easily confirmed by checking the correlation between voice and fingertip position. Third Invention The behavior history display device displays integrated information that combines the position of the worker's gaze point and the position of the fingertip. Third The invention comprises an image information generating means, a storage means, and a display means. The storage means includes: The gaze point position information output from the gaze point position detection means, which indicates the time-series gaze point position at which the worker is gazing, and the fingertip position information output from the fingertip position detection means, which indicates the time-series fingertip position of the worker, are integrated. Integrated information is recorded over time. The image information generating means stores the image data in the storage means. record Based on the time-series integrated information that integrates the time-series gaze point position information and the time-series fingertip position information, Including the worker's gaze point position at any point in time and the worker's fingertip position at that point in time Generate time-series image information, The generated time-series image information and the equipment image in the worker's field of view are The data is output to a display means. The image information at any given time point includes the positions of the worker's gaze point and fingertip at that given time point. Third Invention In this system, the worker's behavior history can be easily confirmed based on the relationship between the gaze point and the fingertip position.

[0006] First invention, second invention In a different form, the image information generating means generates image information including a word sequence indicating speech uttered by the worker within a predetermined period including an arbitrary time point, only for the duration of the section from the start time to the end time of each word included in the speech uttered by the worker within the predetermined period including the arbitrary time point. The duration for generating image information including a word sequence can be set appropriately based on the "start and end times of each word included in the speech uttered by the worker within a predetermined period of time." For example, if a word sequence is made up of multiple words, the duration can be set to the duration of the section from the start time of the first word to the end time of the last word. Alternatively, if a word sequence is made up of a single word, the duration can be set to the duration of the section from the start time of the word to the end time of the word. In this embodiment, it is possible to easily check the relationship between the timing of the worker's speech and the position of the worker's gaze point or fingertip. First invention, second invention In another embodiment, when a specific word is included in the speech uttered by the worker within a predetermined period including any point in time, the image information generating means generates image information in which a word sequence including the specific word is emphasized only for the duration of the section from the start time to the end time of the specific word. For example, a word sequence indicating speech produced by a worker within a predetermined period of time is displayed only for the duration of the section from the start time of the first word constituting the word sequence to the end time of the last word, and specific words are highlighted sequentially from one side of the specific word from the start time to the end time in a display format (different color, different pattern, etc.) different from the display format of the word sequence. Alternatively, a specific word is highlighted only for the duration of the section from the start time to the end time of the specific word in a display format (different color, different pattern, etc.) different from the display format of the word sequence. In this embodiment, the timing of utterance of a particular word can be easily confirmed. First Invention In another embodiment, the image information generating means generates image information showing an image in which the word sequence is arranged in the vicinity of the position of the worker's gaze point. As an example of a mode for arranging the word sequence near the position of the fixation point, the word sequence is displayed in a balloon so that the tip of a triangle is positioned near the position of the fixation point. In this embodiment, the relationship between the timing of utterance of a word sequence and the position of the worker's gaze point can be easily confirmed. First invention, second invention In another embodiment, the image information generating means generates image information showing an image in which the word sequence is placed near the position of the worker's fingertip. Word sequences at your fingertips position As an example of a mode of placing the word sequence near the fingertip, the word sequence is displayed in a balloon so that the tip of the triangle is placed near the position of the fingertip. In this embodiment, the correlation between the utterance timing of the word sequence and the position of the worker's fingertip can be easily confirmed. [Effects of the Invention]

[0007] Behavioral history display device of the first to third inventions By using this, the worker's behavior history can be easily confirmed. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram of a first embodiment of a behavior history recording device or a behavior history display device of the present invention. [Figure 2] FIG. 1 is a diagram illustrating an overview of a first embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a voice uttered by a worker. [Figure 4] FIG. 4 is a diagram showing an example of a display screen of an action history when a worker utters the voice shown in FIG. 3. [Figure 5] FIG. 4 is a diagram showing another example of the behavior history display screen when the worker utters the voice shown in FIG. 3. [Figure 6] FIG. 4 is a diagram showing another example of the behavior history display screen when the worker utters the voice shown in FIG. 3. [Figure 7] FIG. 2 is a block diagram of a second embodiment of the behavior history recording device or behavior history display device of the present invention. [Figure 8] FIG. 10 is a block diagram of a third embodiment of the behavior history recording device or behavior history display device of the present invention. [Figure 9] FIG. 10 is a block diagram of a fourth embodiment of the behavior history recording device or behavior history display device of the present invention. [Figure 10] FIG. 10 is a block diagram of a fifth embodiment of the behavior history recording device of the present invention. [Figure 11] FIG. 10 is a block diagram of a fifth embodiment of the behavior history display device of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First, the behavior history recording device of the present invention will be described. A block diagram of a first embodiment 100 of the behavior history recording device of the present invention is shown in Figure 1. The behavior history recording device 100 of the first embodiment records the behavior history of a worker based on the voice uttered by the worker, the position of the worker's gaze point, and the position of the worker's fingertip. The behavior history recording device 100 of the first embodiment includes a processing means 110 , a voice recognition means 120 , a gaze point position detection means 130 , a fingertip position detection means 140 , a storage means 150 , and a display means 160 .

[0010] The voice recognition means 120 identifies words contained in the voice uttered by the worker. The speech recognition means 100 can use various known speech recognition methods. For example, the speech recognition means 100 has a speech input means 121 and a word discrimination means 122. 120 can be used. The voice input means 121 receives the voice of the worker and outputs analog voice information as time-series digital information. The voice input means 121 may be, for example, a microphone. The word discrimination means 122 receives time-series speech information (speech information series), discriminates words contained in the speech information, and outputs time-series word information including the words and the start and end times of the words. Here, the start time and end time of a word can be determined in the process of searching for a word information sequence from the feature quantities of a speech information sequence in the word determination means 122. For example, by using a combination of an acoustic model based on a Hidden Markov Model (HMM), a language model based on an N-gram, and a One-Pass-Viterbi algorithm, which are existing methods, it is possible to comprehensively evaluate the acoustic similarity and linguistic similarity from the feature quantities of a speech information sequence, and simultaneously determine a word that matches the input speech information, as well as the start time and end time that indicate the temporal location of the word in the speech information. The word discrimination means 122 can use various known speech recognition algorithms, such as an HMM (Hidden Markov Model) or a method based on a neural network and a sequence conversion model. The words identified by the word identifying means 122 are not limited to units such as grammatical parts of speech (morphemes), but may be any predefined character string.

[0011] The gaze point position detection means detects the position of the gaze point of the worker and outputs time-series gaze point position information indicating the gaze point position. The gaze point position information includes the time when the gaze point position is detected. The gaze point position detecting means can use various known gaze point position detecting algorithms, such as the corneal reflex method and the dark pupil method. When displaying the behavior history including the gaze point position on the display means, the gaze point position is displayed in the target area (such as an equipment image acquired by an image sensor such as a CCD) that is displayed on the display screen of the display means and is within the field of view of the worker. In this case, it is necessary to configure the system so that the correspondence between the gaze point position and the coordinate system on the display screen can be determined. For example, the system is configured so that the correspondence between the reference position of the equipment displayed on the display screen and the gaze point position can be determined.

[0012] The fingertip position detection means 140 detects the positions of the worker's fingers and outputs time-series fingertip position information indicating the fingertip positions. The fingertip position information includes the time at which the fingertip positions are detected. The fingertip position detection means can use various known object detection algorithms, such as various object detection methods based on CNN (convolutional neural network). The fingertip position detecting means may be, for example, a fingertip position detecting means configured by an imaging means and an image processing means. The image processing means determines whether or not the worker's fingertip is present within the image capture area based on the captured image captured by the imaging means, and if the worker's fingertip is present within the image capture area, outputs time-series fingertip position information indicating the position of the fingertip. When the action history including the fingertip position is displayed on the display means, the fingertip position is displayed in the target area (such as an equipment image acquired by an image sensor such as a CCD) that is displayed on the display screen of the display means and is within the field of view of the worker. In this case, it is necessary to configure the system so that the correspondence between the fingertip position and the coordinate system on the display screen can be determined. For example, the system is configured so that the correspondence between the reference position of the equipment displayed on the display screen and the fingertip position can be determined.

[0013] The storage means 150 stores programs for executing various processes and various data. The display means 160 is used to display the behavior history of the worker.

[0014] The processing means 110 includes an integration means 111 and an image information generation means 112 . The integration means 111 outputs integrated information in a time series that integrates the word information, gaze point position information, and fingertip position information based on the time series word information (hereinafter referred to as the "word information series") output from the voice recognition means 120, the time series gaze point position information (hereinafter referred to as the "gazing point position information series") output from the gaze point position detection means 130, and the time series fingertip position information (hereinafter referred to as the "fingertip position information series") output from the fingertip position detection means 140. At this time, the integrated information at time t(n) is composed of the words uttered by the worker at time t(n), the positions of the worker's gaze points, and the positions of the fingertips at time t(n). The time-series image information (hereinafter referred to as "image information series") is stored in the storage means 150 as the worker's action history information. record will be done. The processing means 110 may be configured to execute part or all of the processing of the voice recognition means 120, the processing of the gaze point position detection means 130, and the processing of the fingertip position detection means.

[0015] The image information generating means 112 stores the image data in the storage means 150. record An image information sequence is generated based on time-series integrated information (hereinafter referred to as "integrated information sequence") obtained by integrating the word information sequence, the gaze point position information sequence, and the fingertip position information sequence. The image information sequence is output to the display means 160 and displayed. The image information generating means 112 can also generate an image information sequence based on the integrated information sequence output from the integrating means 111, as shown in FIG. Image information at any point in time in the image information series includes the position of the worker's gaze point and the position of his / her fingertip at that point in time, and a word series indicating the voice uttered by the worker within a specified period including that point in time. The "predetermined period including an arbitrary point in time" means, for example, a speech interval between silent intervals of a predetermined duration when the arbitrary point in time is included in the speech interval, or a word speech interval from the start time to the end time when the arbitrary point in time is included in the word speech interval. The "word sequence representing the speech uttered by the worker within a predetermined period" refers to, for example, a combination of words included in the speech if the predetermined period is included in a speech interval sandwiched between silent intervals of a preset duration. Alternatively, if the predetermined period is a word speech interval, it refers to the word itself.

[0016] In this embodiment, the voice recognition means 120, the gaze point position detection means 130, the fingertip position detection means 140, the integration means 111, and the storage means 150 constitute a behavior history recording device that records the behavior history of a worker. In this embodiment, the storage means 150 (or the image information generating means 112) and the display means 160 constitute a behavior history display device that displays the behavior history of the worker. That is, this embodiment is configured as a behavior history recording device having a behavior history display function. As a result, this embodiment is also the first embodiment of the behavior history display device.

[0017] Next, an outline of this embodiment will be described with reference to FIG. The integration means 111 integrates the word information sequence output from the voice recognition means 120, the gaze point position information sequence output from the gaze point position detection means 130, and the fingertip position information sequence output from the fingertip position detection means 140, outputs an integrated information sequence, and stores it in the storage means 150. record do. The image information generating means 112 stores the image data in the storage means 150. record The image information generating means 112 can also be configured to generate an image information sequence based on the integrated information sequence output from the integrating means 111. In Figure 2, the image information series includes integrated images from time t(1) to time t(n). The integrated image at time t(n) shows that when the word series "Pressure normal confirmed" is spoken, the gaze point position (dash-dotted line) is at the position of the numbers "19.50" on display A1, and the fingertip position (dashed line) is at the position of button B1. In this way, by displaying the worker's speech content, gaze point position, and fingertip position in chronological order, the worker's behavioral history can be easily confirmed by the correlation between the voice, gaze point, and fingertip. This makes it easier to confirm the tacit knowledge of skilled workers and can be used to pass on the tacit knowledge of skilled workers.

[0018] next, Specific examples The explanation will be based on the above. The following describes a case where a worker makes the utterance shown in FIG. 3 during recovery from a failure in a power plant. The speech recognition means 120 performs speech recognition on the utterance shown in FIG. 3 and outputs a word information sequence. For example, as shown in Figure 4, the words "serious fault," "," "turbine," "first," "bearing," "vibration," "," "X," "direction," "," "Y," "direction," and "." contained in the word sequence indicating an utterance "serious fault, turbine first bearing vibration, X direction, Y direction" are identified. Note that the word sequence "serious fault, turbine first bearing vibration, X direction, Y direction" is an example of an utterance divided into sections determined to be silent for a predetermined duration. At this time, the start and end times of each word are also identified.

[0019] FIG. 4 shows an example of a display screen displayed by image information. 4, the equipment screen (background screen) on which the gaze point position and fingertip position are displayed may be a screen registered in advance or a screen captured by the worker, as long as it is possible to determine the correspondence between the gaze point position indicated by the gaze point position information and the fingertip position indicated by the fingertip position information and the coordinate system on the equipment screen. Also, an enlarged or reduced screen may be displayed. From the display screen shown in Figure 4, it can be seen that the gaze point position (dash-dotted line) is at the position of the number "19.50" displayed on display unit A1, and the fingertip position (dashed line) is at the position of button B1. It can also be seen that the display timing of the gaze point position and fingertip position shown in Figure 4 is within the utterance period of the utterance "Major failure, turbine first bearing vibration, X direction, Y direction" (the duration of the period from the start time "T_S1" of the first word "Major failure" to the end time "T_E14" of the last word ".").

[0020] FIG. 5 shows different examples of display screens displayed with image information. On the display screen shown in FIG. 5, if a special word is included in the voice uttered by the worker, the special word is displayed in an emphasized manner (highlighted display). For example, if the special word "turbine" is included in the word sequence indicating an utterance, "serious failure, turbine first bearing vibration, X direction, Y direction," the word "turbine" will be highlighted only for the duration of the section from the start time "T_S3" to the end time "T_E3" of the word "turbine." For example, within the display period of the word sequence "serious failure, turbine first bearing vibration, X direction, Y direction" (the duration of the section from the start time "T_S1" of the first word "serious failure" to the end time "T_E14" of the last word "."), the word "turbine" is highlighted sequentially from one end to the other end from the start time "TS3" to the end time "T_E3" (the display mode of the color, pattern, etc. is changed from the display mode of the other parts). Alternatively, the word "turbine" is highlighted only for the duration of the section from the start time "TS3" to the end time "T_E3". The specific word may be a combination of multiple words.

[0021] FIG. 6 shows different examples of display screens displayed with image information. In the display screen shown in Fig. 6, the word sequence indicating the speech, "Major failure, turbine first bearing vibration, X direction, Y direction," is placed near the gaze point position (dashed line). For example, the word sequence is displayed in a speech bubble, and the tip of the triangle in the speech bubble is placed near the gaze point position. It should be noted that a word sequence indicating an utterance can also be placed at the desired fingertip position.

[0022] In the action history recording device of the first embodiment, an integrated information sequence obtained by integrating the word information sequence, the gaze point position information sequence, and the fingertip position information sequence is stored in the storage means as the action history information of the worker. record However, the behavioral history information is not limited to this.

[0023] FIG. 7 shows a second embodiment 200 of the behavior history recording device of the present invention. The behavior history recording device 200 of the second embodiment includes a voice recognition means 220, a gaze point position detection means 230, an integration means 211, an image information generation means 212, a storage means 250, and a display means 260, but does not include the fingertip position detection means of the first embodiment. That is, the behavior history recording device 200 of the second embodiment records an integrated information sequence obtained by integrating a word information sequence and a gaze point position information sequence in the storage means 250 as behavior history information. In addition, the image information generating means 212 stores the image information in the storage means 250. record An image information sequence is generated based on the integrated information sequence stored in the image processing unit 210 (or output from the integration unit 211 ), and is displayed on the display unit 260 . That is, in this embodiment, the storage means 250 (or the image information generating means 212) and the display means 260 constitute a behavior history display device that displays the behavior history of a worker. As a result, this embodiment is also a second embodiment of the behavior history display device.

[0024] FIG. 8 shows a third embodiment 300 of the behavior history recording device of the present invention. The behavior history recording device 300 of the third embodiment includes a voice recognition means 320, a fingertip position detection means 340, an integration means 311, an image information generation means 312, a storage means 350, and a display means 360, but does not include the gaze point position detection means of the first embodiment. That is, the behavior history recording device 300 of the third embodiment records an integrated information sequence obtained by integrating a word information sequence and a fingertip position information sequence in the storage means 350 as behavior history information. The image information generating means 312 also stores the image data in the storage means 350. record An image information sequence is generated based on the integrated information sequence stored in the image processing unit 310 (or output from the integration unit 311 ), and is displayed on the display unit 360 . That is, in this embodiment, the storage means 350 (or the image information generating means 312) and the display means 360 constitute a behavior history display device that displays the behavior history of a worker. As a result, this embodiment is also a third embodiment of the behavior history display device.

[0025] FIG. 9 shows a fourth embodiment 400 of the behavior history recording device of the present invention. The behavior history recording device 400 of the fourth embodiment includes a gaze point position detection means 430, a fingertip position detection means 440, an integration means 411, an image information generation means 412, a storage means 450, and a display means 460, but does not include the voice recognition means of the first embodiment. That is, the behavior history recording device 400 of the fourth embodiment records an integrated information sequence obtained by integrating the gaze point position information sequence and the fingertip position information sequence in the storage means 450 as behavior history information. In addition, the image information generating means 412 stores the image information in the storage means 450. record An image information sequence is generated based on the integrated information sequence stored in the image processing unit 410 (or output from the integration unit 411 ), and is displayed on the display unit 360 . That is, in this embodiment, the storage means 450 (or the image information generating means 412) and the display means 460 constitute a behavior history display device that displays the behavior history of a worker. As a result, this embodiment is also a fourth embodiment of the behavior history display device.

[0026] In the above embodiment, the behavior history recording device is configured to have a behavior history display function, but it can also be divided into a behavior history recording device and a behavior history display device. FIG. 10 shows a block diagram of a fifth embodiment 500 of the behavior history recording device. The behavioral history recording device 500 of the fifth embodiment is a division of the behavioral history recording device 100 of the first embodiment, and records an integrated information series that integrates a word information series, a gaze point position information series, and a fingertip position information series in a storage means as behavioral history information of the worker. The behavior history recording device 500 of the fifth embodiment is composed of a voice recognition means 520, a gaze point position detection means 530, a fingertip position detection means 540, an integration means 51, and a storage means 550. The operation is the same as that of the behavior history recording device information 100 of the first embodiment. The integrated information sequence recorded in the storage means 550 can be used to display the behavior history using a processing device other than the behavior history recording device, as in the behavior history recording devices of the first to fourth embodiments.

[0027] FIG. 11 shows a block diagram of a fifth embodiment 600 of the behavior history display device. The behavior history display device 600 of the fifth embodiment includes an image information generating means 612, a storage means 650, and a display means 660. By It is composed of: The storage means 650 stores an integrated information sequence similar to the integrated information sequence stored in the storage means of the behavior history recording device according to any one of the first to fifth embodiments. The operation is the same as that of the behavior history display devices of the first to fourth embodiments. The behavior history recording devices of the second to fourth embodiments can also be divided into a behavior history recording device and a behavior history display device as shown in FIGS.

[0028] (Aspect 1) A behavior history recording device that records a behavior history of a worker, The system comprises a voice recognition means, a gaze point position detection means, an integration means, and a storage means, the voice recognition means identifies words contained in the voice uttered by the worker and outputs word information indicating the words in a time series; the gaze point position detection means detects a gaze point position at which the worker is gazing, and outputs gaze point position information indicating the gaze point position in chronological order; The behavioral history recording device is characterized in that the integration means outputs integrated information that integrates word information and gaze point position information in a time series based on the time series word information output from the voice recognition means and the time series gaze point position information output from the gaze point position detection means, and records the integrated information in the memory means. (Aspect 2) A behavior history recording device that records a behavior history of a worker, The device comprises a voice recognition means, a gaze point position detection means, a fingertip position detection means, an integration means, and a storage means, the voice recognition means identifies words contained in the voice uttered by the worker and outputs word information indicating the words in a time series; the gaze point position detection means detects a gaze point position at which the worker is gazing, and outputs gaze point position information indicating the gaze point position in chronological order; the fingertip position detection means detects the fingertip positions of the worker and outputs fingertip position information indicating the fingertip positions in a time series; The behavioral history recording device is characterized in that the integration means outputs integrated information that integrates word information, gaze point position information, and fingertip position information in time series based on the time series word information output from the voice recognition means, the time series gaze point position information output from the gaze point position detection means, and the time series fingertip position information output from the fingertip position detection means, and records the integrated information in time series in the storage means. (Aspect 3) A behavior history recording device that records a behavior history of a worker, The device comprises a voice recognition means, a fingertip position detection means, an integration means, and a storage means, the voice recognition means identifies words contained in the voice uttered by the worker and outputs word information indicating the words in a time series; the fingertip position detection means detects the fingertip positions of the worker and outputs fingertip position information indicating the fingertip positions in a time series; The behavioral history recording device is characterized in that the integration means outputs integrated information in time series that integrates word information and fingertip position information based on the time series word information output from the voice recognition means and the time series fingertip position information output from the fingertip position detection means, and records the integrated information in the storage means. (Aspect 4) A behavior history recording device that records a behavior history of a worker, The device includes a gaze point position detection means, a fingertip position detection means, an integration means, and a storage means, the gaze point position detection means detects a gaze point position at which the worker is gazing, and outputs gaze point position information indicating the gaze point position in chronological order; the fingertip position detection means detects the fingertip positions of the worker and outputs fingertip position information indicating the fingertip positions in a time series; The integration means outputs integrated information that integrates the gaze point position information and the fingertip position information in a time series based on the time series gaze point position information output from the gaze point position detection means and the time series fingertip position information output from the fingertip position detection means, and records the integrated information in the storage means. (Aspect 5) A behavior history display device that displays a behavior history of a worker, The apparatus includes an image information generating means, a storage means, and a display means, the storage means stores the integrated information in time series stored in the storage means according to aspect 1; The image information generation means generates image information in chronological order based on the integrated time-series information recorded in the storage means, the image information including the worker's gaze point position at any given time point and a word sequence indicating the voice uttered by the worker within a predetermined period including the given time point, and outputs the image information to the display means. (Aspect 6) A behavior history display device that displays a behavior history of a worker, The apparatus includes an image information generating means, a storage means, and a display means, the storage means stores the time-series integrated information stored in the storage means according to aspect 2; The image information generation means generates image information in chronological order based on the integrated time-series information recorded in the storage means, the image information including the gaze point position and fingertip position of the worker at any given time point and a word sequence indicating the voice uttered by the worker within a predetermined period including the any given time point, and outputs the image information to the display means. (Aspect 7) A behavior history display device that displays a behavior history of a worker, The apparatus includes an image information generating means, a storage means, and a display means, the storage means stores the time-series integrated information stored in the storage means according to aspect 3; The image information generation means generates image information in time series, including a word sequence indicating the fingertip position of the worker at any given time point and the voice uttered by the worker within a predetermined period including the any given time point, based on the time-series integrated information stored in the storage means, and outputs the image information to the display means. (Embodiment 8) A behavior history display device that displays a behavior history of a worker, The apparatus includes an image information generating means, a storage means, and a display means, the storage means stores the time-series integrated information stored in the storage means according to aspect 4; The image information generation means generates image information including the gaze point position and fingertip position of the worker at any point in time in a time series based on the integrated time series information stored in the storage means, and outputs the image information to the display means. (Aspect 9) The behavior history display device according to any one of aspects 5 to 7, The image information generation means generates image information including a word sequence representing the speech uttered by the worker within a predetermined period including the arbitrary time point, for a period set based on the start and end times of each word included in the speech uttered by the worker within the predetermined period including the arbitrary time point. (Aspect 10) The behavior history display device according to aspect 9, The behavioral history display device is characterized in that, if a specific word is included in the voice uttered by the worker within a predetermined period including the arbitrary time point, the image information generation means generates image information in which a word sequence including the specific word is highlighted for a period set based on the start time and end time of the specific word. (Aspect 11) The behavior history display device according to any one of aspects 5 to 7, The behavioral history display device is characterized in that, when a specific word is included in the voice uttered by the worker within a predetermined period including the arbitrary point in time, the image information generation means generates image information including a word sequence including the specific word for a period set based on the start time and end time of the specific word. (Aspect 12) The behavior history display device according to aspect 5 or 6, The behavior history display device is characterized in that the image information generating means generates image information showing an image in which the word sequence is arranged near the worker's gaze point position. (Aspect 13) The behavior history display device according to aspect 6 or 7, The action history display device is characterized in that the image information generating means generates image information showing an image in which the word sequence is placed near the position of the worker's fingertip. (Aspect 14 ) A behavior history recording device that records a behavior history of a worker, The device comprises at least two of a voice recognition means, a gaze point position detection means, and a fingertip position detection means, an integration means, and a storage means, the voice recognition means identifies words contained in the voice uttered by the worker and outputs word information indicating the words in a time series; the gaze point position detection means detects a gaze point position at which the worker is gazing, and outputs gaze point position information indicating the gaze point position in chronological order; the fingertip position detection means detects the fingertip positions of the worker and outputs fingertip position information indicating the fingertip positions in a time series; The behavioral history recording device is characterized in that the integration means outputs integrated information in time series that integrates at least two of the word information, the gaze point position information, and the fingertip position information based on at least two of the time series word information output from the voice recognition means, the time series gaze point position information output from the gaze point position detection means, and the time series fingertip position information output from the fingertip position detection means, and records the integrated information in time series in the storage means. In this aspect, the integration means outputs in chronological order any one of integrated information that integrates word information and gaze point position information, integrated information that integrates word information, gaze point position information and fingertip position information, integrated information that integrates word information and fingertip position information, and integrated information that integrates gaze point position information and fingertip position information, and records it in the storage means. (Aspect 15 ) An action history display device that displays the action history of a worker, The apparatus includes an image information generating means, a storage means, and a display means, The storage means includes 14 The integrated information recorded in the storage means is recorded in time series, The image information generation means generates, in time series based on the integrated time-series information recorded in the storage means, at least two pieces of image information including the gaze position of the worker at an arbitrary time point, the fingertip position of the worker at the arbitrary time point, and a word sequence indicating the voice uttered by the worker within a predetermined period including the arbitrary time point, and outputs the image information to the display means.

[0029] The present invention is not limited to the configurations described in the embodiments, and various modifications, additions, and deletions are possible. Various known methods can be used to determine the words contained in the voice uttered by the worker and the start and end times of the words. As a method for detecting the position of the worker's gaze point, various known methods can be used. As a method for detecting the position of the worker's fingertip, various known methods can be used. The method of displaying the worker's behavior history (integrated information) on the display means is not limited to the method described in the embodiment, and can be changed as appropriate. Each of the configurations described in the embodiments can be used alone or in combination with an appropriately selected plurality of configurations. [Explanation of symbols]

[0030] 100, 200, 300, 400, 500 Behavioral history recording device 110, 210, 310, 410, 510, 610 Processing means 111, 211, 311, 411, 511 Integration means 112, 212, 312, 412, 612 Image information generating means 120, 220, 320, 520 Voice recognition means 121, 221, 321, 521 Voice input means 122, 222, 322, 522 Word discrimination means 130, 230, 430, 530 gaze point position detection means 140, 340, 440, 540 Fingertip position detection means 150, 250, 350, 450, 550 storage means 160, 260, 360, 460, 660 display means 600 Behavioral history display device

Claims

1. An action history display device that displays the action history of a worker, The apparatus includes an image information generating means, a storage means, and a display means, the storage means records time-series integrated information obtained by integrating time-series word information indicating words included in the voice uttered by the worker, which is output from the voice recognition means, time-series gaze point position information indicating the gaze point position at which the worker is gazing, which is output from the gaze point position detection means, and time-series fingertip position information indicating the fingertip positions of the worker, which is output from the fingertip position detection means; The image information generation means generates image information in a time series format based on the time-series integrated information recorded in the storage means, the image information including the position of the worker's gaze point at an arbitrary time point, the position of the worker's fingertip at the arbitrary time point, and a word sequence indicating the voice spoken by the worker within a predetermined period including the arbitrary time point, and outputs the generated time-series image information and an image of equipment that is a target area that came into the worker's field of view to the display means.

2. An action history display device that displays the action history of a worker, The apparatus includes an image information generating means, a storage means, and a display means, the storage means records time-series integrated information obtained by integrating time-series word information indicating words included in the voice uttered by the worker, which is output from the voice recognition means, and time-series fingertip position information indicating fingertip positions of the worker, which is output from the fingertip position detection means; The image information generation means generates image information in a time series, based on the time-series integrated information recorded in the storage means, including the fingertip position of the worker at any given time point and a word sequence indicating the voice spoken by the worker within a predetermined period including the any given time point, and outputs the generated time-series image information and an image of equipment that is a target area within the worker's field of view to the display means.

3. An action history display device that displays the action history of a worker, The apparatus includes an image information generating means, a storage means, and a display means, the storage means records time-series integrated information obtained by integrating time-series gaze point position information, which is output from the gaze point position detection means and indicates a time-series gaze point position at which the worker is gazing, and time-series fingertip position information, which is output from the fingertip position detection means and indicates a fingertip position of the worker; The image information generation means generates image information in a time series, including the worker's gaze point position at any given time and the worker's fingertip position at that given time, based on the time-series integrated information recorded in the storage means, and outputs the generated time-series image information and an image of equipment that is a target area within the worker's field of view to the display means.

4. The behavior history display device according to claim 1, The image information generation means generates image information including a word sequence representing the speech uttered by the worker within a predetermined period including the arbitrary time point, for a period set based on the start and end times of each word included in the speech uttered by the worker within the predetermined period including the arbitrary time point.

5. The behavior history display device according to claim 1 or 4, The behavioral history display device is characterized in that, if a specific word is included in the speech uttered by the worker within a predetermined period including the arbitrary point in time, the image information generation means generates image information in which a word sequence including the specific word is highlighted for a period set based on the start time and end time of the specific word.

6. The behavior history display device according to claim 4 or 5, The behavior history display device is characterized in that the image information generating means generates image information showing an image in which the word sequence is arranged near the worker's gaze point position.

7. The behavior history display device according to claim 4 or 5, The action history display device is characterized in that the image information generating means generates image information showing an image in which the word sequence is placed near the position of the worker's fingertip.

8. The behavior history display device according to claim 2, The image information generation means generates image information including a word sequence representing the speech uttered by the worker within a predetermined period including the arbitrary time point, for a period set based on the start and end times of each word included in the speech uttered by the worker within the predetermined period including the arbitrary time point.

9. The behavior history display device according to claim 2 or 8, The behavioral history display device is characterized in that, if a specific word is included in the speech uttered by the worker within a predetermined period including the arbitrary point in time, the image information generation means generates image information in which a word sequence including the specific word is highlighted for a period set based on the start time and end time of the specific word.

10. The behavior history display device according to claim 8 or 9, The action history display device is characterized in that the image information generating means generates image information showing an image in which the word sequence is placed near the position of the worker's fingertip.

Images