Information processing device, information processing system, information processing method, and information processing program

The information processing device effectively identifies and presents important information from recorded videos by determining importance across multiple subjects, enhancing learning and work improvement through iterative feedback integration.

JP7589438B2Active Publication Date: 2024-11-26OMRON CORP
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
JP2020032229
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-04-09
Filing Date
2020-02-27
Publication Date
2024-11-26
Estimated Expiration
2040-02-27

AI Technical Summary

Technical Problem

Existing technologies lack the ability to automatically identify and present important information from recorded videos for learning and work improvement, particularly in situations where sequential information is involved.

Method used

An information processing device that acquires annotated series information, determines the importance of elements within the information, integrates determination results across multiple subjects, and generates presentation information for effective presentation of important content.

Benefits of technology

Enables the accurate and efficient identification and presentation of important information, allowing for targeted review and improvement in learning and work scenarios by integrating feedback to refine the importance determination logic.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007589438000001
    Figure 0007589438000001
  • Figure 0007589438000002
    Figure 0007589438000002
  • Figure 0007589438000003
    Figure 0007589438000003
Patent Text Reader

Abstract

To provide an information processing apparatus, a terminal device, an information processing system, an information processing method, and an information processing program with which from among pieces of information having series properties, important information can be suitably determined and presented.SOLUTION: In an information processing system, a control unit 110 of an information processing apparatus 100 comprises: a series information acquisition unit 112 that acquires series information with annotations in which elements included in series information on a subject are provided with annotations; an importance determination unit 114 that refers to the series information with annotations on the subject and determines the importance of the elements included in the series information with annotations on the subject; a determination integration unit 116 that refers to a result of determination related to a plurality of subjects made by the importance determination unit and creates determination integration information; and an information for presentation creation unit 118 that refers to at least any one of the result of determination made by the importance determination unit and the determination integration information and creates information for presentation.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Systems that support learning and the acquisition of work skills by learners are known as conventional technologies. For example, Patent Document 1 describes a technology that monitors a learner's rereading frequency and gaze fixation, compares the results with those of other learners, and presents the comparison results to the learner and instructor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-338173 A (Published December 8, 2005) Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, people have been reviewing videos of learning, work, etc. in various situations to help improve the learning, work, etc. In such a technology, it is preferable to automatically identify important scenes from the recorded videos that are useful for improving learning or work.

[0005] In addition, there is a great need for technology that can distinguish and present important information from sequential information, beyond just learning and work.

[0006] In view of the above-mentioned problems, an object of the present invention is to provide a technique that can appropriately identify and present important information from among information having sequentiality. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing device according to one embodiment of the present invention includes a sequence information acquisition unit that acquires annotated sequence information in which annotations are added to each element included in sequence information related to a subject; an importance discrimination unit that refers to the annotated sequence information related to the subject and discriminates the importance of each element included in the annotated sequence information of the subject; a discrimination integration unit that refers to discrimination results by the importance discrimination unit related to multiple subjects and generates discrimination integration information; and a presentation information generation unit that generates presentation information by referring to at least one of the discrimination results by the importance discrimination unit and the discrimination integration information.

[0008] According to the above configuration, important information can be suitably identified and presented from among information having a sequence.

[0009] In the information processing device according to the aspect described above, the importance determining unit may update the determination logic by referring to feedback information from a user.

[0010] According to the above configuration, the determination logic in the importance determining section is updated by referring to feedback information from a user, so that the importance determination of the series information can be performed more accurately each time the importance determination is repeated.

[0011] In the information processing device according to the aspect described above, the series information may include information regarding the subject's line of sight.

[0012] According to the above configuration, by referring to information regarding the line of sight of the subject, it is possible to more appropriately determine the importance.

[0013] In the information processing device according to the aspect described above, the discrimination integration section may extract common information from the discrimination results by the importance discrimination section for each of the plurality of subjects, and include the common information in the discrimination integration information.

[0014] According to the above configuration, even if the series information differs depending on the subject, common information is extracted, so that the importance can be determined more appropriately.

[0015] In the information processing device according to the aspect described above, the importance determining unit may determine a concentration level regarding the subject as the importance level.

[0016] According to the above configuration, the importance of the series information can be determined by referring to the concentration degree related to the subject, so that the importance can be determined more appropriately.

[0017] In order to solve the above problems, a terminal device according to one embodiment of the present invention includes a sequence information acquisition unit that acquires sequence information, an annotation assignment unit that generates annotated sequence information by assigning annotations to each element included in the acquired sequence information, a presentation information acquisition unit that acquires presentation information generated by referring to the annotated sequence information, and a presentation unit that presents the presentation information.

[0018] According to the above configuration, presentation information in which each piece of series information is integrated can be obtained and presented to the user.

[0019] In the terminal device relating to the above-mentioned one aspect, the terminal device may further include a discrimination integration information acquisition unit that acquires discrimination integration information generated by referring to discrimination results regarding annotated series information of multiple subjects, and the annotation assignment unit may update the annotation assignment logic by referring to at least one of feedback information from a user and the discrimination integration information.

[0020] According to the above configuration, by updating the annotation assignment logic with reference to each piece of information, annotations can be assigned more appropriately.

[0021] In the terminal device according to the above aspect, the annotation assignment unit may assign, in addition to the annotation, information indicating a reliability of the annotation to each of the elements.

[0022] According to the above configuration, each process that refers to an annotation can be more suitably executed.

[0023] In order to solve the above problems, an information processing system according to one embodiment of the present invention includes an acquisition unit that acquires sequence information, an annotation assignment unit that generates annotated sequence information by assigning annotations to each element included in the acquired sequence information, an importance discrimination unit that refers to the annotated sequence information related to a certain subject and determines the importance of each element included in the annotated sequence information of the subject, a discrimination and integration unit that refers to the discrimination results by the importance discrimination unit related to multiple subjects and generates discrimination and integration information, and a presentation information generation unit that generates presentation information by referring to at least one of the discrimination results by the importance discrimination unit and the discrimination and integration information.

[0024] According to the above configuration, the same effects as those of the information processing device according to an aspect of the present invention can be obtained.

[0025] In order to solve the above problems, an information processing method according to one embodiment of the present invention includes an annotation adding step of generating annotated series information by adding annotations to each element included in series information related to a subject; an importance determination step of referring to the annotated series information and determining the importance of each element included in the annotated series information of the subject; a discrimination and integration step of generating discrimination and integration information by referring to a discrimination result in the importance determination step related to a plurality of subjects; and a presentation information generating step of generating presentation information by referring to at least one of the discrimination results in the importance determination step and the discrimination and integration information.

[0026] According to the above configuration, the same effects as those of the information processing device according to an aspect of the present invention can be obtained.

[0027] An information processing program according to one embodiment of the present invention is an information processing program for causing a computer to function as any of the information processing devices described above, and causes the computer to function as the series information acquisition unit, the importance discrimination unit, the discrimination and integration unit, and the presentation information generation unit.

[0028] According to the above configuration, the same effects as those of the information processing device according to an aspect of the present invention can be obtained. Effect of the Invention

[0029] According to one aspect of the present invention, important information from among information having a sequence can be suitably identified and presented. [Brief description of the drawings]

[0030] [Figure 1] 1 is a block diagram illustrating a schematic configuration example of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a sequence diagram illustrating an example of the operation of an information processing system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing an example of a test question to be recorded according to an embodiment of the present invention. [Figure 4] FIG. 11 is a diagram showing an example of presentation information according to an embodiment of the present invention. [Diagram 5] FIG. 11 is a diagram showing an example of presentation information according to an embodiment of the present invention. [Figure 6] 1 is a diagram showing an example of a visual target of a subject while learning a customer service task in one embodiment of the present invention; FIG. [Figure 7] 1 is a table showing an example of data on a visual target and speech of a learner according to an embodiment of the present invention. [Figure 8] 11 is a table showing an example of data of a veteran attendant according to an embodiment of the present invention. [Figure 9] FIG. 11 is a diagram showing an example of difference information between a learner and an experienced customer service representative according to an embodiment of the present invention. [Figure 10]11 is a table showing an example of data on a visual target and an action of a learner according to an embodiment of the present invention. [Figure 11] 1 is a table illustrating an example of experienced assembler data according to an embodiment of the present invention. [Figure 12] FIG. 13 is a diagram showing an example of difference information between a learner and an experienced assembler according to an embodiment of the present invention. [Figure 13] 11 is a table showing an example of data on a visual target and an action of a learner according to an embodiment of the present invention. [Figure 14] 1 is a table showing an example of data on an experienced chef according to an embodiment of the present invention. [Figure 15] FIG. 11 is a diagram showing an example of difference information between a learner and an experienced chef in one embodiment of the present invention. [Figure 16] 11 is a table showing an example of data on visual recognition targets and actions of player A according to one embodiment of the present invention. [Figure 17] 13 is a table showing an example of data of player B according to one embodiment of the present invention. [Figure 18] FIG. 2 is a diagram showing an example of difference information between player A and player B according to one embodiment of the present invention. [Figure 19] FIG. 11 is a block diagram illustrating a schematic configuration example of an information processing system according to another embodiment of the present invention. [Figure 20] FIG. 11 is a sequence diagram illustrating an example of the operation of an information processing system according to another embodiment of the present invention. [Figure 21] FIG. 11 is a block diagram illustrating a schematic configuration example of an information processing system according to still another embodiment of the present invention. [Figure 22] FIG. 11 is a sequence diagram illustrating an example of the operation of an information processing system according to still another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment according to one aspect of the present invention (hereinafter, also referred to as "the present embodiment") will be described below with reference to the drawings.

[0032] <Embodiment 1> § 1. Examples of Application First, an example of a situation in which the present invention is applied will be described with reference to Fig. 1. Note that, here, an example will be given in which the subject of the present invention is a learner who takes a lecture or a mock exam at a cram school or the like, but this does not limit the present embodiment, and the subject is not particularly limited as long as he or she has some kind of association with series information.

[0033] FIG. 1 is a block diagram illustrating a schematic configuration example of an information processing system 1 according to an embodiment of the present invention. The information processing system 1 is an information processing system used, for example, in a cram school or the like. As shown in FIG. 1, the information processing system 1 according to this embodiment includes a server 100, learner terminal devices 200A, 200B, and 200C, and a teacher terminal device 1000, which are information processing devices. The server 100, learner terminal devices 200A, 200B, and 200C, and teacher terminal device 1000 are connected via a network and can communicate with each other. The type of network may be any type, such as the Internet, a telephone network, or a dedicated network. In the following, the learner terminal devices 200A, 200B, and 200C may be collectively referred to as the learner terminal device 200.

[0034] The learner's terminal devices 200A, 200B, and 200C according to the present embodiment are computers respectively assigned to learners (hereinafter also referred to as subjects) A, B, and C in a cram school. Each of the learner's terminal devices 200A, 200B, and 200C is equipped with a camera 241, a microphone 242, and the like, and can record the appearance and speech of each subject during a lecture or a mock test. The recorded video is an example of sequence information in the present embodiment. Here, sequence information means information in general in which the order of each element contained in the information has meaning. In addition, sequence information may include video data, audio data, text data, and data showing changes in numerical values ​​over time. In addition, each terminal device 200 can generate annotated sequence information by adding annotations to sequence information and record the annotated sequence information in the respective storage units 230. Here, annotations are meta information added to each element included in the sequence information. As an example, the annotation includes meta-information regarding the state of the subject at each time point or each period during the recording of the series information. Note that the type of annotation is not limited to this embodiment, but as an example, an index indicating the concentration level of the subject at a given time point can be used as the annotation.

[0035] Then, each of the learner's terminal devices 200A, 200B, and 200C transmits the series information for each subject with the annotations to the server 100.

[0036] The server 100 determines the importance of each element included in the series information based on the annotated series information transmitted from each terminal device 200. The server 100 then combines the results of the importance determination based on the annotated series information transmitted from each terminal device 200 to generate presentation information. The server 100 stores the generated presentation information in the storage unit 130 and transmits it to each terminal device 200 or 1000.

[0037] Each terminal device 200 or 1000 presents the presentation information transmitted from the server 100 and transmits feedback information to the server 100. At this time, the subject may check the presentation information together with the instructor, and transmit feedback information regarding an update of the importance determination logic or a determination logic used in the determination integration process described later to the server 100 via each terminal device 200 or 1000. In other words, feedback information as a result of correcting and confirming the presentation information may be transmitted to the server 100 from each terminal device 200 or 1000. The server 100 updates the importance determination logic based on the feedback information transmitted from each terminal device 200 or 1000.

[0038] As described above, the amount of information that can be used for determining the importance level increases each time transmission and reception of sequence information is repeated between the server 100 and each of the terminal devices 200 or 1000, and the importance determination logic and the determination logic used in the determination and integration process are improved. As a result, the importance level of sequence information can be determined more appropriately. This allows the instructor to review the video by focusing on information that is likely to lead to improvements in the content of instruction, and thus allows the instructor to use time efficiently to improve instruction. §2. Example configuration The configurations of the server 100, the learner terminal device 200, and the instructor terminal device 1000, which are information processing devices constituting the information processing system 1 in this embodiment, will be described in detail below with reference to FIG.

[0039] <Student's Terminal Device 200> The learner's terminal devices 200A, 200B, and 200C according to the embodiment have the same configuration, for example. For example, the learner's terminal device 200 has a control unit 210, a communication unit 220, a storage unit 230, a camera 241, a microphone 242, an operation reception unit 243, a display unit (presentation unit) 244, and a speaker 245.

[0040] The communication unit 220 performs communication processing with an external device such as the server 100. The memory unit 230 is a storage device that stores various data. The operation reception unit 243 is an interface that receives input operations from the subject, such as buttons on a keyboard. The display unit 244 is a display panel that displays moving images. The operation reception unit 243 and the display unit 244 may be realized as a touch panel that receives input operations from the subject and displays moving images. The control unit 210 is a control device that controls the entire learner's terminal device 200, and includes a sequence information acquisition unit 212 that acquires sequence information via the camera 241, an annotation assignment unit 214 that generates annotated sequence information by assigning annotations to each element included in the acquired sequence information, a presentation information acquisition unit 216 that acquires presentation information generated with reference to the annotated sequence information via the communication unit 220, and a discrimination and integration information acquisition unit (not shown) that acquires discrimination and integration information generated with reference to a discrimination result regarding the annotated sequence information of one or more subjects.

[0041] <Series information acquisition unit 212> The sequence information acquisition unit 212 acquires sequence information related to the subject. Examples of sequence information acquired by the sequence information acquisition unit 212 are as follows. As an example, these data are acquired as information during a mock test or a lecture, but this is not a limitation of the present embodiment and may be acquired in other situations. A subject captured by the camera 241 (which may include the subject's face) -Voice of the subject picked up by microphone 242 Operation input by the subject accepted by the operation acceptance unit 243 (which may include text data) Data such as images, voice, and text acquired via the communication unit 220 Image, audio, text, and other data read from the storage unit 230 -Temperature changes obtained from a temperature sensor (not shown) - Changes in weather over time acquired via the communication unit 220 Pulse data obtained from a sensor (not shown) for reading the subject's level of tension, stress, concentration, etc.

[0042] The series information may also include information regarding the subject's line of sight. For example, the series information acquisition unit 212 acquires face information of the subject from an image acquired from the camera 241. The face information includes position information indicating the position of each part of the subject's face (for example, eyes, nose, mouth, eyebrows, etc.), shape information indicating the shape, size information indicating the size, etc., and also detects the subject's line of sight as the state of the subject. The line of sight is particularly important as an index indicating the degree of the subject's concentration on a task, etc. The detection of the subject's line of sight will be described later.

[0043] <Annotation assignment unit 214> The annotation assigning unit 214 generates annotated series information by automatically assigning annotations to each element included in the acquired series information or by assigning annotations based on a user instruction. Annotations are assigned to elements included in the series information. As one example, annotation A is assigned to the section from 00:01 to 00:02 in the video data, and annotation B is assigned to the section from 00:05 to 00:06. As another example, annotation X is assigned to phrase AA in sentence A included in the text data, and annotation Y is assigned to phrase BB.

[0044] The annotation assigning unit 214 may also assign information indicating whether or not the subject has viewed a specific region to the series information as an annotation. The operation receiving unit 243 accepts an operation to assign an annotation to the series information and an operation to input an importance level (to be described later) for each portion of the series information. For example, when a user issues an instruction to assign an annotation to a portion of the series information via the operation receiving unit 243, the annotation assigning unit 214 assigns the instructed annotation to that portion of the series information.

[0045] As a more specific example, the annotation assigning unit 214 evaluates the subject's concentration level at each time point in the recorded video on a five-point scale, and assigns the concentration level as an annotation to each time point in the video, which is series information. As another example, from a video of the subject during a mock test, the problem sentence or question that the subject is looking at at each time point is identified, and information indicating which problem sentence or question the subject is looking at is assigned as an annotation to each time point in the video.

[0046] In this way, it is possible to annotate information indicating which problem statement or question is being viewed, for example, by identifying the coordinates indicating the subject's gaze direction as coordinates on the mock test question paper.

[0047] Note that the annotation assignment process by the annotation assignment unit 214 is not limited to the above example, and may be configured to, for example, determine the points to which annotations should be assigned using other image analysis algorithms or audio analysis algorithms, and assign annotations to the determined locations.

[0048] Also, as an example, an annotation may be defined by a start time, an end time, a tag, and a reliability. Here, the tag is information indicating the type of annotation, such as information indicating the concentration level and understanding level of a subject. Also, the reliability is information indicating the likelihood of a target annotation. The annotation assigning unit 214 can calculate the reliability of each annotation assigned by the annotation assigning process, which indicates the degree to which the annotation is reliable. Also, the annotation assigning unit 214 may assign information indicating the reliability of the annotation to each element in the series information in addition to the annotation.

[0049] In addition, for example, the concentration level may be set to a relatively high value when the variation in the visual target of the subject is small (when the variation in the line of sight is small), and may be set to a relatively low value when the variation in the visual target of the subject is large (when the variation in the line of sight is large). The annotation assignment unit 214 can specify the concentration level of the subject by acquiring and referring to the line of sight information of the subject.

[0050] Also, for example, the level of understanding may be set to a high value when the time during which the subject views the object to be understood is relatively long (or short).

[0051] Furthermore, for example, when the sequence information is information recording an object visually recognized by the subject during a mock test, the annotation assigning unit 214 may set the level of understanding by referring to the score of the mock test input by the subject after the mock test via the operation receiving unit 243, or information indicating which question the subject answered correctly. Furthermore, for example, when the subject satisfies a predetermined condition of performing a specific action at a specific timing, the annotation assigning unit 214 may set a high level of understanding for a target part in the sequence information. Furthermore, the information indicating the above-mentioned predetermined condition may be stored in advance in the storage unit 230 or the like.

[0052] The annotation assigning unit 214 may also update the determination logic (annotation assigning logic) used when an annotation is assigned to series information, for example, by referring to information included in the presentation information.

[0053] Furthermore, when the annotation assigning unit 214 assigns annotations based on user input, the annotation assigning unit 214 may be configured to update the annotation assigning logic by using the annotations assigned by the user as feedback information. Furthermore, the annotation assigning unit 214 may update the annotation assigning logic by referring to the discrimination integration information. That is, the annotation assigning unit 214 may update the annotation assigning logic by referring to at least one of the feedback information from the user and the discrimination integration information.

[0054] <Presentation information acquisition unit 216, display unit 244> The presentation information acquisition unit 216 acquires the presentation information generated by the presentation information generation unit 118 of the server 100 with reference to the annotated series information.

[0055] The display unit 244 displays the presentation information acquired by the presentation information acquisition unit 216. The presentation information is not limited to a moving image presented via the display unit 244, and may be audio presented via the speaker 245. The learner's terminal device 200 may be configured to accept feedback information from the user via the operation acceptance unit 243, similar to the instructor's terminal device 1000 described later.

[0056] More specifically, as an example, the subject and the instructor check the presentation information displayed by the display unit 244 and, while having a dialogue, check the importance determination result included in the presentation information. Then, if there is any point that needs to be corrected regarding the importance, What should be fixed, and How to fix it The information including the above is input to the operation receiving unit 243. Here, the point to be corrected in terms of importance is, for example, an importance determination result corresponding to a certain presentation information indicating a relatively high importance, but the presentation information is determined by the subject to be unimportant. The operation receiving unit 243 may be configured to generate feedback information including the input information and transmit it to the server 100 via the communication unit 220.

[0057] <Server 100> 1, the server 100 of the information processing system 1 includes a control unit 110, a communication unit 120, and a storage unit 130. The communication unit 120 enables communication with other information processing devices (learner terminal device 200, instructor terminal device 1000) included in the information processing system 1. The storage unit 130 can store information transmitted from other information processing devices, information integrated by the server, and the like.

[0058] The control unit 110 is a control device that controls the entire server 100, and includes a series information acquisition unit 112, an importance discrimination unit 114, a discrimination integration unit 116, and a presentation information generation unit 118. The communication unit 120 performs communication processing with external devices such as the learner's terminal device 200. The storage unit 130 is a storage device that stores various data.

[0059] <Series information acquisition unit 112> The series information acquiring unit 112 acquires annotated series information in which annotations are added to each element included in the series information about the subject. As an example, the series information acquiring unit 112 acquires, via the communication unit 120, information in which annotations are added to series information about each subject transmitted from each learner's terminal device 200.

[0060] <Importance determination unit 114> The importance determining unit 114 refers to the annotated series information about the subject, and determines the importance of each element included in the annotated series information about the subject.

[0061] The importance determination unit 114 may determine the importance based on the concentration level of the subject. As an example, when the concentration level is equal to or higher than a predetermined value or the concentration level is equal to or lower than a predetermined value for a predetermined period of time or more, the importance determination unit 114 determines that the section has a non-zero importance.

[0062] For example, it is assumed that the server 100 acquires annotated sequence information such as that shown in the table below from each terminal device 200. Time 00:01 00:02 00:03 00:04 00:05 Concentration level Subject A 1 5 5 3 1 Subject B 1 5 5 5 1 Subject C 2 5 5 5 5 In this case, as an example, the importance determination unit 114 determines a section in which the concentration level of 5 continues for 2 seconds or more as a section having an importance level other than 0. For example, since subject A maintains a state of concentration level 5 for 2 seconds from 00:02 to 00:03, the importance determination unit 114 determines that the importance of the section is 1. Furthermore, since subject B maintains a state of concentration level 5 for 3 seconds from 00:02 to 00:04, the importance determination unit 114 determines that the importance of the section is 2. Furthermore, since subject C maintains a state of concentration level 5 for 4 seconds from 00:02 to 00:05, the importance determination unit 114 determines that the importance of the section is 3.

[0063] Similarly, the importance determining section 114 can be configured to determine a section in which a concentration level of 1 or 2 continues for 2 seconds or more as a section having a non-zero importance level.

[0064] Also, for example, the importance determination unit 114 may determine that the lower (or higher) the level of understanding of the subject, the higher the importance of the target part. Alternatively, the importance determination unit 114 may determine that the importance of a part where the gazes of the subject start to disperse (or start to converge) in a lecture or the like is high.

[0065] As another example, the importance determining unit 114 may perform a regression analysis on the annotated sequence information and determine the importance by referring to various parameters obtained by the regression analysis. For example, the importance determining unit 114 may apply a regression analysis to the annotated sequence information of a certain subject to derive parameters that indicate a regression curve representing the annotated sequence information. The importance determining unit 114 may then compare the value of the parameter with a predetermined parameter value and determine the importance according to the comparison result.

[0066] Furthermore, the importance discrimination unit 114 updates the discrimination logic used in the importance discrimination process by referring to feedback information acquired via the communication unit 120. As an example, when the importance discrimination unit 114 acquires feedback information indicating that the importance of an element to which the importance discrimination unit 114 has assigned an importance level of 5 should be lowered, the discrimination logic is updated so that a lower importance level is assigned to the element to which the importance discrimination unit 114 has assigned an importance level of 5.

[0067] The importance determination unit 114 may determine that the more the gaze or movement of the subject in a certain period of time deviates from those of other subjects, the higher the importance. The importance determination unit 114 may determine that the more the gaze or movement of the subject in a certain period of time deviates from the gaze or movement set as a norm, the higher the importance. The information indicating the normative gaze or movement described above may be stored in advance in the storage unit 130 or the like.

[0068] Furthermore, the importance determining unit 114 may perform the importance determining process by determining that a portion that contributes greatly to improving the determination logic when the annotation assigning unit 214 assigns an annotation to series information is of high importance.

[0069] <Discrimination and Integration Unit 116> The discrimination integration unit 116 generates discrimination integration information by referring to the discrimination results by the importance discrimination unit 114 for multiple subjects A, B, C, .... For example, the discrimination integration unit 116 collects sequence information obtained from each terminal device 200 and integrates the sequence information, thereby making it possible to more appropriately determine the importance of the sequence information.

[0070] As an example, the discrimination integration section 116 may extract common information from the discrimination results by the importance discrimination section 114 for each of the plurality of subjects A, B, and C, and include the common information in the discrimination integration information.

[0071] For example, it is assumed that the importance determining section 114 determines the importance of subjects A, B, and C as follows. Time 00:01 00:02 00:03 00:04 00:05 Importance Subject A 0 1 1 0 0 Subject B 0 2 2 2 0 Subject C 0 3 3 3 3 In this case, as an example, the discrimination integration unit 116 determines that 00:02, which is a time assigned a non-zero importance level, is important to subjects A, B, and C, and includes the determination result in the discrimination integration information.

[0072] For example, the discrimination integration unit 116 may include information indicating that the gaze or behavior of the subject in a certain period of time deviates from those of other subjects and information indicating the degree of deviation in the discrimination integration information. The discrimination integration unit 116 may also include information indicating that the gaze or behavior of the subject in a certain period of time deviates from gaze or behavior set as a norm and information indicating the degree of deviation in the discrimination integration information.

[0073] In addition, for example, the discrimination integration unit 116 may include meta-information indicating matters related to the subject, such as the subject's past performance records, or meta-information indicating matters related to the subject's environment, such as mock exams that the subject is taking, in the discrimination integration information. In addition, each of the above-mentioned meta-information may be included in the discrimination result by the importance discrimination unit 114.

[0074] As another example, when the importance discrimination unit 114 is configured to perform a regression analysis on the annotated sequence information, the discrimination integration unit 116 may be configured to extract parameters having common properties from among parameters obtained by the regression analysis on the subjects A, B, and C, and include the parameters in the discrimination integration information. The discrimination integration unit 116 may generate a regression model to be commonly used by referring to regression models corresponding to multiple subjects. In other words, the discrimination integration unit 116 may use multiple discrimination algorithms or parameters referenced by them as input data, and output the integrated discrimination algorithm or parameters referenced by them. When the importance can be input to the sequence information by the subjects or instructors, the discrimination integration unit 116 can generate discrimination integration information that reflects, for example, parts that multiple instructors consider to be important.

[0075] Furthermore, the discrimination integration unit 116 updates the discrimination logic used in the discrimination integration process that generates the discrimination integration process by referring to feedback information acquired via the communication unit 120. As an example, when the discrimination integration unit 116 acquires feedback information indicating that the importance of an element to which an importance level of 5 has been assigned should be lowered, the discrimination integration unit 116 updates the discrimination logic so that a lower importance level is assigned to the element to which an importance level of 5 has been assigned.

[0076] Furthermore, the discrimination integration unit 116 generates information for updating the importance discrimination logic or the updated discrimination logic, and supplies it to the importance discrimination unit 114. The importance discrimination unit 114 updates the importance discrimination logic by referring to the acquired information. This makes it possible to update the result of the importance discrimination process to a more suitable one, for example, by using series information corresponding to multiple subjects.

[0077] Furthermore, the discrimination integration unit 116 generates information for updating the discrimination logic used in adding annotations or the updated discrimination logic, and transmits it to the learner's terminal device 200 via the communication unit 120. The annotation adding unit 214 updates the discrimination logic by referring to the acquired information. This makes it possible to update the annotation adding process to a more suitable one, for example, by using series information corresponding to multiple subjects. Note that the discrimination integration unit 116 may refer to the discrimination integration information generated by itself when updating each of the above-mentioned discrimination logics.

[0078] Note that the annotation assignment logic by the annotation assignment unit 214, the importance determination logic by the importance determination unit 114, and the integration logic by the discrimination integration unit 116 are not limited to the above examples. Furthermore, the processes by the annotation assignment unit 214, the importance determination unit 114, and the discrimination integration unit 116 may use rule-based logic, machine learning such as a neural network, or other methods.

[0079] For example, any one or a combination of the following machine learning techniques can be used:

[0080] Support Vector Machine (SVM) Clustering Inductive Logic Programming (ILP) · Genetic Algorithm (GP: Genetic Programming) ·Bayesian Network (BN) Neural Network (NN) When using a neural network, input data may be processed in advance for input to the neural network. For such processing, in addition to one-dimensional or multidimensional arrangement of data, a method such as data augmentation can be used.

[0081] In addition, when using a neural network, a convolutional neural network (CNN) including a convolution process may be used. More specifically, a convolutional layer that performs a convolution operation may be provided as one or more layers included in the neural network, and a filter operation (product-sum operation) may be performed on input data input to the layer. In addition, when performing the filter operation, a process such as padding may be used in combination, or an appropriately set stride width may be adopted.

[0082] In addition, a multi-layer or super multi-layer neural network having tens to thousands of layers may be used as the neural network. The above-mentioned machine learning may be supervised learning or unsupervised learning.

[0083] <Presentation information generation unit 118> The presentation information generating unit 118 generates presentation information by referring to at least one of the discrimination result by the importance discriminating unit 114 and the discrimination integration information. For example, information processed for each target person from the discrimination integration information integrated by the discrimination integration unit 116 is transmitted to each terminal device 200. The generated integration information and presentation information are stored in the storage unit 130.

[0084] Also, for example, the presentation information generating unit 118 may preferentially include corresponding information in the presentation information as the importance indicated by the determination result by the importance determining unit 114 increases. Also, the presentation information generating unit 118 may preferentially include corresponding information in the presentation information as the information included in the discrimination integration information indicates a high degree of deviation between the gaze and behavior of the subject in a certain period of time and the gaze and behavior of other subjects. Also, the presentation information generating unit 118 may preferentially include corresponding information in the presentation information as the information included in the discrimination integration information indicates a high degree of deviation between the gaze and behavior of the subject in a certain period of time and the gaze and behavior set as a norm.

[0085] <Instructor Terminal Device 1000> The instructor terminal device 1000 includes a control unit 1010 , an operation receiving unit 1043 , a display unit 1044 , a speaker 1045 , a communication unit 1020 , and a storage unit 1030 .

[0086] The control unit 1010 is a control device that controls the entire instructor terminal device 1000 , and includes a presentation information acquisition unit 1014 and a feedback information acquisition unit 1012 .

[0087] The communication unit 1020 performs communication processing with an external device such as the server 100. The memory unit 1030 is a storage device that stores various data. The operation reception unit 1043 is an interface that receives input operations from a lecturer or the like, and is, for example, buttons on a keyboard or the like. The display unit 1044 is a display panel that displays moving images. Note that the operation reception unit 1043 and the display unit 1044 may be configured to be realized as a touch panel that receives input operations from a lecturer or the like and displays moving images.

[0088] <Presentation information acquisition unit 1014, display unit 1044> The presentation information acquisition unit 1014 acquires the presentation information generated by the presentation information generation unit 118 of the server 100 with reference to the annotated series information.

[0089] The display unit 1044 displays the presentation information acquired by the presentation information acquisition unit 1014 .

[0090] <Feedback information acquisition unit 1012> The feedback information acquisition unit 1012 acquires feedback information from the user. As an example, the instructor checks the presentation information displayed by the display unit 1044 and checks the importance determination result included in the presentation information. If there is any point that needs to be corrected regarding the importance, What should be fixed, and How to fix it The operation receiving unit 1043 inputs information including the above to the operation receiving unit 1043. Here, the part of the importance to be corrected is, for example, an importance determination result corresponding to a certain piece of presentation information that indicates a relatively high importance, but the instructor has determined that the piece of presentation information is not important. The operation receiving unit 1043 generates feedback information including the input information and transmits it to the server 100 via the communication unit 1020.

[0091] The importance determination unit 114 of the server 100 refers to feedback information from the user and updates the determination logic.

[0092] The feedback information acquiring section 1012 may be provided in the instructor terminal device 1000 or in the learner terminal device 200 .

[0093] <Gaze detection> The following describes how the series information acquisition unit 212 and the annotation assignment unit 214 detect the gaze of a subject.

[0094] First, the camera 241 captures the subject during the mock test. The series information acquisition unit 212 acquires face information of the subject from the video captured by the camera 241. The face information of the subject includes, for example, position information indicating the positions of each part of the face (for example, eyes, nose, mouth, eyebrows, etc.), shape information indicating the shape, and size information indicating the size. In particular, the eye information can be used to evaluate the subject's concentration on the object that the subject is gazing at. Examples of the eye information include the corners and corners of the eye, edges of the iris and pupil, etc. The series information acquisition unit 212 may perform correction processing such as noise reduction and edge emphasis on the video acquired from the camera 241 as appropriate. The series information acquisition unit 212 transmits the extracted face information to the annotation assignment unit 214.

[0095] The annotation assignment unit 214 detects the state of the subject based on the face information extracted by the series information acquisition unit 212. For example, the state of each part of the subject's face, such as at least one of the subject's gaze, pupil state, number of blinks, eyebrow movement, cheek movement, eyelid movement, lip movement, and jaw movement, is detected.

[0096] The method of detecting the gaze of the subject is not particularly limited, but for example, a method of providing a point light source (not shown) in the terminal device 200 and capturing a corneal reflection image of light from the point light source with the camera 241 for a predetermined period of time to detect the destination of the gaze of the subject is exemplified. The type of point light source is not particularly limited, and examples include visible light and infrared light, but by using an infrared LED, for example, it is possible to detect the gaze without causing discomfort to the subject. In gaze detection, if the gaze does not move for a predetermined period of time or more, it can be said that the subject is gazing at the same place.

[0097] In addition, the method of detecting the pupil state is not particularly limited, but may include, for example, a method of detecting a circular pupil from an image of the eye using a Hough transform. Generally, humans tend to open their pupils when concentrating, so the degree of concentration of a subject can be evaluated by detecting the size of the pupil. For example, the size of the pupil is detected for a predetermined period of time, and if the pupil is enlarged within the predetermined period of time, it can be said that the subject is likely to be gazing at a certain object. A threshold value may be set for the pupil size, and if the pupil size is equal to or larger than the threshold, it may be evaluated as "open," and if the pupil size is less than the threshold, it may be evaluated as "closed."

[0098] In addition, the method of detecting the number of blinks is not particularly limited, but may be, for example, a method of irradiating the eyes of a subject with infrared light and detecting the difference in the amount of infrared light reflected when the eyes are open and when they are closed. Generally, when a person is concentrating, he or she tends to blink at a low frequency and at stable intervals, so the concentration level of the subject can be evaluated by detecting the number of blinks. For example, if the number of blinks is detected for a predetermined period of time and the blinks are performed at stable intervals within the predetermined period of time, it can be said that there is a high possibility that the subject is gazing at a certain object.

[0099] The annotation providing unit 214 may detect at least one of the subject's gaze, pupil state and blink count, eyebrow movement, eyelid movement, cheek movement, nose movement, lip movement, and chin movement, but it is preferable to combine these. By combining the detection methods in this way, the annotation providing unit 214 can appropriately evaluate the subject's concentration level when viewing a certain object.

[0100] Other than the state of the eyes, examples of the state of each part of the face include eyebrow movement such as lifting the inner eyebrow or lifting the outer eyebrow, eyelid movement such as lifting the upper eyelid or tensing the eyelid, nose movement such as wrinkling the nose, lip movement such as lifting the upper lip or pursing the lips, cheek movement such as lifting the cheek, jaw movement such as lowering the chin, etc. The state of a plurality of parts of the face may be combined as the state of the subject.

[0101] In addition to the gaze information, as described above, the concentration level of the subject can be determined more appropriately by referring to the detection results of the pupil state, the number of blinks, the eyebrow movement, the eyelid movement, the cheek movement, the nose movement, the lip movement, and the jaw movement. The concentration level of the subject may also be determined by further referring to the pulse data of the subject.

[0102] The annotation assigning unit 214 evaluates the concentration level of the identified subject on a five-level scale, for example, from 1 to 5, and assigns the evaluation result to the series information of the subject as an annotation. §3. Example of operation <Operation of information processing system 1> Next, a description will be given of the operation of the information processing system 1. Fig. 2 is a sequence diagram illustrating an example of the operation of the information processing system 1 according to this embodiment. With reference to Fig. 2, a process of generating presentation information in the information processing system 1 according to this embodiment will be described.

[0103] (Step S102) In step S102, the annotation assigner 214 generates annotated series information by assigning annotations to each element included in the series information about the subject acquired by the series information acquirer 212. The communication unit 220 transmits the generated annotated series information to the server 100.

[0104] (Step S104) In step S104, the sequence information acquisition unit 112 of the server 100 acquires the sequence information of the annotated sequence information.

[0105] (Step S106) In step S106, the importance determining unit 114 refers to the annotated series information related to the subject, and determines the importance of each element included in the annotated series information of the subject.

[0106] (Step S108) In step S108, the discrimination integration unit 116 generates discrimination integration information by referring to the results of the importance discrimination for the multiple subjects A, B, C, . . . .

[0107] (Step S110) In step S110, the presentation information generating unit 118 generates presentation information by referring to at least one of the discrimination result by the importance discriminating unit 114 and the discrimination integration information. When the presentation information generating unit 118 generates presentation information by referring only to the discrimination result by the importance discriminating unit 114, the process of step S108 does not necessarily have to be executed, and the process of step S110 may be executed following the process of step S106. Thereafter, the communication unit 120 transmits the generated presentation information to each learner's terminal device 200.

[0108] (Step S112) In step S112, the presentation information acquiring unit 216 of each learner's terminal device 200 acquires the generated presentation information.

[0109] (Step S114) In step S114, the presentation unit (display unit) 244 presents the acquired presentation information.

[0110] Furthermore, in step S114, the annotation assigning unit 214 may refer to information included in the presentation information to update the determination logic used when an annotation is assigned to series information.

[0111] (Step S116) In step S116, the terminal device 200 (or the instructor terminal device 1000) acquires feedback information. As described above, the feedback information is generated as a result of the subject confirming the presentation information together with the instructor and confirming and correcting the determined importance. The communication unit 220 of the terminal device 200 transmits the feedback information to the server 100.

[0112] (Step S118) In step S118, the importance discrimination unit 114 or the discrimination integration unit 116 of the server 100 updates each discrimination logic with reference to feedback information from the learner's terminal device 200. For example, the importance discrimination unit 114 updates the importance discrimination logic based on feedback information from the terminal device 200 including the confirmed and corrected importance.

[0113] (Step S120) When the lineage information acquirer 212 of the terminal device 200 acquires lineage information again, the annotation assigner 214 generates annotated lineage information by assigning annotations to the lineage information. The following steps S120 to S136 are the same as the above steps S102 to S118.

[0114] In the information processing system 1, the learner's terminal device 200 repeats the above steps S102 to S118 every time it acquires sequence information.

[0115] Therefore, each time the server 100 acquires series information from a subject, it updates each discrimination logic. As the amount of series information from a subject increases, each discrimination logic is improved, and each discrimination can be performed more appropriately.

[0116] <Another example 1 of annotation assignment processing, importance assignment processing, presentation information generation processing, etc.> The annotation assignment process, the importance assignment process, the presentation information generation process, and the like performed by the information processing system 1 according to the present embodiment are not limited to the above examples. Other examples of the annotation assignment process, the importance assignment process, the presentation information generation process, and the like performed by the information processing system 1 according to the present embodiment will be described below.

[0117] The sequence information in this example is information that records the subject's visual target during the mock test. FIG. 3 is a diagram showing an example of a test question to be recorded. In FIG. 3, areas R1 to R3 are areas that are predefined to distinguish each range. Marker 301 is a marker for identifying the coordinates indicating the subject's line of sight as coordinates on the mock test question paper.

[0118] The annotation assigning unit 214 according to the present embodiment assigns information indicating which part of the test question the subject was looking at while the sequence information was being recorded to the sequence information. For example, the annotation assigning unit 214 assigns information indicating that the subject was looking at region R1 at time 00:05 to the sequence information.

[0119] FIG. 4 is a diagram showing an example of information to be presented. FIG. 4(A) is a diagram showing, in a time series, a model way of moving the gaze when solving a target mock test. FIG. 4(B) is a diagram showing, in a time series, a way of moving the gaze of a subject who solved the mock test. FIG. 4(C) is a diagram showing the importance at each time point assigned to the series information by the importance discrimination unit 114. In each diagram in FIG. 4, the horizontal axis indicates time. Furthermore, the vertical axis in FIG. 4(C) indicates the level of importance.

[0120] For example, in Fig. 4(C), the location 303 indicates a location to which the importance determination unit 114 set a high importance because the variation in the gaze was larger than the normative gaze when the subject read through the entire test question. Also, the location 304 indicates a location to which the importance determination unit 114 set a high importance because the subject's understanding was determined to be low because the subject's visual recognition rate of the area R1, which is the key to the answer, was low when the subject examined the answer.

[0121] The importance discrimination unit 114 may set a high importance to a portion where the difference between the information shown in Fig. 4(A) and the information shown in Fig. 4(B) is large or small, or may set a high importance to a corresponding portion in the series information when the subject has visually recognized an area that is the key to the answer at a high rate. Also, Fig. 4(C) may be configured to include, as the integrated discrimination information generated by the discrimination integration unit 116, information indicating the importance set in the series information corresponding to other users so that the user can compare them.

[0122] FIG. 5 is a diagram showing an example of presentation information displayed on the display unit 244 of the learner's terminal device 200 (or the display unit 1044 of the instructor's terminal device 1000). In FIG. 5, objects 311, 312, and 313 correspond to FIG. 4(A), FIG. 4(B), and FIG. 4(C), respectively.

[0123] A screen 315 is a playback screen of a video of a target being visually recognized by a target person. Also, the marker 301 does not have to be displayed on this screen.

[0124] Object 317 in seek bar 316 and object 314 indicate which part of the video recording the visual target of the subject is being played back on screen 315. The part of the video being played back may be changeable by sliding object 316 or object 317 left or right. Object 318 is an operation panel for controlling the playback of the video.

[0125] Furthermore, the location where the video is played may be changeable by selecting a portion of the text on screen 315. In other words, when a portion of the text is selected, the video may be configured to start playing from the location where the subject was viewing the portion of the text.

[0126] The display unit 244 may display a pie chart or the like showing the proportion of each part of the test question that the subject looked at and to what extent on the screen exemplified in Fig. 5. The same applies to the pie chart or the like regarding the normative gaze described above.

[0127] Furthermore, the operation reception unit 243 receives input of comments and the like corresponding to each part of the video. The input comments and the like are transmitted to the server 100 via the communication unit 220 as feedback information.

[0128] <Another Example 2 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.> Other examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Note that the same description as that already described in the above examples will not be repeated.

[0129] The series information in this example is information that records the visual target of a subject who is learning customer service work in a restaurant or the like. Fig. 6 is a diagram showing an example of the visual target of a subject who is learning customer service work. The image shown in Fig. 6 shows the inside of a restaurant as seen from the perspective of a learner who is a store clerk, and the image shows customers, the learner's hands, other store clerks, tables, etc.

[0130] FIG. 7 is a table showing an example of data regarding the learner's visual target, speech (content of speech and tone of voice), movement and concentration level (blinking, pupil state, gaze fixation time and facial expression), and is a table showing an example of annotations added to series information at each point in time by the annotation adding unit 214 in this example.

[0131] 7, the annotation assignment unit 214 assigns, as an example, the visual target, speech, action, and concentration level at each time point as annotations. An event is a series of decisions (speech, action, etc.) made within a predetermined time period from the time when a learner or an experienced customer recognizes an object. In one example, an event includes the timing, the visual target, speech, action, and concentration level, and each event can be identified by an ID.

[0132] Other than the state of the eyes, examples of the state of each part of the face include eyebrow movement such as lifting the inner eyebrow or lifting the outer eyebrow, eyelid movement such as lifting the upper eyelid or tensing the eyelid, nose movement such as wrinkling the nose, lip movement such as lifting the upper lip or pursing the lips, cheek movement such as lifting the cheek, jaw movement such as lowering the chin, etc. The states of multiple parts of the face may be combined as the state of the learner.

[0133] As a method for judging the concentration level of a learner from facial expressions, for example, if the eyebrows move upward, it can be judged that the learner is concentrating because he or she is paying more attention to the visual target. Also, for example, if the cheeks move upward while gazing at a person's face, it can be judged that the learner is concentrating on the visual target because he or she is making an expression toward the other person.

[0134] Moreover, Fig. 8 is a table showing an example of data of a veteran customer servicer pre-stored in the storage unit 130. Fig. 8 shows, as an example of the data of a veteran customer servicer, the timing at which the object was viewed, the target of the viewing, the content of the utterance, the tone of the voice, the action, and the concentration level.

[0135] FIG. 9 also shows information on the "deviation" between the learner's data shown in FIG. 7 and the veteran customer's data shown in FIG. 8 (quantitative information shown by the value of "+ / -" or "+ / -" (learner-veteran customer) in the table), or information on whether the data matches the veteran customer's data (qualitative information shown by "○ / ×" in the table). As shown in FIG. 9, when the learner's data and the veteran customer's data do not match, the data may include the veteran customer's data ("correct answer" in the table).

[0136] The importance determining unit 114 determines the importance of each element included in the annotated series information of the target person by referring to the information shown in Fig. 7 to Fig. 9. For example, the information corresponding to event ID "4" in Fig. 9 indicates that the target person did not utter an utterance, even though the experienced customer servicer uttered "I will confirm your order." The importance determining unit 114 may set a higher importance to such a portion in the series information where the behavior of the experienced customer servicer and the target person is significantly different than to other portions.

[0137] For example, the information corresponding to event ID "3" indicates that there is a difference of 7 seconds between the experienced customer and the target in the timing of performing the action corresponding to the information. The importance determination unit 114 may set a higher importance to such a portion in the series information where the difference in timing of the action between the experienced customer and the target is large than other portions. For example, the importance determination unit 114 may set a higher importance to a portion where the target's concentration level is lower than a predetermined threshold than other portions, even if the difference in concentration level from the experienced customer is relatively small.

[0138] The configuration of the playback screen of the presentation information exemplified in Fig. 5 can also be applied to this example. The presentation information generating unit 118 may generate the presentation information so that the screen 315 in Fig. 5 includes text or the like indicating that there was a difference in the behavior between the experienced customer servicer and the target person at the target timing and the content of the difference. <Another Example 3 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.> Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Note that the same description as that already described in the above examples will not be repeated.

[0139] The sequence information in this example is, for example, video data capturing the actions of assemblers in FA (factory automation), image data captured by wearable cameras attached to assemblers, assembly sounds, and information recording visual targets. Note that the assemblers may be multiple assemblers who assemble the same product, or assemblers who assemble the same product on different lines.

[0140] 10 is a table showing an example of data on the subject's visual target, action, concentration level (blinking, gaze fixation time, and pupil state), and assembly sounds, and is a table showing an example of annotations added to series information at each time point by the annotation adding unit 214 according to this example. As shown in Fig. 10, the annotation adding unit 214 adds, as an example, the visual target, speech, action, concentration level, and assembly sounds as annotations at each time point.

[0141] FIG. 11 is a table showing an example of data of an experienced assembler stored in advance in the storage unit 130. FIG. 11 shows, as an example of the experienced assembler's data, the timing of visually recognizing an object, the visual target, the action, the concentration level, and the sound of assembly. FIG. 12 shows information on the "deviation" between the subject's data shown in FIG. 10 and the experienced assembler's data shown in FIG. 11 (quantitative information shown by the value of "+ / -" or "+ / -" (subject-experienced assembler) in the table), or information on whether the data matches the experienced assembler's data (qualitative information shown by "○ / ×" in the table). As shown in FIG. 12, when the subject's data does not match the experienced assembler's data, the experienced assembler's data ("correct answer" in the table) may be included.

[0142] 10 to 12, the importance determining unit 114 determines the importance of each element included in the annotated series information of the subject. For example, the information corresponding to event ID "3" in Fig. 10 indicates that the experienced assembler is holding an end portion of part B with his left hand, while the subject is holding the center portion of part B with his right hand. The importance determining unit 114 may set a higher importance for such a portion in the series information where the difference between the actions of the experienced assembler and the subject is large than for other portions.

[0143] For example, the information corresponding to event ID "7" is a process of assembling part B to part A, and indicates that there is an error of 18 seconds between the experienced assembler and the subject. This indicates that the time required for the same assembly operation differs by 18 seconds between the experienced assembler and the subject. The importance determination unit 114 may set a higher importance to a portion in the series information where the difference in the time required for the same operation is large than other portions. For example, the importance determination unit 114 may set a higher importance to a portion where the subject's concentration level is lower than a predetermined threshold than other portions, even if the difference in the concentration level from that of the experienced assembler is relatively small. Alternatively, the importance of each event may be specified by the assembler.

[0144] The presentation information generating unit may integrate the above importance determination results to generate, for example, the following presentation information.

[0145] Presentation information may be generated to present points for improvement and points requiring attention in each process for the target person. Based on the presentation information, a video work manual for new assemblers may be created that shows close-ups or slow-motion playback of difficult processes, processes that are prone to mistakes, and other high-importance processes.

[0146] To allow managers to manage assemblers, display information may be generated that shows time series changes in the performance of each assembler and areas where multiple assemblers make mistakes.

[0147] Furthermore, in order to improve each process, it is possible to generate suggested information that suggests improvements to be made to the process itself, such as changing the order of processes, separating a process into multiple processes, or integrating multiple processes, for the parts where most assemblers make mistakes.

[0148] The configuration of the playback screen of the presentation information exemplified in Fig. 5 can also be applied to this example. The presentation information generating unit 118 may generate the presentation information so that the screen 315 in Fig. 5 includes text or the like indicating that there was a difference in the action between the experienced assembler and the target person at the target timing and the content of the difference. <Another Example 4 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.> Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

[0149] The series information in this example is, for example, video data capturing the actions of a chef in cooking, image data captured by a wearable camera attached to the chef, cooking sounds, and information recording visual targets. The chefs may be multiple chefs who cook the same menu, and may be chefs who want to learn cooking for themselves at home, or employees who cook in the kitchens of restaurants or supermarkets.

[0150] 13 is a table showing an example of data on the subject's visual target, action, concentration level (blinking, gaze fixation time, and pupil state), and cooking sounds, and is a table showing an example of annotations added to series information at each time point by the annotation adding unit 214 according to this example. As shown in FIG. 13, the annotation adding unit 214 adds, as an example, the visual target, speech, action, and concentration level as annotations at each time point.

[0151] FIG. 14 is a table showing an example of data of an experienced cook prestored in the storage unit 130. FIG. 14 shows, as an example of the experienced cook's data, the timing of visualizing an object, the visual target, the action, the concentration level, and the cooking sound. FIG. 15 shows information on the "deviation" between the subject's data shown in FIG. 13 and the experienced cook's data shown in FIG. 14 (quantitative information shown by the value of "+ / -" or "+ / -" (subject-experienced cook) in the table), or information on whether the data matches the experienced cook's data (qualitative information shown by "○ / ×" in the table). As shown in FIG. 15, when the subject's data does not match the experienced cook's data, the experienced cook's data ("correct" in the table) may be included.

[0152] The importance determination unit 114 determines the importance of each element included in the annotated series information of the subject by referring to the information shown in Fig. 13 to Fig. 15. For example, the information corresponding to event ID "3" in Fig. 13 indicates that, when holding a knife, an experienced chef places the index finger on the back of the knife and grasps the knife with the middle finger and thumb, whereas the subject grasps the knife with the thumb and index finger. The importance determination unit 114 may set a higher importance to such parts of the series information where the difference between the actions of the experienced chef and the subject is large than to other parts.

[0153] Also, for example, the information corresponding to the event ID "5" indicates that there is an error of 8 seconds between the veteran cook and the subject when cutting ingredients. Therefore, it indicates that the time required for the same cooking action differs by 8 seconds between the veteran cook and the subject. The importance determination unit 114 may set a higher importance to a part in the series information where the difference in the time required for the same action is large than other parts. Also, for example, the importance determination unit 114 may set a higher importance to a part where the subject's concentration level is lower than a predetermined threshold value than other parts, even if the difference with the concentration level of the veteran cook is relatively small. Alternatively, a high importance may be assigned to a difficult process or a process that affects the taste or quality depending on the cook.

[0154] The presentation information generating unit may integrate the results of the importance determination and present, for example, the following presentation information.

[0155] Presentation information may be generated to present points for improvement and points to be aware of in each cooking situation for the target user. Based on the presentation information, a cooking manual may be created using videos that show close-ups or slow-motion playback of difficult or mistake-prone steps and other important steps.

[0156] If the subject is an employee who cooks in the kitchen of a restaurant or supermarket, presentation information may be generated that shows time-series changes in the performance of each chef and the mistakes made by multiple chefs, in order to enable the manager to manage the chefs.

[0157] Furthermore, in order to improve each process, it is possible to generate presentation information that presents improvements to the process itself, such as changing the order of the processes, separating a process into multiple processes, or integrating multiple processes, for the parts where most cooks make mistakes.

[0158] The configuration of the playback screen of the presentation information exemplified in Fig. 5 can also be applied to this example. The presentation information generating unit 118 may generate the presentation information so that the screen 315 in Fig. 5 includes text or the like indicating that there was a difference in the action between the experienced assembler and the target person at the target timing and the content of the difference. <Another Example 5 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.> Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

[0159] In this example, the subjects include players in a sport. The annotations may be provided by a coach, by the annotation providing unit 214, or a combination thereof.

[0160] Also, one coach may provide annotations to multiple players, A plurality of coaches may provide annotations to one player, and further, a plurality of coaches may provide annotations to a plurality of players.

[0161] Alternatively, a configuration may be provided in which multiple annotation assigning units 214 that have been trained separately are provided, and multiple annotation assigning units may assign annotations to one player. Furthermore, multiple annotation assigning units may assign annotations to multiple players.

[0162] In this example, the series information is, for example, image, voice, and body sensing data of the player. The player may be a player of any sport, but here, for example, a baseball player (batter) will be described. The player may be a professional athlete, a student, or a member of society.

[0163] 16 is a table showing an example of data on the gaze target, action, and concentration level (blink, gaze fixation time, and pupil state) of Player A (a player with a low batting average), and is a table showing an example of annotations added to series information at each time point by the annotation adding unit 214 according to this example. As an example, the annotation adding unit 214 adds the gaze target, speech, action, and concentration level as annotations at each time point.

[0164] FIG. 17 also shows, as an example of data for player B (a player with a high batting average), the timing at which the object was viewed, the viewed object, the action, the concentration level, and the sound of the play.

[0165] FIG. 18 also shows information on the "deviation" between the data of player A shown in FIG. 16 and the data of player B shown in FIG. 17 (quantitative information shown by the value of (player A-player B) using "+ / -" or "+ / -" in the table), or information on whether the data matches the data of player B (qualitative information shown by "○ / ×" in the table). As shown in FIG. 18, when the data of player A and the data of player B do not match, the data of player B ("correct answer" in the table) may be included.

[0166] The importance determination unit 114 determines the importance of each element included in the annotated series information of each player by referring to the information shown in Figures 16 to 18. For example, the information corresponding to event ID "1" in Figure 16 indicates that player B holds the bat short, whereas player A holds the bat long. The importance determination unit 114 may set a higher importance to such parts of the series information where the difference in action between players is large than to other parts.

[0167] Also, for example, the information corresponding to event ID "3" indicates that the swing speed of player A is 135Km / hour, while the swing speed of player B is 148Km / hour, indicating an error of 13km / hour. The importance determination unit 114 may set a higher importance to a portion in the series information where there is a large difference in the speed at which the same action is performed than to other portions. Also, for example, the importance determination unit 114 may set a higher importance to a portion where the player's concentration level is lower than a predetermined threshold than to other portions. Alternatively, a high importance may be designated for actions that vary greatly depending on the player.

[0168] The presentation information generation unit may integrate the results of the importance determination and, for example, generate presentation information for each player by integrating information annotated by multiple coaches, which presents areas in each player's play that need improvement and parts of the play that need improvement (image scenes).

[0169] Also, by integrating information annotated by one or more coaches for multiple players, presentation information may be generated for coaches or other instructors, which presents areas for improvement in instruction. Furthermore, based on the presentation information, a video instruction manual may be created in which important scenes for successful batting are shown in close-up or in slow motion.

[0170] In the above example, the input of plays by individual players, such as batting, was described as an example of sequence information. However, the sequence information may also be directed to plays and actions by multiple players, such as a formation in soccer.

[0171] Furthermore, the input data may include information about spectators of the game. For example, the input data may be video data capturing images of the spectators, information about the line of sight of the spectators, and sounds emitted by the spectators.

[0172] Based on this data about spectators, the spectators' level of attention and excitement for each scene during the game may be added as annotations.

[0173] The presentation information generation unit may generate presentation information for players and coaches that presents scenes that have attracted the attention of spectators.

[0174] Other examples of the annotation assignment process, the importance assignment process, and the presentation information generation process will be described below. <Another Example 6 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.> Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

[0175] In this example, electronic sports (e-sports) in which characters operated by players existing in the real world compete against each other in a virtual space will be taken as an example. Note that the players may be multiple players playing the same e-sport.

[0176] The series information in this example includes, as an example, inputs by the player to a personal computer, a game operation unit, etc. The series information in this example may also include information regarding the movement of a character in a virtual space that is generated based on these inputs. The series information may also include image, audio, and body sensing data of the player. Furthermore, the series information in this example may include information regarding spectators of e-sports, as in the case of sports described above. It may also include information regarding the level of attention and excitement of the spectators.

[0177] The annotation adding unit 214 refers to the sequence information and adds, as an example, the character's action, the spectator's level of attention, and the excitement level at each point in time as annotations.

[0178] The importance determination unit 114 refers to the annotated series information of multiple players and determines the importance of each element included in the annotated series information. For example, the importance determination unit 114 may set a higher importance to a portion of the series information in which there is a large difference in behavior between characters than to other portions.

[0179] Also, for example, the importance determination unit 114 may set a higher importance to a portion where the player's concentration level is lower than a predetermined threshold value than to other portions. Alternatively, a higher importance may be assigned to an action or the like that varies greatly depending on the player.

[0180] The presentation information generation unit may then generate presentation information for the player that presents scenes that attracted high attention from e-sports spectators.

[0181] Furthermore, the presentation information generating unit may integrate the results of the importance determination and generate presentation information for each player, which presents, for example, points in each player's play that need improvement and parts of the play that need improvement. <Another Example 7 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.> Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

[0182] The sequence information in this example is information that records video data of the actions of the office worker in the office work, image data captured by a wearable camera attached to the office worker, work sounds, visual targets, etc. The sequence information may also include operation history of terminal devices such as a keyboard and mouse, application usage history, etc.

[0183] As an example, the annotation adding unit 214 adds, as annotations, information about a visual target, speech, movement, concentration level, working sounds, an operation history of a terminal device, and an application usage history at each time point.

[0184] The importance determination unit 114 determines the importance of each element included in the annotated series information of the target office worker (target person) by referring to the annotated series information of the target office worker and the reference annotated series information. Here, the reference annotated series information may include the annotated series information of other office workers such as veterans.

[0185] For example, the importance determination unit 114 may set a higher importance level when the subject spends more time on text input than other office workers. Also, the importance determination unit 114 may set a higher importance level when the subject spends more time on spreadsheets than other office workers.

[0186] The presentation information generating unit may integrate the above importance determination results to present, for example, the following presentation information.

[0187] Presentation information may be generated to present points for improvement and points to be aware of in each work situation for the target person. Based on the presentation information, a work manual may be created using videos that show close-ups or slow-motion playback of difficult work, work that is prone to mistakes, and other important work.

[0188] Furthermore, in order to improve each task, for tasks that most office workers make mistakes in, suggestion information may be generated that suggests improvements to the tasks themselves, such as changing the order of tasks, separating a task into multiple tasks, or integrating multiple tasks. <Another Example 8 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.> Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

[0189] The sequence information in this example is, for example, video data capturing the actions of the telephone operator (subject), image data captured by a wearable camera attached to the telephone operator, audio data of the conversation with the other party on the telephone, sounds of the work, and information recording the visual target. The sequence information may also include the operation history of terminal devices such as a keyboard and a mouse.

[0190] For example, the annotation assigning unit 214 assigns, as annotations, information on the target's visual target, speech, movement, concentration level, work sounds, operation history of the terminal device, application usage history, etc. at each time point. The annotation assigning unit 214 may also estimate the tension level of the target at each time point and assign an index indicating the tension level as an annotation. Note that a publicly known algorithm can be used as the tension level estimation algorithm.

[0191] For example, the importance determination unit 114 determines the importance of each element included in the annotated sequence information of the target person by referring to the annotated sequence information of the target person and the reference annotated sequence information, which may include the annotated sequence information of another telephone operator, such as a veteran.

[0192] For example, the importance determination unit 114 may set a higher importance level when the target person has a higher level of tension than other telephone operators. Also, the importance determination unit 114 may set a higher importance level when the target person has a higher level of tension toward a specific telephone caller than the other telephone callers.

[0193] The presentation information generating unit may integrate the above importance determination results to present, for example, the following presentation information.

[0194] Presentation information may be generated to present to the target users points that need improvement and points that need attention in each telephone response situation. Based on the presentation information, a work manual may be created using videos that show close-ups or slow-motion playback of important tasks such as difficult telephone responses or telephone responses that are prone to mistakes.

[0195] Furthermore, for the administrator, display information may be generated that indicates the telephone response contents and callers that cause most tension to telephone responders. <Another Example 9 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.> Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

[0196] The sequence information in this example is, for example, video data capturing the movements of a student (subject) at a driving school, image data captured by a wearable camera attached to the subject, audio data of interactions with an instructor, sounds of driving movements, and information recording visual targets. The sequence information may also include sensing data of operating parts such as the steering wheel and brakes.

[0197] For example, the annotation assigning unit 214 assigns, as annotations, information on a visual target, an action, a concentration level, speech, a driving operation sound, an operation history of an operation unit, and the like at each time point. The annotation assigning unit 214 may also estimate a tension level of the subject at each time point and assign an index indicating the tension level as an annotation. A publicly known algorithm may be used as the tension level estimation algorithm.

[0198] For example, the importance determination unit 114 determines the importance of each element included in the annotated series information of the subject by referring to the annotated series information of the subject and reference annotated series information. Here, the reference annotated series information may include annotated series information of an exemplary driver such as an instructor.

[0199] For example, the importance determination unit 114 may set a higher importance when the timing at which the subject driver starts to turn the steering wheel and the timing at which the model driver starts to turn the steering wheel when turning a certain curve are greater than a predetermined value. Also, the importance determination unit 114 may set a higher importance when the degree of tension on a specific course is higher than the degree of tension on other courses.

[0200] The presentation information generating unit may integrate the results of the importance determination and present, for example, the following presentation information.

[0201] Presentation information may be generated to present points for improvement and points to be aware of in each driving scene for the target person. Based on the presentation information, a driving manual may be created using videos that show close-ups or slow-motion playback of difficult driving scenes and other high-importance driving scenes.

[0202] Furthermore, presentation information may be generated for instructors that indicates the courses or driving scenes that are most stressful for most students.

[0203] <Embodiment 2> The configuration of the information processing system in the present invention is not limited to the above. Fig. 10 is a block diagram showing a schematic configuration example of the information processing system 1 according to the second embodiment. The information processing system 1 of the second embodiment is different from the information processing system 1 of the first embodiment in that the importance determination unit 215 is not provided in the server 100 but in the learner's terminal device 200, but is otherwise the same in configuration.

[0204] Next, an operation of the information processing system 1 according to the second embodiment will be described. Fig. 11 is a sequence diagram illustrating an example of the operation of the information processing system 1 according to the second embodiment. A process of generating presentation information in the information processing system 1 according to the second embodiment will be described with reference to Fig. 11.

[0205] (Step S202) In step S202, the annotation assigning unit 214 generates annotated series information by assigning annotations to each element included in the series information related to the subject acquired by the series information acquiring unit 212.

[0206] (Step S206) In step S206, the importance determination unit 215 determines the importance of each element included in the annotated series information of the target person by referring to the annotated series information of the target person generated by the annotation assignment unit 214. The communication unit 220 of the learner's terminal device 200 transmits the importance determination result to the server 100.

[0207] (Step S207) In step S207, series information acquisition section 112 of server 100 receives the result of the importance determination.

[0208] (Step S208) In step S208, the discrimination integration unit 116 of the server 100 generates discrimination integration information by referring to the results of the importance discrimination for the multiple subjects A, B, C, . . . .

[0209] (Step S210) In step S210, the presentation information generating unit 118 generates presentation information by referring to at least one of the discrimination result by the importance discriminating unit 215 and the discrimination integration information. When the presentation information generating unit 118 generates presentation information by referring only to the discrimination result by the importance discriminating unit 215, the process of step S208 does not necessarily have to be executed, and the process of step S210 may be executed following step S207. The communication unit 120 of the server 100 transmits the generated presentation information to each learner's terminal device 200.

[0210] (Step S212) In step S212, the presentation information acquiring unit 216 of each learner's terminal device 200 acquires the generated presentation information.

[0211] (Step S214) In step S214, the presentation unit (display unit) 244 presents the acquired presentation information.

[0212] Furthermore, in step S214, the annotation assigning unit 214 may refer to information included in the presentation information to update the determination logic used when an annotation is assigned to series information.

[0213] (Step S216) In step S216, the terminal device 200 acquires feedback information. As described above, the feedback information is generated as a result of the subject confirming the presentation information together with the instructor and confirming and correcting the determined importance. The feedback information is transmitted from the terminal device 200 to the server 100.

[0214] (Step S218) In step S218, the importance discrimination unit 215 of the learner's terminal device 200 updates its discrimination logic based on the feedback information. For example, the importance discrimination unit 215 updates its importance discrimination logic based on feedback information including the confirmed and corrected importance. The discrimination integration unit 116 of the server 100 also updates its discrimination integration logic based on the feedback information.

[0215] (Step S220) When the lineage information acquirer 212 of the terminal device 200 acquires lineage information again, the annotation assigner 214 generates annotated lineage information by assigning annotations to the lineage information in step S220. The following steps S220 to S236 are the same as the above steps S202 to S218.

[0216] In the information processing system 1 of the second embodiment, the learner's terminal device 200 repeats the above steps S202 to S218 every time it acquires sequence information.

[0217] Therefore, the importance discrimination unit 215 of the learner's terminal device 200 updates the importance discrimination logic every time it acquires sequence information from the subject. As the amount of sequence information from the subject increases, the importance discrimination logic is improved and the importance can be discriminated more appropriately. The same applies to the discrimination logic used by the discrimination and integration unit 116 in the discrimination and integration process.

[0218] <Embodiment 3> 12 is a block diagram illustrating a schematic configuration example of an information processing system 1 according to embodiment 3. The information processing system of embodiment 3 is different from the information processing system of embodiment 1 in that a presentation information generating unit 1013 is not provided in the server 100 but in the instructor terminal device 1000, but the other configurations are the same.

[0219] Next, an operation of the information processing system 1 according to the third embodiment will be described. Fig. 13 is a sequence diagram illustrating an example of the operation of the information processing system 1 according to the third embodiment. With reference to Fig. 13, a process of generating presentation information in the information processing system 1 according to the third embodiment will be described.

[0220] (Step S302) In step S302, the annotation assigner 214 generates annotated series information by assigning annotations to each element included in the series information about the subject acquired by the series information acquirer 212. The communication unit 220 transmits the generated annotated series information to the server 100.

[0221] (Step S304) In step S304, the series information acquirer 112 of the server 100 acquires the annotated series information.

[0222] (Step S306) In step 306, the importance determining unit 114 refers to the annotated series information related to the subject, and determines the importance of each element included in the annotated series information of the subject.

[0223] (Step S308) In step S308, the discrimination integration unit 116 generates discrimination integration information by referring to the discrimination results of the importance levels for the multiple subjects A, B, C, etc. The generated discrimination integration information is transmitted to the instructor terminal device 1000.

[0224] (Step S310) In step S310, the presentation information generating unit 1013 of the instructor terminal device 1000 generates presentation information by referring to at least one of the discrimination result by the importance discrimination unit 114 and the discrimination integration information. When the presentation information generating unit 1013 generates presentation information by referring only to the discrimination result by the importance discrimination unit 114, the process of step S308 does not necessarily need to be executed, and the discrimination result in step S306 may be transmitted to the instructor terminal device 1000, followed by the process of step S310. The generated presentation information is transmitted to each learner terminal device 200.

[0225] (Step S312) In step S312, in each learner's terminal device 200, the presentation information acquiring section 216 acquires the generated presentation information.

[0226] (Step S314) In step S314, the presentation unit (display unit) 244 presents the acquired presentation information.

[0227] Furthermore, in step S314, the annotation assigning unit 214 may refer to information included in the presentation information to update the determination logic used when an annotation is assigned to series information.

[0228] (Step S316) In step S316, the instructor terminal device 1000 acquires feedback information. As described above, the feedback information is generated as a result of the subject confirming the presentation information together with the instructor and confirming and correcting the determined importance. The feedback information is transmitted from the instructor terminal device 1000 to the server 100.

[0229] (Step S318) In step S318, the importance discrimination unit 114 or the discrimination integration unit 116 of the server 100 updates each discrimination logic based on feedback information from the instructor terminal device 1000. For example, the importance discrimination unit 114 updates the importance discrimination logic based on feedback information including the confirmed and corrected importance from the instructor terminal device 1000.

[0230] (Step S320) When the lineage information acquirer 212 of the terminal device 200 acquires lineage information again, the annotation assigner 214 generates annotated lineage information by assigning annotations to the lineage information. The following steps S320 to S336 are the same as the above steps S302 to S318.

[0231] In the information processing system 1, the learner's terminal device 200 repeats the above steps S302 to S318 every time it acquires sequence information.

[0232] Therefore, each time the server 100 obtains series information from a subject, it updates each discrimination logic. As the amount of series information from the subject increases, each discrimination logic is improved, and each discrimination can be performed more appropriately.

[0233] [Software implementation example] The control block of the information processing device 100 (particularly, the series information acquisition unit 112, the importance discrimination unit 114, the discrimination and integration unit 116, and the presentation information generation unit 118) and each terminal device 200 (particularly, the series information acquisition unit 212, the annotation assignment unit 214, and the presentation information acquisition unit 216) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

[0234] In the latter case, the information processing device 100 and each terminal device 200 are provided with a computer that executes instructions of a program, which is software that realizes each function. This computer is provided with, for example, one or more processors, and a computer-readable recording medium that stores the program. In the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention. The processor can be, for example, a CPU (Central Processing Unit). The recording medium can be a "non-transient tangible medium", for example, a ROM (Read Only Memory), as well as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like. The computer may further include a RAM (Random Access Memory) that expands the program. The program may be supplied to the computer via any transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0235] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0236] 1. Information Processing Systems 100 Information processing device 100 Servers 110, 210, 1010 Control unit 112, 212 Series information acquisition section 114, 215 Importance determination unit 116 Discrimination and Integration Division 118, 1013 Presentation information generation unit 120, 220, 1020 Communications Department 130, 230, 1030 storage section 200 Student terminal equipment 214 Annotation section 216, 1014 Presentation information acquisition unit 241 Camera 242 Mike 243, 1043 Operation reception desk 244 Display section (presentation section) 1044 Display section 245, 1045 speakers 1000 Lecturer's terminal device 1012 Feedback information acquisition unit

Claims

1. a sequence information acquiring unit that acquires annotated sequence information in which each element included in sequence information related to a subject is annotated based on at least one of a speech sound and a work sound of the subject; an importance determination unit that determines the importance of each element included in the annotated series information of the subject by referring to the annotated series information of the subject; a discrimination integration unit that generates discrimination integration information by referring to the discrimination results by the importance discrimination unit for a plurality of subjects; a presentation information generating unit that generates presentation information by referring to at least one of a result of the determination by the importance determining unit and the determination integration information; Equipped with The information processing device according to claim 1, wherein the importance determining unit determines, as the importance, a degree of concentration regarding the subject based on movement of the subject's eyebrows, cheeks, nose, lips, or chin.

2. The importance determining unit updates a determination logic by referring to at least one of feedback information from a user and the determination integration information.

2. The information processing apparatus according to claim 1,

3. The sequence information includes information regarding the subject's line of sight 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. The discrimination integration unit extracts common information from the discrimination results by the importance discrimination unit for each of the plurality of subjects, and includes the common information in the discrimination integration information.

4. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises: a first input section;

5. a sequence information acquisition unit for acquiring sequence information relating to a subject; an annotation unit that generates annotated sequence information by adding annotations to each element included in the acquired sequence information in accordance with at least one of the speech of the subject and the work sound; an importance determination unit that determines the importance of each element included in the annotated series information of the subject by referring to the annotated series information of the subject; a presentation information acquisition unit that acquires presentation information generated by referring to the annotated series information and a result of the importance determination unit; A presentation unit that presents the presentation information; Equipped with The importance determination unit is a terminal device that determines the degree of concentration regarding the subject as the importance based on the movement of the subject's eyebrows, cheeks, nose, lips, or chin.

6. The method further includes a discrimination integration information acquisition unit that acquires discrimination integration information generated by referring to discrimination results regarding the annotated series information of a plurality of subjects, The annotation assigning unit updates an annotation assigning logic by referring to at least one of feedback information from a user and the discrimination integration information. The terminal device according to claim 5.

7. The annotation assigning unit assigns, in addition to the annotation, information indicating the reliability of the annotation to each of the elements.

7. The terminal device according to claim 5 or 6.

8. An acquisition unit that acquires sequence information regarding a subject; an annotation unit that generates annotated sequence information by adding annotations to each element included in the acquired sequence information in accordance with at least one of the speech of the subject and the work sound; an importance determination unit that refers to annotated series information related to a certain subject and determines the importance of each element included in the annotated series information of the subject; a discrimination integration unit that generates discrimination integration information by referring to discrimination results by the importance discrimination unit for a plurality of subjects; a presentation information generating unit that generates presentation information by referring to at least one of a result of the determination by the importance determining unit and the determination integration information; Equipped with The importance determination unit determines, as the importance, a degree of concentration regarding the subject based on movement of the subject's eyebrows, cheeks, nose, lips, or chin.

9. 1. An information processing method performed by an apparatus, comprising: an annotation step of generating annotated sequence information by adding annotations to each element included in the sequence information related to the subject in accordance with at least one of a speech sound and a work sound of the subject; an importance determination step of determining the importance of each element included in the annotated series information of the subject by referring to the annotated series information; a discrimination integration step of generating discrimination integration information by referring to the discrimination results in the importance discrimination step for a plurality of subjects; a presentation information generating step of generating at least one of the discrimination result in the importance discrimination step and the discrimination integration information; Including, An information processing method characterized in that in the importance determination step, the importance is determined based on the movement of the subject's eyebrows, cheeks, nose, lips or chin, as a degree of concentration regarding the subject.

10. 5. A program for causing a computer to function as an information processing device according to claim 1, the information processing program causing a computer to function as the series information acquisition unit, the importance discrimination unit, the discrimination and integration unit, and the presentation information generation unit.

Citation Information

Patent Citations

  • Hydraulic circuit structure for four-wheel-drive type working vehicle

    JP1989016428A

  • Gaze point analyzing device

    JP1992282129A

  • Foreign language reading comprehension learning support device

    JP2005338173A

  • Driving aptitude diagnosing method, evaluation standard determining method for driving aptitude diagnosis, and driving aptitude diagnostic program

    JP2008139553A

  • Vehicle control unit

    JP2008213823A