Training support apparatus, training support method, and computer-readable recording medium

Abstract

To enable effective education support.SOLUTION: The training support apparatus 100 constructs a three dimensional structure in the virtual space based on the acquired image data. The training support apparatus 100 collects history information of the user's operation on the constructed three dimensional structure. The training support apparatus 100 outputs information about the user's motion based on the collected history information about the user's motion.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an education support device, an education support method, and an education support program. [Background technology]

[0002] Education and guidance may be provided in various fields such as medicine, industry, sports, etc. Examples of education and guidance include classroom education using textbooks and videos, and practical training in which the educational content is put into practice.

[0003] Therefore, as a technology for efficiently providing education and guidance, for example, a technology is known in which an education screen is automatically displayed according to the student's movements obtained from measurement information for grasping the student's movements and photographing information taken by a photographing unit (see, for example, Patent Document 1). The above-mentioned conventional technology makes it possible to provide highly effective education to students. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2024-070364 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with conventional technologies, it is sometimes difficult to provide effective educational support. For example, in the medical field, education on medical procedures is sometimes conducted through simulated surgeries, but with conventional technologies, it is difficult to accurately reproduce teaching materials such as organs, skeletons, and blood vessels to perform the simulated surgeries. Also, in sports, instruction is sometimes provided in which an instructor demonstrates a movement and the trainee imitates it, but with conventional technologies, it is difficult to accurately reproduce the instructor's movements and enable the trainee to trace the instructor's reproduced movements. [Means for solving the problem]

[0006] Therefore, in order to solve the above-mentioned problems and achieve the objectives, the educational support device of the present invention is characterized by having a construction unit that constructs a three-dimensional structure in a virtual space based on acquired image data, a collection unit that collects historical information of a user's actions relative to the three-dimensional structure constructed by the construction unit, and an output unit that outputs information regarding the user's actions based on the historical information of the user's actions collected by the collection unit. [Effects of the Invention]

[0007] The present invention has the effect of enabling effective educational support. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an overall view of the processing of an education support device according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating a configuration of the education support device according to the embodiment. [Figure 3] FIG. 3 is a table illustrating an example of image data according to the embodiment. [Figure 4] FIG. 4 is a table diagram illustrating an example of sensor data according to the embodiment. [Figure 5] FIG. 5 is a table diagram illustrating an example of operation history information according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of educational support according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of educational support according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating a process performed by the education support device according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating IOWN. [Figure 10] FIG. 10 is a diagram showing an example of generation of a work record according to a modified example. [Figure 11]FIG. 11 is a diagram illustrating an example of a computer that realizes the education support device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention (hereinafter referred to as "embodiments") will be described with reference to the drawings. Note that the embodiments are not limited to the following description.

[0010] <Overview> (background) When education or instruction is provided in various fields such as medicine, industry, and sports, it is done through classroom education using textbooks, videos, etc., or practical education in which the educational content is put into practice, etc. Therefore, as a technology for efficiently providing education or instruction, for example, a reference technology is known that automatically displays an educational screen in accordance with the user's (student's) movements detected using measurement information on the user's (student's) movements and video information captured by a camera, etc.

[0011] The above-mentioned reference technology can provide highly effective education to students (users), but it may be difficult to provide effective educational support in the fields of medicine and sports. For example, in the case of simulated surgery in the medical field, the conventional reference technology has difficulty in accurately reproducing teaching materials such as organs, skeletons, and blood vessels, making it difficult to realize an effective simulated surgery.

[0012] On the other hand, in sports where instruction is given by an instructor demonstrating exemplary movements and the trainee imitating them, it is difficult with conventional reference technology to accurately reproduce the instructor's movements, and it is also difficult for the trainee to trace the instructor's reproduced movements.

[0013] (Processing by the education support device 100) Therefore, the education support device 100 according to this embodiment displays to the user a three-dimensional structure in a virtual space constructed using still images and moving images (hereinafter, sometimes simply referred to as "image data") captured by an imaging device such as a camera. The education support device 100 then collects historical information on the user's actions relative to the three-dimensional structure and outputs feedback and the like regarding the actions to the user.

[0014] Here, an overview of the processing by the education support device 100 will be described with reference to Fig. 1. Fig. 1 is a diagram illustrating an overview of the processing by the education support device 100 according to an embodiment. The education support device 100 shown in Fig. 1 is an example of a computer that executes the processing described below.

[0015] The education support device 100 constructs a three-dimensional structure based on the acquired image data ((1) in FIG. 1). Then, the education support device 100 displays the virtual space and the three-dimensional structure in the virtual space to the user ((2) in FIG. 1).

[0016] Next, the educational support device 100 collects historical information about the user's actions relative to the constructed three-dimensional structure based on image data of the user's actions or sensor data acquired by sensors attached to the user or to tools, machines, etc. used by the user ((3) in Figure 1).

[0017] The action history information is information about the trajectory of the user's action relative to the three-dimensional structure, the action speed, the amount of force applied to the action, etc. Hereinafter, "action history information" may be referred to as "action history information."

[0018] Then, based on the collected user behavior history information, the educational support device 100 outputs information about the user's behavior (hereinafter, sometimes referred to as "behavior information"), such as differences in behavior from other users and behaviors that the user is not good at (weak behaviors) ((4) in Figure 1).

[0019] In this way, the education support device 100 according to this embodiment constructs a three-dimensional structure using image data and displays it to the user, and generates movement information using movement history information for the three-dimensional structure.

[0020] As a result, the education support device 100 enables the user to perform more realistic simulated movements when learning simulated movements, and also enables feedback of the differences in the simulated movements between the user and other users, movements that the user is weak at, etc. As a result, the education support device 100 has the effect of enabling effective education support.

[0021] <Explanation of the Educational Support Device 100> Next, the configuration of the education support device 100 according to this embodiment will be described. Fig. 2 is a diagram showing the configuration of the education support device 100 according to this embodiment. As shown in Fig. 2, the education support device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Although not shown in Fig. 2, the education support device 100 can also have an input unit such as a keyboard or a mouse for receiving input such as operations by an administrator or the like.

[0022] (Communication unit 110) The communication unit 110 performs data communication related to input of information such as image data captured by the imaging device 200 or the like, operation history information collected by the sensor 300 or the like, and predetermined user commands input from the terminal device 400. The communication unit 110 also performs data communication related to output of information related to the virtual space and the constructed three-dimensional structure.

[0023] The communication unit 110 is realized by a NIC (Network Interface Card) or the like, and controls communication via an electric communication line such as a LAN (Local Area Network), the Internet, etc. The communication unit 110 is connected to a network by wire or wirelessly as necessary, and can transmit and receive information bidirectionally with the imaging device 200, the sensor 300, the terminal device 400, etc.

[0024] (Storage unit 120) The storage unit 120 stores data and programs used for various processes by the control unit 130, and various data acquired by the operation of the control unit 130. The storage unit 120 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 2 , the storage unit 120 has an image data DB 121, a sensor data DB 122, a virtual space DB 123, a three-dimensional structure DB 124, and an operation history information DB 125.

[0025] (Image data DB121) The image data DB 121 is a database that stores image data including still images and moving images captured by the imaging device 200. An example of image data stored in the image data DB 121 will now be described with reference to FIG.

[0026] Fig. 3 is a table diagram showing an example of image data according to the embodiment. As shown in Fig. 3, the image data DB 121 stores imaging targets and imaging data in association with "No.", which is information for identifying individual image data. For example, as shown in Fig. 3, the image data DB 121 stores imaging target "A" and imaging data "video data a" in association with No. "1." Note that "A," "video data a," and the like shown in Fig. 3 are legends for the data.

[0027] The "image capture subject" refers to the target space or object that is captured by the image capture device 200, and includes, for example, actual surgery scenes, sports play scenes, rescue scenes at disaster sites, and dangerous work scenes using power saws.

[0028] "Imaging data" refers to still image data or moving image data captured by imaging device 200, and includes, for example, JPEG (Joint Photographic Experts Group), GIF (Graphics Interchange Format), PNG (Portable Network Graphics), etc. Moving image data also includes data saved in formats such as MP4 (MPEG-4 Part 14), AVI (Audio Video Interleave), MOV, WMV (Windows (registered trademark) Media Video), FLV, WebM, MPEG (Moving Picture Experts Group), MPEG2 (Generic coding of moving pictures and associated audio information), and MKV (Matroska Video File).

[0029] (Sensor data DB122) The sensor data DB 122 is a database that stores sensor data relating to user actions acquired by the sensor 300 etc. An example of the sensor data stored in the sensor data DB 122 will now be described with reference to FIG.

[0030] Fig. 4 is a table diagram showing an example of sensor data according to the embodiment. As shown in Fig. 4, the sensor data DB 122 stores a sensor name and acquired data in association with "No." This "No." is information that identifies individual sensor data. For example, as shown in Fig. 4, the sensor data DB 122 stores a sensor name "C" and acquired data "c" in association with No. "1." Note that the letters "C" and "c" shown in Fig. 3 are legends for the data.

[0031] "Sensor name" is information that identifies a sensor attached to a user or an appliance or machine operated by the user, and includes, for example, the name or identification symbol of an acceleration sensor (e.g., a 3-axis acceleration sensor), a gyro sensor (e.g., a 3-axis gyro sensor), a magnetic sensor, a pressure sensor, a temperature sensor, a motion sensor, etc. "Acquired data" is individual sensor data acquired by the various sensors described above. Note that the data format of the acquired data is not particularly limited.

[0032] (Virtual Space DB123) The virtual space DB 123 is a database that stores data on the virtual space constructed by the construction unit 132. For example, the virtual space data stores information on the size of the virtual space, the functions of the virtual space, the materials that make up the virtual space, the interactions between the materials, and the like.

[0033] The virtual space data is not particularly limited and may include various information used to configure a virtual space constructed using known technology. The virtual space may also include a digital twin constructed based on Digital Twin Computing (DTC) of the Innovative Optical and Wireless Network (IOWN), which will be described later.

[0034] (Three-dimensional structure DB124) The three-dimensional structure DB 124 is a database that stores data of three-dimensional structures to be placed in a virtual space constructed by the construction unit 132. For example, the data of the three-dimensional structures may include, but is not limited to, data on a 3DCG (3 Dimensional Computer Graphics) model constructed by a well-known technique such as photogrammetry.

[0035] (Operation history information DB125) The operation history information DB 125 is a database that stores history information of user operations (operation history information) collected by the later-described collection unit 134. Here, an example of image data stored in the operation history information DB 125 will be described with reference to FIG.

[0036] 5 is a table diagram showing an example of operation history information according to an embodiment. As shown in FIG. 5, the operation history information DB 125 stores user identification information and history information in association with "No." This "No." is information that identifies individual operation history information. For example, as shown in FIG. 5, the operation history information DB 125 stores user identification information "E" and history information "e" in association with No. "1." Note that the letters "E" and "e" shown in FIG. 5 are legends for the data.

[0037] "User identification information" is information for identifying a user, and includes, for example, a username, a combination of unique numbers, text, symbols, etc. assigned to a user, and the like.

[0038] "History information" refers to history information related to the user's actions collected by the collection unit 134, which will be described later. For example, the history information includes information such as the coordinates (starting point) from which the user's body moved to which coordinates (ending point) in the target space, and at what speed. The history information also includes information such as the amount of force with which the user applied the action to the target three-dimensional structure.

[0039] (control unit 130) Now, returning to Fig. 2, the explanation will be continued. The control unit 130 has an internal memory for temporarily storing programs defining various processing procedures and the like of the education support device 100 and processing data, and is realized by electronic circuits such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit), and integrated circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). As shown in Fig. 2, the control unit 130 has an acquisition unit 131, a construction unit 132, a display unit 133, a collection unit 134, a calculation unit 135, and an output unit 136.

[0040] (Acquisition part 131) The acquisition unit 131 acquires predetermined information from the imaging device 200, the sensor 300, etc. Specifically, the acquisition unit 131 acquires image data including still image data and moving image data captured by the imaging device 200, and stores the image data in the image data DB 121. For example, the acquisition unit 131 acquires imaging data "moving image data a" about imaging target "A," associates it with No. "1," and stores it in the image data DB 121.

[0041] Furthermore, the acquiring unit 131 acquires sensor data from the sensor 300 and stores it in the sensor data DB 122. For example, the acquiring unit 131 acquires sensor data "c" of a sensor name "C", associates it with No. "1", and stores it in the sensor data DB 122.

[0042] (Construction Section 132) The construction unit 132 constructs a virtual space and a three-dimensional structure, and stores them in the virtual space DB 123 and the three-dimensional structure DB 124, respectively.

[0043] Specifically, the construction unit 132 constructs a three-dimensional structure in the virtual space based on the image data acquired from the image capture device 200. For example, the construction unit 132 can use the acquired image data to construct a three-dimensional structure to be placed in the virtual space based on photogrammetry, which is a well-known technique.

[0044] The construction unit 132 also constructs a virtual space based on known technology using image data captured by the imaging device 200 and image data input via an external device.

[0045] (Display section 133) The display unit 133 displays to the user the virtual space and the three-dimensional structure constructed by the construction unit 132. Specifically, the display unit 133 displays the virtual space and the three-dimensional structure to the user via a head-mounted display worn by the user, a space reproduction display, or the like.

[0046] In addition to the head-mounted display and spatial reproduction display described above, the display unit 133 can display virtual spaces and three-dimensional structures to the user via devices that output information via the five human senses, such as visual devices and tactile devices.

[0047] (Collection Department 134) The collection unit 134 collects operation history information based on data acquired by a sensor or the imaging device 200, etc. Then, the collection unit 134 stores the collected operation history information in the operation history information DB 125.

[0048] The collection unit 134 collects operation history information for the three-dimensional structure constructed by the construction unit 132. For example, the collection unit 134 collects information collected by a sensor attached to an instrument operated by a user, or operation history information of a predetermined machine operated by a user, as the operation history information.

[0049] Specifically, the collection unit 134 collects the user's operation history information based on the sensor data obtained by the sensor 300 or the like in the image data DB 121. Then, the collection unit 134 stores the collected user's operation history information in the operation history information DB 125.

[0050] For example, the collection unit 134 inputs image data stored in the image data DB 121 into a model trained to extract the movements of a target person included in the image data, and extracts the user's movements included in the image data. Then, the collection unit 134 collects user movement history information based on the extracted user movements. Note that the method by which the collection unit 134 extracts the movement history information from the image data is not particularly limited.

[0051] For example, the collection unit 134 uses the sensor data stored in the sensor data DB 122 to collect user movement history information, such as the distance the user's body has moved, the moving speed, and the amount of force applied to a three-dimensional structure, based on known technology.

[0052] For example, the collection unit 134 extracts user actions included in the sensor data by inputting the actions of a target person included in the sensor data into a model trained to extract the actions, and then collects user action history information based on the extracted user actions.

[0053] In the medical field, examples of the above-mentioned specific machines include surgical robots and rehabilitation robots. A surgical robot is a robot that a doctor operates remotely while watching a camera image. A rehabilitation robot is a robot that supports the rehabilitation of patients who have difficulty moving normally due to illness.

[0054] (Calculation unit 135) The calculation unit 135 compares the multiple pieces of operation history information collected by the collection unit 134 between the multiple users and calculates the difference. Specifically, the calculation unit 135 extracts the operation of a first user from the operation history information of a first user among the multiple users. The calculation unit 135 also extracts the operation of a second user from the operation history information of a second user among the multiple users. Then, the calculation unit 135 compares the operation of the first user with the operation of the second user and calculates the difference in operation between the users.

[0055] Furthermore, the calculation unit 135 calculates specific actions of the user, such as actions that the user is not good at or actions that are significantly different from those of other users, from the action history information collected by the collection unit 134.

[0056] An example of the difference calculated by the calculation unit 135 and the specific user action will be described later in the sections of FIGS. 6 and 7.

[0057] (output unit 136) Based on the operation history information collected by the collection unit 134, the output unit 136 outputs user operation information including differences in operation between multiple users, operations that the user is not good at, and work records that are operation history information for the target task to a terminal device (wearable terminal, laptop computer, etc.) operated by the user.

[0058] Furthermore, when outputting motion information including a difference between motions of a first user and a second user, if the difference related to the second user calculated by the calculation unit 135 satisfies a predetermined condition, the output unit 136 outputs the motion with emphasis on the target motion. For example, if the difference between the motions exceeds a predetermined threshold, the output unit 136 can output the motion information to the user after applying a predetermined color or highlighting to the trajectory of the motion or an area including the motion in the image data.

[0059] Furthermore, the output unit 136 highlights and outputs the specific action calculated by the calculation unit 135. For example, when the user's action is an action (specific action) that satisfies a predetermined condition, the output unit 136 can output the image data to the user after applying a predetermined color or highlighting to an area in the image data that includes the specific action. Note that the above-mentioned "predetermined condition" is an arbitrarily determined condition, and may be a condition such as "the action is determined to be the same or similar by a machine learning model" or "the trajectory of the specific action is the same or similar."

[0060] (imaging device 200) The imaging device 200 is a device that captures still images or moving images in a target space. For example, it may be a digital camera, a digital video camera, or other device fixed at a predetermined position. The imaging device 200 is not particularly limited as long as it is a device that can acquire still images or moving images used by the education support device 100 for processing.

[0061] (Sensor 300) The sensor 300 is a device that is attached to a user or a machine operated by the user, and acquires historical information about the operation of the user or the machine. For example, the sensor 300 includes an acceleration sensor (e.g., a three-axis acceleration sensor), a gyro sensor (e.g., a three-axis gyro sensor), a magnetic sensor, a pressure sensor, a temperature sensor, a motion sensor, etc. The sensor 300 is not particularly limited as long as it is a device that can acquire historical information about the operation used by the education support device 100 for processing.

[0062] (Terminal device 400) The terminal device 400 is an information processing terminal device operated by a user. Specifically, the terminal device 400 includes a visual device (for example, VR (Virtual Reality) goggles, etc.) worn by the user, a tactile device (for example, a haptic device, etc.), a thermal device (for example, a thermo device, etc.), etc.

[0063] In addition to a wearable terminal, the terminal device 400 may include a spatial reproduction display that displays a three-dimensional shape without wearing special goggles, etc. The terminal device 400 may also be a smartphone, a tablet, a PDA (Personal Digital Assistant), a PC (Personal Computer), a notebook PC, etc.

[0064] (Examples of educational support) An example of educational support by the educational support device 100 according to this embodiment will now be described with reference to Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are diagrams showing an example of educational support according to this embodiment. Fig. 6 shows an example (first example) of educational support in the field of medicine. Fig. 7 shows an example (second example) of educational support in the field of sports.

[0065] (First example) First, the first example, "an example of educational support in the medical field," will be described with reference to Fig. 6. Fig. 6 shows an example in which a junior doctor 11 performs a simulated surgery (practice surgery) using a three-dimensional structure reproduced based on image data relating to a surgery (movement) performed by a veteran doctor 10 (a predetermined user).

[0066] First, the education support device 100 (acquisition unit) acquires image data relating to an actual surgery performed by the experienced doctor 10, captured by the imaging devices 200 (200a and 200b) ((1) in FIG. 6).

[0067] Next, the education support device 100 (construction unit) constructs a virtual organ (three-dimensional structure) using the acquired image data related to the actual surgery ((2) in Figure 6). Note that the "virtual organ (three-dimensional structure)" is a virtual organ constructed by the construction unit 132, and allows the user to touch the virtual organ, modify it, and perform other virtual tasks via a wearable device or the like.

[0068] Next, the education support device 100 (display unit) transmits information for displaying the constructed virtual organ to the terminal device 400 (VR goggles or the like) worn by the junior doctor 11 ((3-1) in FIG. 6).

[0069] Then, the terminal device 400 (VR goggles or the like) worn by the young doctor 11 uses the information for displaying the virtual organ transmitted by the education support device 100 (display unit) to display the virtual organ to the young doctor 11 ((3-2) in FIG. 6). At this time, the terminal device 400 may display the virtual organ to the young doctor 11 as XR (cross reality). The terminal device 400 may also display to the user a virtual space constructed in advance and a virtual organ placed in the virtual space.

[0070] Next, the education support device 100 (collection unit) collects action history information of the young doctor 11 (user) for each of the multiple users based on the displayed virtual organ ((4-1) in FIG. 6). For example, the education support device 100 (collection unit) collects information such as the speed, pressure, and accuracy of the action of the instrument used by the young doctor 11 when incising, suturing, etc. the virtual organ, which is obtained by sensors attached to the body of the young doctor 11 and the medical instrument operated by the young doctor 11.

[0071] Furthermore, the education support device 100 (collection unit) collects operation history information such as the operation speed and accuracy of instruments during actual surgery by the veteran doctor 10 from image data relating to actual surgery acquired by the imaging devices 200 (200a and 200b) based on known image analysis techniques, etc. ((4-2) in FIG. 6). Note that the operation history information relating to the veteran doctor 10 described above may be acquired by sensors attached to the body of the veteran doctor 10 or to the medical instruments operated by the veteran doctor 10.

[0072] The education support device 100 (calculation unit) uses the collected action history information to calculate the differences and difficult actions ((5) in FIG. 6). Specifically, the education support device 100 (calculation unit) compares the collected action history information of the veteran doctor 10 with that of the junior doctor 11 to calculate the differences between their actions. For example, the education support device 100 (calculation unit) can compare action history information including information such as the organ resection speed by the veteran doctor 10 and the junior doctor 11, whether the resection speed is constant, the resection position, the suturing position, the suturing angle, the depth of insertion of the suturing needle, and the pressure applied to the organ, to calculate the differences between their actions.

[0073] Furthermore, the education support device 100 (calculation unit) compares the collected motion history information of the veteran doctor 10 with that of the young doctor 11, and calculates the parts of the motion of the young doctor 11 where there is a large difference compared to the motion of the veteran doctor 10 as "motions that the young doctor 11 is not good at." For example, the education support device 100 (calculation unit) compares motion history information including information such as the organ resection speed by the veteran doctor 10 and the young doctor 11, whether the resection speed is constant, the resection position, the suturing position, the suturing angle, the depth of insertion of the suturing needle, and the pressure applied to the organ, and can calculate the motion where the difference exceeds a threshold (for example, "pressure applied to the organ") as the motion that the young doctor 11 is not good at.

[0074] The education support device 100 (output unit) outputs the movement information including the calculated difference and information about the difficult movements to the young doctor 11 as feedback ((6) in FIG. 6). For example, the education support device 100 (output unit) can output to the young doctor 11 a list of movements that show differences compared to the experienced doctor 10, a comparison video of the movements, etc. Furthermore, the education support device 100 (output unit) can color or highlight movements with particularly large differences (difficult movements) by superimposing them on the virtual organ, and output this to the young doctor 11 via the terminal device 400 (VR goggles, etc.) ((6-1) in FIG. 6).

[0075] (Second example) Next, the second example, "An example of educational support in a sports field," will be described with reference to Fig. 7. Fig. 7 shows an example in which a sports practitioner 21 practices using a reproduced three-dimensional structure reproduced based on image data relating to sports movements (movements) performed by a role model 20 (a predetermined user), such as a sports instructor or a professional athlete.

[0076] First, the education support device 100 (acquisition unit) acquires image data relating to the sports movements of the role model 20 captured by the imaging device 200 (200a and 200b) ((1) in FIG. 7). Next, the education support device 100 (construction unit) constructs a model athlete (three-dimensional structure) using the acquired image data relating to the sports movements of the role model 20 ((2) in FIG. 7). Note that the "model athlete (three-dimensional structure)" refers to an athlete who is to be used as a virtual role model constructed by the construction unit 132, and enables a user to observe the movements of the model athlete via a wearable device or the like, and to imitate the movements of the model athlete.

[0077] Next, the training support device 100 (display unit) transmits information for displaying the constructed model player to the terminal device 400 (VR goggles or the like) worn by the trainee 21 ((3-1) in FIG. 7).

[0078] Then, the terminal device 400 (VR goggles or the like) worn by the trainee 21 displays the model player to the trainee 21 based on the information for displaying the model player transmitted by the education support device 100 (display unit) ((3-2) in FIG. 7). At this time, the terminal device 400 may display a virtual space constructed in advance and the model player placed in the virtual space to the trainee 21. The terminal device 400 may also display the model player to the trainee 21 as XR (cross reality).

[0079] Next, the education support device 100 (collection unit) collects movement history information based on the displayed model player. For example, the education support device 100 (collection unit) collects movement history information related to sports movements such as the swing of a bat by the trainee 21, which is acquired by sensors 21a to 21d attached to the body of the trainee 21 who imitates the movement of the displayed model player and to tools used by the trainee 21 ((4-1) in FIG. 7).

[0080] Furthermore, the education support device 100 (collection unit) collects movement history information such as the movement speed and accuracy of the actual sports movements of the role model 20 from image data relating to sports movements acquired by the imaging devices 200 (200a and 200b) based on known image analysis techniques, etc. ((4-2) in FIG. 7). Note that the movement history information relating to the role model 20 described above may be acquired by a sensor attached to the body of the role model 20 or to an implement operated by the role model 20.

[0081] The education support device 100 (calculation unit) uses the collected action history information to calculate differences and weak actions ((5) in FIG. 7). Specifically, the education support device 100 (calculation unit) compares the collected action history information of the role model 20 with that of the practicer 21 to calculate differences between the actions of the two. For example, the education support device 100 (calculation unit) can compare action history information including information such as the position at which the role model 20 and the practicer 21 hold the bat, the swing speed, and the trajectory of the swing, to calculate differences in the position at which the bat is held, the swing speed, the trajectory of the swing, and the like.

[0082] Furthermore, the education support device 100 (calculation unit) compares the collected movement history information of the role model 20 with the movement history information of the practice person 21, and calculates, as a "movement that the practice person 21 is weak at," any part or portions where there is a large difference between the movement of the practice person 21 and the movement of the role model 20. For example, the education support device 100 (calculation unit) can compare movement history information including information such as the position at which the bat is held, the swing speed, and the swing trajectory of the role model 20 and the practice person 21, and calculate, as a movement where the difference exceeds a threshold value (for example, "bat swing speed"), the movement.

[0083] The training support device 100 (output unit) outputs the movement information including the calculated difference and information about the weak movements to the trainee 21 as feedback ((6) in FIG. 7). For example, the training support device 100 (output unit) can output to the trainee 21 a list of movements with differences between the role model 20 and the trainee 21, a comparison video of the movements, etc. Furthermore, the training support device 100 (output unit) can color or highlight movements with particularly large differences (weak movements) and superimpose them on the movements of the model player before outputting them to the trainee 21.

[0084] (Processing Procedure by Education Support Device 100) Next, the procedure of the process realized by the education support device 100 according to this embodiment will be described with reference to Fig. 10. Fig. 8 is a diagram showing a flowchart of the process performed by the education support device 100 according to this embodiment.

[0085] The acquisition unit 131 acquires image data from the imaging device 200 (S101). Next, the construction unit 132 constructs a three-dimensional structure using the acquired image data (S102). The display unit 133 displays the three-dimensional structure to the user (S103).

[0086] If the user has performed an action (Yes in S104), the collection unit 134 collects the action history information (S105). On the other hand, if the user has not performed an action (No in S104), the education support device 100 waits for processing.

[0087] The calculation unit 135 calculates the action information using the collected action history information (S106). Next, the output unit 136 outputs the action information (S107). Then, the education support device 100 ends the process.

[0088] (effect) Next, we will explain the effects of the education support device 100 according to this embodiment. When education or instruction is provided in various fields such as medicine, industry, and sports, it may be difficult to provide appropriate education using conventional technology.

[0089] Therefore, the construction unit 132 of the education support device 100 according to this embodiment constructs a three-dimensional structure in a virtual space based on the acquired image data. The collection unit 134 of the education support device 100 collects operation history information for the three-dimensional structure constructed by the construction unit 132. The output unit 136 of the education support device 100 outputs operation information based on the operation history information collected by the collection unit 134.

[0090] As described above, the education support device 100 can construct three-dimensional structures that replicate organs used in simulated surgeries and the movements of skilled practitioners, which are difficult to accurately replicate using conventional technologies. For example, in cases where it is difficult to prepare replicas of human organs, human teeth, or other similar objects that are close to the real thing, the education support device 100 can construct highly accurate virtual organs (three-dimensional structures), thereby improving the effectiveness of education support compared to conventional methods.

[0091] Furthermore, the education support device 100 can collect user actions (action history information) based on the constructed three-dimensional structure and output feedback to the user. As a result, the education support device 100 according to this embodiment has the effect of enabling effective education support.

[0092] Furthermore, the education support device 100 according to this embodiment achieves predetermined effects by executing the processes described below.

[0093] The collection unit 134 collects motion history information based on a three-dimensional structure constructed using image data relating to motions by a specific user. The calculation unit 135 compares the motion history information between multiple users and calculates the difference. Specifically, the calculation unit 135 extracts the motion of a first user from the motion history information of a first user among the multiple users. The calculation unit 135 also extracts the motion of a second user from the motion history information of a second user among the multiple users. Next, the calculation unit 135 compares the motion of the first user with the motion of the second user and calculates the difference in motion between the users.

[0094] Through the above-described processing, the education support device 100 can collect action history information related to the collected user actions on the three-dimensional structure and user actions imitating the actions of the three-dimensional structure, and calculate differences in actions between multiple users, differences in action accuracy, etc. As a result, the education support device 100 has the effect of enabling effective education support, such as allowing users to improve their own actions based on the differences in actions.

[0095] Furthermore, when the difference related to the second user calculated by the calculation unit 135 satisfies a predetermined condition, the output unit 136 outputs the target action with emphasis. By performing the above-described process, when a user's action is significantly different from the actions of other users (i.e., the difference is large, the accuracy is low, etc.), the education support device 100 highlights the action, thereby achieving the effect of enabling effective education support, such as enabling the user to improve their own action by themselves.

[0096] The calculation unit 135 calculates the specific action of the user from the action history information of the user collected by the collection unit 134. The output unit 136 outputs the specific action calculated by the calculation unit 135 in an emphasized manner.

[0097] Through the above-described processing, the education support device 100 enables the user to easily grasp actions that the user is not good at or actions that the user is less proficient at than other users, etc. As a result, the education support device 100 has the effect of enabling effective education support, such as allowing the user to improve their own actions based on the output highlighting of the specific actions.

[0098] The collection unit 134 collects, as operation history information, information collected by a sensor attached to a tool operated by a user or operation history information of a predetermined machine operated by a user. By performing the above-described processing, the education support device 100 can easily collect operation history information of the user.

[0099] For example, the education support device 100 can collect action history information based on sensors attached to tools and machines, and can calculate the differences in actions between users and weak actions using accurately traced action history information.

[0100] <Modification> The following describes modified examples realized by the education support device 100 according to this embodiment.

[0101] (Using IOWN) The education support device 100 according to this embodiment can realize education support processing based on the IOWN technology. The IOWN technology will now be described. Figure 9 is a diagram for explaining IOWN.

[0102] As shown in Figure 9, IOWN technology consists of three main technology areas: All-Photonics Network (APN), Digital Twin Computing (DTC), and Cognitive Foundation (CF).

[0103] (APN (All-Photonics Network)) The APN related to IOWN technology is a technology that enables the construction of high-speed networks by processing all network transfer functions in the optical domain. Specifically, the APN related to IOWN technology is a technology that realizes low-power, high-quality, large-capacity, and low-latency communications based on optical-based (photonics-based) technologies such as "photonics-electronic convergence technology," "large-capacity optical transmission system and device technology," "optical Ising machine," and "optical lattice clock network."

[0104] For example, the construction unit 132 constructs a three-dimensional structure based on photogrammetry using image data with high image quality and a large amount of data acquired by multiple imaging devices connected via the APN of the IOWN. The display unit 133 can then display the highly accurate three-dimensional structure constructed by the construction unit 132 to the user via the APN.

[0105] By the above-described processing, the education support device 100 according to this embodiment can use the APN of the IOWN described above to construct a three-dimensional structure with higher accuracy than conventionally possible and display it to the user.

[0106] (Digital Twin Computing) DTC, which is related to IOWN technology, is a technology that maps individual objects in the real world onto a virtual space using the vast amount of data collected by devices connected to the APN described above.

[0107] Conventional digital twin frameworks are used by mapping individual objects, such as automobiles and robots, into a virtual space, performing analysis and predictions on them, and then mapping the results of the analysis and predictions back onto the real world.

[0108] On the other hand, DTC related to IOWN technology expands on the conventional concept of digital twins, freely combining digital twins of various industries, objects, and people to perform calculations, thereby reproducing with high accuracy the combination of multiple objects, such as people and automobiles in a city. Furthermore, DTC related to IOWN technology enables not only the expression of a person's external appearance, but also the digital expression of their internal state, such as consciousness and thoughts, by combining technologies that enable "speech recognition," "speech synthesis," "understanding of emotions and intentions," etc. to collect information and build a digital twin environment.

[0109] In this way, DTC related to IOWN technology is a technology that enables the creation of digital twins that do not exist in the real world by combining multiple entities that are single in the real world and replicating them as digital twins in a virtual space, or by exchanging or merging some of the components between multiple digital twins.

[0110] For example, the construction unit 132 constructs a virtual space as a digital twin based on IOWN's DTC (Digital Twin Computing). The display unit 133 can display the digital twin constructed by the construction unit 132 and the three-dimensional structure of the digital twin to the user via the APN.

[0111] Through the above-described processing, the education support device 100 according to this embodiment can construct a virtual space and a three-dimensional structure based on the DTC of IOWN and display it to the user.

[0112] (Cognitive Foundation) CF related to IOWN technology is a technology that centrally performs the deployment, configuration, linkage, management, and operation of ICT (Information and Communication Technology) resources at different layers, from the cloud to edge computers, network services, user equipment, etc. Specifically, CF related to IOWN technology treats various targets as a group of virtualized ICT resources, and optimally integrates multiple resources at different layers using multi-orchestration functions as a hub.

[0113] Furthermore, as shown in FIG. 9, the IOWN technology provides high-value-added services by linking the above-mentioned APN, DTC, and network services provided by operators.

[0114] For example, as shown in (1) of Figure 9, IOWN technology provides a technology for transmitting information collected via APN to other terminal devices at high speed and with low latency. Also, as shown in (2) of Figure 9, IOWN technology provides a technology for collecting large amounts of information from terminal devices and outputting information such as analysis results from the service provided by the operator at high speed and with low latency in services such as information analysis. Also, as shown in (3) of Figure 9, IOWN technology provides a technology for transmitting large amounts of information at high speed and with low latency, using information obtained from surveillance cameras, automobile sensors, etc. to build a digital twin environment, make future predictions, and output the prediction results to the user.

[0115] By using the above-described IOWN technology, the education support device 100 can construct three-dimensional structures and virtual spaces with higher accuracy than conventional ones. As a result, the education support device 100 displays highly accurate three-dimensional structures to the user, enabling the user to perform simulated surgery or imitate sports movements with a greater sense of reality.

[0116] Furthermore, the education support device 100 can create virtual spaces and three-dimensional structures that recreate various situations. For example, the education support device 100 can provide real-time, interactive education support by using a digital twin created based on the above-mentioned IOWN technology. For example, the education support device 100 allows multiple trainees (e.g., medical interns, students, sports trainees, etc.) to simultaneously view the work of a single instructor and perform mock training, etc.

[0117] Therefore, the education support device 100 has an effect of enabling a user to easily experience situations that can only be experienced under actual circumstances, and as a result, the education support device 100 can realize effective education support that has been difficult to achieve with conventional technologies.

[0118] (Data, etc.) The image data, three-dimensional structures, virtual space, sensor data, simulated surgeries, veteran doctors, young doctors, instructors, professional athletes, trainees, work records, names of functional parts of the educational support device 100, steps, processes, names of steps or processes, etc. used in the description of the above embodiments are merely examples and can be changed as desired.

[0119] For example, the image data DB 121 stores imaging targets and imaging data in association with "No.", which is information identifying individual image data, but the items and contents to be stored are not particularly limited. The sensor data DB 122 stores sensor names and acquired data in association with "No.", which is information identifying individual sensor data, but the items and contents to be stored are not particularly limited. The operation history information DB 125 stores user identification information and history information in association with "No.", which is information identifying individual operation history information, but the items and contents to be stored are not particularly limited.

[0120] (Modifications related to the creation of work records) The education support device 100 according to this embodiment can perform a process of generating a work record by modifying the education support process. Hereinafter, an example of a surgical record (work record) in a medical field will be described with reference to FIG.

[0121] Fig. 10 is a diagram showing an example of generating a work record according to a modified example. Fig. 10 shows an example of outputting a work record using a three-dimensional structure related to work constructed by a construction unit based on acquired image data related to surgery (work) performed by a doctor 30 (a predetermined user).

[0122] First, the education support device 100 (acquisition unit) acquires image data relating to an actual surgery performed by the doctor 30, captured by the imaging devices 200 (200a and 200b) ((1) in FIG. 10). Next, the education support device 100 (construction unit) constructs a virtual organ related to the work using the acquired image data relating to the actual surgery ((2) in FIG. 10). In addition, the education support device 100 (collection unit) collects operation history information of the doctor 30 from the image data acquired by the imaging devices 200 (200a and 200b) ((3) in FIG. 10).

[0123] The education support device 100 (output unit) outputs the virtual organs related to the work constructed by the construction unit and the action history information related to the surgery (work) of the doctor 30 collected by the collection unit as the work record 31 ((4) in FIG. 10). For example, the work record 31 output by the education support device 100 (output unit) includes information related to the surgery, such as the patient name, the name of the surgery, and the date and time of the surgery, as well as surgery history information showing how the actual surgery was carried out (i.e., the procedure of the surgery, the positions of the organs that were actually treated, the contents of the treatment, etc.) using a three-dimensional structure.

[0124] Through the above-described processing, the education support device 100 can output, as an operation record, a three-dimensional structure constructed from image data of a predetermined operation such as an actual surgery or an actual sports game, and action history information about the predetermined operation. Therefore, the education support device 100 can easily generate an operation record by modifying the education support processing.

[0125] (Flowcharts, etc.) The steps in the flowcharts may be interchanged as long as there is no contradiction, and some steps may not be performed. In addition, conjunctions such as "next," "continue," "further," "at this time," and "on this occasion" used in the explanation of the flowcharts do not limit the order or timing of the execution of the processes in the flowcharts.

[0126] <Hardware configuration> The components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of each device can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

[0127] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can also be performed manually using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data, and parameters shown in the drawings can be changed as desired unless otherwise specified.

[0128] <Program> In one embodiment, the various devices constituting the education support device 100 can be implemented by installing an education support program as package software or online software on a desired computer. For example, by executing the education support program on an information processing device, the various devices constituting the education support device 100 can function. The information processing device referred to here includes desktop and notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones and mobile phones, and even slate terminals such as PDAs (Personal Digital Assistants).

[0129] 11 is a diagram showing an example of a computer that realizes an education support device 100 according to an embodiment. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0130] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0131] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, application programs 1092, program modules 1093, and program data 1094. That is, programs that define the processes of the various devices that make up the education support device 100 are implemented as program modules 1093 in which computer-executable code is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, program modules 1093 for executing processes similar to those of the functional configurations of the various devices that make up the education support device 100 are stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0132] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0133] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a LAN or a WAN (Wide Area Network)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0134] <Other> Although the present embodiment has been described above, the present embodiment is not limited by the descriptions and drawings that form part of the disclosure. In other words, other embodiments, examples, operational techniques, etc. that are made by those skilled in the art based on the present embodiment are all included in the scope of the present embodiment. [Explanation of symbols]

[0135] 100 Educational support equipment 110 Communications Department 120 Storage section 121 Image Data DB 122 Sensor Data DB 123 Virtual Space DB 124 Three-dimensional structure DB 125 Operation History Information DB 130 control section 131 Acquisition Department 132 Construction Department 133 Display section 134 Collection Department 135 Calculation Unit 136 Output section 200 Imaging device 300 sensors 400 Terminal Equipment

Claims

1. a construction unit that constructs a three-dimensional structure in a virtual space based on the acquired image data; a collection unit that collects history information of user actions with respect to the three-dimensional structure constructed by the construction unit; an output unit that outputs information about the user's actions based on the history information about the user's actions collected by the collection unit; An educational support device comprising:

2. The collecting unit Collecting history information of a user's actions based on the three-dimensional structure constructed using image data relating to actions by the user; The system further includes a calculation unit that compares the user action history information between a plurality of users and calculates a difference.

2. The educational support device according to claim 1.

3. The calculation unit extracting a motion of a first user from history information of the motions of a first user among the plurality of users; extracting a motion of a second user from history information of the motion of a second user among the plurality of users; comparing the first user's motion with the second user's motion to calculate a difference in motion between the users; The output unit When the difference related to the second user calculated by the calculation unit satisfies a predetermined condition, a target motion is emphasized and output.

3. The educational support device according to claim 2.

4. a calculation unit that calculates a specific action of the user from the history information of the user's actions collected by the collection unit; The output unit the specific action calculated by the calculation unit is output in an emphasized manner; 2. The educational support device according to claim 1.

5. The construction unit constructing a three-dimensional structure relating to the work by the predetermined user using the acquired image data relating to the work; The output unit outputting the three-dimensional structure related to the task constructed by the construction unit and history information of the user's actions related to the task collected by the collection unit; 2. The educational support device according to claim 1.

6. The collecting unit Collecting information collected by a sensor attached to an appliance operated by the user or operation history information of a predetermined machine operated by the user as operation history information of the user; 6. The educational support device according to claim 1, wherein the educational support device is a computer.

7. The construction unit The three-dimensional structure is constructed based on photogrammetry using image data acquired by a plurality of imaging devices connected via an APN (All-Photonics Network) of an IOWN (Innovative Optical and Wireless Network), The system further includes a display unit that displays the three-dimensional structure constructed by the construction unit to the user via the APN.

6. The educational support device according to claim 1, wherein the educational support device is a computer.

8. The construction unit Based on IOWN's DTC (Digital Twin Computing), the virtual space is constructed as a digital twin, The display unit The digital twin constructed by the construction unit and a three-dimensional structure in the digital twin are displayed to the user via the APN.

8. The educational support device according to claim 7.

9. An education support method executed by an education support device, a construction process of constructing a three-dimensional structure in a virtual space based on the acquired image data; a collection step of collecting history information of user actions with respect to the three-dimensional structure constructed by the construction step; an output step of outputting information about the user's actions based on the history information about the user's actions collected by the collection step; An educational support method comprising:

10. a construction step of constructing a three-dimensional structure in a virtual space based on the acquired image data; a collection step of collecting history information of user actions on the three-dimensional structure constructed by the construction step; an output step of outputting information about the user's actions based on the history information about the user's actions collected by the collection step; An educational support program that runs on a computer.

Images