Projection device and image processing system

Abstract

This projection device comprises: a photographing unit that photographs a form on which information on which guidance is to be given is filled in at a plurality of times including a first time and a second time later than the first time; an image processing unit that extracts a partial feature amount of the form at the first time and a partial feature amount of the from at the second time, and calculates a movement amount of the form from the first time at the second time; and a projection unit that projects guidance information for the information on which guidance is to be given filled in on the form onto a coordinate position taking into account the movement amount of the form calculated by the image processing unit.

Description

Projection device and image processing system

[0001] The present disclosure relates to a projection device and an image processing system.

[0002] There is a system that photographs an answer sheet on which answers are written as information to be taught, and electronically grades (provides guidance) based on the photographed image. A system has also been proposed that projects the graded results as guidance information onto the answer sheet (see Patent Documents 1 and 2).

[0003] JP 2015-167001 A JP 2006-4010 A

[0004] In real-time interactive instruction, for example, when the answering party (student) receives real-time instruction from the grading party (teacher) while answering, it is expected that the answer sheet will be moved while answering, which makes it difficult to properly project the grading results onto the answer sheet.

[0005] It is desirable to provide a projection device and an image processing system that can appropriately project onto a sheet of paper guidance information for guidance information written on the sheet of paper.

[0006] A projection device according to one embodiment of the present disclosure includes an imaging unit that images a sheet of paper on which instructed information has been written at a plurality of times including a first time and a second time that is later than the first time; an image processing unit that extracts partial characteristics of the sheet of paper at the first time and partial characteristics of the sheet of paper at the second time and calculates the amount of movement of the sheet of paper from the first time to the second time; and a projection unit that projects instruction information for the instructed information written on the sheet of paper at a coordinate position that takes into account the amount of movement of the sheet of paper calculated by the image processing unit.

[0007] An image processing system according to one embodiment of the present disclosure includes an external device that generates instruction information for the instruction target information written on a sheet of paper based on image data of the sheet of paper on which the instruction target information is written, and a projection device that projects the instruction information generated by the external device onto the sheet of paper, wherein the projection device includes: an imaging unit that images the sheet of paper on which the instruction target information is written at a plurality of times including a first time and a second time that is later than the first time; an image processing unit that extracts partial feature amounts of the sheet of paper at the first time and partial feature amounts of the sheet of paper at the second time and calculates the amount of movement of the sheet from the first time to the second time; and a projection unit that projects the instruction information for the instruction target information written on the sheet of paper at a coordinate position that takes into account the amount of movement of the sheet calculated by the image processing unit.

[0008] In the projection device or image processing system according to an embodiment of the present disclosure, guidance information for guidance information written on a sheet is projected at a coordinate position that takes into account the amount of movement of the sheet.

[0009] FIG. 1 is a configuration diagram showing an overview of an image processing system according to a first embodiment of the present disclosure. FIG. 2 is an explanatory diagram showing an example of a state in which a grading result (grading information) is projected onto an answer sheet in the image processing system according to the first embodiment. FIG. 3 is a block diagram showing an example of a configuration of a projection device in the image processing system according to the first embodiment. FIG. 4 is a block diagram showing an example of a configuration of a terminal device in the image processing system according to the first embodiment. FIG. 5 is a block diagram showing an example of a configuration of a server in the image processing system according to the first embodiment. FIG. 6 is an explanatory diagram showing an example of a display of the number of projectable pixels displayed on a terminal device. FIG. 7 is an explanatory diagram showing an overview of distortion correction in the image processing system according to the first embodiment. FIG. 8 is an explanatory diagram showing an example of a configuration of an answer sheet. FIG. 9 is an explanatory diagram showing an example of a command written on an answer sheet. FIG. 10 is a configuration diagram showing an example of a situation in which a plurality of projection devices (a plurality of students) exist. FIG. 11 is an explanatory diagram showing an example of a display state of a terminal device when a plurality of projection devices (a plurality of students) exist. FIG. 12 is a flowchart showing an example of the flow of the overall processing operation in the image processing system according to the first embodiment. Fig. 1 is a flowchart showing an example of the flow of processing operations until an image captured by a projection device is displayed on a terminal device in the image processing system according to the first embodiment. Fig. 2 is a flowchart showing an example of the flow of processing operations until marking information generated on a terminal device is projected by a projection device in the image processing system according to the first embodiment. Fig. 3 is a flowchart showing an example of the flow of processing operations when archiving image data of an answer sheet and marking information in the image processing system according to the first embodiment.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The description will be made in the following order: 1. First embodiment (FIGS. 1 to 15) 1.1 Configuration 1.2 Operation 1.3 Effects 2. Other embodiments

[0011] 1. First Embodiment [1.1 Configuration] (Outline of Image Processing System) FIG. 1 shows an outline of an image processing system according to a first embodiment of the present disclosure.

[0012] The image processing system according to the first embodiment includes a projection device 1, a terminal device 2, and a server 30. The server 30 is provided, for example, on a cloud 3. The terminal device 2 and the server 30 are external devices to the projection device 1. The projection device 1, the terminal device 2, and the server 30 are capable of communicating with each other wirelessly or via a wired connection.

[0013] The projection device 1 includes a projection unit 10 , an image capturing unit 11 , a microphone 51 , and a speaker 52 .

[0014] The terminal device 2 includes a display unit 20, a microphone 71, and a speaker 72. The display unit 20 is configured, for example, as a touch panel. The terminal device 2 is capable of accepting operation inputs on the screen of the display unit 20 using an input device 4 such as a touch pen.

[0015] The projection device 1 is placed on the side of the student who is the subject of instruction (grading). The terminal device 2 is placed on the side of the teacher who will provide the instruction (grading). Furthermore, this image processing system allows instruction (grading) not only by a human (teacher), but also by the server 30 using AI (artificial intelligence).

[0016] In addition, in this image processing system, the projection device 1 is equipped with a microphone 51 and a speaker 52, and the terminal device 2 is equipped with a microphone 71 and a speaker 72, so that students can receive voice guidance from the teacher. In addition, in this image processing system, voice data can be sent and received between the projection device 1 and the server 30, so that students can receive voice guidance from the AI.

[0017] In this image processing system, when a student writes an answer as guidance information for a question written on an answer sheet 100, the answer sheet 100 is photographed by the photographing unit 11 of the projection device 1. The image data of the answer sheet 100 photographed by the photographing unit 11 is sent to the server 30 and the terminal device 2, which are external devices. The server 30 or the terminal device 2 generates grading information as guidance information for the answer written on the answer sheet 100 based on the image data of the answer sheet 100, and transmits the grading information to the projection device 1. The generation of the grading information by the server 30 is performed by AI. The generation of the grading information by the terminal device 2 is performed, for example, based on writing by the teacher using the input device 4 on the screen of the display unit 20 on which an image of the answer sheet 100 is displayed.

[0018] The scoring information as instruction information may include not only the result of determining whether an answer is correct or incorrect, but also information for guiding the answer, such as an explanation for the question. Furthermore, in this image processing system, the subject of instruction is not limited to graded subjects such as English or mathematics, but may also be instruction that does not involve grading, such as art instruction (how to draw, color, etc.) or instruction on the stroke order of kanji characters. Furthermore, the subject of instruction is not limited to school education, but may also be instruction on hobbies such as Go or Shogi. The projection device 1 projects the scoring information generated by the server 30 or the terminal device 2 onto the answer sheet 100 using the projection unit 10.

[0019] FIG. 2 shows an example of a state in which the marking results (marking information) are projected onto the answer sheet 100 in the image processing system according to the first embodiment.

[0020] It is preferable that the imaging range 111 of the imaging unit 11 is larger than the projection range 110 of the projection unit 10. For this reason, it is preferable that the imaging unit 11 is disposed above the projection unit 10.

[0021] The photographing unit 11 of the projection device 1 photographs the answer sheet 100 at a first time T=t1 (FIG. 1) in order to transmit image data of the answer sheet 100. The photographing unit 11 also photographs the answer sheet 100 at a second time T=t2 (FIG. 2). The projection device 1 calculates the amount of movement of the answer sheet 100 from the first time T=t1 at the second time T=t2. The projection unit 10 projects the marking information onto a coordinate position that takes into account the calculated amount of movement of the answer sheet 100. This makes it possible to properly project the marking information onto the answer sheet 100 even if the answer sheet 100 moves between the first time T=t1 and the time the marking information is projected.

[0022] (Configuration of Projection Device 1) FIG. 3 shows a schematic configuration example of the projection device 1 in the image processing system according to the first embodiment.

[0023] The projection device 1 includes a projection unit 10 , an imaging unit 11 , an image processing unit 12 , a control unit 13 , a storage unit 14 , an operation input unit 15 , a call unit 50 , and a communication unit 60 .

[0024] The projection device 1 may have a microcomputer including, for example, a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). In this case, for example, the processing by the image processing unit 12 and the control unit 13 in the projection device 1 may be realized by the CPU executing processing based on a program stored in the ROM or RAM. Alternatively, the processing by the image processing unit 12 and the control unit 13 may be realized by the CPU executing processing based on a program supplied from an external source via, for example, a wired or wireless network.

[0025] The photographing unit 11 may have a function equivalent to the processing performed by the image processing unit 12 .

[0026] The photographing unit 11 photographs the answer sheet 100 on which answers have been written at a plurality of times including a first time T=t1 and a second time T=t2 that is later than the first time T=t1. The photographing unit 11 also photographs the answer sheet 100 before answers have been written and a predetermined pattern image for distortion correction projected by the projection unit 10, for example, in order to calculate in advance correction parameters used for distortion correction of the photographed image and the projected image.

[0027] The image processing unit 12 has a feature extraction unit 121. The image processing unit 12 extracts partial feature amounts of the answer sheet 100 at a first time T = t1 and partial feature amounts of the answer sheet 100 at a second time T = t2 using the feature extraction unit 121. The image processing unit 12 calculates the amount of movement of the answer sheet 100 at the second time T = t2 from the first time T = t1 based on the partial feature amounts extracted by the feature extraction unit 121.

[0028] The storage unit 34 stores the partial feature amounts extracted by the feature amount extraction unit 121 .

[0029] The projection unit 10 projects the marking information for the answer written on the answer sheet 100 at a coordinate position that takes into account the amount of movement of the answer sheet 100 calculated by the image processing unit 12. The projection unit 10 also projects a predetermined pattern image for distortion correction, for example, to calculate in advance correction parameters used for distortion correction of the captured image and the projected image.

[0030] The communication unit 50 has a microphone 51 and a speaker 52 as an audio output unit.

[0031] The communication unit 60 has a receiving unit 61 and a transmitting unit 62. The communication unit 60 transmits and receives various types of data wirelessly or via a wired connection between the terminal device 2 and the server 30. The transmitting unit 62 transmits, for example, image data of the answer sheet 100 photographed by the photographing unit 11, audio data input from the microphone 51, etc. to an external device. The receiving unit 61 receives, for example, grading information generated in the external device based on the image data of the answer sheet 100, audio data from the external device, etc.

[0032] The operation input unit 15 has an archive instruction receiving unit 151. The operation input unit 15 receives various operation inputs from the students who are users. The archive instruction receiving unit 151 is a receiving unit that receives instructions to store (archive), for example, image data of the answer sheet 100 and grading information in an external device. When the archive instruction receiving unit 151 receives an archive instruction, it transmits an archive instruction command to the external device via the transmission unit 62.

[0033] (Configuration of Terminal Device 2) FIG. 4 shows a schematic configuration example of the terminal device 2 in the image processing system according to the first embodiment.

[0034] The terminal device 2 includes a display unit 20, an image processing unit 22, a control unit 23, a memory unit 24, an operation input unit 25, a teaching mode notification unit 26, a displayable pixel count calculation unit 27, a call unit 70, and a communication unit 80.

[0035] The terminal device 2 may have a microcomputer having, for example, a CPU, a ROM, and a RAM. In this case, for example, the processes performed by the image processing unit 22, the control unit 23, the instruction mode notification unit 26, and the displayable pixel number calculation unit 27 in the terminal device 2 may be realized by the CPU executing processes based on programs stored in the ROM or RAM. Furthermore, the processes performed by the image processing unit 22, the control unit 23, the instruction mode notification unit 26, and the displayable pixel number calculation unit 27 may be realized by the CPU executing processes based on programs supplied from an external device via, for example, a wired or wireless network.

[0036] The display unit 20 is configured with, for example, a touch panel, and displays an image of the answer sheet 100 captured by the capturing unit 11 of the projection device 1. The image of the answer sheet 100 displayed on the display unit 20 is an image after distortion correction in the server 30, as shown in Fig. 7(B) described later.

[0037] The operation input unit 25 has a grading information input unit 251 as a teaching information input unit. The grading information input unit 251 accepts operation input on the screen of the display unit 20 using an input device 4 ( FIG. 1 ) such as a touch pen, and generates write data for generating grading information. The teacher operating the terminal device 2 can use the input device 4 to input data for generating grading information based on the image of the answer sheet 100 after distortion correction displayed on the display unit 20.

[0038] The image processing unit 12 generates the scoring information based on the writing data generated by the operation input unit 25 .

[0039] In this image processing system, as shown in FIGS. 10 and 11 , which will be described later, it is possible to provide guidance to multiple students 201 (multiple projection devices 1). The guidance mode notification unit 26 is a notification unit that notifies a specific student 202 (specific projection device 1) that the student 202 has become a target for guidance. The notification by the guidance mode notification unit 26 is made via the call unit 70 or the communication unit 80.

[0040] The displayable pixel number calculation unit 27 is a calculation unit that calculates information on the number of pixels that can be displayed (projected) by the projection device 1 and causes the display unit 20 to display the calculated information on the number of pixels.

[0041] 6 is a schematic diagram showing an example of a projectable pixel count display 300 displayed on the terminal device 2. As shown in FIG. 6, the displayable pixel count calculation unit 27 determines in advance an upper limit on the number of pixels that can be displayed (projected) by the projection device 1, and displays the projectable pixel count display 300 so that the number of pixels that can be displayed (projected) decreases as more scoring information is written. By determining an upper limit on the number of pixels that can be displayed (projected) by the projection device 1, it is possible to reduce the power consumption of the projection device 1.

[0042] The communication unit 70 has a microphone 71 and a speaker 72 .

[0043] The communication unit 80 has a receiving unit 81 and a transmitting unit 82. The communication unit 80 transmits and receives various types of data wirelessly or via a wired connection between the projection device 1 and the server 30. The transmitting unit 82 transmits, for example, the marking information generated by the image processing unit 12 and the audio data input from the microphone 71 to the projection device 1 or the server 30. The receiving unit 81 receives, for example, image data of the answer sheet 100 from the projection device 1 or the server 30 and the audio data from the projection device 1.

[0044] (Configuration of Server 30) FIG. 5 shows an outline of an example of the configuration of the server 30 in the image processing system according to the first embodiment.

[0045] The server 30 includes an image processing unit 32 , a control unit 33 , a memory unit 34 , a command processing unit 35 , a scoring information generation unit 36 ​​, a training target identification unit 37 , a command recognition unit 38 , and a communication unit 90 .

[0046] The server 30 may include a microcomputer having, for example, a CPU, a ROM, and a RAM. In this case, for example, the processes performed by the image processing unit 32, the control unit 33, the command processing unit 35, the scoring information generation unit 36, the training target identification unit 37, and the command recognition unit 38 in the server 30 may be realized by the CPU executing processes based on programs stored in the ROM or RAM. Furthermore, the processes performed by the image processing unit 32, the control unit 33, the command processing unit 35, the scoring information generation unit 36, the training target identification unit 37, and the command recognition unit 38 may be realized by the CPU executing processes based on programs supplied from an external source, for example, via a wired or wireless network.

[0047] The image processing unit 32 includes a distortion correction unit 321 .

[0048] The communication unit 90 includes a receiving unit 91 and a transmitting unit 92 .

[0049] The distortion correction unit 321 corrects distortion of the image data of the answer sheet 100 photographed by the photographing unit 11 of the projection device 1 .

[0050] The marking information generation unit 36 ​​generates marking information using AI based on the image data of the answer sheet 100 after distortion correction by the distortion correction unit 321.

[0051] FIG. 7 shows an overview of distortion correction in the image processing system according to the first embodiment.

[0052] In the image processing system, an initial distortion correction process is performed in advance. In the image processing system, the projection unit 10 of the projection device 1 projects a predetermined pattern image for distortion correction. The capture unit 11 of the projection device 1 captures the predetermined pattern image for distortion correction. FIG. 7A shows an example of the predetermined pattern image for distortion correction (the pattern image as seen by the capture unit 11) projected by the projection unit 10 of the projection device 1. The transmission unit 62 of the projection device 1 transmits data of the predetermined pattern image captured by the capture unit 11 to the server 30. The distortion correction unit 321 of the server 30 performs a predetermined projective transformation on the data of the predetermined pattern image to perform distortion correction. FIG. 7B shows an example of the pattern image after distortion correction.

[0053] The distortion correction unit 321 stores the transformation parameters of the projective transformation used when performing the initial distortion correction process as initial distortion correction parameters in the storage unit 34. Thereafter, the server 30 can use the initial distortion correction parameters to perform distortion correction on the image data of the answer sheet 100 from the projection device 1. The transmission unit 92 of the server 30 transmits the image data of the answer sheet 100 after the distortion correction to the terminal device 2. This allows the terminal device 2 to display the image of the answer sheet 100 after the distortion correction on the display unit 20.

[0054] On the other hand, when the scoring information generated in the terminal device 2 or the server 30 is projected by the projection unit 10 of the projection device 1, the distortion correction unit 321 performs inverse distortion correction on the image data of the scoring information, for example, by inverse projection transformation. This allows the projection unit 10 of the projection device 1 to project the scoring information at an appropriate coordinate position. For example, if the display unit 20 of the terminal device 2 indicates the position X=3, Y=3 in the XY coordinate system after distortion correction shown in FIG. 7B, a point can be displayed at the coordinate position corresponding to X=3, Y=3 in the projection device 1 shown in FIG. 7A.

[0055] The storage unit 34 stores, for example, the above-mentioned initial distortion correction parameters and various data based on command processing by the command processing unit 35 .

[0056] The command recognition unit 38 recognizes various commands sent from the projection device 1. The command recognition unit 38 recognizes, for example, archive instruction commands and commands input on the answer sheet 100.

[0057] If the command recognized by the command recognition unit 38 is an archive instruction command, the command processing unit 35 stores the image data of the answer sheet 100 and the grading information in the memory unit 34. If the command recognized by the command recognition unit 38 is an archive instruction command, the command processing unit 35 stores, for example, the following first to third data. By storing the following first to third data, for example, the student can later check the level of understanding of the instruction. Note that, for example, only the first data may be archived. First data: Image data and grading information of the answer sheet 100 before the grading information is projected (before receiving instruction). Second data: Image data and grading information of the answer sheet 100 after the grading information is projected (after receiving instruction). Third data: Image data of the answer sheet 100 after the grading information is projected (after receiving instruction).

[0058] FIG. 8 shows a schematic diagram of an example of the structure of the answer sheet 100. As shown in FIG.

[0059] The answer sheet 100 may include a question and answer entry area 101 and a command information display area 102, which is an area for entering command information. The photographing range 111 of the photographing unit 11 may be an area including the question and answer entry area 101 and the command information display area 102. The projection range 110 of the projection unit 10 may be an area including the question and answer entry area 101. The command recognition unit 38 performs image recognition on the projection device 1 when an operation such as pointing at a command entered in the command information display area 102 is performed, and identifies the command. The command processing unit 35 performs processing according to the identified command.

[0060] The command recognition unit 38 also recognizes that a command has been written in the question and answer entry area 101 of the answer sheet 100 .

[0061] FIG. 9 shows an example of a command written on the answer sheet 100. As shown in FIG.

[0062] For example, when a written question in English or the like is written and a box is written around the sentence, the command recognition unit 38 recognizes this as a command to register the word (character information) in the boxed area. Based on the recognition result of the command recognition unit 38, the command processing unit 35 stores the data of the word in the boxed area in the storage unit 34.

[0063] Furthermore, for example, when a written question in English or the like is written and there is a note underlining the text, the command recognition unit 38 recognizes this as a command to instruct the audio output of the underlined words (text information). Based on the recognition result of the command recognition unit 38, the command processing unit 35 instructs the projection device 1 to output audio corresponding to the text information on the answer sheet 100 instructed by the command.

[0064] (Instruction to Multiple Students) This image processing system makes it possible to give instruction to multiple students (multiple projection devices 1).

[0065] Fig. 10 shows an example of a situation where multiple projection devices 1 (multiple students) are present. Fig. 11 shows an example of a display state of the terminal device 2 when multiple projection devices 1 (multiple students) are present.

[0066] As shown in FIG. 10 , when there are multiple students 201 in a classroom 200, a projection device 1 is provided for each student 201. In such a case, it is difficult for a teacher to simultaneously view the answer sheets 100 of all students 201 using a terminal device 2 and provide guidance (grading) to all students 201 at the same time. Therefore, the terminal device 2 first displays simplified images corresponding to the answer sheets 100 of the multiple students 201 on the display unit 20. Then, guidance (grading) is provided only to a specific student 202 who is the subject of guidance. For example, when a tap is performed on the display area corresponding to the answer sheet 100 of a specific student 202 on the screen of the display unit 20, the terminal device 2 displays an enlarged image of the answer sheet 100 captured by a specific projection device 1 provided for the specific student 202 on the display unit 20. This allows the teacher to generate grading information for the answers written on the answer sheet 100 of the specific student 202.

[0067] The training target identification unit 37 of the server 30 identifies at least one of the multiple projection devices 1 that is capturing an image of the answer sheet 100 that is the target of training, based on the image data of the answer sheet 100 from each of the multiple projection devices 1. The training target identification unit 37 notifies the terminal device 2 of information about the specific projection device 1 (specific student 202) via the transmission unit 92. A student is considered a target of training, for example, if there is an error in the answer or if the student took a long time to write their answer. Based on the information about the specific projection device 1 from the server 30, the terminal device 2 changes the display state of the display area corresponding to the answer sheet 100 of the specific student 202 on the screen of the display unit 20. For example, the display color may be changed or the display may flash. The terminal device 2 generates grading information for the answers entered on the answer sheet 100 captured by the specific projection device 1 identified by the training target identification unit 37. The training mode notification unit 26 of the terminal device 2 notifies the specific projection device 1 that the student has become a target of training. The notification by the instruction mode notification unit 26 is made via the call unit 70 or the communication unit 80 .

[0068] [1.2 Operation] FIG. 12 is a flowchart showing an example of the flow of the overall processing operation in the image processing system according to the first embodiment.

[0069] First, the projection device 1 photographs the answer sheet 100 using the photographing unit 11 (step S11). Next, the projection device 1 transmits the image data of the answer sheet 100 photographed by the photographing unit 11 (step S12). The terminal device 2 displays a page image based on the image data of the answer sheet 100 photographed by the photographing unit 11 (step S22). At this time, the terminal device 2 displays the page image based on the image data of the answer sheet 100 after projective transformation (distortion correction) (step S32) has been performed in the server 30.

[0070] If there are multiple students 201 (multiple projection devices 1), the terminal device 2 selects which of the multiple projection devices 1 the terminal device 2 will use to display a paper image based on image data of the answer sheet 100 captured by (step S21). The instruction target identification unit 37 of the server 30 identifies at least one projection device 1 that is capturing an image of the answer sheet 100 to be trained, and notifies the terminal device 2 of information about the specific projection device 1 (specific student 202) via the transmission unit 92 (step S31). Based on the information about the specific projection device 1 from the server 30, the terminal device 2 selects which projection device 1 the terminal device 2 will use to display a paper image based on image data of the answer sheet 100 captured by.

[0071] The terminal device 2 generates grading information based on the written data using the input device 4 (step S23). Next, the terminal device 2 transmits the grading information generated based on the written data (step S24). The server 30 performs inverse distortion correction on the image data of the grading information, for example, by inverse projection transformation (step S33). The projection device 1 receives the grading information after the inverse distortion correction (step S13) and projects the grading information onto the answer sheet 100 (step S14). If an instruction to delete the grading information is received from the terminal device 2 (step S25), the projection device 1 deletes the grading information from the projected image (step S15) and returns to the processing of step S11.

[0072] FIG. 13 is a flowchart showing an example of the flow of processing operations until an image captured by the projection device 1 is displayed on the terminal device 2 in the image processing system according to the first embodiment.

[0073] First, the projection device 1 performs an image capture using the image capture unit 11 (step S101). The image capture unit 11 may capture a predetermined pattern image for distortion correction that has been projected in advance by the projection unit 10. The server 30 performs an initial distortion correction process based on the data of the predetermined pattern image captured by the image capture unit 11, and calculates initial distortion correction parameters (step S200).

[0074] Next, the image processing unit 12 of the projection device 1 performs a paper surface recognition (image recognition) process for the answer sheet 100 in the image captured by the image capturing unit 11 (step S102). Next, the image processing unit 12 transmits the image data captured by the image capturing unit 11 and the paper surface recognition metadata for the answer sheet 100 (step S106). In parallel with this, the image processing unit 12 of the projection device 1 performs a process for calculating partial feature amounts of the image region corresponding to the answer sheet 100 for the image captured by the image capturing unit 11 at the first time T = t1. As part of the feature amount calculation process, the image processing unit 12 of the projection device 1 first performs a partial cutout of the image captured by the image capturing unit 11 at the first time T = t1 (step S103). The partial cutout is performed, for example, by cutting out at least two rectangular regions from the image region corresponding to the answer sheet 100. Next, the image processing unit 12 extracts feature points and feature amounts from the cutout image (step S104). Next, the image processing unit 12 stores the coordinate data of the extracted feature points and the feature amount data in the storage unit 14 (step S105). Note that the answer sheet 100 may be blank initially, for example, when providing painting instruction. To recognize the answer sheet 100, a recognition marker may be written on the answer sheet 100.

[0075] The server 20 receives the captured image data and the paper surface recognition metadata of the answer sheet 100 from the projection device 1 (step S201). Next, the image processing unit 32 of the server 20 performs projective transformation (distortion correction) on the captured image data by referring to the initial distortion correction parameters calculated in advance (step S202). Next, the image processing unit 32 performs a process to erase all parts of the captured image except the paper surface image corresponding to the answer sheet 100 based on the paper surface recognition metadata (step S203). Next, the server 20 transmits data of the paper surface image corresponding to the answer sheet 100 to the terminal device 2 (step S204). The terminal device 2 displays the paper surface image corresponding to the answer sheet 100 (step S301).

[0076] FIG. 14 is a flowchart showing an example of the flow of processing operations until the scoring information generated by the terminal device 2 is projected by the projection device 1 in the image processing system according to the first embodiment.

[0077] First, in the terminal device 2, the teacher writes using the input device 4 on the screen of the display unit 20, which displays an image of the answer sheet 100. The terminal device 2 generates grading information based on the written data (step S311). Next, the terminal device 2 transmits the grading information data generated based on the written data (step S312). The transmitted grading information data may be image data such as bitmap data or vector data.

[0078] The projection device 1 receives the grading information data (step S111). Next, the image processing unit 12 of the projection device 1 calculates partial feature quantities of the image region corresponding to the answer sheet 100 for the image captured by the imaging unit 11 at the second time T = t2 (step S112). As part of the feature quantity calculation process, the image processing unit 12 of the projection device 1 first partially cuts out the image captured by the imaging unit 11 at the second time T = t2. The partial cutout is performed, for example, by cutting out at least two rectangular regions from the image region corresponding to the answer sheet 100. Next, the image processing unit 12 extracts feature points and feature quantities from the cutout image. Next, the image processing unit 12 stores the coordinate data and feature quantity data of the extracted feature points in the storage unit 14. Next, the image processing unit 12 compares the partial feature quantities of the answer sheet 100 at the first time T = t1 with the partial feature quantities of the answer sheet 100 at the second time T = t2. Based on the comparison result, the image processing unit 12 calculates the amount of movement of the answer sheet 100 from the first time T=t1 to the second time T=t2, and stores the amount of movement in the storage unit 14 (step S113). Next, the image processing unit 12 transmits the calculated amount of movement of the answer sheet 100 (step S114).

[0079] Meanwhile, the server 30 receives the grading information data from the terminal device 2 (step S211). If the image data of the grading information is vector data, the image processing unit 32 of the server 30 rasterizes it into bitmap data. In step S211, the server 30 also receives the movement amount of the answer sheet 100 from the projection device 1. The image processing unit 32 of the server 30 references the initial correction parameters and creates new correction parameters that take into account the movement amount of the answer sheet 100. Next, the image processing unit 32 of the server 30 references the new correction parameters and performs inverse distortion correction, for example, by inverse projection transformation, on the image data of the grading information (step S212). The server 30 transmits the image data of the grading information after the inverse distortion correction to the projection device 1 (step S213). The projection device 1 projects and displays the image data of the grading information from the server 30 on the answer sheet 100 (step S115). Furthermore, by looping the processing of the above steps S112 to S115 and steps S211 to S213, it is possible to continue projecting and displaying the grading information at the appropriate coordinate position on the answer sheet 100 even if the answer sheet 100 is moved.

[0080] FIG. 15 is a flowchart showing an example of the flow of processing operations when archiving image data of the answer sheet 100 and marking information in the image processing system according to the first embodiment.

[0081] In the projection device 1, when the archive instruction receiving unit 151 receives an archive instruction, the photographing unit 11 photographs the answer sheet 100 (step S121). The image processing unit 12 of the projection device 1 performs a paper surface recognition (image recognition) process for the answer sheet 100 in the image photographed by the photographing unit 11 (step S122). Next, the image processing unit 12 transmits the image data photographed by the photographing unit 11 and the paper surface recognition metadata for the answer sheet 100 (step S123).

[0082] The server 20 receives the captured image data and the paper surface recognition metadata of the answer sheet 100 from the projection device 1 (step S221). Next, the image processing unit 32 of the server 20 performs projective transformation (distortion correction) on the captured image data by referring to the initial distortion correction parameters calculated in advance (step S222). Next, the image processing unit 32 performs a process to erase all parts of the captured image except the paper surface image corresponding to the answer sheet 100 based on the paper surface recognition metadata (step S223). Next, the image processing unit 32 stores the data of the paper surface image corresponding to the answer sheet 100 in the storage unit 34 (step S223).

[0083] In addition, when the server 20 receives an archive instruction, the data to be stored in the memory unit 34 may be the following first to third data, as described above. First data: Image data and marking information of the answer sheet 100 before marking information is projected (before receiving instruction). Second data: Image data and marking information of the answer sheet 100 after marking information is projected (after receiving instruction). Third data: Image data of the answer sheet 100 after marking information is projected (after receiving instruction).

[0084] [1.3 Effects] As described above, the projection device 1 and image processing system according to the first embodiment project marking information as guidance information for answers as guidance information entered on the answer sheet 100 at a coordinate position that takes into account the amount of movement of the answer sheet 100. This makes it possible to appropriately project marking information for answers entered on the answer sheet 100 onto the answer sheet 100. This makes it possible to appropriately project marking information in real time in accordance with the movement of the answer sheet 100, even when the answer sheet 100 moves.

[0085] The effects described in this specification are merely examples and are not limiting, and other effects may also be achieved. The same applies to the effects of other embodiments described below.

[0086] 2. Other Embodiments The technology according to the present disclosure is not limited to the above-described embodiments, and various modifications are possible.

[0087] For example, the present technology can be configured as follows. According to the present technology configured as follows, instruction information for instruction information written on a sheet of paper is projected at a coordinate position that takes into account the amount of movement of the sheet of paper. This makes it possible to appropriately project instruction information for instruction information written on a sheet of paper onto the sheet of paper.

[0088] (1) A projection device comprising: an imaging unit that images a sheet of paper on which instruction-receiving information has been written, at a plurality of times including a first time and a second time that is later than the first time; an image processing unit that extracts partial feature amounts of the sheet of paper at the first time and partial feature amounts of the sheet of paper at the second time, and calculates an amount of movement of the sheet of paper from the first time at the second time; and a projection unit that projects instruction information for the instruction-receiving information written on the sheet of paper at a coordinate position that takes into account the amount of movement of the sheet of paper calculated by the image processing unit. (2) The projection device described in (1) above, further comprising: a transmission unit that transmits image data of the sheet of paper captured by the imaging unit to an external device; and a reception unit that receives the instruction information generated in the external device based on image data of the sheet. (3) The projection device described in (2) above, further comprising: a reception unit that receives an input instructing the external device to store image data of the sheet of paper and the instruction information. (4) The projection device according to any one of (1) to (3), wherein the imaging range of the imaging unit is larger than the projection range of the projection unit. (5) An image processing system including: an external device that generates, based on image data of a sheet of paper on which instruction target information is written, instruction information for the instruction target information written on the sheet of paper; and a projection device that projects the instruction information generated by the external device onto the sheet of paper, wherein the projection device comprises: an imaging unit that images the sheet of paper on which the instruction target information is written at a plurality of times including a first time and a second time that is later than the first time; an image processing unit that extracts partial feature amounts of the sheet of paper at the first time and partial feature amounts of the sheet of paper at the second time and calculates an amount of movement of the sheet of paper from the first time at the second time; and a projection unit that projects the instruction information for the instruction target information written on the sheet of paper at a coordinate position that takes into account the amount of movement of the sheet calculated by the image processing unit.(6) The image processing system according to (5) above, wherein the projection device further comprises: a transmitting unit that transmits image data of the paper captured by the photographing unit to the external device; and a receiving unit that receives the instruction information generated in the external device based on the image data of the paper, the external device comprising: a distortion correction unit that corrects distortion of the image data of the paper captured by the photographing unit, and generates the instruction information based on the image data of the paper after distortion correction by the distortion correction unit. (7) The image processing system according to (6) above, wherein the external device includes: a server and a terminal device, the server comprising: the distortion correction unit, and the terminal device comprising: a display unit that displays the image data of the paper after distortion correction by the distortion correction unit, and an instruction information input unit that accepts input of the instruction information based on the image data of the paper displayed on the display unit. (8) The image processing system according to (7) above, comprising a plurality of the projection devices, wherein the server further comprises a training target identification unit that identifies at least one projection device among the plurality of projection devices that is photographing the paper sheet to be a training target based on image data of the paper sheet from each of the plurality of projection devices and notifies the terminal device of information about the specific projection device, wherein the terminal device generates the training information for the training target information written on the paper sheet photographed by the specific projection device identified by the training target identification unit. (9) The image processing system according to (8) above, wherein the terminal device further comprises a notification unit that notifies the specific projection device that it has become a training target. (10) The image processing system according to any one of (5) to (9) above, wherein the external device further comprises a storage unit that stores image data of the paper sheet and the training information. (11) The image processing system described in (10) above, wherein the storage unit stores first data including image data of the paper sheet before the instruction information is projected and the instruction information, second data including image data of the paper sheet after the instruction information is projected and the instruction information, and third data including image data of the paper sheet after the instruction information is projected.(12) The image processing system according to any one of (5) to (11) above, wherein the external device comprises: a command recognition unit that recognizes commands input on the paper; and a command processing unit that performs processing based on the commands recognized by the command recognition unit. (13) The image processing system according to (12) above, wherein the external device further comprises: a storage unit that stores character information on the paper specified by the command. (14) The image processing system according to (12) or (13) above, wherein the projection device further comprises: an audio output unit, and the command processing unit of the external device instructs the projection device to output audio corresponding to the character information on the paper specified by the command. (15) The image processing system according to any one of (12) to (14) above, wherein the paper includes areas for writing questions and information to be taught, and an area for writing information of the command. (16) The image processing system according to any one of (5) to (15) above, wherein the terminal device further includes a calculation unit that calculates information about the number of pixels that can be displayed by the projection device and causes the calculated information about the number of pixels to be displayed on the display unit.

[0089] This application claims priority based on Japanese Patent Application No. 2021-13770, filed on January 29, 2021, in the Japan Patent Office, the entire contents of which are incorporated herein by reference.

[0090] Those skilled in the art will recognize various modifications, combinations, subcombinations, and variations depending on design requirements and other factors, and it is understood that these are within the scope of the appended claims and their equivalents.

Claims

1. A projection device comprising: an imaging unit that images a sheet of paper on which information to be taught is written at a plurality of times including a first time and a second time that is later than the first time; an image processing unit that extracts partial feature amounts of the sheet of paper at the first time and partial feature amounts of the sheet of paper at the second time, and calculates the amount of movement of the sheet of paper from the first time to the second time; and a projection unit that projects instruction information for the information to be taught written on the sheet of paper at a coordinate position that takes into account the amount of movement of the sheet of paper calculated by the image processing unit.

2. The projection device according to claim 1, further comprising: a transmitting unit that transmits image data of the paper captured by the photographing unit to an external device; and a receiving unit that receives the instruction information generated in the external device based on the image data of the paper.

3. The projection device according to claim 2, further comprising a receiving unit that receives an input instructing the external device to store the image data of the paper and the instruction information.

4. The projection device according to claim 1, wherein the imaging range of said imaging unit is larger than the projection range of said projection unit.

5. An image processing system including: an external device that generates instruction information for the taught information written on a form based on image data of the form on which the taught information is written; and a projection device that projects the instruction information generated by the external device onto the form, wherein the projection device is equipped with: an imaging unit that images the form on which the taught information is written at a plurality of times including a first time and a second time after the first time; an image processing unit that extracts partial feature amounts of the form at the first time and partial feature amounts of the form at the second time and calculates the amount of movement of the form from the first time to the second time; and a projection unit that projects the instruction information for the taught information written on the form at a coordinate position that takes into account the amount of movement of the form calculated by the image processing unit.

6. The image processing system of claim 5, wherein the projection device further comprises a transmitting unit that transmits image data of the paper captured by the photographing unit to the external device, and a receiving unit that receives the instruction information generated in the external device based on the image data of the paper, and the external device comprises a distortion correction unit that performs distortion correction on the image data of the paper captured by the photographing unit, and generates the instruction information based on the image data of the paper after distortion correction by the distortion correction unit.

7. The image processing system according to claim 6, wherein the external devices include a server and a terminal device, the server having the distortion correction unit, and the terminal device having a display unit that displays the image data of the paper after distortion correction by the distortion correction unit, and a training information input unit that accepts input of the training information based on the image data of the paper displayed on the display unit.

8. An image processing system as described in claim 7, comprising a plurality of the projection devices, wherein the server further comprises a training target identification unit that identifies at least one projection device among the plurality of projection devices that is photographing the paper that is the target of training based on image data of the paper from each of the plurality of projection devices, and notifies the terminal device of information about the identified projection device, and wherein the terminal device generates the training information for the training target information written on the paper that was photographed by the specific projection device identified by the training target identification unit.

9. The image processing system according to claim 8, wherein the terminal device further comprises a notification unit that notifies the specific projection device that it has become a target for guidance.

10. The image processing system according to claim 5, wherein the external device further comprises a storage unit that stores the image data of the paper and the instruction information.

11. The image processing system of claim 10, wherein the memory unit stores first data including image data of the paper before the instruction information is projected and the instruction information, second data including image data of the paper after the instruction information is projected and the instruction information, and third data including image data of the paper after the instruction information is projected.

12. The image processing system according to claim 5, wherein the external device comprises: a command recognition unit that recognizes commands input on the paper; and a command processing unit that performs processing based on the commands recognized by the command recognition unit.

13. The image processing system according to claim 12, wherein the external device further comprises a storage unit that stores character information on the paper sheet instructed by the command.

14. The image processing system according to claim 12, wherein the projection device further comprises an audio output unit, and the command processing unit of the external device instructs the projection device to output audio corresponding to the character information on the paper as instructed by the command.

15. The image processing system according to claim 12, wherein the form includes an area for writing questions and information to be taught, and an area for writing information about the command.

16. The image processing system according to claim 7, wherein the terminal device further comprises a calculation unit that calculates information about the number of pixels that can be displayed by the projection device and displays the calculated information about the number of pixels on the display unit.