Image processing device, image processing method, and program

Abstract

An image processing device according to the present disclosure is provided with: a movement detection unit for detecting the movement of a writing medium on the basis of moving-image data in which a movable writing medium and a scribe writing to the writing medium are imaged; and a written portion detection unit for detecting a written portion on the writing medium. When the movement of the writing medium is detected, the written portion detection unit executes a process that corresponds to the movement of the writing medium.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] A technology has been proposed that extracts the written portion on a writing medium from video image data that captures a state in which a writer is writing on the writing medium (see, for example, Patent Document 1). Such image processing requires even greater accuracy. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 163977 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an image processing device, an image processing method, and a program that are capable of performing image processing with high accuracy even when writing is performed on a movable writing medium. [Means for solving the problem]

[0005] The image processing device according to the present disclosure includes a movement detection unit that detects movement of a movable writing medium based on video image data capturing an image of a writer writing on the writing medium, and a writing portion detection unit that detects the writing portion on the writing medium, and when the movement of the writing medium is detected, the writing portion detection unit performs processing corresponding to the movement of the writing medium. Effect of the Invention

[0006] According to the image processing device, image processing method, and program of the present disclosure, image processing can be performed with high accuracy even when writing is performed on a movable writing medium. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an example of a configuration of an image processing system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment of the present disclosure. [Diagram 3] FIG. 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment of the present disclosure. [Figure 4] 1 is a diagram showing a result of detection of movement of a board surface by the image processing device according to the first embodiment of the present disclosure together with moving image data. FIG. [Diagram 5] 4A to 4C are diagrams illustrating a method for distinguishing between a written portion and a non-written portion by the image processing device according to the first embodiment of the present disclosure. [Figure 6] FIG. 2 is a diagram showing a detection result of a handwritten portion by the image processing device according to the first embodiment of the present disclosure together with moving image data. [Figure 7] FIG. 4 is a flowchart showing an example of a procedure of image processing by the image processing device according to the first embodiment of the present disclosure. [Figure 8] 11 is a diagram showing a result of detection of movement of the board surface by an image processing device according to Modification 1 of Embodiment 1 of the present disclosure, together with moving image data. FIG. [Figure 9] FIG. 13 is a diagram showing a result of detection of movement of the board surface by an image processing device according to Modification 2 of Embodiment 1 of the present disclosure, together with moving image data. [Figure 10] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing device according to a second embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram showing a generation result of handwritten data and composite video data by an image processing device according to a second embodiment of the present disclosure, together with video data. [Figure 12] FIG. 11 is a flowchart showing an example of a procedure of image processing by an image processing device according to a second embodiment of the present disclosure. [Figure 13] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing device according to a third embodiment of the present disclosure. [Figure 14] FIG. 13 is a diagram showing a result of generation of a writing log by an image processing device according to a third embodiment of the present disclosure, together with moving image data. [Figure 15] FIG. 11 is a flowchart showing an example of a procedure of image processing by an image processing device according to a third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, similar components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0009] [Embodiment 1] The first embodiment will be described with reference to the drawings.

[0010] (Example of image processing system configuration) 1 is a diagram showing an example of a configuration of an image processing system 1 according to the first embodiment of the present disclosure. As shown in FIG 1, the image processing system 1 according to the first embodiment includes an image processing device 10, an imaging device 20, an input / output device 30, and a recording device 40.

[0011] The imaging device 20 is a video camera or the like that records a lecture, training, seminar, conference, or the like that uses a writing medium such as a blackboard, a whiteboard, an electronic blackboard, or the like. In the example of Fig. 1, the imaging device 20 captures an image of a scribe 300 giving an explanation while writing on the whiteboard 200.

[0012] Here, the transcriber 300 is a lecturer, teacher, presenter, or the like who gives a lecture, training session, seminar, conference, or the like.

[0013] The whiteboard 200 as a group of writing media includes a plurality of board surfaces 210a, 210b, 210c, and 210d as writing media. The whiteboard 200 itself is fixed to, for example, a wall of a building, but is configured as a sliding whiteboard in which the board surfaces 210a and 210b, or the board surfaces 210c and 210d arranged above and below can be interchanged by raising and lowering. The scribe 300 can write on each of the board surfaces 210a to 210d or erase the written contents while raising and lowering the upper and lower board surfaces 210a and 210b, or the board surfaces 210c and 210d.

[0014] However, whiteboard 200 may be of a type in which board surfaces 210a and 210c, or board surfaces 210b and 210d arranged on the left and right, slide left and right and are interchangeable. Also, the number of board surfaces 210a to 210d that move at one time may be one, or three or more.

[0015] Furthermore, the board surfaces 210a to 210d may be white, black, green, or other colors. Characters, graphs, figures, and the like may be written on the board surfaces 210a to 210d. The color of the writing may be any color that is distinguishable on the board surfaces 210a to 210d, and multiple colors may be used.

[0016] Hereinafter, when there is no need to distinguish between the individual plate surfaces 210a, 210b, 210c, and 210d, they may be simply referred to as plate surfaces 210.

[0017] The imaging device 20 is set at a position and angle of view that allows it to simultaneously capture images of the scribe 300 and the whiteboard 200. The imaging device 20 generates video data by capturing images of the scribe 300 and the whiteboard 200, and outputs the generated video data to the image processing device 10.

[0018] The image processing device 10 performs a process of extracting the handwritten portion from the moving image data generated by the imaging device 20. At this time, the image processing device 10 performs a process according to the vertical movement of the board surface 210.

[0019] The input / output device 30 is configured to allow a user to input instructions and the like to the image processing device 10. Furthermore, the input / output device 30 acquires the extraction result of the handwritten portion from the image processing device 10 and presents it to the user, for example.

[0020] The recording device 40 records the moving image data processed by the image processing device 10, the extracted results of the handwritten portion, and the like.

[0021] (Example of configuration of image processing device) FIG. 2 is a block diagram showing an example of a functional configuration of the image processing device 10 according to the first embodiment of the present disclosure.

[0022] 2, the image processing device 10 is connected to one or more imaging devices 20, an input / output device 30, and a recording device 40. The image processing device 10 may be directly connected to the imaging devices 20, the input / output devices 30, and the recording device 40, or may be connected to them via a network.

[0023] The image processing device 10 receives moving image data output from one or more imaging devices 20, and detects the movement of each of the board surfaces 210a to 210d of the whiteboard 200, and executes other processes.

[0024] To realize such functions, the image processing device 10 includes, as its functional components, a moving image input unit 11, a control information input unit 12, a movement detection unit 13, a handwritten portion detection unit 14, and an output unit 16, for example.

[0025] The video image input unit 11 receives at least one video image data output from one or more imaging devices 20 as a video signal, and passes it to the movement detection unit 13 and the handwritten portion detection unit 14.

[0026] The control information input unit 12 receives control information, which is an instruction from the user input to the input / output device 30, as a rule signal, and passes it to the movement detection unit 13 and the handwritten portion detection unit .

[0027] The movement detection unit 13 receives moving image data from the moving image input unit 11, detects straight line portions from the frames of each of the board surfaces 210a-210d, and detects the presence or absence of movement, the direction and speed of the movement, etc., for each of the board surfaces 210a-210d from the movement of these straight line portions. At this time, the movement detection unit 13 refers to the control information received from the control information input unit 12. The movement detection unit 13 passes the movement detection results of each of the board surfaces 210a-210d to the handwriting portion detection unit 14.

[0028] The handwritten portion detection unit 14 detects handwritten portions written by the writer 300 on the touch screens 210a-210d from the video data received from the video input unit 11. At this time, the handwritten portion detection unit 14 executes a process for each touch screen 210a-210d according to the movement detection result for each touch screen 210a-210d based on the movement detection result for each touch screen 210a-210d received from the movement detection unit 13. At this time, the handwritten portion detection unit 14 also refers to the control information received from the control information input unit 12. The handwritten portion detection unit 14 passes the detection result of the handwritten portions on the touch screens 210a-210d to the output unit 16.

[0029] The output unit 16 outputs the detection result of the handwriting portion by the handwriting portion detection unit 14 to one or more input / output devices 30 and one or more recording devices 40 connected to the image processing device 10. In addition, the output unit 16 may output the moving image data captured by the imaging device 20 to the recording device 40 as is.

[0030] The user can check the detection result of the handwritten portion output from the image processing device 10 on the input / output device 30. The user may also be able to view video data on the input / output device 30. The user can also input control information from the input / output device 30. However, the input / output device 30 does not have to be an independent device, and the image processing device 10 may have the function of the input / output device 30.

[0031] The control information that the user can input includes, for example, information used to determine movement of the plates 210a to 210d, the details of processing and control based on the movement determination result, the details of processing for video data, and supplementary information used to detect movement of the plates 210a to 210d.

[0032] Information used to determine the movement of board surfaces 210a to 210d includes, for example, information on straight line portions detected from the frames of board surfaces 210a to 210d. The contents of the processing and control based on the movement determination results may differ between board surfaces 210 where movement has been detected and board surfaces 210 where movement has not been detected.

[0033] The supplemental information used to detect the movement of the board surfaces 210a to 210d includes, for example, the movement direction of each of the board surfaces 210a to 210d, the number, configuration, arrangement, and color of the board surfaces 210a to 210d.

[0034] The storage device 40 records the detection result of the handwritten portion obtained from the image processing device 10. However, the storage device 40 does not have to be an independent device, and the image processing device 10 may have the function of the storage device 40.

[0035] FIG. 3 is a block diagram showing an example of a hardware configuration of the image processing device 10 according to the first embodiment of the present disclosure.

[0036] As shown in FIG. 3, the image processing device 10 is configured as a computer having, as its hardware configuration, for example, a CPU (Central Processing Unit) 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, and an I / O (In / Out) port 140.

[0037] The CPU 110 reads out a program related to image processing stored in the ROM 120, for example, and develops it on the RAM 130, and operates according to the program. In this way, the CPU 110 can realize various functions of the image processing device 10.

[0038] The I / O port 140 is connected to the imaging device 20, the input / output device 30, and the recording device 40, which are external devices of the image processing device 10, and transmits and receives various data and information such as video image data, detection results of the handwritten portion, and control information to and from these external devices.

[0039] (Example of board movement detection operation) Next, the operation of detecting movement of the board surfaces 210a to 210d by the image processing device 10 of the first embodiment will be described with reference to FIG.

[0040] Fig. 4 is a diagram showing the results of detection of movement of the surfaces 210a-210d by the image processing device 10 according to the first embodiment of the present disclosure together with moving image data. Fig. 4 shows frames of the moving image data at times t1, t2, and t3, and detection results of the surfaces 210a-210d in those frames. Here, one frame refers to one frame of a still image constituting the moving image data.

[0041] At time t1 in FIG. 4, the movement detection unit 13 of the image processing device 10 detects the board surfaces 210a to 210d by detecting the straight line portions 261 of the frames of the individual board surfaces 210a to 210d.

[0042] The straight line portions 261 of the frames of the plate surfaces 210a to 210d can be detected, for example, by extracting edges based on differences in luminance values ​​in the video data or by binarizing the video data, and then performing a Hough transform on the obtained data.

[0043] However, in addition to the frames of the board surfaces 210a to 210d, for example, straight lines included in the handwritten portion on the board surfaces 210a to 210d may also be detected as straight line portions. By referring to control information from the input / output device 30, the movement detection unit 13 can determine whether the detected straight line portion is the frame of the board surfaces 210a to 210d based on the length of the straight line portion 261 of the frame of each of the board surfaces 210a to 210d and the position at which the straight line portion 261 of the frame of each of the board surfaces 210a to 210d is detected.

[0044] For example, if the detected straight line portion is a straight line of the handwritten portion, it can be determined that it is not the straight line portion 261 of the frame of the board surfaces 210a to 210d because it is shorter than the frame of the board surfaces 210a to 210d and is detected inside the position of the frame of the board surfaces 210a to 210d.

[0045] When the straight line portions 261 of the individual board surfaces 210a to 210d are detected in this manner, the movement detection unit 13 thereafter starts tracking the detected straight line portions 261 of the board surfaces 210a to 210d.

[0046] At time t2 in Fig. 4, the board surfaces 210a, 210b arranged above and below are being raised and lowered by the scribe 300 or the like. The movement detection unit 13 tracks the straight line portion 261, for example, by taking a difference from the detection result of the straight line portion 261 in the previous frame. The movement detection unit 13 detects that the straight line portion 261 is moving and that its movement has ended, based on the fact that a difference has occurred in the detection result of the straight line portion 261 in the current frame from the previous frame.

[0047] 4, the lifting and lowering of the board surfaces 210a, 210b is completed, and the positions of the board surfaces 210a, 210b are switched, with the board surface 210a located above and the board surface 210b located below. The movement detection unit 13 continues to track the straight line portions 261 of each of the board surfaces 210a to 210d.

[0048] (Example of detection of handwritten portion) Next, the operation of detecting a written portion on the board surface 210 by the image processing device 10 according to the first embodiment will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram for explaining a method of distinguishing a written portion from a non-written portion by the image processing device 10 according to the first embodiment of the present disclosure.

[0049] As shown in Fig. 5(a), the moving image data includes a handwritten portion 250 written on the keyboard surface 210. As shown in Fig. 5(b), the moving image data also includes, for example, a writer 300 standing in front of the keyboard surface 210, superimposed on the keyboard surface 210. The handwritten portion detection unit 14 of the image processing device 10 identifies and detects the handwritten portion 250 from the writer 300, etc., reflected on the keyboard surface 210 in this manner.

[0050] When detecting the handwritten portion 250, the handwritten portion detection unit 14 refers to the movement detection results of the plate surfaces 210a to 210d detected by the movement detection unit 13, and varies the detection operation of the handwritten portion 250 for each of the plate surfaces 210a to 210d according to the movement detection results.

[0051] When the movement detection unit 13 does not detect movement on a given board surface 210 that is the target of the detection operation for the handwritten portion 250, the handwritten portion detection unit 14 attempts to detect the handwritten portion 250 on that board surface 210.

[0052] The detection operation of the handwritten portion 250 by the handwritten portion detection unit 14 is performed, for example, using a reference frame. The reference frame is a frame that is updated by the handwritten portion detection unit 14 for each predetermined operation and serves as a reference for the subsequent operation. For example, the earliest reference frame may be a frame that shows the plate surfaces 210a to 210d before writing is started and before they are replaced by vertical movement.

[0053] The handwriting detection unit 14 calculates the difference for each pixel between the current frame and the reference frame, extracts pixels with a large difference, that is, pixels that have undergone a large change from the state of the reference frame, and detects clusters where such pixels exist consecutively.

[0054] Such a collection portion may include, for example, a writing portion 250 added by the writer 300, and the writer 300 performing some action in front of the board surface 210. In order to distinguish between these, the writing portion detection unit 14 performs shape evaluation, extraction time evaluation, and stillness time evaluation, which will be described below.

[0055] First, for each detected group part, the writing part detection unit 14 sets an outer rectangle, which is the smallest rectangle that contains the entire group part, and an inner rectangle, which is the largest rectangle that fits within the group part. The writing part detection unit 14 also calculates a shape evaluation value α from the size of the inner rectangle relative to the set outer rectangle. The shape evaluation value α is set to a higher value as the size of the inner rectangle relative to the outer rectangle is smaller.

[0056] 5(a), when the collection portion detected by the writing portion detection unit 14 is the writing portion 250, the writing portion 250 is mainly composed of a combination of line segments, so that the inner quadrangle Pi is relatively small compared to the outer quadrangle Po. Therefore, the shape evaluation value α is high.

[0057] 5(b), when the collection part detected by the writing part detection unit 14 is the writer 300, the writer 300 is composed of parts having a large size such as the head and the torso, so the inner rectangle Qi is relatively large compared to the outer rectangle Qo. Therefore, the shape evaluation value α is low.

[0058] Next, the writing portion detection unit 14 checks the time during which changes continued to occur in the pixels included in each detected group portion. In other words, the writing portion detection unit 14 calculates the difference between the current frame and a reference frame in multiple frames prior to the current frame for at least some or all of the pixels included in the group portion.

[0059] The handwriting detection unit 14 also accumulates frames that have a large difference from the reference frame, and the number of consecutive frames among the accumulated frames is set as an evaluation value β of the extraction time, which is the time during which a change continues to be extracted in the pixels included in the collection part. In other words, the evaluation value β of the extraction time becomes higher as the number of consecutive frames increases and the time during which the change is extracted becomes longer.

[0060] When the collection portion detected by the writing portion detection unit 14 is the writing portion 250, each time writing is added, frames having a large difference are continuously accumulated, and therefore the evaluation value β of the extraction time becomes high.

[0061] When the collection part detected by the writing part detection unit 14 is the writer 300 moving in front of the board surface 210, the frames with a large difference are cut off when the writer 300 passes in front of the board surface 210. Therefore, the number of consecutive frames decreases, and the evaluation value β of the extraction time becomes low. However, when the writer 300 stops moving in front of the board surface 210, the evaluation value β of the extraction time may become high even if the collection part is the writer 300.

[0062] Next, the handwriting detection unit 14 checks the time when the pixels included in each detected group part did not change. In other words, the handwriting detection unit 14 calculates the difference between the current frame and the frame immediately before the current frame for at least some or all of the pixels included in the group part.

[0063] The handwriting detection unit 14 also accumulates frames that have a large difference from the previous frame, and sets the reciprocal of these differences as the evaluation value γ of the still time, which is the time during which no change appears in the pixels included in the collection part. In other words, the evaluation value γ of the still time becomes higher as there are more frames with a small difference from the previous frame and the still time of each pixel becomes longer.

[0064] When the collection portion detected by the writing portion detection unit 14 is the writing portion 250, since a writing portion that has been written once does not normally move, the difference with the previous frame is zero or extremely small, and the evaluation value γ of the still time becomes high.

[0065] When the collection part detected by the writing part detection unit 14 is a writer 300 or the like who is moving in front of the writing surface 210 or making gestures or other movements, the difference with the previous frame becomes large, and the evaluation value γ of the still time becomes low.

[0066] The handwriting portion detection unit 14 substitutes each of the evaluation values ​​α, β, and γ obtained as described above into the evaluation function E(α, β, γ) to obtain a solution, and detects the collection portion where the solution of the evaluation function E(α, β, γ) is greater than the threshold value THr as the handwriting portion.

[0067] Then, for each detected piece of writing, the writing portion detection unit 14 determines whether to write and retain each pixel value of the piece of writing at the corresponding pixel position in the current frame, or to delete the piece of writing without writing it at the corresponding pixel position in the current frame.

[0068] For example, if the pixel position of the handwriting detected in the current frame matches the pixel position of a handwriting in the previous frame, the handwriting detection unit 14 deletes the pixel values ​​of the handwriting without writing them to the corresponding pixel positions in the current frame. The reason is as follows.

[0069] To be detected as a handwritten portion in a given frame, the solution of the evaluation function E(α,β,γ) must be greater than the threshold value THr, which means that a large change has occurred at that pixel position. If such a large change occurs again in the current frame at a pixel position that was previously held as a handwritten portion, it is considered to mean that the previously written portion has been erased by the writer 300 or the like in the current frame.

[0070] For the above reasons, the handwriting detection unit 14 deletes from the current frame any handwriting detected at the same pixel position as a previously detected handwriting.

[0071] For any other handwriting detected in the current frame, the handwriting detection unit 14 writes and holds the pixel values ​​of the handwriting in the current frame. Note that, instead of writing the pixel values ​​of the handwriting in the corresponding pixel positions in the current frame, the handwriting detection unit 14 may hold only the pixel position information of each pixel included in the handwriting.

[0072] The handwriting portion detection unit 14 sets the current frame, which reflects the results of retaining and deleting the pixel values ​​of the handwriting portion, as a new reference frame. Such updating of the reference frame is performed, for example, for each of the board surfaces 210a to 210d.

[0073] In the case of a type in which a plurality of board surfaces 210a to 210d move and change places simultaneously, like whiteboard 200 of the embodiment, the reference frame may be updated for each of board surfaces 210a, 210b that move at one time, or for each of board surfaces 210c, 210d.

[0074] The handwriting detection unit 14 uses the newly updated reference frame when detecting handwriting in the frame next to the current frame.

[0075] By performing such processing, the handwriting portion detection unit 14 can perform different processing depending on the result of the movement determination of the plate surfaces 210a to 210d. A specific example is shown in FIG.

[0076] Fig. 6 is a diagram showing the detection result of the handwritten portion by the image processing device 10 according to the first embodiment of the present disclosure together with video data. Fig. 6 shows frames of video data at times t1, t2, and t3, and the detection result of the handwritten portion in each frame. Here, the video data at times t1, t2, and t3 is the same as the video data at times t1, t2, and t3 in Fig. 4 described above.

[0077] 6, the writing portion detection unit 14 detects the writing portion 250 for each of the board surfaces 210a to 210d. At time t1, no movement has occurred on any of the board surfaces 210a to 210d. In this case, the writing portion detection unit 14 detects the writing portion 250 for all of the board surfaces 210a to 210d.

[0078] At time t2 in Fig. 6, movement occurs on the upper and lower board surfaces 210a, 210b. In this case, the writing portion detection unit 14 performs different detection processes for the writing portion 250 depending on the movement detection results of the individual board surfaces 210a to 210d.

[0079] More specifically, in the frame at time t2 of the video image data, the writing portion detection unit 14 detects the writing portion 250 in the area corresponding to the board surfaces 210c, 210d where no movement was detected by the movement detection unit 13, and appropriately retains and deletes each pixel value of the detected writing portion 250.

[0080] Furthermore, in the frame at time t2 of the video image data, the handwriting detection unit 14 does not detect the handwriting 250 in the region corresponding to the board surfaces 210a, 210b whose movement has been detected by the movement detection unit 13. Furthermore, the handwriting detection unit 14 deletes from the current frame each pixel value that has been held as the handwriting 250 on the board surfaces 210a, 210b in the previous frame. Furthermore, the handwriting detection unit 14 deletes the reference frame at that time for the board surfaces 210a, 210b.

[0081] As described above, when a movement occurs on any of the board surfaces 210, each pixel included in that area changes significantly. For this reason, if a difference in a frame after the movement of the board surface 210 occurs is taken with reference to a reference frame before the movement of the board surface 210 occurs, a large difference is detected in areas other than the written area 250, making it difficult to correctly separate the written area 250 from the non-written area. When the movement of the board surface 210 is detected, the reference frame corresponding to that board surface 210 is deleted, thereby making it possible to suppress erroneous detections such as those described above.

[0082] Through these processes, the detection result of the handwriting portion 250 at time t2 is a state in which the left side area corresponding to the board surfaces 210a and 210b does not have the handwriting portion 250, and the right side area corresponding to the board surfaces 210c and 210d has the handwriting portion 250. This state in the detection result of the handwriting portion 250 is maintained until the movement of the board surfaces 210a and 210b is completed.

[0083] 6, when the lifting and lowering of the plate surfaces 210a, 210b is completed, the handwritten portion detection unit 14 starts detecting the handwritten portion 250 on each of the plate surfaces 210a to 210d again. The handwritten portion detection unit 14 also newly registers the frame at that time as a reference frame.

[0084] (Example of processing by image processing device) Next, an example of image processing by the image processing device 10 according to the first embodiment will be described with reference to Fig. 7. Fig. 7 is a flow diagram showing an example of a procedure of image processing by the image processing device 10 according to the first embodiment of the present disclosure.

[0085] In the flow diagram of FIG. 7, one cycle corresponds to processing for one predetermined frame constituting moving image data, and when processing for one frame is completed, the cycle is repeated sequentially for the subsequent frames.

[0086] As described above, when performing image processing, the image processing device 10 registers a reference frame in which the board surfaces 210a to 210d are reflected before writing is started and before replacement due to vertical movement as the initial reference frame. In other words, this process can be regarded as the process of step S114 in FIG. 7.

[0087] That is, when the movement detection unit 13 detects movement of the board surfaces 210a-210d (step S101), no movement of any of the board surfaces 210a-210d is detected (step S102: No), and since it is the very beginning of processing by the image processing device 10 and there is no reference frame (step S106: No), the handwriting detection unit 14 registers the frame of the video image data at that time as the reference frame (step S114).

[0088] As a result, for example, in the next frame, the processing returns to the beginning again, and the movement detection unit 13 detects and tracks straight portions of the board surfaces 210a to 210d to detect movement of the board surfaces 210a to 210d (step S101), and the writing portion detection unit 14 checks whether the movement detection unit 13 has detected movement of any of the board surfaces 210a to 210d (step S102).

[0089] At the very beginning of processing by the image processing device 10, since there is no writing on the lower surface 210a, 210c or there is still ample space to add writing, it is considered that the writer 300 or the like will not move the surface 210a to 210d.

[0090] In this way, for example, if the movement detection unit 13 does not detect any movement of the board surfaces 210a to 210d (step S102: No), the handwriting detection unit 14 determines whether or not there is a reference frame that has already been registered or updated (step S106).

[0091] If there is a reference frame (step S106: Yes), the writing portion detection unit 14 extracts a group of pixels with a large difference between the reference frame and the current frame (step S107). At this time, the extracted group may include not only a writing portion but also a non-writing portion such as the writer 300 performing some action.

[0092] The writing part detection unit 14 compares the outer rectangle and the inner rectangle for each aggregate part to calculate a shape evaluation value α (step S108). The shape evaluation value α tends to be higher for a writing part than for a non-writing part, such as the writer 300 extracted as an aggregate part.

[0093] The writing portion detection unit 14 calculates an evaluation value β of the extraction time for each group portion from the number of consecutive frames that have a large difference from the reference frame among the current frame and the multiple frames preceding the current frame (step S109). The extraction time at this time represents the time during which a change continues to be extracted in a specific group portion, and the evaluation value β of the extraction time tends to be higher in the writing portion than in the non-writing portion.

[0094] The handwriting portion detection unit 14 calculates a stillness evaluation value γ for each group portion from the reciprocal of the difference between the current frame and the frame immediately preceding the current frame, which has the largest difference from the current frame and the frame immediately preceding the current frame, among the frames preceding the current frame (step S110). The stillness time in this case represents the time during which no change appears in the pixels included in the group portion, and the stillness evaluation value γ tends to be higher in the handwriting portion than in the non-handwriting portion.

[0095] The handwriting portion detection unit 14 finds a solution of the evaluation function E(α, β, γ) for each collection portion, and determines that a collection portion for which the solution of the evaluation function E(α, β, γ) is greater than a threshold value THr is a handwriting portion (step S111).

[0096] The handwriting portion detection unit 14 determines whether to hold or delete each collection portion determined to be a handwriting portion (step S112). For example, when the pixel position of the handwriting portion detected in the frame currently being processed matches the pixel position of the handwriting portion held in the previous frame, the handwriting portion detection unit 14 determines that the current detection result indicates that the previously written portion has been erased by the writer 300, and deletes the collection portion without holding it.

[0097] The handwriting detection unit 14 reflects the results of keeping and deleting the individual collection parts determined to be handwriting parts in the frame to be processed this time, and updates the reference frame using this as a new reference frame (step S113).

[0098] As described above, for a while after the image processing device 10 starts processing, writing is performed exclusively on the board surfaces 210a, 210c located below, so that no movement of the board surfaces 210a to 210d occurs, and the processing of steps S106 to S113 is repeated while the reference frame is updated sequentially.

[0099] As this happens, the lower plate surfaces 210a and 210c are filled with writing, and at a predetermined timing, one of the plate surfaces 210a to 210d is switched.

[0100] In this way, for example, if the movement detection unit 13 detects movement of any of the board surfaces 210a to 210d (step S102: Yes), the handwriting detection unit 14 branches the processing between the board surface 210 where movement has occurred and the board surface 210 where no movement has occurred (step S103).

[0101] That is, when the area to be processed in a given frame corresponds to the board surface 210 where the current movement has occurred (step S103: Yes), the handwriting portion detection unit 14 deletes the handwriting portion stored in the area corresponding to the board surface 210 where the current movement has occurred in the previous frame (step S104).

[0102] The writing portion detection unit 14 deletes the reference frame for the board surface 210 where the current movement has occurred (step S105).

[0103] On the other hand, if the area to be processed in a given frame does not correspond to the board surface 210 where the current movement occurred (step S103: No), the writing portion detection unit 14 performs the processes of steps S106 to S113 as described above.

[0104] In this way, by the writing portion detection unit 14 performing the branched processing from steps S103 to S113, the previous data is temporarily reset in the area corresponding to the board surface 210 where movement has occurred, and the previous data is maintained in the area corresponding to the board surface 210 where no movement has occurred.

[0105] After that, for example, when the processing returns to the beginning again in the next frame, if no movement occurs on any of the board surfaces 210a to 210d (step S102: No), or if no movement occurs on the board surface 210 on which movement has occurred this time (step S103: No), then no reference frame exists for the board surface 210 on which movement has occurred this time (step S105: No), and a new reference frame is registered for the board surface 210 on which movement has occurred this time (step S114).

[0106] With the above, the image processing by the image processing device 10 of the first embodiment is completed.

[0107] (Overview) In recent years, there have been an increasing number of opportunities to record lectures, training sessions, seminars, conferences, and the like in various settings, including educational settings.

[0108] In these recordings, rather than using presentation materials that can be easily distributed as digital data, lecturers, teachers, or presenters are often recorded explaining by actually writing on a whiteboard, blackboard, or electronic blackboard.

[0109] For this reason, various image analysis and image processing techniques have been proposed for recorded video that includes whiteboards, blackboards, electronic blackboards, and the like.

[0110] According to the image processing device 10 of the first embodiment, when separating the written portion from the non-written portion based on the change between frames of the video image data, if movement of the board surface 210 is detected, processing is executed according to the movement of the board surface 210. This makes it possible to perform image processing with high accuracy even when writing is performed on the movable board surface 210.

[0111] According to the image processing device 10 of the first embodiment, when movement of the tablet surface 210 is detected, the detection result of the handwritten portion on the tablet surface 210 is deleted. This makes it possible to suppress changes between frames caused by the movement of the tablet surface 210 from affecting the entire tablet surface 210 or the entire screen of the video data, and to accurately distinguish between the handwritten portion and the non-handwritten portion.

[0112] According to the image processing device 10 of the first embodiment, when movement of the board surface 210 is detected, the detection result of the handwriting portion on the board surface 210 where the movement was detected is deleted, and the detection result of the handwriting portion on the board surface 210 where the movement was not detected is retained. This makes it possible to more accurately distinguish between changes between frames due to movement of the board surface 210 and changes between frames due to the addition of handwriting, etc.

[0113] According to the image processing device 10 of the first embodiment, the movement of the board surface 210 is detected by detecting the straight line parts of the frame of the board surface 210 and detecting the movement of the straight line parts. This makes it possible to detect the movement of the board surface 210 simply and with high accuracy without requiring, for example, a huge amount of data analysis. In addition, since the board surface 210 itself is directly detected, the movement of the board surface 210 can be detected even if, for example, there is no writing on the board surface 210.

[0114] According to the image processing device 10 of the first embodiment, for example, a straight line included in a handwritten portion and a frame of the board surface 210 are distinguished based on the length of the straight line portion. The length of the straight line portion of the frame of the board surface 210 can be determined based on the control information input from the input / output device 30 and supplemental information including information on the direction of movement of the board surface 210, the number, configuration, arrangement, color, etc. This makes it possible to suppress erroneous detection.

[0115] According to the image processing device 10 of the first embodiment, the detection result of the handwritten portion extracted with high accuracy as described above can be used for various purposes.

[0116] In the above-described first embodiment, the whiteboard 200 has a plurality of board surfaces 210a, 210c. However, a whiteboard or the like on which a writer writes may have only one board surface.

[0117] In the above-described first embodiment, the whiteboard 200 is fixed to, for example, a wall of a building. However, the whiteboard on which the writer writes may be a whiteboard with casters that can be moved freely.

[0118] In the above-described first embodiment, a blackboard, an electronic blackboard, or the like is used in addition to the whiteboard 200. However, the object on which writing is performed may be the whiteboard 200, a wall of a building, or a sheet-like paper medium such as a construction paper attached to a blackboard or the like.

[0119] In this case, the paper medium may be attached to a whiteboard with casters or the like so as to be movable. Alternatively, a plurality of paper media may be attached in layers, and each time writing on one paper medium is completed, each paper medium is turned over. Even in such a configuration, in the above-mentioned embodiment 1, the writing portion on the paper medium can be correctly detected continuously in response to a large change that occurs in the image each time a paper medium filled with writing is turned over and a new paper medium appears.

[0120] In the above-described first embodiment, the detection result of the handwritten portion according to the movement of the board surface 210 is output to the recording device 40. However, the image processing device may output the moving image data received from the imaging device and the movement detection result of the board surface 210 to the recording device in a state in which they are associated with each other.

[0121] (Variation 1) Next, an image processing device according to a first modification of the first embodiment will be described with reference to Fig. 8. In the image processing device according to the first modification, the method of detecting the movement of the board surfaces 210a to 210d is different from that of the first embodiment described above.

[0122] The movement detection unit of the image processing device of the first modification detects the movement of the display surfaces 210a-210d by estimating the optical flow of the entire screen in a predetermined frame constituting the moving image data. In the optical flow, the movement of an object detected by, for example, a luminance gradient is represented by vectors, and the movement of the object is estimated by the direction and magnitude of these vectors.

[0123] When detecting the movement of the predetermined board surface 210 using optical flow, the brightness gradient is used to detect the handwritten portion that moves with the movement of the predetermined board surface 210. Normally, the movement of the board surface 210 occurs after the board surface 210 is filled to a certain extent with the handwritten portion, so the movement of the board surface 210 can also be detected by detecting the movement of the handwritten portion of the board surface 210 in this way.

[0124] However, at this time, the writer 300 performing some action in front of the tablet surface 210 may also be detected. Therefore, the movement detection unit of the first modification group the optical flows expressed by vectors into vector groups each containing a predetermined number of vectors. The movement detection unit of the first modification also determines whether the detected movement is global based on the size of the grouped vector groups.

[0125] For example, if the grouped vectors are a handwritten portion that covers substantially the entirety of the board surface 210, the vector group will have a predetermined uniform size, and the detected movement will be global.

[0126] On the other hand, if the grouped vectors are of a person such as the writer 300 standing in front of the board surface 210, the vectors will be small and the detected movement will be localized.

[0127] From the above, the movement detection unit of the first modified example can distinguish between the moving tablet surface 210 and the writer 300 performing some kind of action, and can correctly detect the movement of the tablet surface 210.

[0128] In the case where multiple board surfaces 210 move simultaneously by being raised and lowered, or move simultaneously in the left-right direction by sliding sideways, as in the case of the whiteboard 200 described above, the movement detection unit of variant example 1 may detect the movement of the board surfaces 210 based on the global detection of optical flows having vectors in opposite directions.

[0129] FIG. 8 is a diagram showing the results of detection of movement of the surfaces 210a to 210d by the image processing device according to the first modification of the first embodiment of the present disclosure, together with moving image data.

[0130] 8 shows frames of video data at times t1, t2, and t3, and the results of detecting movement of the board surfaces 210a to 210d in those frames, as well as the results of detecting handwritten portions for reference. Note that the video data at times t1, t2, and t3 is the same as the video data at times t1, t2, and t3 in FIG. 4 and FIG. 6 described above.

[0131] At time t1 in FIG. 8, writing is being performed on the board surfaces 210a to 210d, but since no movement occurs on any of the board surfaces 210a to 210d, the movement detection unit of Modification 1 does not detect movement of the board surfaces 210a to 210d due to optical flow.

[0132] Since the movement of the plate surfaces 210a to 210d is not detected, the handwritten portion detection unit of the first modified example detects the handwritten portion 250 on all of the plate surfaces 210a to 210d. As a result, the detection result of the handwritten portion 250 at the time t1 is obtained.

[0133] At time t2 in FIG. 8, when the upper and lower keyboard surfaces 210a, 210b are moving by being raised and lowered, the movement detection unit of variant example 1 detects an optical flow 262 having vectors in opposite directions up and down from the movement of the actual handwritten part recorded in the video image data, and since this is a global one, it determines that movement of the keyboard surfaces 210a, 210b has occurred.

[0134] Since the movement of the board surfaces 210a and 210b has been detected, the handwriting detection unit of the first modified example deletes the detection result of the handwriting portion 250 on the board surfaces 210a and 210b, and holds the detection result of the handwriting portion 250 on the stationary board surfaces 210c and 210d. This provides the detection result of the handwriting portion 20 at time t2.

[0135] 8, the moving image data includes the board surfaces 210a-210d, all of which are again in a stationary state, and the scribe 300, who moves in front of these board surfaces 210a-210d. Therefore, the movement detection unit of the first modification detects the optical flow 262s caused by the movement of the scribe 300.

[0136] However, this optical flow 262s is not global and its movement direction is horizontal, which is different from the movement of the board surfaces 210a to 210d that move in the vertical direction. Therefore, the movement detection unit of the first modification determines that the optical flow 262s is not the movement of the board surfaces 210a and 210b.

[0137] At time t3, the movement of the plate surfaces 210a and 210b has ended, so the handwritten portion detection unit of Modification 1 again detects the handwritten portion 250 on all of the plate surfaces 210a to 210d, thereby obtaining the detection result of the handwritten portion 250 at time t3.

[0138] As described above, detection of the movement of the plate surfaces 210a to 210d using optical flow also provides detection results of the written portion 250 similar to those in the first embodiment described above.

[0139] According to the image processing device of the first modification, the movement of the board surface 210 is detected by estimating an optical flow for the movement of the writing portion accompanying the movement of the board surface 210. This makes it possible to easily detect the movement of the board surface 210 without, for example, needing a huge amount of data analysis. Furthermore, by using the optical flow, it is possible to detect even small movements of less than one pixel and slow movements. Furthermore, it is possible to detect the movement of the board surface even when the frame of the board surface does not appear as a clear edge on the image and when the board surface is curved.

[0140] According to the image processing device of the first modification, the optical flows are grouped, and the movement of the writer 300 and the movement of the board surface 210 are distinguished based on the size and direction of movement of the grouped area. The size and direction of movement of the board surface 210 can be determined based on the control information input from the input / output device 30, and supplemental information including information on the direction, number, configuration, arrangement, color, etc. of the movement of the board surface 210. This makes it possible to suppress erroneous detection.

[0141] (Variation 2) Next, an image processing device according to a second modification of the first embodiment will be described with reference to Fig. 9. In the image processing device according to the second modification, the method of detecting the movement of the board surfaces 210a to 210d is different from that of the first embodiment described above.

[0142] The movement detection unit of the image processing device of the second modification detects the movement of the display surfaces 210a to 210d by generic object recognition in a predetermined frame constituting the video data. In generic object recognition, for example, an object included in a video is recognized by a generic name, and the position of the recognized object is tracked to detect the movement of the object. In order to recognize an object by a generic name, for example, a method such as machine learning is used.

[0143] When detecting the movement of the board surfaces 210a-210d using generic object recognition, the board surfaces 210a-210d that can be moved individually are recognized as generic names such as "whiteboard" and their positions are tracked based on the changes in their positions between frames. However, the movement detection unit of the second modification may detect the movement of the "whiteboard" by tracking the center of gravity, that is, the center of gravity of the board surfaces 210a-210d, instead of the position of the object recognized as the "whiteboard."

[0144] At this time, it is preferable to have the system learn about things that may be included in the video data other than the surfaces 210a to 210d, for example by having the system recognize the transcriber 300 as a general name such as "person."

[0145] FIG. 9 is a diagram showing the results of detection of movement of the surfaces 210a to 210d by the image processing device according to the second modification of the first embodiment of the present disclosure, together with moving image data.

[0146] 9 shows frames of video data at times t1, t2, and t3, and the results of detecting movement of the board surfaces 210a to 210d in those frames, as well as the results of detecting handwritten portions for reference. Note that the video data at times t1, t2, and t3 is the same as the video data at times t1, t2, and t3 in FIG. 4 and FIG. 6 described above.

[0147] 9, the movement detection unit of the second modification recognizes each of the board surfaces 210a-210d included in the video data as a "whiteboard" 263, and similarly recognizes the scribe 300 included in the video data as a "person" 263s. After this, the movement detection unit of the second modification starts tracking the positions of these "whiteboards" 263 and "person" 263s.

[0148] At time t1, no movement of board surfaces 210a-210d recognized as "whiteboard" 263 has been detected, so the handwritten portion detection unit of Modification 2 detects handwritten portions 250 on all board surfaces 210a-210d. This provides a detection result of handwritten portions 250 at time t1.

[0149] At time t2 in FIG. 9, when the board surfaces 210a, 210b arranged above and below are moving by rising and falling, the movement detection unit of variant example 2 determines that movement of board surfaces 210a, 210b has occurred due to a change in the position of "whiteboard" 263 corresponding to board surfaces 210a, 210b, etc.

[0150] Since the movement of the board surfaces 210a and 210b has been detected, the handwriting portion detection unit of the second modification deletes the detection result of the handwriting portion 250 on the board surfaces 210a and 210b, and holds the detection result of the handwriting portion 250 on the stationary board surfaces 210c and 210d. This provides the detection result of the handwriting portion 250 at time t2.

[0151] 9, the video data includes board surfaces 210a-210d which are all again stationary, and scribe 300 moving in front of board surfaces 210a-210d. Therefore, the movement detection unit of Modification 2 determines that movement of "person" 263s, which is scribe 300, has occurred and that movement of "whiteboard" 263, which is board surfaces 210a-210d, has ended.

[0152] Since the movement of the plate surfaces 210a and 210b has been completed, the handwritten portion detection unit of the second modified example again detects the handwritten portion 250 on all of the plate surfaces 210a to 210d. As a result, the detection result of the handwritten portion 250 at the time t3 is obtained.

[0153] As described above, detection of the movement of the plate surfaces 210a to 210d using generic object recognition also provides detection results for the handwritten portion 250 similar to those in the first embodiment described above.

[0154] According to the image processing device of the second modification, the board surfaces 210a to 210d are recognized by general object recognition, and the positions or centers of gravity of the board surfaces 210a to 210d are tracked to detect the movement of the board surfaces 210a to 210d. This makes it possible to detect the movement of the board surfaces 210a to 210d with high accuracy.

[0155] According to the image processing device of the second modification, the movement of the writer 300 is also recognized by general object recognition, and thus the movement of the writer 300 and the movement of the board surfaces 210a to 210d are distinguished. This makes it possible to suppress erroneous detection.

[0156] (Variation 3) Next, a description will be given of an image processing device according to Modification 3 of Embodiment 1. The image processing device according to Modification 3 differs from the above-described embodiment 1 in that it has a function of tracking a moving plate surface.

[0157] The movement detection unit of the image processing device of the third modified example detects the movement of a given board surface by, as described above, detecting straight lines of the board surface frame, detecting the optical flow of the handwritten portion on the board surface, or general object recognition of the board surface.

[0158] Furthermore, the movement detection section of the third modification changes the angle of view of the imaging device 20 by a control signal for controlling the imaging device 20 when the plate surface whose movement has been detected continues to move outside the frame of the moving image data.

[0159] More specifically, the movement detection unit of the third modification causes the imaging device 20 to capture an image with a wider angle of view. Such control is possible when the imaging device 20 has, for example, an optical or electronic zoom-up function and a zoom-back function. As described above, by widening the angle of view of the imaging device 20, a wider range can be captured, and a plate surface that has fallen out of the frame of the video image data can be brought back into the frame.

[0160] Alternatively, the movement detection unit of the third modification causes the imaging device 20 to move the angle of view in at least one of the horizontal and vertical directions. Such control is possible when the imaging device 20 has, for example, a function of mechanically panning and tilting. The direction of the movement (pan / tilt) of the angle of view can be determined based on the movement direction of the board surface detected by the movement detection unit of the third modification just before the board surface is cut off from the frame. As described above, by moving the angle of view of the imaging device 20, the angle of view can be directed in the direction of the movement of the board surface, and the board surface that has fallen out of the frame of the video data can be brought back into the frame.

[0161] Alternatively, the movement detection unit of the third modification causes the imaging device 20 to capture an image at a wider angle of view while moving the angle of view in at least one of the horizontal and vertical directions. This makes it possible to bring the board surface that has fallen out of the frame of the video data back into the frame, thereby enabling the board surface to be captured at a more appropriate angle of view.

[0162] The configuration of the movement detection unit of variant example 3 is preferably applied to cases where, for example, a written portion on a specific board surface 210 of the above-mentioned whiteboard 200 is enlarged and displayed, making it difficult to fit the entire whiteboard 200 within the frame.

[0163] The configuration of the movement detection unit of the third modified example is preferably applied to cases where the direction and range of movement are irregular and relatively wide, such as a whiteboard with casters.

[0164] According to the image processing device of the third modification, when the board surface moves out of the frame of the video data, the angle of view of the imaging device 20 is changed. This makes it possible to keep the board surface within the frame of the video data, and to continue detecting the handwritten portion on the board surface.

[0165] [Embodiment 2] A second embodiment will be described with reference to the drawings. In the second embodiment, as an example of a use of the detection result of the handwritten portion obtained as in the first embodiment or the first to third modifications, an image processing device having a function of superimposing the detected handwritten portion on video image data captured by the imaging device 20 will be described. Such a superimposing process improves the visibility of the handwritten portion.

[0166] (Example of configuration of image processing device) FIG. 10 is a block diagram showing an example of a functional configuration of an image processing device 10a according to the second embodiment of the present disclosure.

[0167] 10, the image processing device 10a is connected to one or more imaging devices 20, an input / output device 30, and a recording device 40, and constitutes a part of an image processing system 2 of the second embodiment. The image processing device 10a of the second embodiment is also a computer equipped with, for example, a CPU, a ROM, a RAM, and an I / O port.

[0168] In addition, the image processing device 10 of the second embodiment includes a video input unit 11, a control information input unit 12, a movement detection unit 13, a handwritten portion detection unit 14, and an output unit 16, which are functional configurations similar to those of the first embodiment described above, as well as a data generation unit 15a.

[0169] However, the image processing device 10a may be provided with a movement detection unit according to variant 1 or variant 2 of the above-mentioned embodiment 1 that detects movement of the board surfaces 210a to 210d by optical flow or general object recognition, instead of the movement detection unit 13 that detects movement of the board surfaces 210a to 210d by straight line detection.

[0170] The data generating unit 15a generates handwriting data based on the handwriting portion detection result obtained from the handwriting portion detection unit 14. The data generating unit 15a also generates composite video data by superimposing the generated handwriting data on the video data received from the video input unit 11. The data generating unit 15a refers to the control information received from the control information input unit 12 throughout this series of processes.

[0171] The output unit 16 outputs the composite video data generated by the data generation unit 15a to one or more input / output devices 30 and one or more recording devices 40 connected to the image processing device 10a. This allows the user to view the composite video data on the input / output device 30.

[0172] (Example of data generation behavior) Next, the operation of generating handwritten data and composite video data by the image processing device 10a according to the second embodiment will be described with reference to FIG.

[0173] 11 is a diagram showing handwriting data and composite video data generated by the image processing device 10a according to the second embodiment of the present disclosure together with video data. FIG 11 shows frames of video data at times t1, t2, and t3, as well as handwriting data and composite video data generated for the frames.

[0174] The video data at times t1, t2, and t3 are the same as the video data at times t1, t2, and t3 in Figures 4 and 6 of the above-mentioned embodiment 1, and it is assumed that the movement detection results of the board surfaces 210a-210d and the generation results of the handwriting data are the same as those in Figures 4 and 6 of the above-mentioned embodiment 1.

[0175] 11, the data generating unit 15a of the image processing device 10a extracts the detection result of the handwritten portion 250 by the handwritten portion detection unit 14 and generates handwritten data. At time t1, no movement has occurred on any of the board surfaces 210a to 210d, so the data generating unit 15a generates handwritten data including the handwritten portion 250 on all of the board surfaces 210a to 210d. The data generating unit 15a also superimposes the generated handwritten data on the video data at time t1 to generate composite video data.

[0176] The moving image data at time t1 includes stationary board surfaces 210a-210d and scribe 300 standing in front of board surfaces 210a-210d. Furthermore, the handwriting data at time t1 includes handwritten portions 250 written on each of board surfaces 210a-210d as described above. This handwriting data is superimposed on the moving image data so as to overlap with the actual handwritten portions captured in the moving image data.

[0177] As a result, in the synthetic video data at time t1, the handwritten portion hidden by the handwritten portion 250 is superimposed on the image of the handwritten portion 300 while maintaining the positional relationship between the handwritten portion 250 and the handwritten portion, thereby improving the visibility of the handwritten portion 250.

[0178] Similarly, at time t2 in FIG. 11, the data generating unit 15a extracts the detection result of the handwritten portion 250 by the handwritten portion detecting unit 14 and generates handwritten data.

[0179] At time t2, movement occurs on the board surfaces 210a and 210b arranged above and below. Therefore, the data generating unit 15a extracts data of each pixel value of the handwriting portion 250 held by the handwriting portion detecting unit 14 from the areas corresponding to the board surfaces 210c and 210d where no movement was detected, and reflects the extracted data in the handwriting data to be generated.

[0180] Furthermore, since the handwritten portion 250 has been deleted by the handwritten portion detection unit 14 from the areas corresponding to the board surfaces 210a and 210b where the movement was detected, the data generation unit 15a reflects this state as it is in the handwritten data.

[0181] As a result, the handwriting data at time t2 does not have handwriting portion 250 in the left region corresponding to board surfaces 210a and 210b, but has handwriting portion 250 in the right region corresponding to board surfaces 210c and 210d. This state of the handwriting data is maintained until the movement of board surfaces 210a and 210b is completed.

[0182] The data generating unit 15a superimposes the handwritten data thus generated on the moving image data at time t2 to generate a composite moving image data.

[0183] The video data at time t2 includes the moving board surfaces 210a and 210b, the stationary board surfaces 210c and 210d, and the scribe 300 standing in front of these board surfaces 210a to 210d. As described above, the handwriting data at time t2 does not include the handwriting portion written on the moving board surfaces 210a and 210b, but only includes the handwriting portion 250 written on the board surfaces 210c and 210d.

[0184] Therefore, by superimposing the handwriting data at time t2 on the video data at time t2, the handwriting portion is not superimposed on the board surfaces 210a and 210b in the composite video data at time t2, and the handwriting portion reflected in the video data is displayed as is. Therefore, the handwriting portion behind the scribe 300 standing in front of the board surfaces 210a and 210b remains hidden by the scribe 300.

[0185] On the other hand, on the surfaces 210c and 210d, a handwritten portion 250 included in the handwritten data is also superimposed on the image of the writer 300.

[0186] 11, data generation unit 15a also generates handwriting data by extracting the detection result of handwriting portion 250 by handwriting portion detection unit 14. At time t3, since the raising and lowering of board surfaces 210a, 210b has finished, data generation unit 15a again generates handwriting data including handwriting portions 250 on all board surfaces 210a to 210d. Data generation unit 15a also superimposes the generated handwriting data on the video data at time t3 to generate composite video data.

[0187] The moving image data at time t3 includes the board surfaces 210a-210d, all of which are again in a still state, and the scribe 300 standing in front of these board surfaces 210a-210d. Also, the handwriting data at time t3 includes the handwriting portions 250 of all of the board surfaces 210a-210d, as described above.

[0188] Therefore, by superimposing the handwriting data at time t3 on the moving image data at time t3, the handwriting portion 250 is superimposed on all of the plate surfaces 210a to 210d in the composite moving image data at time t3. As a result, the handwriting portion hidden by the writer 300 is superimposed on the image of the writer 300, improving the visibility of the handwriting portion 250.

[0189] (Example of processing by image processing device) Next, an example of image processing by the image processing device 10a according to the second embodiment will be described with reference to Fig. 12. Fig. 12 is a flow diagram showing an example of a procedure of image processing by the image processing device 10a according to the second embodiment of the present disclosure.

[0190] In the flow diagram of FIG. 12, the processes in steps S101 to S114 are the same as the processes in steps S101 to S114 in FIG. 8 of the above-mentioned first embodiment.

[0191] That is, when the movement detection unit 13 detects the movement of the board surfaces 210a-210d (step S101), but no movement is detected for all or part of the board surfaces 210 (step S102 or step S103: No), and there is no reference frame because it is the very beginning of the processing by the image processing device 10a before the first reference frame is registered, or it is immediately after the movement of the board surfaces 210a-210d occurs in the previous frame and the reference frame is deleted (step S106: No), the handwriting detection unit 14 registers the frame of the video image data at that point as the reference frame (step S114). As a result, the processing returns to the beginning again in the next frame, for example.

[0192] Also, in the above case (step S102 or step S103: No), if there is a reference frame (step S106: Yes), the handwriting part detection unit 14 detects the handwriting part, and updates the reference frame by reflecting the results of retaining and deleting each collection part determined to be the handwriting part in the frame to be processed (steps S107 to S113).

[0193] The data generating unit 15a generates handwritten data based on the processes in steps S107 to S113 performed by the handwritten portion detecting unit 14 (step S115).

[0194] The data generating unit 15a superimposes the generated handwriting data on the currently processed frame of the video data (step S116), whereby the process returns to the beginning again, for example, in the next frame.

[0195] Furthermore, if movement of any of the board surfaces 210a to 210d is detected (step S102: Yes) and the area to be processed next corresponds to the board surface 210 where the movement has currently occurred (step S103: Yes), the writing portion detection unit 14 deletes the writing portion that was held in the area corresponding to the board surface 210 where the movement has currently occurred in the previous frame and the reference frame for the board surface 210 where the movement has currently occurred (steps S104 to S105).

[0196] Based on the processing of steps S104 to S105 by the handwriting portion detection unit 14 and the processing of steps S107 to S113, the composite moving image generation unit 15 generates handwriting data that does not have a handwriting portion in an area corresponding to the board surface 210 where movement has occurred, and that has a handwriting portion in an area corresponding to the board surface 210 where no movement has occurred (step S115).

[0197] The composite video generating unit 15 superimposes the generated handwritten data on the current frame of the video data to be processed (step S116).

[0198] With the above, the moving image processing by the image processing device 10a of the second embodiment is completed.

[0199] (Overview) According to the image processing device 10a of the second embodiment, handwritten data is generated based on the detection result of the handwritten portion, and the handwritten data is superimposed on the video data to generate composite video data. This makes it possible to improve the visibility of the handwritten portion when viewing the recorded contents such as lecture notes and meeting notes.

[0200] [Embodiment 3] A third embodiment will be described with reference to the drawings. In the third embodiment, an image processing device having a function of generating a handwriting log will be described as another example of use of the detection result of the handwriting portion obtained as in the first embodiment or the first to third modifications described above.

[0201] (Example of configuration of image processing device) FIG. 13 is a block diagram showing an example of a functional configuration of an image processing device 10b according to the third embodiment of the present disclosure.

[0202] 13, the image processing device 10b is connected to one or more imaging devices 20, an input / output device 30, and a recording device 40, and is configured as a part of the image processing system 3 of the third embodiment. The image processing device 10b of the third embodiment is also a computer including, for example, a CPU, a ROM, a RAM, and an I / O port.

[0203] In addition, the image processing device 10b includes a video input unit 11, a control information input unit 12, a movement detection unit 13, a handwritten portion detection unit 14, and an output unit 16, which have the same functional configuration as the image processing device 10 of the above-mentioned embodiment 1, as well as a handwritten log generation unit 15b.

[0204] However, the image processing device 10b may be provided with a movement detection unit according to variant 1 or variant 2 of the above-mentioned embodiment 1 that detects movement of the board surfaces 210a to 210d by optical flow or general object recognition, instead of the movement detection unit 13 that detects movement of the board surfaces 210a to 210d by straight line detection.

[0205] The writing log generating unit 15b acquires the movement detection result of the touch screen 210a-210d from the movement detecting unit 13, and at the timing when the movement of at least any of the touch screens 210a-210d is detected, acquires the detection result of the handwritten portion by the handwritten portion detecting unit 14 in the immediately preceding frame, and stores it as a writing log. The writing log generating unit 15b adds the writing log thus generated to the video data acquired from the video input unit 11, and passes it to the output unit 16. This allows the user to view the video data to which the writing log has been added, on the input / output device 30.

[0206] (Example of writing log generation behavior) Next, the operation of generating a writing log by the image processing device 10b according to the third embodiment will be described with reference to FIG.

[0207] Fig. 14 is a diagram showing a result of generating a writing log by the image processing device 10b according to the third embodiment of the present disclosure together with video data. Fig. 14 shows frames of video data at times t1, t2, t3, t4, and t5, and a result of generating a writing log at a specific timing among times t1, t2, t3, t4, and t5.

[0208] At time t1 in FIG. 14, the scribe 300 is writing down explanations or the like using only the tablet surface 210a among the tablet surfaces 210a to 210d.

[0209] 14, the writer 300 completes writing on the tablet surface 210a, and switches the tablet surfaces 210a and 210b by raising and lowering them. The movement detection unit 13 detects the movement of the tablet surfaces 210a and 210b.

[0210] When movement of the plate surfaces 210a, 210b is detected, the writing log generating unit 15b acquires the detection result of the writing portion 250 in the immediately preceding frame from the writing portion detecting unit 14. In the frame immediately preceding the frame in which the movement of the plate surfaces 210a, 210b is detected, no movement has occurred on any of the plate surfaces 210a to 210d, so the writing portion detecting unit 14 detects and holds the writing portion 250 written on the plate surface 210a.

[0211] The writing log generating unit 15b generates a writing log 170 including the writing portion 250 written on the tablet surface 210a from the detection result of the writing portion 250 acquired from the writing portion detecting unit 14, and links the writing log 170 to the frame at time t2.

[0212] At time t3 in FIG. 14, the scribe 300 is writing down explanations or the like using the tablet surface 210b.

[0213] At time t4 in FIG. 14, the scribe 300 has completed writing on the tablet surface 210b, and is now exclusively using the tablet surface 210c to write explanations and the like.

[0214] 14, the writer 300 completes writing on the tablet surface 210c, and switches the tablet surfaces 210c and 210d by raising and lowering them. The movement detection unit 13 detects the movement of the tablet surfaces 210c and 210d.

[0215] When movement of the plate surfaces 210c, 210d is detected, the handwriting log generating unit 15b acquires the detection result of the handwriting portion 250 in the immediately preceding frame from the handwriting portion detecting unit 14. Since no movement has occurred on any of the plate surfaces 210a to 210d in the frame immediately preceding the detection of the movement of the plate surfaces 210c, 210d, the handwriting portion detecting unit 14 detects and holds the handwriting portions 250 written on the plate surfaces 210a, 210b, and 210c, respectively.

[0216] The writing log generating unit 15b generates a writing log 170 including the writing portions 250 written on the respective board surfaces 210a, 210b, and 210c from the detection result of the writing portions 250 acquired from the writing portion detecting unit 14, and links the writing log 170 to the frame at time t5.

[0217] As described above, the handwriting log generating unit 15b generates the handwriting log 170 at times t2 and t5, triggered by the movement detecting unit 13 detecting the movement of each of the board surfaces 210a to 210d.

[0218] Here, it is considered that the timing when the scribe 300 or the like moves a given board surface 210 is often the timing when the writing on that board surface 210 is completed. Therefore, the writing log generated at the timing of the movement of each of the board surfaces 210a to 210d can be used as a record of, for example, lecture records, meeting records, etc., in which the contents of the writing on the board surfaces 210a to 210d are recorded.

[0219] (Example of processing by image processing device) Next, an example of image processing by the image processing device 10b according to the third embodiment will be described with reference to Fig. 15. Fig. 15 is a flow diagram showing an example of a procedure of image processing by the image processing device 10b according to the third embodiment of the present disclosure.

[0220] In the flow diagram of FIG. 15, except for the newly inserted step S102b, the processing in steps S101 to S114 is the same as the processing in steps S101 to S114 in FIG. 8 of the above-mentioned first embodiment.

[0221] That is, when the movement detection unit 13 detects the movement of the board surfaces 210a-210d (step S101), but no movement is detected for all or part of the board surfaces 210 (step S102 or step S103: No), and there is no reference frame because it is the very beginning of the processing by the image processing device 10a before the first reference frame is registered, or it is immediately after the movement of the board surfaces 210a-210d occurs in the previous frame and the reference frame is deleted (step S106: No), the handwriting detection unit 14 registers the frame of the video image data at that point as the reference frame (step S114). As a result, the processing returns to the beginning again in the next frame, for example.

[0222] In the above case (step S102 or S103: No), if there is a reference frame (step S106: Yes), the handwriting detection unit 14 detects the handwriting and updates the reference frame by reflecting the results of keeping and deleting each group part determined to be the handwriting in the frame to be processed (steps S107 to S113). This causes the process to return to the beginning again in the next frame, for example.

[0223] Furthermore, if movement of any of the board surfaces 210a to 210d is detected (step S102: Yes) and the area to be processed next corresponds to the board surface 210 where the movement has currently occurred (step S103: Yes), the writing portion detection unit 14 deletes the writing portion that was held in the area corresponding to the board surface 210 where the movement has currently occurred in the previous frame and the reference frame for the board surface 210 where the movement has currently occurred (steps S104 to S105).

[0224] On the other hand, when the movement of any of the plate surfaces 210a to 210d is detected (step S102: Yes), the writing log generating unit 15b generates a writing log (step S102b).

[0225] Specifically, the writing log generating unit 15b acquires a detection result of the writing portion in the frame immediately before the movement of any of the touch screens 210a to 210d is detected. The writing log generating unit 15b also extracts the writing portion from the acquired detection result of the writing portion to generate a writing log. The writing log generating unit 15b also stores the generated writing log by linking it to the frame at that time.

[0226] With the above, the moving image processing by the image processing device 10b of the third embodiment is completed.

[0227] (Overview) According to the image processing device 10b of the third embodiment, a handwriting log is generated when the movement detection unit 13 detects a predetermined movement of the board surface 210. This makes it possible to keep a record of the handwriting contents on the board surfaces 210a to 210d.

[0228] In the above-described embodiment 3, the image processing device 10b has a function of generating a handwriting log, but an image processing device having a handwriting log generating function may have a function of generating composite video data by superimposing a handwritten portion on video data, like the image processing device 10a in the above-described embodiment 2. In this case, the generated handwriting log is saved, for example, linked to a corresponding frame of the composite video data.

[0229] (Variation 1) Next, a description will be given of an image processing device according to Modification 1 of Embodiment 3. The image processing device according to Modification 1 differs from the above-described embodiment 3 in that it determines whether or not to generate a handwriting log depending on the moving speed of the keyboard surfaces 210a to 210d.

[0230] The writing log generating unit of the image processing device of the first modification acquires information on the movement speed of board surface 210 where movement has been detected, together with the movement detection result of board surfaces 210a-210d, from the movement detection unit of the first modification. Furthermore, the writing log generating unit of the first modification determines whether or not to generate a writing log, based on the information on the movement speed of board surface 210.

[0231] For example, when the moving speed of the board surface 210 is less than a predetermined value, the writing log generating unit of the first modified example may be configured not to generate a writing log. When the moving speed of the board surface 210 is equal to or greater than a predetermined value, the writing log generating unit of the first modified example may be configured to generate a writing log.

[0232] The user performs the above-mentioned settings in advance on the image processing device of the modified example 1. When the transcriber 300 or the like moves the predetermined keyboard surface 210 during recording of the moving image data, the transcriber 300 can move the keyboard surface 210 slowly at a timing when the handwriting log is deemed unnecessary. Also, the transcriber 300 can move the keyboard surface 210 quickly at a timing when the handwriting log is deemed necessary.

[0233] According to the image processing device of the first modification, when the movement detection unit of the first modification detects the movement of the board surface at a predetermined speed or more, a writing log is generated. This allows the writing log to be generated at a timing desired by the transcriber 300, for example, and allows a more accurate record to be kept.

[0234] (Variation 2) Next, a description will be given of an image processing device according to Modification 2 of Embodiment 3. The image processing device according to Modification 2 differs from the above-described Embodiment 3 in that it determines whether or not to generate a handwriting log depending on the progress of writing on the board surface 210.

[0235] When a specific movement of the board surface 210 is detected, the handwriting log generation unit of the image processing device of variant example 2 obtains the detection results of the handwriting portion in the immediately preceding frame and multiple frames prior to the immediately preceding frame from the handwriting portion detection unit of variant example 2.

[0236] Furthermore, the writing log generating unit of the second modification determines the direction of writing of the writing part on the board surface 210 whose movement has been detected, from the detection results of the writing part in a plurality of frames including the frame immediately before the detection of the movement of the board surface 210. Possible writing directions include a case in which the writing part progresses from top to bottom when the writing part is written horizontally, or a case in which the writing part progresses from right to left when the writing part is written vertically.

[0237] Furthermore, the writing log generating unit of the second modification determines whether or not the writing portion detected in the frame immediately prior to the detection of movement of the board surface 210 reaches from one end to the other end of the board surface 210 in the writing direction.

[0238] That is, for example, when the writing direction is from top to bottom, it is determined whether or not the writing portion reaches from the top end to the bottom end of the board surface 210. Also, for example, when the writing direction is from right to left, it is determined whether or not the writing portion reaches from the right end to the left end of the board surface 210.

[0239] Whether or not the writing portion reaches from one end of the plate surface 210 to the other end can be determined based on, for example, information on the vertical or horizontal width of the plate surface 210 given by the user.

[0240] The writing log generating unit of the modified example 2 generates a writing log when the writing portion reaches from one end to the other end of the board surface 210. The writing log generating unit of the modified example 2 does not generate a writing log when the writing portion does not reach from one end to the other end of the board surface 210.

[0241] Here, if the writing portion has reached from one end to the other end of the tablet surface 210, it can be considered that writing on that tablet surface has been completed. Also, if the writing portion has not reached from one end to the other end of the tablet surface 210, the writer 300 can still add writing to that tablet surface 210, and may, for example, pull back the tablet surface 210 that was once moved upward to add to the writing portion, etc.

[0242] Therefore, by adding the condition for generating a writing log that the writing portion reaches from one end of the board surface 210 to the other end, the writing log can be generated more accurately at the timing when the writing on the board surface 210 is completed.

[0243] According to the image processing device of the second modification, when the movement detection unit of the second modification detects the movement of the board surface 210 and the writing portion reaches from one end of the board surface 210 to the other end, a writing log is generated. This makes it possible to more reliably determine the timing at which writing on a given board surface 210 is completed, and generate a writing log at a more appropriate timing. Also, even if additional writing is made on the board surface 210 that has been pulled back midway, it is possible to prevent a writing log from being generated in duplicate for similar writing content.

[0244] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0245] [Other embodiments] The image processing device of the above-mentioned first to third embodiments and their first to third modifications is, for example, a computer equipped with a CPU, a ROM, a RAM, and an I / O port. However, the image processing device may have a configuration equipped with an ASIC (Application Specific Integrated Circuit) dedicated for the application described in the above-mentioned first to third embodiments and their first to third modifications.

[0246] The image processing devices of the above-mentioned first to third embodiments and their first to third modifications have an independent device configuration separate from the imaging device 20. However, the functions of the image processing device may be incorporated in the imaging device.

[0247] The image processing device of the above-mentioned embodiments 1 to 3 and their modified examples 1 to 3 is provided with a ROM in which a program related to image processing is stored. Such a program can be provided by being stored in a recording medium or the like so as to be readable by a computer. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. The above-mentioned program may also be distributed, for example, via a network without using a recording medium. In this case, the program does not necessarily have to be installed in the image processing device.

[0248] The present technology can also be configured as follows. (1) a movement detection unit that detects movement of the writing medium based on moving image data of a movable writing medium and a writer writing on the writing medium; a writing portion detection unit that detects a writing portion on the writing medium, The handwriting portion detection unit When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed. Image processing device. (2) The movement detection unit detecting a straight portion of a frame of the writing medium and detecting the movement of the straight portion, thereby detecting the movement of the writing medium; The image processing device according to (1) above. (3) The movement detection unit Distinguishing between a straight line included in the writing portion and the frame of the writing medium based on the length of the straight line portion; The image processing device according to (2) above. (4) The movement detection unit detecting the movement of the writing medium by estimating an optical flow that represents, as a vector, the movement of the writing portion associated with the movement of the writing medium; The image processing device according to (1) above. (5) The movement detection unit grouping the optical flows and distinguishing between the movement of the writer and the movement of the writing medium based on the size and direction of the movement of the grouped areas; The image processing device according to (4) above. (6) The movement detection unit Recognizing the writing medium by generic object recognition that recognizes objects by generic names, and detecting the movement of the writing medium by tracking a position or a center of gravity of the writing medium; The image processing device according to (1) above. (7) The movement detection unit and recognizing the writer by the generic object recognition to distinguish between the movement of the writer and the movement of the writing medium. The image processing device according to (6) above. (8) deleting the detection result of the writing portion on the writing medium when the movement of the writing medium is detected; An image processing device according to any one of (1) to (7) above. (9) the writing medium is one of a plurality of individually movable writing media included in a writing medium group; The movement detection unit Detecting the movement of each of the writing media; The handwriting portion detection unit when the movement of at least one of the plurality of writing media is detected, deleting the detection result of the writing portion on the writing medium where the movement is detected, and retaining the detection result of the writing portion on the writing medium where the movement is not detected. The image processing device according to (8) above. (10) The movement detection unit changing an angle of view of an imaging device that captures the moving image data when the writing medium moves out of a frame of the moving image data; An image processing device according to any one of (1) to (9) above. (11) The movement detection unit When changing the angle of view of the imaging device, the imaging device is caused to capture an image with a wider angle of view. The image processing device according to (10) above. (12) A data generating unit that generates handwriting data based on the detection result of the handwriting portion. An image processing device according to any one of (1) to (9) above. (13) The data generation unit generating composite video data by superimposing the handwriting data on the video data; The image processing device according to (12) above. (14) A writing log generating unit that generates a writing log based on a detection result of the movement detecting unit. An image processing device according to any one of (1) to (13) above. (15) The writing log generation unit includes: When the movement detection unit detects the movement of the writing medium, the writing log is generated. The image processing device according to (14) above. (16) The writing log generation unit includes: generating the writing log when the movement detection unit detects the movement of the writing medium at a predetermined speed or faster; The image processing device according to (14) above. (17) The writing log generation unit includes: generating the writing log when the movement detection unit detects the movement of the writing medium and the writing portion reaches from one end of the writing medium to the other end of the writing medium; The image processing device according to (13) above. (18) The movement detection unit detecting the movement of the writing medium based on information about the writing medium input by a user; 18. An image processing device according to any one of claims 1 to 17. (19) Detecting movement of the writing medium based on video image data capturing a movable writing medium and a writer writing on the writing medium; Detecting a portion of writing on the writing medium; In detecting the handwritten portion, When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed. Image processing methods. (20) On the computer, A process of detecting movement of the writing medium based on video image data of a movable writing medium and a writer writing on the writing medium; and detecting a portion of writing on the writing medium. In the process of detecting the handwritten portion, When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed. program.

[0249] Although some embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as the scope and spirit of the invention. [Explanation of symbols]

[0250] 1,2 Image processing system 10, 10a, 10b Image processing device 13 Movement detection unit 14 Writing part detection unit 15a Data Generation Section 15b Writing log generation unit 20 Imaging device 30 Input / Output Devices 40 Recording Device

Claims

1. a movement detection unit that detects movement of the writing medium based on moving image data of a movable writing medium and a writer writing on the writing medium; a writing portion detection unit that detects a writing portion on the writing medium, The handwriting portion detection unit When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed; deleting the detection result of the writing portion on the writing medium when the movement of the writing medium is detected; Image processing device.

2. The writing medium is one of a plurality of individually movable writing media included in a writing medium group, The movement detection unit Detecting the movement of each of the writing media; The handwriting portion detection unit when the movement of at least one of the plurality of writing media is detected, deleting the detection result of the writing portion on the writing medium where the movement is detected, and retaining the detection result of the writing portion on the writing medium where the movement is not detected. The image processing device according to claim 1 .

3. A movement detection unit that detects movement of a movable writing medium based on video image data of a movable writing medium and a writer writing on the writing medium; a writing portion detection unit that detects a writing portion on the writing medium; a data generating unit that generates handwriting data based on the detection result of the handwriting portion, The handwriting portion detection unit When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed; The data generation unit generating composite video data by superimposing the handwriting data on the video data; Image processing device.

4. A movement detection unit that detects movement of a movable writing medium based on video image data of a movable writing medium and a writer writing on the writing medium; a writing portion detection unit that detects a writing portion on the writing medium; a writing log generating unit that generates a writing log based on a detection result of the movement detecting unit, The handwriting portion detection unit When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed. Image processing device.

5. The writing log generation unit When the movement detection unit detects the movement of the writing medium, the writing log is generated. The image processing device according to claim 4.

6. The writing log generation unit generating the writing log when the movement detection unit detects the movement of the writing medium at or above a predetermined speed; The image processing device according to claim 4.

7. The writing log generation unit generating the writing log when the movement detection unit detects the movement of the writing medium and the writing portion reaches from one end of the writing medium to the other end of the writing medium; The image processing device according to claim 4.

8. The movement detection unit detecting a straight portion of a frame of the writing medium and detecting the movement of the straight portion, thereby detecting the movement of the writing medium; The image processing device according to claim 1 .

9. The movement detection unit Distinguishing between a straight line included in the writing portion and the frame of the writing medium based on the length of the straight line portion; The image processing device according to claim 8.

10. The movement detection unit detecting the movement of the writing medium by estimating an optical flow that represents, in vector form, the movement of the writing portion that accompanies the movement of the writing medium; The image processing device according to claim 1 .

11. The movement detection unit grouping the optical flows and distinguishing between the movement of the writer and the movement of the writing medium based on the size and direction of the movement of the grouped areas; The image processing device according to claim 10.

12. The movement detection unit Recognizing the writing medium by generic object recognition that recognizes objects by generic names, and detecting the movement of the writing medium by tracking a position or a center of gravity of the writing medium; The image processing device according to claim 1 .

13. The movement detection unit and recognizing the writer by the generic object recognition to distinguish between the movement of the writer and the movement of the writing medium. The image processing device according to claim 12.

14. The movement detection unit changing an angle of view of an imaging device that captures the moving image data when the writing medium moves out of a frame of the moving image data; The image processing device according to any one of claims 1 to 13.

15. A computer comprising: Detecting movement of the writing medium based on video image data capturing a movable writing medium and a writer writing on the writing medium; Detecting a portion of writing on the writing medium; In detecting the handwritten portion, When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed; deleting the detection result of the writing portion on the writing medium when the movement of the writing medium is detected; Image processing methods.

16. A computer comprising: Detecting movement of the writing medium based on video image data capturing a movable writing medium and a writer writing on the writing medium; Detecting a portion of writing on the writing medium; generating handwriting data based on the detection result of the handwriting portion; In detecting the handwritten portion, When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed; In generating the handwritten data, generating composite video data by superimposing the handwriting data on the video data; Image processing methods.

17. A computer comprising: Detecting movement of the writing medium based on video image data capturing a movable writing medium and a writer writing on the writing medium; Detecting a portion of writing on the writing medium; generating a writing log based on the detection result of the movement of the writing medium; In detecting the handwritten portion, When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed. Image processing methods.

18. On the computer, A process of detecting movement of the writing medium based on video image data of a movable writing medium and a writer writing on the writing medium; and detecting a portion of writing on the writing medium. In the process of detecting the handwritten portion, When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed; executing a process of deleting the detection result of the writing portion on the writing medium when the movement of the writing medium is detected; program.

19. A computer comprising: A process of detecting movement of the writing medium based on video image data of a movable writing medium and a writer writing on the writing medium; detecting a portion of writing on the writing medium; and generating handwriting data based on the detection result of the handwriting portion. In the process of detecting the handwritten portion, When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed; In the process of generating the handwritten data, executing a process of generating composite video data by superimposing the handwriting data on the video data; program.

20. A computer comprising: A process of detecting movement of the writing medium based on video image data of a movable writing medium and a writer writing on the writing medium; detecting a portion of writing on the writing medium; and generating a writing log based on the detection result of the movement of the writing medium. In the process of detecting the handwritten portion, When the movement of the writing medium is detected, a process corresponding to the movement of the writing medium is executed. program.

Images