Image processing device, image processing method

Abstract

To provide a technique capable of outputting contents of a background hidden by a foreground even if the background is updated.SOLUTION: An image processing apparatus is configured to: identify, based on a difference between a corresponding area corresponding to a foreground area of a frame of interest in a background image of the frame of interest in a buffered frame group and a corresponding area corresponding to the foreground area in background images of frames after the frame of interest in the buffered frame group, a background image to be displayed among the background images of the frames; and output the background image to be displayed.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to image processing technology.

Background Art

[0002] In recent years, due to the development of Internet technology, in the field of education, the introduction of e-learning systems through real-time distribution and on-demand distribution of lectures has been progressing. In e-learning systems, there is some content such as displaying slides on a PC or digitizing blackboard writing on an electronic whiteboard, but there is also a lot of content that is just the filming of lectures on a general blackboard with a camera. Also, for lectures on a general blackboard, technologies for displaying the content of the blackboard writing in an easy-to-view manner have been disclosed.

[0003] For example, in the technology disclosed in Patent Document 1, vector data is created from the blackboard writing area extracted from the video, and the blackboard writing information is made easier to view by synthesizing and displaying it in separate windows.

[0004] Also, in the technology disclosed in Non-Patent Document 1, the already written characters and figures are extracted, and the blackboard writing information that has been blocked and cannot be seen by the instructor is made visible by overlaying and displaying them.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006]

Non-Patent Document 1

Summary of the Invention

[0007] However, these prior art technologies cannot display the content of the whiteboard in an easily visible manner when the writing is obscured by the instructor's head or body. When the instructor remains in place or writes on the whiteboard while speaking, it takes some time before the content of the whiteboard becomes visible. As a result, by the time the content of the whiteboard becomes visible, the instructor's lecture has already progressed, which hinders the viewer from understanding the lecture. This disclosure provides a technology that can output the content of the background even when the background, which is obscured by the foreground, is updated. [Means for solving the problem]

[0008] One aspect of this disclosure is a means for identifying a target background image from among the background images of a frame, based on the difference between a first region corresponding to the foreground region of a target frame in a background image corresponding to a target frame in a buffered set of frames, and a second region corresponding to the foreground region in the background images of frames later than the target frame in the buffered set of frames. The identified background image A semi-transparent composite frame is superimposed onto the aforementioned target frame. Output means and It is characterized by having the following features. [Effects of the Invention]

[0009] According to this disclosure, it is possible to provide a technology that can output the content of the background even if the background, which is obscured by the foreground, is updated. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram showing an example of the system configuration. [Figure 2] A block diagram showing an example of the system's hardware configuration. [Figure 3] A block diagram showing an example of the functional configuration of the image processing device 103. [Figure 4] A flowchart of the processing performed when the image processing device 103 acquires a frame captured by the camera 101. [Figure 5] A flowchart illustrating the process performed by the image processing device 103 to play back a frame from a group of frames buffered in main memory 202. [Figure 6] A diagram showing an example of a group of frames. [Figure 7] A diagram showing an example of a specific comparison operation in the comparison unit 305. [Figure 8] A diagram showing an example of a composite frame. [Figure 9] A flowchart illustrating the process performed by the image processing device 103 to play back a frame from a group of frames buffered in main memory 202. [Figure 10] A diagram showing an example of a composite frame. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the scope of this disclosure as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to this disclosure, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] [First Embodiment] In this embodiment, a system applied to an e-learning system will be described. In the system according to this embodiment, as shown in FIG. 1, the camera 101 captures an image of a lecturer 105 located in front of the blackboard 104 (the front side as viewed from the camera 101) giving a lecture using the blackboard 104. The camera 101 transmits each captured frame (captured image) to the image processing device 103 via a wired or wireless network 102. The image processing device 103 displays each frame received from the camera 101 via the network 102. At that time, when there is a difference in the area corresponding to the foreground in a subsequent frame after the frame to be played back with respect to the background hidden by the foreground (lecturer 105) in the frame to be played back, the image processing device 103 displays the subsequent frame.

[0013] Note that the configuration shown in FIG. 1 is an example, and the present system is not limited to such a configuration. For example, the number of cameras 101 is not limited to 1, and may be 2 or more. Further, the system may include devices such as a server device that controls the operation of the camera 101 and image transmission, and a device that relays data communication.

[0014] Next, an example of the hardware configuration of the system according to this embodiment will be described using the block diagram of FIG. 2. In this embodiment, for the sake of specific description, as shown in FIG. 2, it is assumed that the camera 101 and the image processing device 103 are connected to a LAN / WAN which is an example of the network 102. However, the network configuration between the camera 101 and the image processing device 103 is not limited to a specific network configuration.

[0015] First, the camera 101 will be described. As shown in FIG. 1, the camera 101 captures an image of a lecturer 105 giving a lecture using a blackboard 104. The camera 101 may be a device that captures a moving image of this scene, or it may be a device that captures this scene periodically or irregularly. In the former case, the camera 101 transmits each frame in the captured moving image to the image processing device 103 via the network 102. In the latter case, the camera 101 transmits the still images captured periodically or irregularly as frames to the image processing device 103 via the network 102.

[0016] Next, the image processing device 103 will be described. For the image processing device 103, computer devices such as a PC (personal computer), a smartphone, and a tablet terminal device are applicable.

[0017] The CPU 201 executes various processes using the computer programs and data stored in the main memory 202. As a result, the CPU 201 controls the operation of the entire image processing device 103 and executes or controls various processes described as being performed by the image processing device 103.

[0018] The main memory 202 has an area for storing computer programs and data loaded from the storage unit 203, and an area for storing (buffering) the frames received from the camera 101 via the communication unit 206. Further, the main memory 202 has a work area used when the CPU 201 executes various processes. Thus, the main memory 202 can appropriately provide various areas.

[0019] The memory unit 203 is a large-capacity information storage device such as a hard disk drive. The memory unit 203 stores the OS (operating system), computer programs and data for the CPU 201 to execute or control various processes described as being performed by the image processing device 103. The computer programs and data stored in the memory unit 203 are loaded into the main memory 202 as appropriate according to the control of the CPU 201 and become the target of processing by the CPU 201. For example, non-volatile memory such as a silicon disk can be applied to the memory unit 203.

[0020] The control unit 204 is a user interface consisting of a keyboard, mouse, buttons, mode dial, switches, levers, and a touch panel screen, which allows the user to input various instructions to the CPU 201 through user operation.

[0021] The display unit 205 is a device having an LCD screen or a touch panel screen, and can display the processing results of the CPU 201 as images, text, etc. If the display unit 205 has a touch panel screen, various instructions entered by the user through operation of the touch panel screen are notified to the CPU 201. The display unit 205 may also be a projection device such as a projector that projects images, text, etc.

[0022] The communication unit 206 is a device compliant with communication standards such as Ethernet and IEEE 802.11, and is a device for connecting the image processing device 103 to the network 102. The image processing device 103 can communicate data with the camera 101 on the network 102 via the communication unit 206.

[0023] The CPU 201, main memory 202, storage unit 203, operation unit 204, display unit 205, and communication unit 206 are all connected to the bus 207.

[0024] Figure 3 shows a block diagram illustrating an example of the functional configuration of the image processing apparatus 103 according to this embodiment. Each functional unit shown in Figure 3 is assumed to be implemented using software (computer programs). In the following description, the functional units in Figure 3 will be described as the main processing units; however, in reality, the functions of these functional units are realized by the CPU 201 executing the corresponding computer programs. Note that one or more of the functional units shown in Figure 3 may be implemented using hardware.

[0025] First, the processing performed by the image processing device 103 when it acquires a frame captured by the camera 101 will be explained according to the flowchart in Figure 4.

[0026] In step S401, the acquisition unit 301 acquires frames captured by the camera 101 and stores (buffers) the acquired frames in the main memory 202. The acquisition unit 301 may acquire the frames from the camera 101 via the network 102 and the communication unit 206, or it may acquire them from an external device such as a server that holds frames captured by the camera 101. In other words, the method of acquiring frames is not limited to a specific method. Furthermore, the acquisition unit 301 may acquire each frame obtained by decoding the encoded moving image, or it may acquire still images captured periodically or irregularly as frames.

[0027] In the following example, we will describe the case in which frames 601 to 604 are acquired in the order of frames 601, 602, 603, and 604, as shown in Figure 6. Frames 601, 602, 603, and 604 are frames captured at imaging times t1, t2, t3, and t4, respectively, with imaging time t1 being the earliest imaging time and imaging time t4 being the most recent imaging time. Frames 601, 602, 603, and 604 are frames capturing the scene in which the instructor 105 writes the letters "AB" on a blackboard 104 that is blank.

[0028] Next, in step S402, the extraction unit 302 extracts (identifies) the foreground region in the frame acquired in step S401. The foreground region is the region of the foreground (subject) that obscures the background in the frame. In this embodiment, the extraction unit 302 extracts the region of the instructor 105 that obscures the blackboard 104 as the foreground region. A well-known method for extracting the foreground region from a frame is the background subtraction method, which extracts the region with a difference (difference region) between a reference background image and the frame as the background region. The reference background image can be generated by methods such as calculating the time average for each pixel from continuously captured images, or by capturing an image in advance that does not contain a subject. The background image may also be an captured image that does not include the foreground. Furthermore, the foreground region may be extracted using a detector that detects specific regions such as a person's face or body, or the foreground region may be extracted based on phase difference information (defocus amount) at an arbitrary pixel using an image sensor capable of phase difference focus detection.

[0029] In this embodiment, the foreground region is extracted using the background subtraction method based on the background image of the previous frame (the previously acquired frame). If a background image for the previous frame does not exist, a previously captured background image is used. Furthermore, by calculating image features such as the histogram and shape of the foreground region in the frame and performing clustering, a foreground region with less noise can be extracted. However, the method of extracting the foreground region applicable to step S402 is not limited to a specific extraction method.

[0030] Next, in step S403, the generation unit 303 processes the foreground region of the frame acquired in step S401 to remove the foreground and generates a processed frame as a background image, and stores the generated background image in the main memory 202. For example, the generation unit 303 generates a processed frame as a background image in which the pixel values ​​of pixels in the foreground region of the frame acquired in step S401 are replaced with other pixel values. "Other pixel values" may be, for example, the average value of the pixel values ​​of pixels located around the foreground region in the frame acquired in step S401. Alternatively, "other pixel values" may be, for example, the average value of the pixel values ​​of pixels in a corresponding region corresponding to the foreground region in another image (for example, a background image in which a background without a subject has been previously captured). In this embodiment, a partial image of the foreground region in the frame acquired in step S401 is replaced with a partial image of the corresponding region corresponding to the foreground region in the previously acquired frame. However, there are various methods for generating a processed frame as a background image by removing the foreground from the frame acquired in step S401, and it is not limited to a specific method.

[0031] Thus, when the image processing device 103 acquires a frame, it generates a background image corresponding to that frame and stores it in the main memory 202. Therefore, for each frame buffered in the main memory 202, the image processing device 103 generates a corresponding background image and stores it in the main memory 202.

[0032] Next, the process performed by the image processing device 103 to play back the frame to be played back from the frame group buffered in the main memory 202 will be explained according to the flowchart in Figure 5. The process according to the flowchart in Figure 5 shows the process for playing back one frame. The CPU 201 enables the playback of multiple frames by repeatedly executing the process according to the flowchart in Figure 5.

[0033] In this embodiment, when the CPU 201 receives a frame via the network 102 and the communication unit 206, it stores (buffers) the received frame in the main memory 202. After a predetermined time has elapsed since buffering began, the CPU 201 displays on the display unit 205 that a frame playback instruction can be entered. Alternatively, the CPU 201 may display on the display unit 205 that a frame playback instruction can be entered when a predetermined number of frames have been stored in the main memory 202 since buffering began. When the CPU 201 detects that a user has entered a frame playback instruction by operating the operation unit 204, it starts processing according to the flowchart in Figure 5. Alternatively, the CPU 201 may start processing according to the flowchart in Figure 5 without receiving a frame playback instruction from the user after a predetermined time has elapsed since buffering began, or when a predetermined number of frames have been stored in the main memory 202 since buffering began.

[0034] In any case, in this embodiment, when playing back a frame stored in the main memory 202, the main memory 202 is assumed to have one or more frames stored in it that are later than the frame being played back (frames with a more recent imaging time than the frame being played back).

[0035] In step S501, the acquisition unit 301 acquires the frame to be played back from the buffered frame group in the main memory 202. The frame to be played back may be a frame specified by the user from the buffered frame group by operating the operation unit 204, or it may be a frame selected by the CPU 201 from the buffered frame group based on appropriate criteria. The frame to be played back may also be the oldest frame in the buffered frame group.

[0036] In step S502, the extraction unit 302 extracts the foreground region of the frame to be played back, acquired in step S501, as an occluding region (the region of the subject that occludes the background), in the same manner as in step S402 above.

[0037] In step S503, the acquisition unit 304 acquires a background image corresponding to the frame to be played back acquired in step S501 (a background image generated for the frame to be played back according to the flowchart in Figure 4) from the main memory 202.

[0038] In step S504, the comparison unit 305 determines whether there are any subsequent frames remaining in the buffered frame group that are later than the frame to be played back (frames with a newer imaging time than the frame to be played back) that have not been selected as the selected frame. The subsequent frames may be N (N is a natural number) frames in the buffered frame group that are later than the frame to be played back (N frames with a newer imaging time than the frame to be played back).

[0039] As a result of this judgment, if "there are any subsequent frames in the buffered frame group that are later than the frame to be played back but have not been selected as selected frames," the process proceeds to step S505. On the other hand, if "all subsequent frames in the buffered frame group that are later than the frame to be played back have been selected as selected frames," the process proceeds to step S508.

[0040] In step S505, the comparison unit 305 selects the latest subsequent frame with the earliest acquisition time from among the unselected subsequent frames that are later than the frame to be played back in the buffered frame group, and obtains a background image generated for the selected frame by processing according to the flowchart in Figure 4.

[0041] In step S506, the comparison unit 305 calculates the difference between the first corresponding region corresponding to the occluded region in the background image acquired in step S503 and the second corresponding region corresponding to the occluded region in the background image acquired in step S505. The comparison unit 305 calculates the difference between the first and second corresponding regions, for example, by summing the difference values ​​(absolute values) of pixel values ​​between pixels that are positionally corresponding in the first and second corresponding regions. The difference between the first and second corresponding regions may be a value obtained by another method, as long as it is a value that quantifies the magnitude of the difference between the first and second corresponding regions.

[0042] The comparison unit 305 then determines that there is a difference between the first corresponding area and the second corresponding area if the calculated difference is greater than or equal to a threshold, and proceeds to step S507. On the other hand, if the comparison unit 305 determines that there is no difference between the first corresponding area and the second corresponding area if the calculated difference is less than a threshold, and proceeds to step S504. The method used to determine whether or not there is a difference between the first corresponding area and the second corresponding area is not limited to a specific determination method.

[0043] In step S507, the display control unit 306 identifies the background image acquired in step S505 (the background image that was determined to have a difference between the first corresponding area and the second corresponding area) as the background image to be displayed, and displays the identified background image on the display unit 205. Various display methods are possible for displaying the background image acquired in step S505.

[0044] For example, as shown in Figure 8, the display control unit 306 may generate a composite frame 801 by superimposing the background image acquired in step S505 semi-transparently onto the frame to be played back acquired in step S501 (for example, by controlling the alpha value of the background image to process it into a semi-transparent image, and then superimposing the processed image onto the frame to be played back), and then display the generated composite frame on the display unit 205. In this case, the display control unit 306 may display the corresponding area to the occluded area and the area other than the corresponding area in the composite frame using different display methods. For example, the display control unit 306 may change the brightness and color tone of the corresponding area to the occluded area and the area other than the corresponding area, or it may make the corresponding area to the occluded area more noticeable by displaying the edges sharply. It may also display the difference of the occluded area using PinP (Picture In Picture) or on a separate screen. This allows the display control unit 306 to retrieve the updated occlusion area from a buffered frame at a later time and add it to the frame to be played back for display, if the occlusion area is updated between frames. Alternatively, the display control unit 306 may display the frame to be played back and the background image acquired in step S505 in separate display areas on the display screen of the display unit 205.

[0045] On the other hand, in step S508, the display control unit 306 displays the frame to be played back, acquired in step S501, on the display unit 205. In step S508, the background image is not displayed.

[0046] The specific comparison operation in the comparison unit 305 will be explained using Figure 7 as an example. Background image 701 is the background image of frame 601, which was captured at imaging time t1, and background image 702 is the background image of frame 602, which was captured at imaging time t2. Also, background image 703 is the background image of frame 603, which was captured at imaging time t3, and background image 704 is the background image of frame 604, which was captured at imaging time t4. Below, we will explain the case in which frame 601 is played back as the frame to be played back when frames 601 to 604 are buffered in the main memory 202.

[0047] The area 705 indicated by the dotted line in background image 701 corresponds to the occluded area in frame 601. The area 706 indicated by the dotted line in background image 702 corresponds to the occluded area in frame 601. The area 707 indicated by the dotted line in background image 703 corresponds to the occluded area in frame 601. The area 708 indicated by the dotted line in background image 704 corresponds to the occluded area in frame 601.

[0048] In this case, the comparison unit 305 first determines whether there is a difference between region 705 and region 706. In Figure 7, there is no difference between region 705 and region 706, so next, the comparison unit 305 determines whether there is a difference between region 705 and region 707. In Figure 7, there is a difference between region 705 and region 707, so the background image 703 is selected as the display target. Figure 8 shows an example of a composite frame 801 in which the background image 703 is superimposed semi-transparently on the frame 601 to be played (for example, the alpha value of the background image 703 is controlled to process it into a semi-transparent image, and the processed image is superimposed on the frame 601 to be played).

[0049] Thus, according to this embodiment, even in situations where the writing on the blackboard is obscured by the instructor's head or part of their body, the writing on the board can be displayed in a way that is easy to see. As a result, the writing on the board is displayed in sync with the instructor's speech, making it easier for viewers to understand the lecture.

[0050] In this embodiment, the background image acquired in step S505 is described as being displayed on the display unit 205, but the output destination of the background image is not limited to the display unit 205. For example, the background image, or a composite frame obtained by combining the background image with the frame to be played back in a semi-transparent manner, may be transmitted to an external device via the communication unit 206 and the network 102.

[0051] [Second Embodiment] In this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, it will be assumed to be the same as the first embodiment.

[0052] In the case shown in Figure 6, when the instructor is writing a cohesive piece of information on the board, such as a diagram, mathematical formula, or text, it may be more convenient for the viewer to display the entire piece. Therefore, in this embodiment, we will describe a configuration in which a set of differing regions is obtained from a later frame and combined with the frame to be played back for display.

[0053] The process performed by the image processing device 103 to reproduce the frame to be reproduced from the frame group buffered in the main memory 202 will be explained according to the flowchart in Figure 9. In Figure 9, the same processing steps as those shown in Figure 5 are given the same step numbers, and the explanation for these processing steps will be omitted.

[0054] In step S901, the comparison unit 305 determines whether the difference in the shielded area has been completed as a whole. Here, the method for determining whether the difference in the shielded area has been completed as a whole will be explained.

[0055] For example, the comparison unit 305 may determine whether the difference in the occluded area has been completed as a whole based on the orientation of the instructor's face in the frame. For example, when the instructor is writing on the blackboard, they often have their back to the camera 101, so only the back of their head is visible. Also, when the instructor finishes writing on the blackboard, they often turn their gaze back towards the camera 101, so they are facing forward. Therefore, the comparison unit 305 determines that the difference in the occluded area has been completed as a whole when it determines that in a later frame in the buffered frame group, the face is facing forward (facing the camera 101) than the frame to be played back. The technique for determining the orientation of a person's face in a frame is a well-known technique, so an explanation related to it will be omitted.

[0056] Furthermore, the comparison unit 305 may also determine whether the difference in the occluded area has been completed as a whole based on the recognition result of the instructor's voice. For example, when the instructor is writing on the board while explaining, the content of the voice and the content of the board match, so it is preferable to display the difference in the occluded area for each chunk of the voice content. Therefore, when the comparison unit 305 recognizes the instructor's voice collected through a voice collection device such as a microphone in the image processing device 103 and detects the end of a sentence or the break between words, it determines that the difference in the occluded area has been completed as a whole.

[0057] Furthermore, the comparison unit 305 may also determine whether the difference in the occluded area has been completed as a whole based on the detection results of the instructor's gestures and other movements. For example, when an instructor writes on the blackboard, they hold chalk in their hand, and when they have finished writing on the blackboard, they return the chalk to its original position. Therefore, the comparison unit 305 determines that the difference in the occluded area has been completed as a whole when it determines that the instructor lowered their arm in a frame later than the frame to be played back in the buffered frame group. The technology for determining a person's movements within a frame is well known, so an explanation of it will be omitted.

[0058] Alternatively, the system may pre-register an audio or action indicating that "the difference in the occluded area has been completed as a unit" in the memory unit 203, and the comparison unit 305 may determine that "the difference in the occluded area has been completed as a unit" when it detects that the instructor has made that audio or performed that action. This allows the instructor to control which divisions they want to display as units, and thus enables the instructor to notify viewers of the units of whiteboard writing they want to convey. In addition, if an action such as erasing the blackboard occurs, the system can continue to display the whiteboard writing before it was erased until the next whiteboard writing begins, providing a display that is convenient for the viewers.

[0059] If, as a result of this judgment, it is determined that "the difference in the shielded area has been completed as a whole," the process proceeds to step S507. If it is not determined that "the difference in the shielded area has been completed as a whole," the process proceeds to step S504.

[0060] In step S507, the display control unit 306 displays the background image acquired in the most recent step S505 on the display unit 205. The "background image acquired in the most recent step S505" is the background image of the frame at the imaging time corresponding to the timing at which it was determined that "the difference of the occluded area has been completed as a whole." For example, the background image of the frame at the imaging time corresponding to the timing at which it was determined that "the difference of the occluded area has been completed as a whole" is the background image of the frame at the imaging time that is after the timing at which it was determined that "the difference of the occluded area has been completed as a whole" and is closest to that timing.

[0061] In the example in Figure 7, there is no difference between region 705 and region 706. Next, the comparison unit 305 determines whether there is a difference between region 705 and region 707. In Figure 7, there is a difference between region 705 and region 707, but at this point, the instructor is not facing forward, and it has not been determined that "the difference in the occluded region has been completed as a whole." Therefore, the comparison unit 305 then determines whether there is a difference between region 705 and region 708. In Figure 7, there is a difference between region 705 and region 708, and at this point, the instructor is facing forward, and it has been determined that "the difference in the occluded region has been completed as a whole." Therefore, background image 704 is selected as the display target.

[0062] Figure 10 shows an example of a composite frame 1001 in which a background image 704 is superimposed semi-transparently on the frame 601 to be played back (for example, the alpha value of the background image 704 is controlled to process it into a semi-transparent image, and this processed image is superimposed on the frame 601 to be played back).

[0063] Thus, according to this embodiment, when the instructor is writing a cohesive piece of information on the board, such as a diagram, mathematical formula, or text, the entire piece can be displayed, making it easier for viewers to understand the lecture. In other words, when the image processing device 103 detects the occurrence of a specific event, it outputs the background image of the frame with the imaging time corresponding to the timing of the detection from the background images acquired in step S505.

[0064] In the first and second embodiments, we explained using the example of displaying a frame showing an instructor giving a lecture using a blackboard, but the invention is not limited to such cases.

[0065] Furthermore, the numerical values, processing timing, processing order, processing entity, data (information) destination / source / storage location, etc., used in each of the above embodiments are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0066] Furthermore, some or all of the embodiments described above may be used in appropriate combinations. Alternatively, some or all of the embodiments described above may be used selectively.

[0067] (Other embodiments) This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0068] The disclosures herein include the following image processing apparatus, image processing methods, and computer programs.

[0069] (Item 1) A means for identifying the background image to be displayed from among the background images of the frame in question, based on the difference between a corresponding region in the background image of the frame in question within a buffered set of frames that corresponds to the foreground region of the frame in question, and a corresponding region in the background image of a frame after the frame in question within the buffered set of frames that corresponds to the foreground region of the frame in question. Output means for outputting the background image of the object to be displayed An image processing apparatus characterized by comprising:

[0070] (Item 2) The image processing apparatus according to item 1, characterized in that the identifying means calculates the difference for background images of frames later than the frame of interest in the buffered frame group, starting from the background images of earlier frames, and identifies the background image whose calculated difference is greater than or equal to a threshold as the background image to be displayed.

[0071] (Item 3) moreover, The image processing apparatus according to item 1 or 2, characterized by comprising a generation means for generating a background image that does not include the foreground in a buffered frame.

[0072] (Item 4) The image processing apparatus according to item 3, characterized in that the generation means generates a processed frame as a background image by replacing the pixel values ​​of pixels in the foreground region of the buffered frame with other pixel values.

[0073] (Item 5) The image processing apparatus according to item 4, characterized in that the other pixel values ​​are the average value of the pixel values ​​of pixels located around the foreground region in the buffered frame.

[0074] (Item 6) The image processing apparatus according to item 4, characterized in that the other pixel values ​​are the average value of the pixel values ​​of pixels in a corresponding region corresponding to the foreground region of the buffered frame in a background image in which a background without a subject has been captured in advance.

[0075] (Item 7) The image processing apparatus according to item 3, characterized in that the generation means generates a processed frame as a background image by replacing a partial image in the foreground region of a buffered frame with a partial image in the corresponding region corresponding to the foreground region in the previously acquired frame.

[0076] (Item 8) The image processing apparatus according to any one of items 1 to 7, characterized in that the output means outputs a composite frame obtained by semi-transparently compositing the background image of the object to be displayed onto the frame of interest.

[0077] (Item 9) The image processing apparatus according to item 8, characterized in that the output means displays the corresponding region corresponding to the foreground region and the region other than the corresponding region in the composite frame using different display methods.

[0078] (Item 10) moreover, Equipped with detection means for detecting the occurrence of a specific event, The image processing apparatus according to any one of items 1 to 7, characterized in that the output means outputs a background image of a frame of the background image of the display target identified by the identification means, corresponding to the timing at which the detection means detected a specific event.

[0079] (Item 11) The image processing apparatus according to item 10, characterized in that the output means outputs a background image of a frame at an imaging time corresponding to the timing at which the detection means detects that a person's face is facing forward in a frame later than the frame of interest in the buffered frame group.

[0080] (Item 12) The image processing apparatus according to item 10, characterized in that the output means outputs a background image of the frame at the imaging time corresponding to the timing at which the detection means detects the end of a sentence or a word break in the speech recognition result.

[0081] (Item 13) The image processing apparatus according to item 10, characterized in that the output means outputs a background image of a frame at an imaging time corresponding to the timing when the detection means detects a specific sound.

[0082] (Item 14) The image processing apparatus according to item 10, characterized in that the output means outputs a background image of the frame at the imaging time corresponding to the timing when the detection means detects a specific operation.

[0083] (Item 15) An image processing method performed by an image processing device, The identification means of the image processing apparatus includes an identification step of identifying the background image to be displayed from among the background images of the frame, based on the difference between a corresponding region in the background image of the frame of interest in the buffered frame group that corresponds to the foreground region of the frame of interest, and a corresponding region in the background image of a frame later than the frame of interest in the buffered frame group that corresponds to the foreground region. The output means of the image processing apparatus includes an output step of outputting the background image of the object to be displayed. An image processing method characterized by comprising:

[0084] (Item 16) A computer program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 1 through 14.

[0085] This disclosure is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of this disclosure. Accordingly, the claims are attached to make the scope of this disclosure public. [Explanation of symbols]

[0086] 301: Acquisition unit 302: Extraction unit 303: Generation unit 304: Acquisition unit 305: Comparison unit 306: Display control unit

Claims

1. A means for identifying a target background image from among the background images of a frame, based on the difference between a first region corresponding to the foreground region of the target frame in the background image corresponding to the target frame in the buffered frame set, and a second region corresponding to the foreground region in the background image of a frame later than the target frame in the buffered frame set. Output means for outputting a composite frame obtained by semi-transparently compositing the identified background image onto the target frame. An image processing apparatus characterized by comprising:

2. The image processing apparatus according to claim 1, wherein the identifying means calculates the difference for background images of frames later than the target frame in the buffered frame group, starting from the background images of earlier frames, and identifies background images for which the calculated difference is equal to or greater than a threshold.

3. moreover, The image processing apparatus according to claim 1, further comprising a generation means for generating a background image that does not include the foreground in a buffered frame.

4. The image processing apparatus according to claim 3, characterized in that the generation means generates a processed frame as a background image by replacing the pixel values ​​of pixels in the foreground region of the buffered frame with other pixel values.

5. The image processing apparatus according to claim 4, characterized in that the other pixel values ​​are the average value of the pixel values ​​of pixels located around the foreground region in the buffered frame.

6. The image processing apparatus according to claim 4, characterized in that the other pixel values ​​are the average of the pixel values ​​of pixels in the region corresponding to the foreground region of the buffered frame in a background image in which a background without a subject has been captured in advance.

7. The image processing apparatus according to claim 3, characterized in that the generation means generates a processed frame by replacing a partial image in the foreground region of a buffered frame with a partial image in the region corresponding to the foreground region in the previously acquired frame.

8. The image processing apparatus according to claim 1, characterized in that the output means displays the region corresponding to the foreground region and the region other than the foreground region in the composite frame using different display methods.

9. A means for identifying a target background image from among the background images of a frame, based on the difference between a first region corresponding to the foreground region of a target frame in a background image corresponding to a target frame in a buffered group of frames, and a second region corresponding to the foreground region in the background images of frames later than the target frame in the buffered group of frames, Output means for outputting the identified background image, A detection means for detecting the occurrence of a specific event and Equipped with, The output means outputs the background image of the frame whose imaging time corresponds to the timing at which the detection means detected that a person's face is facing forward in a frame later than the target frame in the buffered frame group from among the identified background images. An image processing apparatus characterized by the following:

10. A means for identifying a target background image from among the background images of a frame, based on the difference between a first region corresponding to the foreground region of a target frame in a background image corresponding to a target frame in a buffered group of frames, and a second region corresponding to the foreground region in the background images of frames later than the target frame in the buffered group of frames, Output means for outputting the identified background image, A detection means for detecting the occurrence of a specific event and Equipped with, The output means is characterized by outputting a background image of a frame from among the identified background images, corresponding to the timing at which the detection means detects the end of a sentence or a word break in the speech recognition result.

11. A means for identifying a target background image from among the background images of a frame, based on the difference between a first region corresponding to the foreground region of a target frame in a background image corresponding to a target frame in a buffered group of frames, and a second region corresponding to the foreground region in the background images of frames later than the target frame in the buffered group of frames, Output means for outputting the identified background image, A detection means for detecting the occurrence of a specific event and Equipped with, The output means is characterized in that it outputs a background image of a frame from among the identified background images, corresponding to the timing at which the detection means detected a specific sound.

12. An image processing method performed by an image processing device, The identification means of the image processing apparatus includes an identification step of identifying a target background image from among the background images of the frame, based on the difference between a first region corresponding to the foreground region of the target frame in the background image corresponding to the target frame in the buffered frame group, and a second region corresponding to the foreground region in the background image of a frame later than the target frame in the buffered frame group. The output means of the image processing apparatus outputs a composite frame obtained by semi-transparently compositing the identified background image onto the target frame. An image processing method characterized by comprising:

13. A computer program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 11.

Images