Recording system, recording / playback device, and image processing method

Abstract

To provide a video recording system which can display a first image during playback even when video recording is performed in such a state that a second image is superimposed on the first image.SOLUTION: The present invention provides a video recording system 60 comprising: a communication unit which receives first image data from another base; an image processing unit which moves the position of the first image data or second image data within a frame of a video in which the second image data received from the other base or prepared in the own base is embedded in the first image; an image composition unit which generates a video file in which the second image data is embedded in the first image data; an image construction unit which constructs the first image data that does not include the second image data from the video file; and a display control unit which displays the first image data constructed by the image construction unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a recording system, a recording / playback device, and an image processing method. [Background technology]

[0002] There are known communication systems that transmit and receive video and audio between remote locations via a network. In such systems, a communication terminal at a location where one of the parties participating in a conference is located captures images and collects audio, such as speech, and converts these into digital data and transmits it to the other party's communication terminal. The other party's communication terminal then displays the images on a display and outputs audio from a speaker, allowing for a video call.

[0003] A technology for recording images taken during a meeting has been devised (see, for example, Patent Document 1). Patent Document 1 discloses a system that allows for efficient reproduction of a meeting by temporarily recording unprocessed images, thereby enabling later editing. Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technology has the problem that because the second image is superimposed on the first image, the first image cannot be displayed even when the user plays back the recorded image later. That is, the communication terminal displays not only the images from the cameras at multiple locations (second images) but also the document (first image) on the display, but because the document often contains text information, users generally display the document as large as possible. The images from the cameras at multiple locations are displayed in a reduced size, similar to a picture-in-picture display, superimposed on the document. This enables the following uses. - View all screens at the same time. Important screens and materials (shared documents, video of the location being spoken, etc.) can be displayed large.

[0005] However, even if the recording system records documents displayed on the screen and camera images from each location, part of the first image (the image displayed large) is hidden, and the user cannot view any parts or areas of the first image that they are interested in.

[0006] In view of the above-mentioned problems, an object of the present invention is to provide a recording system that can display a first image during playback even if a second image is recorded with the first image superimposed on it. [Means for solving the problem]

[0007] In view of the above problems, the present invention provides a recording system having a communication unit that receives first image data from another location, an image processing unit that moves the position of the first image data or the second image data within a frame of a video in which second image data received from the other location or prepared at the own location is embedded in the first image, an image synthesis unit that generates a video file in which the second image data is embedded in the first image data, an image construction unit that constructs the first image data that does not include the second image data from the video file, and a display control unit that displays the first image data constructed by the image construction unit. [Effects of the Invention]

[0008] It is possible to provide a recording system that can display the first image during playback even if the second image is recorded with the first image superimposed on it. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a system configuration of a communication system. [Figure 2] This is an example of a configuration diagram of a recording system at one location. [Figure 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication terminal. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a recording and playback device. [Figure 5]1 is an example of a functional block diagram illustrating functions of a communication terminal and a recording and playback device, divided into blocks. [Figure 6] 10A and 10B are diagrams illustrating information stored in a composite image storage unit. [Figure 7] FIG. 2 is a diagram illustrating information stored in a camera image storage unit. [Figure 8] 10A and 10B are diagrams illustrating information stored in a document image storage unit. [Figure 9] FIG. 1 is a diagram showing an example of a conventional composite image created by a communication terminal. [Figure 10] 10A and 10B are diagrams illustrating an outline of image processing for moving the position of a camera image in a composite image. [Figure 11] FIG. 10 is a flowchart illustrating an example of a recording process for a composite image. [Figure 12] FIG. 10 is a diagram showing the movement of a camera image in a composite image as a screen display image. [Figure 13] FIG. 12 is an example of a flowchart illustrating the process of step S7 in FIG. 11. [Figure 14] FIG. 10 is a diagram showing an example of metadata attached to a frame of a composite image. [Figure 15] 10A and 10B are diagrams illustrating a method for playing back a document image in a recording / playback device. [Figure 16] 10 is a flowchart illustrating an example of a process in which the recording and playback device saves camera images (video files) and document images. [Figure 17] 10A and 10B are diagrams illustrating an outline of image processing for creating a composite image while sliding document images upward. [Figure 18] FIG. 10 is a diagram illustrating the up and down sliding of a document image. [Figure 19] FIG. 12 is a flowchart illustrating the process of step S7 in FIG. 11 in the process of sliding the document image upward. [Figure 20] 10 is an example of a flowchart illustrating a process in which a second document image constructing unit constructs a document image in a process of sliding the document image up and down. [Figure 21]FIG. 10 is a diagram showing three types of images that can be used when reproducing a composite image. [Figure 22] FIG. 10 is a diagram showing an example of an image displayed using two displays. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A recording system and an image processing method performed by the recording system will be described below as an example of an embodiment of the present invention with reference to the accompanying drawings.

[0011] <Outline of recording system operation> In the recording system of this embodiment, at least the following two images are transmitted and received between the locations. 1. Camera images (videos) of the meeting participants and other people at the site 2. Images of materials to be shared in meetings (generally still images) The communication terminals at each location fit images 1 and 2 onto one screen (one screen frame) (so that they can be viewed simultaneously) while maintaining the readability of the materials. In this case, users have no choice but to use a Picture-in-Picture-like layout (hereinafter referred to as embedding 1 into 2).

[0012] When a communication terminal records the entire screen on which the camera image (an example of the second image data) and the document image (an example of the first image data) are laid out, if a person who was unable to attend the meeting wants to play it back and review it, Careful review of all footage may help identify blind spots in the source image, but this takes time. Or, it cannot be denied that the blind spot area may remain invisible. This is an inconvenience.

[0013] <Outline of the processing of this embodiment> Therefore, in this embodiment, the following process is performed so that the recording system can play back areas that were not visible (hidden) during the conference at any timing during playback.

[0014] ◆ During recording (processing on communication terminal) The communication terminal performs at least one of the following processes during recording. A. If a camera image is embedded in the document image, the display position of the image in the foreground (usually the camera image) will continue to move slowly. B. When the image at the back (usually a document image) is changed to a new page or the document itself is changed, slide the document image up and down or left and right to switch to the next page.

[0015] ◆ During playback (processing by recording / playback device) C (corresponding to A). The display position of the camera image in the foreground moves, allowing the recording / playback device to capture areas hidden behind the document image. The recording / playback device temporarily saves this as a separate file during playback, allowing the entire document image to be displayed during playback or the camera image to be moved to any position within the document image. "Temporary during playback" means that the file is saved as a single file, like a regular recording device, and the recording / playback device generates a separate file from that single file during playback.

[0016] Furthermore, since document images are often used in meetings, they can be treated almost as still images, which makes this type of playback possible. D (corresponding to B). By using the sliding transition of the document image, information on blind spots can be obtained. The recording / playback device temporarily saves this information as a separate file during playback, allowing the entire document image to be displayed during playback, and the camera image to be moved to any position on the document image.

[0017] Unlike during a conference, the processes in C and D can be realized because the composite image in which the camera image is embedded in the document image can be recorded in advance and does not need to be displayed in real time. In other words, the composite image can be played back and displayed because it is processed as if an already recorded composite image were being edited.

[0018] In this way, when a user who was unable to attend a conference reviews the composite image of the conference, the communication system of this embodiment does not end playback without being able to see the blind spot area, and there is no need to carefully track the entire duration of the composite image.

[0019] <Terminology> The blind spot is an area of ​​the first image data where the second image data overlaps, and is an area that is not visible to the user.

[0020] Real-time means that if a certain process is executed now, the result of the process will be obtained within a certain delay.

[0021] <Recording system configuration example> FIG. 1 is a schematic diagram of the system configuration of a communication system 100. In the communication system 100, communication terminals 10A to 10D located at points A to D communicate with each other via a communication network N such as the Internet. Each of the points A to D has a recording system 60 (a communication terminal 10 and a recording and playback device 30). It is not necessary to have a recording and playback device 30 located at every point. The number of communication terminals 10A to 10D (four) in FIG. 1 is an example, and there may be two or more (two or more points). Any of the communication terminals 10A to 10D will be referred to as a "communication terminal 10."

[0022] The communication terminals 10A to 10D transmit captured camera images and audio to the communication management system 50. The communication terminals 10A to 10D also receive camera images from the communication management system 50. Note that camera images from their own bases do not need to be received at their own bases. Any of the communication terminals 10A to 10D also transmits document images to the communication management system 50. The document images are images of conference documents that the communication terminals 10A to 10D have imported from an external PC or that they have internally stored. The communication terminals 10A to 10D receive the document images from the communication management system 50. The document images may have been originally registered in the communication management system 50.

[0023] The communication terminals 10A to 10D at the locations A to D display the camera images captured by the communication terminals 10A to 10D and the above-mentioned reference images. The same applies to the audio data. Note that the camera images transmitted by the communication terminals 10A to 10D are assumed to be moving images, but they may also be still images. Furthermore, the communication terminals 10A to 10D may simply receive the camera images and audio without transmitting them.

[0024] Each of the communication terminals 10A to 10D may be a dedicated video conference terminal or a PC (Personal Computer) running an application. That is, a PC is normally used as a general-purpose information processing device, and operates as a communication terminal 10 when a video conference application is executed.

[0025] Furthermore, the communication terminal 10 as a general-purpose information processing device may be a smartphone, a tablet terminal, a PDA (Personal Digital Assistant), a mobile phone, a projector, an electronic whiteboard, a car navigation system, or the like, in addition to a PC.

[0026] In addition, a communication management system 50 is connected to the communication network N. The communication management system 50 manages and controls communications between the communication terminals 10A to 10D. The communication management system 50 may be constructed by a single computer, or may be constructed by multiple computers in which each unit (function, means, or storage unit) is divided and arbitrarily assigned.

[0027] The communication management system 50 manages, for example, communication terminals 10 participating in the same conference by associating them with session IDs. Based on the session ID, the communication management system 50 transmits images (document images, camera images) and audio from each location to other locations participating in the same conference. This enables video conferences between different locations.

[0028] The communication management system 50 is realized by one or more information processing devices. The communication management system 50 may be realized by cloud computing or by a single information processing device. Cloud computing refers to a form in which resources on a network are used without being aware of specific hardware resources. The communication management system 50 may exist on the Internet or on-premise.

[0029] 2 shows a configuration diagram of a recording system 60 at one location. The recording system 60 has a communication terminal 10 and a recording / playback device 30. A display 20 may be connected to the recording / playback device 30. The display 20 can also be connected to the communication terminal 10.

[0030] A camera and a microphone may be externally connected to the communication terminal 10. The camera captures an image of the area within the field of view of the own base and generates a camera image, and the microphone collects sound from the own base and generates audio data. The communication terminal 10 displays the camera image on a built-in or external display 20 and transmits it to other bases. The communication terminal 10 transmits the audio data to other bases.

[0031] The communication terminal 10 generates a composite image by embedding camera images from its own location and camera images from other locations into document images prepared by its own location or other locations. The communication terminal 10 transmits the composite image and audio data to the recording and playback device 30. The recording and playback device 30 records the composite image and audio data generated during the conference and creates a video file.

[0032] The recording and playback device 30 is installed at each base or department, and is used mainly to record a digest of the conference so that users who did not attend the conference can understand the contents of the conference.

[0033] After the conference ends, the recording and playback device 30 plays the moving image file of the composite image in response to a user operation. The played back video is displayed on the display 20. Alternatively, the recording and playback device 30 may function as a web server and transmit the played back moving image file of the composite image to a user terminal. The recording and playback device 30 may also upload the moving image file to the cloud. The recording and playback device 30 may be located on the cloud.

[0034] Although the communication terminal 10 and the recording and playback device 30 are separate entities in FIG. 2, the communication terminal 10 may have the functions of the recording and playback device 30.

[0035] <Hardware configuration example> The hardware configurations of the communication terminal 10 and the recording / playback device 30 included in the recording system 60 according to this embodiment will be described with reference to FIGS.

[0036] <<Communication terminal>> 3 is a diagram showing an example of the hardware configuration of communication terminal 10. Communication terminal 10 includes a camera module 301, a video processing unit 303, a video CODEC unit 314, a video output processing unit 310, a microphone array 304, an audio output unit 307, an audio processing unit 308, a network processing unit 313, an overall processing unit 315, an operation unit 316, RAM 306, a recording device I / F unit 317, a video characteristic analysis unit 302, RAM 305, and a capture processing unit 312. Note that camera module 301 may be configured to connect to an external general-purpose camera.

[0037] Camera module 301 is an example of an "imaging device." Camera module 301 captures video of a conference scene. Camera module 301 has lens 301a, imaging unit 301b (image sensor), and DSP 301c. Imaging unit 301b generates video data (RAW data) by converting video light collected through lens 301a into an electrical signal. DSP 301c generates video data (YUV data) by performing known camera video processing such as Bayer conversion and 3A control on the video data (RAW data) output from imaging unit 301b.

[0038] The video processing unit 303 performs various types of video processing, such as cropping and scaling processing 303a, on the video data (YUV data) output from the camera module 301 according to the purpose. For example, the video processing unit 303 acquires face detection information from the video characteristic analysis unit 302 and beamforming information from the audio processing unit 308, and generates a close-up video of a speaker. The generated video is transferred to the video output processing unit 310. The video processing unit 303 uses the RAM 306 as a buffer when performing various types of video processing.

[0039] The video CODEC unit 314 encodes and decodes video data (video stream data) transmitted and received between the other communication terminals 10. For example, the video CODEC unit 314 encodes video data output from the video processing unit 303 using the video encoder 314a, and transmits the encoded video data to the other communication terminals 10 via the network processing unit 313. Alternatively, the video CODEC unit 314 encodes video data that has been layout processed by the video output processing unit 310, and transmits the encoded video data to the other communication terminals 10 via the network processing unit 313.

[0040] Furthermore, for example, the video CODEC unit 314 acquires video data transmitted from another communication terminal 10 (video data encoded by the other communication terminal 10) via the network processing unit 313, and decodes the video data using the video decoder 314b. Then, the video CODEC unit 314 outputs the decoded video data to the video output processing unit 310. The video CODEC unit 314 is configured by a CODEC circuit or software using a compression standard such as H.264 / 265, for example.

[0041] The video output processing unit 310 displays a video based on the video data on a display provided in the touch panel unit 309. The display may be an external general monitor.

[0042] For example, the video output processing unit 310 displays video based on video data decoded by the video CODEC unit 314 (i.e., video from another location) on a display provided in the touch panel unit 309. There may be multiple videos from another location.

[0043] Furthermore, for example, the video output processing unit 310 displays an image based on the video data output from the camera module 301 (i.e., an image of the own location) on a display provided in the touch panel unit 309. The image from the camera module 301 may be processed by the video processing unit 303 to produce a close-up image of the speaker.

[0044] Furthermore, for example, the video output processing unit 310 displays an image based on the video data output from the capture processing unit 312 (i.e., an image of a document screen or the like displayed on the external PC 311) on a display provided in the touch panel unit 309.

[0045] As described above, the video output processing unit 310 displays a wide variety of images, but because it is necessary to fit all of these video conference-related images into a single frame, layout processing is performed within the video output processing unit 310. The displayed images may be simply displayed side-by-side, or the shared materials may be displayed enlarged and images of participants at each location may be displayed in picture-in-picture format. The video output processing unit 310 changes the layout as needed depending on the characteristics of the image being displayed and the current layout. Layout changes (movement of camera images, etc.) in this embodiment are performed by this module.

[0046] The microphone array 304 includes a microphone array 304a and an A / D converter 304b. The microphone array 304a collects the voices of the participants in the video conference and outputs an audio signal (analog signal). The A / D converter 304b converts the audio signal (analog signal) of the voice output from the microphone array 304a into a digital signal and outputs the converted audio signal (digital signal) to the audio processing unit 308.

[0047] The audio output unit 307 has a D / A converter 307B and a speaker 307a. The D / A converter 307B converts audio signals (digital signals) transmitted from other communication terminals 10 into analog signals. The speaker 307a receives the audio signals (analog signals) converted by the D / A converter 307B and outputs the audio of the video conference participants collected at other locations.

[0048] Audio processing unit 308 has DSP 308a, audio CODEC function 308b, noise cancellation (NR / EC) function 308c, audio discrimination function 308d, and beamforming function 308e, and performs predetermined audio processing (e.g., audio CODEC processing, noise cancellation (NR / EC), audio discrimination, beamforming, etc.) on audio data constituting video data received from other communication terminals 10. Audio processing unit 308 then outputs the processed audio data to audio output unit 307. At the same time, audio processing unit 308, while grasping the audio data to be output to audio output unit 307, performs echo cancellation (EC) processing on audio data that has been routed around and input to microphone array 304. Audio processing unit 308 then outputs the processed audio data to network processing unit 313. Audio processing unit 308 also identifies the direction of sound using its beamforming function, and video processing unit 303 generates a close-up video of the speaker based on that information.

[0049] The network processing unit 313 has a NIC 313a that transmits encoded video data output from the video CODEC unit (encoder) 314 to another destination communication terminal 10 via the network. The network processing unit 313 also receives encoded video data transmitted from another communication terminal 10 via the network. The network processing unit 313 then outputs the video data to the video CODEC unit (decoder) 314. The network processing unit 313 also has a function (network status detection unit 313b) of monitoring the network bandwidth in order to determine encoding parameters (QP value, etc.). The network processing unit 313 also has a function (other station function determination unit 313c) of acquiring information about the functions and performance of the other station in order to optimize the settings of the encoding parameters (QP value, etc.) and the transmission mode.

[0050] The overall processing unit 315 performs overall control of the communication terminal 10. The overall processing unit 315 is configured with a CPU 315a, a ROM 315b, an SSD 315c, a RAM 315d, etc. For example, the overall processing unit 315 performs mode setting and status management for each module and block according to instructions from an operator. The overall processing unit 315 also has a function of arbitrating the right to use the system memory (RAM) and the access right to the system bus.

[0051] Moreover, the overall processing unit 315 sets the imaging mode of the camera module 301. The imaging mode setting of the camera module 301 may include automatic setting items (e.g., photometric conditions, etc.) that are automatically set according to the environment, and manual setting items that are manually set by an operator through operational input.

[0052] The overall processing unit 315 also performs settings related to layout processing performed by the video output processing unit 310. The overall processing unit 315 selects and sets a predetermined display format, such as a Picture-in-Picture display for preferentially displaying shared materials, or a large image of a specific location.

[0053] These settings are made by an operator operating the operation unit 316 and are stored in a memory (RAM) provided in the communication terminal 10. These settings are then used by the video processing unit 303.

[0054] The operation unit 316 includes various input devices (for example, a touch panel, operation buttons, a remote control, etc.) The operation unit 316 accepts various inputs (for example, various settings, calling conference participants, etc.) by an operator operating the various input devices.

[0055] The recording device I / F unit 317 has an I / F function for combining the audio data output from the audio processing unit 308 with the video data generated by the video output processing unit 310 to form recording data, and for outputting the combined data to the recording / playback device 30.

[0056] Video characteristics analysis unit 302 has detection unit 302a and motion determination unit 302b. Detection unit 302a detects areas where a human face is present from frame images constituting the video data output from camera module 301. Motion determination unit 302b detects areas where a human is moving from frame images constituting the video data output from camera module 301. Video characteristics analysis unit 302 outputs the detection results for each area to video processing unit 303. Note that video characteristics analysis unit 302 uses RAM 305 as a buffer when detecting each area.

[0057] The capture processing unit 312 captures video input from the external PC 311 and transfers it to the video output processing unit 310. The capture processing unit 312 has a page break (update) detection function as a function related to this embodiment.

[0058] The Capture processing unit 312 detects whether a page change (update) has occurred in the image for sharing materials (screen) transferred from the external PC 311.

[0059] There are no particular limitations on the detection method, but since the image data is from a PC screen and does not contain any noise, a simple verify check or sum check between frames may be used. In order to reduce the amount of calculation, the capture processing unit 312 may perform the above processing after lowering the resolution of the captured image.

[0060] The screen page change (update) mentioned here also includes the page change trigger when the external PC 311 is connected after the conference has started and the external PC 311 starts transferring video data (screen) for sharing.

[0061] When the Capture processing unit 312 detects a page change (update), it notifies the video output processing unit 310 of this.

[0062] The video transferred from the external PC 311 is treated as an image for sharing documents (screens), and is not treated as a moving image like the video from the camera module. Therefore, an upper limit is set on the frame rate (i.e., a certain interval is secured for detecting page breaks (updates)).

[0063] If the user wishes to transfer video being played on the external PC 311 as a normal video without any frame rate restrictions, the user can disable the video input from the camera module. The user inputs this setting into the Capture control unit when specifying the system operating mode.

[0064] <<Recording and playback device>> Fig. 4 is a hardware configuration diagram of the recording and playback device 30. As shown in Fig. 4, the recording and playback device 30 has a CPU 501, a RAM 502, an operation unit 503, a ROM 504, an input I / F 505, a CODEC 506, a recording and playback circuit 507, an encryption circuit 508, an output I / F 510, and a NIC 511.

[0065] The CPU 501 controls each block of the recording / playback device 30 in accordance with a predetermined program stored in the ROM 504 to realize the recording function described in this embodiment.

[0066] The RAM 502 is used as a working area for the CPU 501, and is also used as a storage area for various processing programs stored in the ROM 504. The RAM 502 is used as a temporary storage destination for image data and audio data transferred from the communication terminal 10. The RAM 502 is also used as a work memory for the CODEC 506 and the recording / playback circuit 507.

[0067] The operation unit 503 is a general user interface that is configured with hard keys or a remote control, and is used to start up the recording / playback device 30, set modes, and so on.

[0068] The input I / F 505 is an interface used when inputting image data and audio data transferred from the communication terminal 10. The I / F may be a dedicated interface, or may be a general interface such as HDMI (registered trademark) or DisplayPort.

[0069] The CODEC 506 is configured with a CODEC circuit such as H.264 / 265 or software to encode / decode frames of image data (video stream data) input via the input I / F 505. The data coded by the encoding process is encrypted by an encryption circuit 508 and then stored in a storage device 509 (HDD, SSD, SD memory card, etc.). The data stored in the storage device 509 is the moving image file of the composite image itself.

[0070] The recording / playback circuit 507 performs a series of image processing steps described in this embodiment to facilitate reuse of recorded data.

[0071] The output I / F 510 outputs image data transferred from the communication terminal 10 to a display. Alternatively, the output I / F outputs a composite image reproduced from composite image data to a display. I / F standards include HDMI (registered trademark) and DisplayPort (registered trademark). The image recording data also includes audio data.

[0072] The NIC 511 can be connected to the Internet via a network such as a LAN (Local Area Network), and transfers the composite image data to an external server or NAS.

[0073] <About the function> FIG. 5 is an example of a functional block diagram illustrating functions of the communication terminal 10 and the recording and playback device 30, divided into blocks.

[0074] <<Communication terminal>> The communication terminal 10 has a camera image acquisition unit 11, an audio acquisition unit 12, a communication unit 13, a composite image transmission unit 14, a movement unit 15, a page switching unit 16, an image composition unit 17, and an operation reception unit 18. These functions of the communication terminal 10 are realized by the hardware circuit of the communication terminal 10 shown in Fig. 3, but may also be realized by a CPU executing a program. Also, the functions in Fig. 5 merely show the main functions of the communication terminal 10, and the communication terminal 10 may have functions other than those shown in the figure.

[0075] The camera image acquisition unit 11 acquires in real time from the camera module 301 camera images captured by the camera module 301. The audio acquisition unit 12 converts the audio collected by the microphone array 304 into PCM data to generate audio data.

[0076] The communication unit 13 transmits camera images to the communication terminal 10 at the other base via the communication management system, and also receives camera images from the communication terminal 10 at the other base via the communication management system. The communication unit 13 transmits audio data to the communication terminal 10 at the other base via the communication management system 50, and also receives audio data from the communication terminal 10 at the other base via the communication management system 50. When the communication terminal 10 at the other base shares a document image, the communication unit 13 receives the document image via the communication management system 50. When the communication terminal 10 at the own base shares a document image, the communication unit 13 transmits the document image to the communication terminal 10 at the other base via the communication management system 50.

[0077] The moving unit 15 continuously moves the camera image slowly relative to the document image in the image processing of the above-mentioned method A.

[0078] The image synthesis unit 17 generates one frame of a synthesized image by arranging a camera image from the own location, a camera image from another location, and a document image, either in response to a user operation or automatically. A frame is an individual still image in a video. The user can reduce these camera images and document images and arrange them in a tiled pattern, or embed the camera image in the document image (or vice versa). In this embodiment, a case where a camera image is embedded and arranged in the document image will be described.

[0079] Furthermore, the image synthesis unit 17 attaches the position information and size of the camera image in the synthesized image and the identification information of the base to the frame as metadata or the like.

[0080] The page switching unit 16 detects the switching of the document image in the image processing B above, and switches from the document image before the switching to the document image after the switching by sliding the document image up and down or left and right. The document image during the switching is recorded as it slides in each frame of the composite image. In addition, the switching of the document image itself is performed by a user operation. The page switching unit 16 detects the switching of the document image by a user operation, and prepares the document image after the switching for processing B (the document image before the switching has already been acquired). The page switching unit 16 switches between the prepared document image after the switching and the document image before the switching by sliding it.

[0081] The shifting unit 15 and the page switching unit 16 function as the image processing unit 21. Either the shifting unit 15 or the page switching unit 16 may operate, or both may operate.

[0082] The image synthesis unit 17 generates one frame of a synthesized image by arranging the camera image of the own site and the camera image of the other site on the document image generated by the page switching unit 16 and recorded while sliding.

[0083] The composite image transmission unit 14 repeatedly transmits the composite image to the recording and playback device 30 in the same way as a moving image. In the case of a moving image, for example, 30 or more frames are transmitted one after another per second. Note that the composite image transmission unit 14 may transmit the composite image in real time, or may accumulate a certain amount of images before transmitting them (if the recording and playback device 30 is not displaying images during a conference).

[0084] The operation receiving unit 18 receives user operations on the communication terminal 10.

[0085] <<Recording and playback device>> The recording and playback device 30 has a receiving unit 31, a video file creating unit 32, a first material image creating unit 33, an operation accepting unit 34, a switching detection unit 35, a second material image creating unit 36, a display control unit 37, and a storage unit 49. These functions of the recording and playback device 30 are realized by the hardware shown in Fig. 4, but may also be realized by a CPU executing a program.

[0086] The receiving unit 31 receives the composite image from the communication terminal 10 and stores it in the composite image storage unit 41. In this way, the composite image storage unit 41 stores the composite image in which the camera image is embedded in the document image.

[0087] The video file creation unit 32 extracts camera images from the composite image based on the position information and size of the camera images, and creates a video file (an example of a second video file) made up of the camera images. Extracting camera images from a composite image is sometimes called trimming. The video file creation unit 32 stores camera images from the same location in chronological order in the camera image storage unit 42 based on the location identification information. Since camera images can be obtained from each frame of the composite image, frames at the same fps (frames per second) as the composite image are stored for each camera image (for each location).

[0088] As the process C above, the first reference image constructing unit 33 constructs a reference image by performing an OR operation on composite images captured at different times (composite images from which camera images have been cut out are called intermediate images). The first reference image constructing unit 33 stores the constructed reference image in the reference image storage unit 43. This reference image is constructed for each time required for the camera image to move until blind spots are eliminated.

[0089] As the process D above, the change detection unit 35 detects a change in the document image itself or a page of the document image (hereinafter, these are referred to as a page change without distinction). The change detection unit 35 compares the area of ​​the composite image other than the camera image for each frame captured at a different time. If the difference is equal to or greater than a certain level, the change detection unit 35 detects a change in the page of the document image. Alternatively, the change detection unit 35 detects a change in the page of the document image by an operation signal for changing the page transmitted from the communication terminal 10.

[0090] When a page change is detected, the second document image constructing unit 36 ​​constructs a document image by combining the document images of the portions where the camera images do not overlap, as the process D above. The second document image constructing unit 36 ​​stores the constructed document image in the document image storage unit 43. This document image is constructed each time a page change occurs.

[0091] Note that processing may be performed by either the first document image constructing unit 33 or the second document image constructing unit 36, or by both. When processing is performed by both, the same document image may be created, but the same image may be deleted.

[0092] The operation receiving unit 34 receives playback operations and recording operations for the recording and playback device 30. The display control unit 37 displays the composite image, the camera image, or the document image on the display 20.

[0093] Fig. 6 is a diagram illustrating information stored in the composite image storage unit 41. In the composite image storage unit 41, recording from start to finish is saved as one video file. In Fig. 6, the following items are managed for each video file: recording ID, shooting start time, shooting end time, number of locations, size, recording time, and file name. The recording ID is identification information that identifies a video file, and is information that associates camera images and document images with composite images. The shooting start time is the start time of the recording. The shooting end time is the end time of the recording. The number of locations is the number of locations participating in the conference, and is transmitted from the communication terminal 10. The number of locations includes the own location. *Size is the capacity of the video file. Recording time is the time elapsed from the start time to the end time. The file name is the name of the composite image saved along with the file path. There are no restrictions on the format of the video file.

[0094] 7 is a diagram illustrating information stored in the camera image storage unit 42. Camera images from each location are saved in one video file in the camera image storage unit 42. In FIG. 7, the following items are managed for each video file: recording ID, shooting start time, shooting end time, location ID, size, and recording time. The recording ID, shooting start time, shooting end time, size, and recording time may be the same as in FIG. 6. The location ID is identification information for identifying the location. The recording and playback device 30 may acquire the location name from the communication terminal 10, or may assign a unique number.

[0095] Although file names are omitted in Fig. 7, camera images from each location are saved as a video file. Also, camera images from each location may be saved in a single video file.

[0096] 8A and 8B are diagrams illustrating information stored in the document image storage unit 43. Fig. 8A shows information on a document image constructed by the first document image construction unit 33, and Fig. 8B shows information on a document image constructed by the second document image construction unit 36.

[0097] The document image storage unit 43 in Fig. 8(a) stores one or more document images constructed from a moving image file of a composite image. In Fig. 8(a), the following items are managed for each document image: recording ID, construction start time, construction end time, document image ID, size, and file name. The construction start time is the time when the construction of the document image is started, based on the start time of the recording. The construction start time can be the time of State A, which will be described later. The construction end time is the time when the construction of the document image is completed, based on the start time of recording. The construction may be completed at the time of state C, which will be described later. The document image ID is identification information that identifies the document image. *The size is the capacity of the image data. The file name is the file name of the document image. In Fig. 8(a), the document image is a still image, but it may be saved as a video.

[0098] In the document image storage unit 43 of Fig. 8(b), the items of recording ID, page switching time, document image ID, size, and file name are managed for each document image. The items of recording ID, document image ID, size, and file name may be the same as those of Fig. 8(a). The page change time is the time when a page change is detected, recorded based on the start time of the recording.

[0099] <Conventional image synthesis method> Fig. 9 is an example of a conventional composite image created by communication terminal 10. The composite image in Fig. 9 has the following configuration. The document image 201 under discussion is displayed on the entire screen. Since it contains small text, it is desirable to display it as large as possible. The bottom left of the screen is a panoramic image 202 (camera image) capturing all the participants at a certain location. The bottom center of the screen shows individual images 203 (camera images) of individuals who are participating alone remotely due to working from home, etc. One of these images is a camera image taken by a camera at the individual's own location. The bottom right of the screen is a speaker camera image 204 in which the person currently speaking is displayed slightly larger.

[0100] If the recording / playback device 30 records the composite image of FIG. 9 as is, the following problem occurs when a person who was unable to attend the conference plays back the image. · It is inevitable that blind spots will occur in the data. Displaying document images in a smaller size may eliminate blind spots, but documents often contain text, making the content difficult to read when displayed in a smaller size, which can hinder the progress of the meeting. There may be times when the blind spot is visible, but if the time is unknown, the user may have to watch the entire video during playback. In that case, it may be difficult to skip a part of the video to save time, but in the end, there may be cases where the user does not see the entire video.

[0101] Therefore, the composite image was a recording that was difficult to use for those who were unable to attend the meeting.

[0102] <Outline of Image Processing in This Embodiment> Below, two types of image processing for the composite image will be explained.

[0103] A. Changing the camera image position First, FIG. 10 is a diagram for explaining an outline of image processing for moving the position of the camera image in the composite image.

[0104] The movement unit 15 of the communication terminal 10 slowly and subtly changes the position of the camera images from each location as shown in FIG. 10(a) → FIG. 10(b) → FIG. 10(c) to composite them. As can be seen from FIGS. 10(a) to 10(c), the panoramic image 202, the individual images 203 of the remote participants at home, and the speaker video are moving. The movement unit 15 moves by changing the position of the camera images so as to leave no areas completely invisible. For example, the image composition unit 17 keeps track of the "size (of the camera image window)" and the "display position coordinates (position information)" and slowly changes its position (by a larger amount than the size) so as to leave no blind spots.

[0105] The composite image is recorded after compression encoding in the recording / playback device 30. During playback (or between recording and playback), the first material image constructing unit 33 of the recording / playback device 30 uses the changing display position of the camera image to separately create a material image without blind spots as process C. This type of operation is possible because the conference has already ended (there is no need for real time).

[0106] During playback, the recording / playback device 30 can also display only the document images without blind spots. If there are two displays, the document images and the camera images can be displayed simultaneously on separate displays.

[0107] The above operations are expected to significantly reduce the time it takes to understand the contents of a meeting when playing it back.

[0108] <Processing Procedure> FIG. 11 is a flowchart illustrating the recording process of the composite image.

[0109] The user of the communication terminal 10 starts up the communication terminal 10 or an application. At the time of startup, the user performs initial settings related to the recording system 60 (S1). The initial settings include, for example, specifying the connection destination location and the layout of the conference screen.

[0110] Next, the communication terminal 10 initializes input / output devices such as a camera, microphone, and speaker, sets their operation modes, and starts up each device (S2). For example, the communication terminal 10 sets the camera imaging mode, such as photometric conditions, to suit the conference environment.

[0111] When the main body is ready in steps S1 and S2, the communication unit 13 requests the communication management system 50 to start communication and starts the conference (S3). Alternatively, the communication unit 13 may start communication upon receiving a communication request from the other station. If the communication terminal 10 is to record a composite image, the recording / playback device 30 is also started at this point.

[0112] If the composite image is in a recording state, the process moves to step S7, otherwise the process proceeds to step S5.

[0113] If the recording / playback device is in a READY state, the process proceeds to step S6, otherwise the process proceeds to step S8.

[0114] The composite image transmission unit 14 of the communication terminal 10 transmits the composite image to the recording and playback device 30, which then starts the recording process (S6).

[0115] The composite image transmission unit 14 executes or continues the recording process (S7). When the conference ends, the communication terminal 10 transitions to a standby state (S8). The process returns to step S4, and the processes from S4 to S7 are repeated until the conference ends.

[0116] <<Moving the camera image position>> 12 is a diagram showing the movement of the camera image of the composite image in the form of a screen display image. In other words, if the movement unit 15 moves the camera image as shown in FIG. 12, it is possible to eliminate blind spots in the document image. 1. During the transition from state A to state B, the panoramic image 202 and the speaker camera image 204 move as shown in the figure (moving upward). 2. During the transition from state B to state C, the individual image 203 and the speaker camera image 204 move as shown in the figure (the individual image 203 moves to the left, and the speaker camera image 204 moves upward). 3. During the transition from state C to state D, the individual image 203 and the speaker camera image 204 move as shown in the figure (the individual image 203 moves to the right, and the speaker camera image 204 moves downward). In other words, each camera image tries to return to its original position. 4. During the transition from state D to state A, the panoramic image 202 and the speaker camera image 204 move as shown in the figure (moving downward, trying to return to their initial positions). This state transitions as follows: State A → State B → State C → State D → State A → State B → State C → State D →... (Repeat)

[0117] Fig. 13 is a flowchart illustrating the processing of step S7 in Fig. 11. In Fig. 13, the movement direction of the camera image is specified according to the current movement state. As explained in Fig. 12, states A to C during the movement of the camera image are any of three states (state D to A is a return path and can be omitted as a state), so the camera image is moved as follows:

[0118] If the camera image is gradually moving from state A to state B (Yes in S71), the movement unit 15 gradually moves the camera image from state A to state B (S72). If step S71 is No, the process proceeds to step S73.

[0119] If the camera image is gradually moving from state B to state C (Yes in S73), the movement unit 15 gradually moves the camera image from state B to state C (S74). If step S73 is No, the process proceeds to step S75.

[0120] If the camera image is gradually moving from state C to state D (Yes in S75), the movement unit 15 gradually moves the camera image from state C to state D (S76). If step S75 is No, the process proceeds to step S77. The movement unit 15 gradually moves the camera image from state D to state A (S77).

[0121] The moving part 15 of the communication terminal 10 will always perform any one of the processes in steps S72, 74, 76, and 77. For example, the time from state A to state C is set in advance, and a slow time such as 30 seconds or 1 minute (a time when the user does not care about the movement of the camera image) may be used.

[0122] FIG. 14 is an example of metadata attached to a frame of a composite image. Frame ID, region ID, position information, size, and status are attached to the frame. · The frame ID is, for example, identification information of the frame. It is preferably included with the imaging time or the elapsed time from the start of imaging. · The base ID is identification information for identifying the base. · The position information is the coordinates of the camera image in the composite image. The position information is a value controlled by the moving part 15. · The size is the width and height of the camera image. The size may be set by the user or fixed, but in any case, it is known.

[0123] The base ID, position information, and size are included for the number of camera images. · When the arrangement of the camera images is in the above A to D, states A to D are stored in the status. It is a value controlled by the moving part 15.

[0124] <C. Construction of Material Images in a Recording and Reproducing Apparatus> FIG. 15 is a diagram for explaining a method of reproducing a material image in the recording and reproducing apparatus 30. The first material image construction unit 33 of the recording and reproducing apparatus 30 constructs a material image without dead angles by utilizing the recorded composite image.

[0125] The first material image construction unit 33 acquires each frame of state A, state B, and state C from the image data of the composite image. Whether it is a frame of state A, state B, or state C may be determined by the recording and reproducing apparatus 30 based on the position of the camera image, or may be determined by the image synthesizing unit 17 from the metadata in FIG. 14.

[0126] The first material image construction unit 33 creates an intermediate image 250 by replacing the area where the camera image was displayed with black pixels (0 in 256 gradations) (Figs. 15(a) to (c)). The intermediate image 250 may be a composite image after the video file creation unit 32 trims the camera image. The first material image construction unit 33 may replace it with white pixels. The positions and sizes of the camera images in states A, B, and C can be obtained from the metadata.

[0127] If the first material image construction unit 33 synthesizes the intermediate images 250 in states A', B', and C' pixel by pixel using the OR operation, a material image 251 without dead angles can be constructed (Fig. 15(d)). This OR operation is, for example, a process of discarding a pixel value of 0 among the three intermediate images 250 and adopting the average (or any one of the images) if it is not 0.

[0128] Note that since there is a possibility that the material image has switched between states A to C (while the position of the camera image is changing), the first material image construction unit 33 may divide the area of the intermediate image 250 without a camera image into blocks, compare the blocks of the three intermediate images 250, and detect the switch. If the material image has switched, the first material image construction unit 33 does not construct the material image. Alternatively, if there are two non-switched material images, a material image may be constructed (in this case, there is a possibility that the entire material image cannot be constructed).

[0129] Also, the composite images used above are three in states A to C, but as long as there are composite images that eliminate dead angles, two or four or more may be used.

[0130] <Creation of Material Image and Video File by Process C> Fig. 16 is a flowchart for explaining the process in which the recording and playback device 30 stores the camera image (video file) and the material image. The process in Fig. 16 can be executed when the video file of the composite image is stored in the composite image storage unit 41. It may also start when the user starts an operation to play back the composite image on the recording and playback device 30 (ordinary data storage may remain in the composite image).

[0131] First, the video file creation unit 32 acquires a composite image from the composite image storage unit 41 (S11). The acquired composite image may be a video file specified by the user, or a video file of a camera image or a video file in which a material image is not constructed.

[0132] The video file creation unit 32 cuts out the camera image from the frame based on the position information and size of the camera image included in the metadata of each frame. The video file creation unit 32 stores these as video files for each base in the camera image storage unit 42 (S12).

[0133] Next, the first material image construction unit 33 identifies frames in states A to C from the video file of the composite image (S13). Here, it is assumed that the intermediate image has been created by the video file creation unit 32.

[0134] The frames in states A to C appear at substantially regular intervals in time series. The first material image construction unit 33 constructs a material image by performing an OR operation on one set of the frames in states A to C (S14).

[0135] The first material image construction unit 33 associates a recording ID, a construction start time, a construction end time, a material image ID, a size, and a file name with the material image and stores it in the material image storage unit 43 (S15). Note that the construction start time is the imaging time of the frame in state A, and the construction end time is the imaging time of the frame in state C.

[0136] <B. Vertical Slide of Material Image> Subsequently, generation of a composite image and construction of a recorded material image by "B. Vertical Slide of Material Image" will be described.

[0137] FIG. 17 is a diagram for explaining an outline of image processing for creating a composite image while sliding a material image upward.

[0138] The page switching unit 16 of the communication terminal 10 moves the document image upward as shown in FIG. 17(a) → FIG. 17(b) → FIG. 17(c) to move to the next document image (next page). The sliding speed is set to a speed at which the recording / playback device 30 can capture the document image. In FIG. 17, the camera image is at the bottom of the document image, so the page switching unit 16 slides the document image upward, but if the camera image is to the side of the document image, the page switching unit 16 slides the document image left or right. If the camera image is at the top of the document image, the page switching unit 16 slides the document image downward. In this way, the page switching unit 16 determines the sliding direction taking into account the layout of the camera image.

[0139] In the sequence of FIG. 17(a) → FIG. 17(b) → FIG. 17(c), the document image A before the switch and the document image B after the switch gradually slide upward.

[0140] This will be explained in detail using FIG. 18. FIG. 18 is a diagram illustrating the up and down sliding of a document image. First, the page switching unit 16 detects a change in document image as shown in FIG. 18(a). The page switching unit 16 can detect a change in document image, for example, by comparing document images transmitted from other locations at regular intervals. The page switching unit 16 may also detect this by an operation signal indicating that a page has been switched. The page switching unit 16 prepares document image B after the switch.

[0141] When a change in document image is detected, the page switching unit 16 connects the two document images A and B vertically and cuts out the portion that fits into one frame 220 as a document image, as shown in Figures 18(b) → 18(c) → 18(d). The page switching unit 16 gradually moves the cutting range from top to bottom of the two document images A and B. For example, in Figure 18(b), the entire document image A before the change is placed in one frame 220, in Figure 18(c), the lower half of the document image A before the change and the upper half of the document image B after the change are placed in one frame 220, and in Figure 18(d), the entire document image B after the change is placed in one frame 220. The image synthesis unit 17 places camera images in these frames to create a synthesized image.

[0142] Then, in the recording / playback device 30, the composite image described in Figures 17 and 18 is recorded after compression encoding. This composite image becomes a moving image in which the document images slide up and down and gradually switch. Camera images are also embedded in the document images. When the document images are played back, the second document image creation unit 36 ​​of the recording / playback device 30, in process D, takes advantage of the fact that the document images are displayed while sliding, and separately creates a document image without blind spots.

[0143] That is, the second document image construction unit 36 ​​constructs a document image by combining the document images in the portions where the camera images do not overlap. If the area where the camera images exist is defined as L from the bottom, then the area equal to or larger than L should be cut out. For example, the second document image construction unit 36 ​​cuts out the upper half 221 (part of the frame) of the screen (frame) in FIG. 18(b) and the upper half 222 (part of the frame) of the screen (frame) in FIG. 18(c). The second document image construction unit 36 ​​combines the two cut-out document images to construct a document image. If the switching time from FIG. 18(b) to FIG. 18(d) is T seconds, then the second document image construction unit 36 ​​cuts out the upper halves 221 and 222 of the frame immediately before the switching is detected and when T / 2 seconds have elapsed since the switching was detected.

[0144] The image processing shown in Figures 17 and 18 is suitable for cases where there are many conference participants (due to the large number of participating locations), where it is difficult to move the image of camera A, or where page breaks in the document occur at fairly short intervals.

[0145] <Processing Procedure>> FIG. 19 is a flowchart illustrating the process of step S7 in FIG. 11 in the process of sliding the document image upward.

[0146] The page switching unit 16 determines whether the document image (page) has been switched based on the document image transmitted from another location and the document image currently being displayed (S21). When the communication terminal 10 at the own location is displaying a document image, the page switching unit 16 can detect a page switch by a user operation.

[0147] Also, the page switching unit 16 determines the slide direction based on the position of the camera image in the document image (S22). Here, it is assumed that the page switching unit 16 determines that it slides upward.

[0148] Next, the page switching unit 16 connects the two document images vertically with the document image before switching on top and the document image after switching below (S23).

[0149] The page switching unit 16 determines the slide speed by dividing the height of the upper document image by the time T (S24). The time T is set in advance. It is preferable that the time T is as fast as possible within the range where the capture process can keep up (for example, 10 seconds, etc.). This is to display the document image so that the user does not fall behind in the discussion during the meeting.

[0150] For each frame of the composite image, the page switching unit 16 determines the range to be included in the frame based on the slide speed, with reference to the upper end of the upper document image (S25).

[0151] <D. Construction of Document Images in the Recording / Playback Device>

[0152] <D. Construction of Document Images in the Recording / Playback Device> Next, referring to FIG. 20, the construction of document images in the recording / playback device 30 will be described. FIG. 20 is a flowchart illustrating the process in which the second document image construction unit 36 constructs a document image in the process of sliding the document image vertically.

[0153] First, the second document image construction unit 36 acquires the video file of the composite image from the composite image storage unit 41 (S31). The composite image to be acquired may be a video file specified by the user, or a video file of a camera image or a video file in which no document image has been constructed. <00,00584>

[0154] The switching detection unit 35 monitors the frames of the composite image and determines whether the document image has been switched (S32).

[0155] If the determination in step S32 is Yes, first, the second material image constructing unit 36 ​​captures the upper half of the material image before switching (S33). Because the composite image has already been saved, the second material image constructing unit 36 ​​can identify the frame immediately before the start of switching.

[0156] Next, the second material image constructing unit 36 ​​captures the upper half of the material image T / 2 seconds after the switch was detected (S34).

[0157] The second document image constructing unit 36 ​​constructs a document image by connecting the upper half and the lower half of the document image (S35).

[0158] The second material image creating unit associates the recording ID, page change time, material image ID, size, and file name with the material image and stores the image in the material image storage unit 43 (S36).

[0159] The second material image constructing unit 36 ​​repeats the process of FIG. 20 until the end of the moving image file of the composite image (S37).

[0160] <Viewing document images by a user who did not participate in the meeting> 21A and 21B show three types of images that can be used when reproducing a composite image.

[0161] Figure 21(b) is a file for reference images. FIG. 21(b) is a file of a document image 251 that has no blind spots and is created by the process of C or D for the documents etc. that were shared during the meeting.

[0162] FIG. 21(c) is a video file that displays only the camera images 252 such as the panoramic image 202, the individual images 203, and the speaker camera image 204. If there is no need to display the document at the same time as the image during playback, the image can be enlarged, making it easier to see people's facial expressions, etc.

[0163] FIG. 21(d) is a moving image file of a composite image 260 similar to that of FIG. 21(a). · Figure 21(d) is the original composite image, which can be displayed as a video.

[0164] An example use case will be described. A user who is not participating in the conference connects their PC to the recording / playback device 30 and selects a video file of the composite image. The user can determine which composite image is the desired conference image based on the start time of the recording, etc. As shown in Figure 21(d), the user views the composite image from the beginning, but if they notice a blind spot in the document, they can, for example, double-click the document. The operation reception unit 34 detects a request for the document. The display control unit 37 detects the display time of the composite image for which the document is requested and retrieves the document image corresponding to the display time from the document image storage unit 43. For example, if the document is requested 1 minute 15 seconds after the start of the composite image, the document image 001-003.jpeg constructed from frames between 1 minute and 1 minute 30 seconds is retrieved. The display control unit 37 can display this document image (without blind spots) large on the display.

[0165] Similarly, if the user is interested in the facial expressions of the participants, he or she may, for example, double-click on any camera image. The operation reception unit 34 detects the request for a camera image. The display control unit 37 detects the display time of the composite image for which the camera image was requested, and retrieves the camera image corresponding to the display time from the camera image storage unit 42. For example, if a camera image is requested 1 minute 15 seconds after the start of the composite image, the camera images (video files) from each location are played back from 1 minute 15 seconds after the start of the composite image.

[0166] Furthermore, as shown in Fig. 22, if there are two displays 20A and 20B, the recording and playback device 30 can simultaneously display a camera image 252 and a document image 251. Fig. 22 shows an example of displaying a camera image 252 and a document image 251 created from a composite image 260 using two displays. By preparing the camera image and the document image separately as described in Fig. 21, the following playback method becomes possible. If the recording / playback device 30 has two output I / Fs, the images in FIGS. 21(b) and 21(c) can be output simultaneously to two displays 20A and 20B, respectively. Alternatively, by using Dual Display and the DisplayPort multi-stream function (which allows displays to be connected to each other), the images in Figure 21(b) and (c) can be output to the monitor simultaneously.

[0167] <Major Effects> As described above, the recording system 60 of this embodiment allows the recording and playback device 30 to capture areas hidden behind the image by at least one of gradually moving the camera image and sliding the document image to change pages. The recording and playback device 30 saves this as a separate file, making it possible to display the entire document image during playback or to move the camera image to any position on the document image.

[0168] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0169] For example, in this embodiment, an example has been described in which a camera image is embedded in a document image, but this embodiment can be applied regardless of what is shown in the image. For example, the document image may be a landscape image, or the camera image may be an image provided by a video posting site. Furthermore, the document image may be a web page, and the camera image may be an advertisement.

[0170] Furthermore, in this embodiment, the recording and playback device 30 creates the material image, but the server may create the material image.

[0171] Furthermore, the configuration examples shown in Fig. 5 and the like are divided according to main functions to facilitate understanding of the processing by the communication terminal 10 and the recording and playback device 30. The method of dividing the processing units and the names of the processing units do not limit the present invention. The processing by the communication terminal 10 and the recording and playback device 30 can be divided into even more processing units depending on the processing content. Furthermore, the processing can be divided so that one processing unit includes even more processes.

[0172] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions. [Explanation of symbols]

[0173] 10. Communication terminals 30 Recording and playback device 60 Recording System 100 Communication Systems [Prior art documents] [Patent documents]

[0174] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-18298

Claims

1. a communication unit that receives the first image data from another location; an image processing unit that moves a position of the first image data or the second image data within a frame of a moving image in which second image data received from another location or prepared at the own location is embedded in the first image data; an image synthesis unit that generates a moving image file by embedding the second image data in the first image data; an image constructing unit that constructs the first image data that does not include the second image data from the moving image file; a display control unit that displays the first image data constructed by the image construction unit; A recording system having the above.

2. a moving unit that arranges the first image data in a frame of a moving image and moves the second image data relative to the first image data by a size greater than the size of the second image data over a predetermined time period; The recording system described in claim 1, characterized in that the image construction unit obtains multiple first image data from the video file, each having a different position of the second image data, and constructs the first image data that does not include the second image data by cutting out the second image data from the first image data and combining them.

3. The recording system described in claim 2, characterized in that the image construction unit constructs the first image data that does not include the second image data by performing an OR operation on multiple first image data obtained by removing the second image data from the first image data.

4. When it is detected that the first image data has been switched, a page switching unit that slides the first image data before switching, arranged in a frame of the video, over a preset time period, and switches to the first image data after switching; the image synthesis unit generates a moving image file in which the second image data is embedded in the first image data in the middle of a slide; When it is detected that the first image data has been switched from the video file, The recording system described in claim 1, characterized in that the image construction unit constructs the first image data that does not include the second image data by cutting out and combining a portion of the first image data before the switch and a portion of the first image data after the slide.

5. When it is detected that the first image data has been switched from the video file, The recording system described in claim 4, characterized in that the image construction unit constructs the first image data that does not include the second image data by cutting out half of the first image data before the switch and half of the first image data after half the time has passed and combining them.

6. the video file has position information and size of the second image data in the first image data, A recording system as described in any one of claims 1 to 5, characterized in that it has a video file creation unit that creates a second video file of the second image data imported from the first image data based on the location information and size.

7. 7. The recording system according to claim 1, wherein the first image data not including the second image data is displayed in response to a user operation.

8. 7. The recording system according to claim 6, wherein the second moving image file is displayed in response to a user operation.

9. a communication unit that receives the first image data from another location; an image processing unit that moves a position of the first image data or the second image data within a frame of a moving image in which second image data received from another location or prepared at the own location is embedded in the first image data; an image synthesis unit that generates a moving image file in which the second image data is embedded in the first image data, a receiving unit that receives the video file; an image constructing unit that constructs the first image data not including the second image data from the moving image file received by the receiving unit; a display control unit that displays the first image data constructed by the image construction unit; A recording and playback device having the above.

10. a step of receiving first image data from another location by a communication unit; an image processing unit moving a position of the first image data or the second image data within a frame of a moving image in which second image data received from another location or prepared at the own location is embedded in the first image data; an image synthesis unit generating a moving image file in which the second image data is embedded in the first image data; an image constructing unit constructing the first image data that does not include the second image data from the moving image file; a step in which a display control unit displays the first image data constructed by the image construction unit; An image processing method comprising:

Images