Imaging method, electronic device and video conference system

Abstract

The invention discloses an imaging method, an electronic device and a video conference system. The imaging method is used for a real-time video conference performed by a display. The imaging method includes the following steps. At least one video conference stream is received. When a three-dimensional modeling instruction is received, a plurality of two-dimensional pictures of the display are continuously acquired. Establishing at least one three-dimensional model according to the two-dimensional pictures; when an imaging instruction corresponding to a viewing angle message is received, an adjusted image corresponding to the viewing angle message is imaged according to the at least one stereoscopic model.

Description

Technical Field

[0001] The present disclosure relates to a processing method, an electronic device and an electronic system, and more particularly to an imaging method, an electronic device and a video conferencing system. Background Art

[0002] With the advancement of video technology, people can use video conferencing software to communicate and discuss over long distances. During a video conference, as long as the speaker turns on the camera, his or her image can be transmitted to the participants.

[0003] During a video conference, the presenter can show an object, an indoor environment or a beautiful view through the camera. In order to allow the attendees to see the object in its entirety, the presenter can rotate the object or shoot around it. However, even if the presenter tries his best to capture all the details of the object, the attendees may still want to take a closer look at a certain angle, but they cannot keep asking the presenter to show the object again at that angle. Therefore, researchers are working on developing a technology that allows attendees to actively adjust the angle without asking the presenter to show it again. Summary of the invention

[0004] The present disclosure relates to an imaging method, an electronic device and a video conference system. In the process of a video conference, in addition to displaying the video conference stream on a display, a three-dimensional model can be directly created locally using the two-dimensional image displayed on the display, and an adjusted image corresponding to a certain viewing angle can be imaged accordingly. Regardless of the rotation angle of the object by the presenter, each participant can rotate the object in the imaging window by himself using an input device. Therefore, it is as if the participant is holding the object in his hand and looking at it carefully, and has an immersive feeling.

[0005] According to one aspect of the present disclosure, an imaging method is provided. The imaging method is used for a video conference conducted in real time on a display. The imaging method comprises the following steps. At least one video conference stream is received. When a stereo modeling instruction is received, several two-dimensional images of the display are captured. At least one stereo model is established based on these two-dimensional images. When an imaging instruction corresponding to a viewing angle message is received, an adjusted image corresponding to the viewing angle message is displayed based on at least one stereo model.

[0006] According to another aspect of the present disclosure, an electronic device is provided. The electronic device includes a transmission module, a display, and a processing module. The transmission module is used to receive at least one video conference stream. The display is used to display the video conference stream. The processing module includes a screen capture unit, a modeling unit, and an imaging unit. When a stereoscopic modeling instruction is received, the screen capture unit captures several two-dimensional images of the display. The modeling unit is used to establish at least one three-dimensional model based on these two-dimensional images. When an imaging instruction corresponding to a viewing angle message is received, the imaging unit displays an adjusted image corresponding to the viewing angle message based on the three-dimensional model.

[0007] According to another aspect of the present disclosure, a video conferencing system is proposed. The video conferencing system includes a first electronic device and a second electronic device. The first electronic device is used to shoot at least one video conferencing stream and upload it to a network. The second electronic device includes a transmission module, a display and a processing module. The transmission module is used to receive a video conferencing stream through the network. The display is used to display the video conferencing stream. The processing module includes a screen capture unit, a modeling unit and an imaging unit. When a three-dimensional modeling instruction is received, the screen capture unit captures several two-dimensional images of the display. The modeling unit is used to continuously establish at least one three-dimensional model based on these two-dimensional images. When an imaging instruction corresponding to a viewing angle message is received, the imaging unit displays an adjusted image corresponding to the viewing angle message based on the three-dimensional model.

[0008] In order to better understand the above and other aspects of the present disclosure, embodiments are given below and described in detail with reference to the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of a video conferencing system according to an embodiment is shown.

[0010] Figure 2 A block diagram of a presenter's electronic device and a participant's electronic device according to an embodiment is shown.

[0011] Figure 3 A flow chart of an imaging method according to an embodiment is shown.

[0012] Figure 4 Draw Figure 3 The timing relationship diagram of steps S110, S130, S140, and S160 is shown in FIG.

[0013] Figure 5 An example illustrating the process of real-time stereo imaging.

[0014] Figures 6 to 8 Examples illustrating stereoscopic imaging at different time points.

[0015] Description of Reference Numerals

[0016] 100,200n:Electronic device

[0017] 170: Lens

[0018] 210:Transmission module

[0019] 220: Display

[0020] 230: Processing module

[0021] 231: Screen capture unit

[0022] 232: Modeling Unit

[0023] 233: Imaging unit

[0024] 300: Input device

[0025] 800: Object

[0026] 900: Network

[0027] 1000: Video conferencing system

[0028] A1, A2, Ak: Viewing angle message

[0029] CM1: Solid Modeling Instructions

[0030] CM2: Imaging instructions

[0031] FM1,FM2,FM3,FM4,FM8,FMi: 2D screen

[0032] FPij: Feature Points

[0033] IM31,IM82,IMtk: Adjust the image

[0034] LD: Light source direction

[0035] MD3,MD8,MDt: 3D Model

[0036] RS: Relative angle relationship

[0037] S110, S120, S130, S140, S141, S142, S150, S160: Steps

[0038] T0, T1, T2, T3: time points

[0039] W1: Video conference window

[0040] W2: Imaging window

[0041] VS: Video conference streaming DETAILED DESCRIPTION

[0042] Please refer to Figure 1 , which illustrates a schematic diagram of a video conferencing system 1000 according to one embodiment. The video conferencing system 1000 includes a speaker's electronic device 100 and a participant's electronic device 200n. During the video conference, the speaker's electronic device 100 and the participant's electronic device 200n can conduct long-distance video communication and discussion through a network 900. The speaker can display an object 800 in front of a lens 170. The lens 170 is, for example, the front lens of the electronic device 100, or the lens of a mobile phone connected to the electronic device 100, or an external lens. In order for the participant to fully see the appearance of the object 800, the speaker can rotate the object 800 or shoot around the object 800.

[0043] The video conference stream VS is transmitted from the speaker's electronic device 100 to the electronic device 200n in real time. The video conference window W1 on the display 220 of the participant's electronic device 200n continuously displays the video conference stream VS.

[0044] In this embodiment, the electronic device 200n of the participant can display the adjusted image IMtk of the object 800 in the imaging window W2. The imaging window W2 is, for example, overlapped on the video conference window W1. Regardless of the rotation angle of the object 800 by the presenter, each participant can rotate the object 800 in the imaging window W2 by himself using a mouse or keyboard. Therefore, it is as if the participant is holding the object 800 in his hand and looking at it carefully, and has an immersive feeling.

[0045] The video conference window W1 displays the synchronized video conference stream VS, while the content displayed in the imaging window W2 is selected by the participants themselves by adjusting the rotation angle of the object 800 to select the part / position of interest.

[0046] Please refer to Figure 2, which depicts a block diagram of the electronic device 100 of the presenter and the electronic device 200n of the participant according to an embodiment. The electronic device 200n includes a transmission module 210, a display 220 and a processing module 230. The transmission module 210 is used to transmit data, such as a wired network module, a wireless network module or an LTE module. The display 220 is used to display images and data, such as a liquid crystal display panel, an OLED display panel or an electronic paper display. The processing module 230 is used to execute various processing and control programs, such as a chip, a circuit, a circuit board or a storage device storing program codes. The processing module 230 includes a screen capture unit 231, a modeling unit 232 and an imaging unit 233. The screen capture unit 231 is used to capture the screen of the display 220. The modeling unit 232 is used to execute a three-dimensional modeling program. The imaging unit 233 is used to render the required screen. The image capture unit 231, the modeling unit 232 and / or the imaging unit 233 are, for example, a chip, a circuit, a circuit board or a storage device storing program codes. In this embodiment, after the transmission module 210 receives the video conference stream VS, it can display the video conference stream VS on the display 220. The processing module 230 can directly use the two-dimensional picture displayed by the display 220 to establish a three-dimensional model MDt locally, and image an adjusted image IMtk corresponding to a certain viewing angle message Ak. The following is a flowchart to explain in detail the operation of each component.

[0047] Please refer to Figure 3 , which depicts a flow chart of an imaging method according to an embodiment. In step S110, the electronic device 200n receives a video conference stream VS. In this step, the video conference stream VS is directly received by the network 900, and the content of the video conference stream VS is not stored in the electronic device 200n, but is directly displayed on the display 220 after being decoded and adjusting the parameters of the corresponding display 220. Although each electronic device 200n receives the same video conference stream VS, the display 220 of each electronic device 200n will present a different screen according to the different hardware parameters (such as size and resolution) and display parameters (such as color temperature and chroma) of the display 220.

[0048] Next, in step S120, the screen capture unit 231 of the processing module 230 determines whether a three-dimensional modeling instruction CM1 is received. The three-dimensional modeling instruction CM1 is, for example, triggered by an input device 300. The input device 300 is, for example, a mouse, a keyboard, or a microphone. For example, the participant can use the mouse to move the cursor to a modeling button on the screen, and trigger the three-dimensional modeling instruction CM1 after clicking the modeling button. Alternatively, the participant can press a shortcut key combination on the keyboard to trigger the three-dimensional modeling instruction CM1. Alternatively, the participant can input a specific voice through the microphone to trigger the three-dimensional modeling instruction CM1. If the three-dimensional modeling instruction CM1 is received, enter step S130.

[0049] In this embodiment, even if the process enters step S130 , step S110 will still be performed continuously.

[0050] In step S130, the image capture unit 231 captures a plurality of two-dimensional images FMi of the display 220. The two-dimensional image FMi is, for example, the entire image of the display 220, or the image of the video conference window W1, or the image of the video sub-window in the video conference window W1. The image capture unit 231 captures the two-dimensional image FMi according to a certain capture frequency, which is not necessarily the same as the display frequency of the video conference stream.

[0051] Then, the process enters step S140. In this embodiment, even if the process enters step S140, step S110 and step S130 will continue to be performed.

[0052] In step S140, the modeling unit 232 of the electronic device 200n creates at least one three-dimensional model MDt according to the two-dimensional images FMi. Step S140 includes steps S141 and S142. In step S141, the modeling unit 232 of the electronic device 200n captures a plurality of feature points FPij from the two-dimensional images FMi.

[0053] Next, in step S142, the modeling unit 232 of the electronic device 200n overlaps the feature points FPij to link the two-dimensional images FMi. After the two-dimensional images FMi are linked, a three-dimensional model MDt can be established using triangular mesh technology.

[0054] When step S140 is executed, step S110 and step S130 are still being executed, so the electronic device 200n will gradually accumulate more and more two-dimensional images FMi. Generally speaking, more two-dimensional images FMi can create a more detailed three-dimensional model MDt.

[0055] In this step, the modeling unit 232 of the electronic device 200n establishes a relative angle relationship RS on the three-dimensional model MDt. The relative angle relationship RS is, for example, pitch, yaw, and roll.

[0056] In addition, the modeling unit 232 of the electronic device 200n can further establish a light source direction LD. The light source direction LD is a certain direction in the three-dimensional space.

[0057] Next, in step S150, the imaging unit 233 of the electronic device 200n determines whether an imaging instruction CM2 corresponding to a viewing angle message Ak is received. The imaging instruction CM2 is provided by the input device 300, for example. For example, the input device 300 is, for example, a mouse, a keyboard, or a microphone. For example, the participant can use the mouse to click on the object 800 and then rotate the object 800 to generate an imaging instruction CM2 corresponding to a certain viewing angle message Ak. Alternatively, the participant can press the arrow keys on the keyboard to generate an imaging instruction CM2 corresponding to a certain viewing angle message Ak. Alternatively, the participant can input a specific voice through the microphone to generate an imaging instruction CM2 corresponding to a certain viewing angle message Ak. If the imaging instruction CM2 is received, proceed to step S160.

[0058] Then, the process enters step S160. In this embodiment, even if the process enters step S160, step S110, step S130 and step S140 will continue to be performed.

[0059] In step S160, the imaging unit 233 of the electronic device 200n displays an adjustment image IMtk corresponding to the viewing angle message Ak according to the three-dimensional model MDt, and images it in the imaging window W2. In this step, the imaging unit 233 of the electronic device 200n can present light and shadow on the adjustment image IMtk according to the light source direction LD.

[0060] Please refer to Figure 4 , which shows Figure 3The timing relationship diagram of steps S110, S130, S140, and S160. Starting from time point T0, step S110 of receiving the video conference stream VS is continuously performed without interruption. When the three-dimensional modeling instruction CM1 is received from time point T1, step S130 of capturing the two-dimensional image FMi and step S140 of establishing the three-dimensional model MDt begin to be performed continuously. As time goes on, more and more two-dimensional images FMi are captured, and the three-dimensional model MDt will become more and more detailed and clear. When the imaging instruction CM2 is received at time point T2, step S160 will be briefly executed to image the adjustment image IMtk. When another imaging instruction CM2 is received at time point T3, step S160 will also be briefly executed to image another adjustment image IMtk. As time goes on, the later adjusted images IMtk imaged will become clearer and clearer.

[0061] Since the present embodiment synchronously executes step S130 of capturing the two-dimensional image FMi, step S140 of establishing the three-dimensional model MDt and step S160 of imaging the adjustment image IMtk, even if the participant's electronic device 200n receives the imaging instruction CM2 very early, it can still successfully image the adjustment image IMtk with a certain degree of clarity.

[0062] Please refer to Figure 5 , which illustrates the process of real-time stereoscopic imaging. The two-dimensional images FMi are continuously collected and accumulated. The electronic device 200n can continuously establish a three-dimensional model MDt based on the collected two-dimensional images FMi. As the two-dimensional images FMi are accumulated, the three-dimensional model MDt is also continuously modified. With the three-dimensional model MDt, an adjusted image IMtk corresponding to the viewing angle message Ak can be generated based on the viewing angle message Ak.

[0063] Please refer to Figures 6 to 8 , which illustrates stereoscopic imaging at different time points. Figure 6 As shown, when only three 2D images FM1, FM2, FM3 are collected, the electronic device 200n can create a 3D model MD3 based on the collected 2D images FM1, FM2, FM3. Then, based on the viewing angle message A1, an adjustment image IM31 corresponding to the viewing angle message A1 can be generated.

[0064] like Figure 7 As shown, when eight two-dimensional images FM1, FM2, FM3, FM4, ..., FM8 are collected, the electronic device 200n can create a three-dimensional model MD8 based on the eight collected two-dimensional images FM1, FM2, FM3, FM4, ..., FM8. Then, based on the viewing angle message A2, an adjustment image IM82 corresponding to the viewing angle message A2 can be generated.

[0065] like Figure 8As shown, when a plurality of two-dimensional images FM1, FM2, FM3, FM4, ..., FM8, ..., FMi, ... are collected, the electronic device 200n can establish a three-dimensional model MDt based on the collected two-dimensional images FM1, FM2, FM3, FM4, ..., FM8, ..., FMi, .... Then, based on the viewing angle message Ak, an adjustment image IMtk corresponding to the viewing angle message Ak can be generated.

[0066] As mentioned above Figures 6 to 8 As shown in the example, as time goes by, more and more two-dimensional images FMi are captured, and the three-dimensional model MDt will become more and more detailed and clear, so the generated adjusted image IMtk will also become clearer and clearer.

[0067] According to the above embodiment, during the video conference, not only can the video conference stream VS be displayed on the display 220, but also the 3D model MDt can be directly established locally using the 2D image displayed on the display 220, and the adjusted image IMtk corresponding to a certain viewing angle message Ak can be imaged accordingly. Regardless of the rotation angle of the object 800 by the presenter, each participant can rotate the object 800 in the imaging window W2 by himself using the input device 300. Therefore, it is as if the participant holds the object 800 in his hand and looks at it carefully, and feels as if he is actually there.

[0068] In summary, although the present disclosure has been disclosed as above by way of embodiments, it is not intended to limit the present disclosure. A person with ordinary knowledge in the technical field to which the present disclosure belongs may make various changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be determined by the definition of the appended claims.

Claims

1. An imaging method for real-time video conferencing on a display, characterized in that: The imaging method includes: receiving at least one video conference stream; When a three-dimensional modeling instruction is received, a plurality of two-dimensional images of the display are captured; Establishing at least one three-dimensional model based on the plurality of two-dimensional images; and When an imaging instruction corresponding to a viewing angle message is received, an adjusted image corresponding to the viewing angle message is displayed according to the at least one three-dimensional model.

2. The imaging method according to claim 1, characterized in that The at least one video conference stream is directly received from the network.

3. The imaging method according to claim 1, characterized in that: The step of establishing the at least one three-dimensional model and the step of displaying the adjusted image corresponding to the viewing angle message are performed simultaneously.

4. The imaging method according to claim 1, wherein: In the step of establishing the at least one three-dimensional model, a relative angle relationship is further established.

5. The visual imaging method according to claim 1, wherein: In the step of establishing the at least one three-dimensional model, a light source direction is further established.

6. The imaging method according to claim 1, characterized in that: The viewing angle information is adjusted by an input device, and the adjusted image is displayed in an imaging window.

7. The imaging method according to claim 1, characterized in that: The step of establishing the at least one three-dimensional model comprises: capturing a plurality of feature points from the plurality of two-dimensional images; and The plurality of characteristic points are superimposed to link the plurality of two-dimensional images.

8. An electronic device, characterized in that: include: a transmission module, configured to receive at least one video conference stream; a display for displaying the at least one video conference stream; as well as A processing module, comprising: a picture capturing unit, which captures a plurality of two-dimensional pictures of the display when receiving a three-dimensional modeling instruction; A modeling unit, used for establishing at least one three-dimensional shape according to the plurality of two-dimensional images; and An imaging unit, when receiving an imaging instruction corresponding to a viewing angle message, displays an adjusted image corresponding to the viewing angle message according to the at least one three-dimensional model.

9. The electronic device as claimed in claim 8, characterized in that: The transmission module directly receives the at least one video conference stream from the network.

10. The electronic device as claimed in claim 8, characterized in that The action of the modeling unit establishing the at least one three-dimensional model and the action of the imaging unit displaying the adjusted image corresponding to the viewing angle message are performed simultaneously.

11. The electronic device according to claim 8, wherein: The modeling unit is further used to establish a relative angle relationship.

12. The electronic device as claimed in claim 8, characterized in that: The modeling unit is further used to establish a light source direction.

13. The electronic device as claimed in claim 8, characterized in that: The viewing angle information is adjusted by an input device, and the adjusted image is displayed in an imaging window.

14. The electronic device as claimed in claim 8, wherein: The modeling unit is used for capturing a plurality of characteristic points from the plurality of two-dimensional images and superimposing the plurality of characteristic points to link the plurality of two-dimensional images.

15. A video conferencing system, characterized in that: include: A first electronic device for capturing at least one video conference stream and uploading it to a network; as well as A second electronic device, comprising: a transmission module, configured to receive the at least one video conference stream via the network; a display for displaying the at least one video conference stream; and A processing module, comprising: a picture capturing unit, which captures a plurality of two-dimensional pictures of the display when receiving a three-dimensional modeling instruction; a modeling unit, used for establishing at least one three-dimensional model according to the plurality of two-dimensional images; and An imaging unit, when receiving an imaging instruction corresponding to a viewing angle message, displays an adjusted image corresponding to the viewing angle message according to the at least one three-dimensional model.

Images