Selective distortion or deformation correction in images from cameras with wide-angle lenses

Abstract

A method includes receiving, at a conference endpoint, a video captured using a wide angle lens. The method further includes selecting a view area in a frame of the video. The method further includes selectively applying a warping correction or a distortion correction to the view area based on a size of the view area to generate a corrected video frame. The method further includes transmitting the corrected video frame to a remote endpoint.

Description

Technical Field

[0001] The present disclosure relates to video processing, and more particularly to selectively correcting distortion or deformation caused by wide-angle lenses. Background Art

[0002] In video conferencing, a single camera (such as a wide angle camera or a panoramic camera) may be used to capture a view feed that includes multiple conference participants, (a "room view"). A copy of the captured feed may be cropped to include a single person at an endpoint (such as a person who has been talking for a while) or a zoomed-in view of a group of people. Another copy of the captured feed may be cropped to include other people (or things), for example, a person who spoke before the person in the first copy. Various views based on data captured by a wide angle curved lens will be distorted and deformed when displayed on a flat surface. Therefore, there is room for improvement in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] For the purpose of illustration, certain embodiments described in the present disclosure are shown in the accompanying drawings. In the drawings, like numbers refer to like elements throughout. It should be understood that the full scope of the invention disclosed herein is not limited to the precise arrangements, dimensions and instruments shown. In the drawings:

[0004] Figure 1 The operating environment of the embodiment of the present disclosure is illustrated.

[0005] Figure 2 Pictured Figure 1 components of your operating environment.

[0006] Figure 3A A method of selectively correcting deformation or distortion in an image according to an embodiment of the present disclosure is illustrated.

[0007] Figure 3B An alternative embodiment of a method for selectively correcting deformation or distortion in an image is illustrated.

[0008] Figure 4 A full frame image corrected by the distortion correction logic is illustrated.

[0009] Figure 5 A full frame image corrected by the distortion correction logic is illustrated.

[0010] Figure 6 The distortion and warping corrected view area is illustrated.

[0011] Fig. 7A Another method of selectively correcting deformation or distortion in an image is illustrated.

[0012] Figure 7BAn alternative embodiment of another method for selectively correcting deformation or distortion in an image is illustrated.

[0013] Figure 8 A full frame image corrected by the distortion correction logic is illustrated.

[0014] Fig. 9 A full frame image corrected by the distortion correction logic is illustrated.

[0015] Fig.10 Both the warping and distortion corrected versions are illustrated for a view area relatively close to the center of the image.

[0016] Fig.11 Both warping-corrected and distortion-corrected versions of view regions relatively close to the image border are illustrated.

[0017] Fig.12 Electronic devices are illustrated that can be used to practice the concepts and methods of the present disclosure. DETAILED DESCRIPTION

[0018] In the drawings and description of the drawings herein, certain terms are used for convenience only and should not be considered as limiting the embodiments of the present disclosure. In the drawings and the following description, like numbers always refer to like elements.

[0019] Images captured using wide-angle lenses inherently include distortion effects and deformation effects. As used herein, distortion refers to the bending of light, making straight lines in the image appear to be bent. As used herein, deformation refers to "stretching" in a portion of an image, making an object appear larger than it naturally is in one or more dimensions. Distortion or deformation in an image can be corrected by applying a transformation to the image. However, distortion and deformation may not be corrected in the image at the same time. Distortion and deformation may be relatively more obvious in different views of the image. For example, in a cropped view of the image, the deformation may be more obvious than in the full view of the image. Further, the deformation may be more obvious at the edge of the image compared to the center. Systems and methods for selectively correcting distortion and deformation in an image are disclosed. Although the disclosed systems and methods are described in conjunction with a teleconferencing system, it should be noted that according to the present disclosure, the disclosed systems and methods can be used in other environments according to the present disclosure.

[0020] Figure 1 The diagram illustrates a possible operating environment of the example circuit of the present disclosure. Specifically, Figure 1 A conferencing device or endpoint 10 is illustrated according to an embodiment of the present disclosure. Figure 1The conference device or endpoint 10 communicates with one or more remote endpoints 60 via a network 55. The endpoint 10 includes an audio module 30 having an audio codec 32 and a video module 40 having a video codec 42. These modules 30 / 40 are operatively coupled to a control module 20 and a network module 50. The modules 30 / 40 / 20 / 50 include dedicated hardware, software executed by one or more processors, or a combination thereof. In some examples, the video module 40 corresponds to a graphics processing unit (GPU), software executable by a graphics processing unit, a central processing unit (CPU), software executable by a CPU, or a combination thereof. In some examples, the control module 20 includes a CPU, software executable by a CPU, or a combination thereof. In some examples, the network module 50 includes one or more network interface devices, a CPU, software executable by a CPU, or a combination thereof. In some examples, the audio module 30 includes a CPU, software executable by a CPU, a sound card, or a combination thereof.

[0021] In general, endpoint 10 may be a conference device, a video conference device, a personal computer with audio or video conferencing capabilities, or any similar type of communication device. Endpoint 10 is configured to generate near-end audio and video and receive far-end audio and video from remote endpoint 60. Endpoint 10 is configured to transmit near-end audio and video to remote endpoint 60 and initiate local presentation of far-end audio and video.

[0022] Microphone 120 captures audio and provides the audio to audio module 30 and codec 32 for processing. Microphone 120 can be a table or ceiling microphone, part of a microphone pod, or an integrated microphone of an endpoint, etc. Additional microphone 121 can also be provided. Unless otherwise indicated, all descriptions related to microphone 120 throughout this disclosure apply to any additional microphone 121. Endpoint 10 mainly uses the audio captured with microphone 120 for near-end audio. Camera 46 captures video and provides the captured video to video module 40 and codec 42 for processing to generate near-end video. For each frame of near-end video captured by camera 46, control module 20 selects a view area, and control module 20 or video module 40 crops the frame to the view area. The view area can be selected based on near-end audio generated by microphone 120 and additional microphone 121, other sensor data, or a combination thereof. For example, control module 20 can select an area in the frame depicting the participant currently speaking as the view area. As another example, control module 20 may select the entire frame as the view area in response to determining that no one has spoken for a period of time. Thus, control module 20 selects the view area based on the context of the communication session.

[0023] Camera 46 includes a wide angle lens. Due to the nature of the wide angle lens, the video (and still images) captured by camera 46 include both distortion effects and deformation effects. Video module 40 includes deformation correction logic 72 and distortion correction logic 74. In some examples, deformation correction logic 72 and distortion correction logic 74 correspond to a mapping table that identifies adjustments to the images captured by camera 46. The mapping table can be based on the properties of the lens of camera 46, such as focal length, etc. For each frame of video captured by camera 46, video module 40 selects deformation correction logic 72 or distortion correction logic 40 based on the size of the view area selected by control module 20 for the frame, as further described herein below. Video module 40 then applies the selected correction logic to the view area of ​​the frame to generate a corrected near-end video frame. Therefore, each corrected near-end video frame corresponds to a potentially cropped and corrected version of the video frame. The corrected near-end video frames taken together include the corrected near-end video.

[0024] Endpoint 10 uses codec 32 / 42 to encode near-end audio and corrected near-end video according to any public coding standard (such as MPEG-1, MPEG-2, MPEG-4, H.261, H.263, and H.264). Network module 50 then outputs the encoded near-end audio and corrected video to remote endpoint 60 via network 55 using any appropriate protocol. Similarly, network module 50 receives far-end audio and video from remote endpoint 60 via network 55 and sends these to their corresponding codec 32 / 42 for processing. Finally, speaker 130 outputs the far-end audio (received from the remote endpoint), and display 48 outputs the far-end video. In some embodiments, display 48 also outputs the corrected near-end video.

[0025] therefore, Figure 1 An example of a device for selectively correcting deformation or distortion in a video captured by a camera having a wide-angle lens is illustrated. In particular, Figure 1 The equipment can be referenced as follows Figure 3A -B and Fig. 7A -B further described in one of the methods. As described below, these methods can improve the video quality during a communication session.

[0026] Figure 2 Detailed illustration Figure 1 The endpoint 10 has a processing unit 110, a memory 140, a network interface 150, and a general input / output (I / O) interface 160 coupled via a bus 100. As above, the endpoint 10 has a basic microphone 120, a speaker 130, a camera 46, and a display 48.

[0027] Processing unit 110 includes a CPU, a GPU, or both. Memory 140 may be any conventional memory (such as SDRAM) and may store modules 145 in software and firmware form for controlling endpoint 10. Stored modules 145 include various video and audio codecs 32 / 42 and software components of other modules 20 / 30 / 40 / 50 / 200 discussed previously. In addition, modules 145 may include an operating system, a graphical user interface (GUI) that enables a user to control endpoint 10, and other algorithms for processing audio / video signals.

[0028] Network interface 150 provides communication between endpoint 10 and remote endpoints (60). In contrast, general I / O interface 160 can provide data transfer with local devices such as keyboard, mouse, printer, overhead projector, display, external speakers, additional camera, microphone, etc.

[0029] As described above, endpoint 10 captures frames of video, selectively crops the frames into view areas, and selectively applies deformation or distortion correction to the view areas based on the size of the view areas. Because distortion may be more noticeable in relatively large view areas and deformation may be more noticeable in relatively small view areas, selectively using one of the correction techniques improves video quality during a communication session by addressing irregularities that may be more noticeable to communication session participants. Thus, Figure 2 An example physical configuration of an apparatus for selectively correcting warping or distortion to improve video quality is illustrated.

[0030] refer to Figure 3A , a flow chart illustrating a method 300 for selectively applying corrections to an image is shown. The method 300 includes receiving an image at 302. For example, the endpoint 10 may receive a frame of video via the camera 46. The method 300 further includes selecting a view area at 304. For example, the control module 20 may select a view area in the frame based on the context of the communication session, such as which participant is currently speaking. If the view area is smaller than the frame, the video module 40 may then crop the frame to the view area.

[0031] Method 300 further includes measuring the size of the view area at 306. For example, video module 40 may determine the size of the view area at an absolute level (e.g., in length and width of a certain unit, such as pixels) or compared to the size of the frame. Method 300 further includes determining whether the size of the view area is less than a threshold at 310. An illustrative example of the threshold is 1 / 2 of the view of the entire frame.

[0032] If the size of the view area is not less than the threshold, method 300 includes performing distortion correction to generate a corrected view area at 312. For example, video module 40 may apply distortion correction logic 74 to the view area in response to determining that the view area is greater than the threshold.

[0033] If the size of the view area is less than the threshold, method 300 includes performing distortion correction to generate a corrected view area at 314. For example, video module 40 may apply distortion correction logic 72 to the view area in response to determining that the view area is less than the threshold.

[0034] Method 300 further includes outputting the corrected view area at 316. For example, endpoint 10 may output the corrected view area as a corrected near-end video frame to remote endpoint 60 or display device 48. In some implementations, method 300 includes fewer or additional steps. For example, Figure 3B Method 350 is illustrated, which is an alternative version of method 300 that includes an image stabilization step. For example, before performing distortion correction at 312, method 350 includes determining at 440 whether the previous frame was distortion corrected. If the previous frame was distortion corrected, the method includes performing distortion correction at 312 as described above. If the previous frame was deformation corrected, method 350 includes determining at 444 whether the size of the view area is within a gap range. The gap range is determined based on the size of the size threshold used at 310. In some embodiments, the gap range corresponds to the size threshold plus or minus half of the size threshold. If the size of the view area is within the gap range, method 350 includes performing deformation correction at 314 instead of performing distortion correction at 312. If the size of the view area is outside the gap range, method 350 includes performing distortion correction at 312. Similarly, before performing deformation correction at 314, method 350 includes determining at 442 whether the previous frame was deformation corrected. If the previous frame was deformation corrected, the method 350 includes performing deformation correction at 314 as described above. If the previous frame was distortion corrected, method 350 includes determining at 446 whether the size of the view area is within the gap range. If the size of the view area is within the gap range, method 350 includes performing distortion correction at 312 instead of performing deformation correction at 314. If the size of the view area is outside the gap range, method 350 includes performing deformation correction at 314. Therefore, method 350 includes an additional step of increasing the stability of the video output generated by method 300 by avoiding rapid switching between deformation and distortion correction between frames. It should be noted that in some embodiments, the image stabilization step may include a timeout feature. In such embodiments, method 350 includes performing the commanded correction by comparing the size of the view area to a threshold value based on a timer or counter regardless of whether the size of the view area is within the gap range.

[0035] Figure 4-Figure 6 The benefits of method 300 are illustrated. In particular, Figure 4-Figure 6 It is shown that the distortion effect may be more obvious in a full-frame image, while the deformation effect may be more obvious in a cropped (eg, enlarged) view area. Figure 4 , a distortion-corrected full frame 400 is shown. The distortion-corrected full frame 400 corresponds to a first video frame captured by camera 46 as corrected by distortion correction logic 72. The distortion-corrected full frame 400 depicts light fixture 400 and participant 406. As illustrated, the lines of light fixture 406 are distorted (e.g., artificially curved) due to the wide angle lens used to capture the first video frame. In contrast, Figure 5 A distortion-corrected full frame 500 is shown. The distortion-corrected full frame 500 corresponds to the captured first video frame as corrected by the distortion correction logic 74. As illustrated, the distortion of the light fixture 404 (and other distortion effects) are corrected in the distortion-corrected full frame 500, but the deformation effects are present. In particular, the face of the participant 406 is radially stretched in the distortion-corrected full frame 500 as compared to the deformation-corrected full frame 400. However, the distortion effects shown in the deformation-corrected full frame image 400 may be more noticeable to a viewer as compared to the deformation effects shown in the deformation-corrected full frame image 500.

[0036] on the other hand, Figure 6 The illustration shows that the distortion may be more noticeable in the cropped view area. Figure 6 Deformation-corrected view area 600 and distortion-corrected view area 700 are shown. Deformation-corrected view area 600 corresponds to a view area centered on participant 406 as corrected by deformation correction logic 72. Distortion-corrected view area 700 corresponds to a view area centered on participant 406 as corrected by distortion correction logic 74. As shown, the deformation effects (e.g., radial stretching) on ​​the face of participant 406 in distortion-corrected view area 700 may be more apparent as compared to the distortion effects shown in deformation-corrected view area 600.

[0037] therefore, Figure 4-Figure 6 It is illustrated that performing distortion correction on a relatively large view area (e.g., greater than a threshold) and performing deformation correction on a relatively small view area (e.g., less than a threshold) can improve the quality of near-end video output to a display or a remote endpoint. Thus, as in method 300, selectively performing distortion correction or deformation correction on a view area based on the size of the view area can result in improved video output to a display device or a remote endpoint.

[0038] Fig. 7AA flow chart illustrating an alternative method 700 for selectively applying corrections to an image is shown. Method 700 corresponds to method 300, except that method 700 includes determining the location of a face in the image at 708 after measuring the size of the view area at 306. Further, rather than simply determining whether the size of the view area is less than the threshold at 310, method 700 includes determining whether the size of the view area is less than a threshold and whether the detected face is near a border of the image at 710. For example, video module 40 may perform face detection to identify the location of a face depicted in the view area within the image. Video module 40 may further determine whether the location is within a threshold distance of the border (or outside a threshold distance of the center of the image). If the size of the area is greater than the threshold or the detected face is not near a border of the image, method 700 includes performing distortion correction at 312. If the size of the area is less than the threshold and the detected face is near a border of the image, method 700 includes performing deformation correction at 314.

[0039] Alternative implementations of method 700 may include the above referenced Figure 3B Image stabilization techniques described. For example, Figure 7B A method 750 is illustrated corresponding to an alternative version of the method 700 including the image stabilization techniques described above. It should be noted that Figure 3A -B and Fig. 7A - Some of the method steps illustrated in B may be performed in an order different from that illustrated.

[0040] Figure 8-Figure 11 Benefits of method 700 are illustrated. Figure 8 A distortion-corrected full frame 800 is shown that is generated by applying distortion correction logic 72 to a second full video frame captured by camera 46. Distortion-corrected full frame 800 depicts wall 902, first participant 904, and second participant 906. In distortion-corrected full frame 800, straight lines are distorted due to the properties of the wide-angle lens of camera 46. For example, the lines of wall 902 are curved rather than straight. Fig. 9 A distortion corrected full frame 900 generated by applying the distortion correction logic 74 to the second full video frame is shown. As shown, the lines of the wall 902 may appear more natural, while the distortion effects may be difficult to notice in the distortion corrected full frame 900, as compared to the distortion corrected full frame 800. Thus, distortion correction may be more desirable in relatively larger viewing areas.

[0041] Fig.10 The figure shows that the deformation effect is not strong near the center of the frame captured by the wide-angle lens. In particular, Fig.10Depicted are a distortion-corrected view area 1000 centered on a second participant 906 and a distortion-corrected view area 1100 centered on the second participant 906. Although lines of background 1102 are distorted in the distortion-corrected view area 1000 centered on the second participant 906, the distortion effects on the second participant 906 may be difficult to detect in the distortion-corrected image 1100 because the second participant's face is relatively close to the center of the second frame (e.g., a threshold distance from the center of the second frame). Thus, distortion correction may be more ideal for a view area focused on a face close to the center of the frame.

[0042] Fig.11 The illustration shows that the distortion effect is stronger away from the center of the frame captured by the wide-angle lens. In particular, Fig.11 Depicted are a distortion-corrected view area 1300 centered on a first participant 904 and a distortion-corrected view area 1400 centered on the first participant 904. As illustrated, facial deformations of the first participant 906 may be more extreme in the distortion-corrected view area 1400 because the first participant 904 is farther from the center of the second frame, as compared to facial deformations of the second participant 906 in the distortion-corrected view area 1100. Thus, distortion correction may be more ideal for relatively small (e.g., less than a threshold) view areas that focus on faces that are closer to the center of the frame (e.g., within a threshold distance of a frame border).

[0043] therefore, Figure 8-Figure 11 It is illustrated that performing distortion correction on a relatively large view area (e.g., greater than a threshold) or a smaller view area near the center of the image and performing deformation correction on a relatively small view area (e.g., less than a threshold) near the border of the image can improve the quality of near-end video output to a display or a remote endpoint. Thus, selectively performing distortion correction or deformation correction based on the size of the view area and the positioning of the face in the view area in the image can result in improved video output to a display device or a remote endpoint, as in method 700.

[0044] refer to Fig.12, a diagram illustrating an example of a device 1600 configured to perform selective distortion or deformation correction on a video frame is shown. Device 1600 includes speakers 1602A, 1062B, a microphone array 1604, and a camera 1606. Although not shown, device 1600 further includes an output interface configured to output video to a display and / or a communication interface configured to exchange data with a remote endpoint. Device 1600 is configured to capture video using camera 1606 and generate a distortion or deformation corrected version of the video output as described herein. In some embodiments, device 1600 corresponds to endpoint 10. Therefore, Figure 16 illustrates an example of a device that can selectively correct distortion or deformation in an image captured by a camera with a wide-angle lens.

[0045] The various embodiments described above are provided by way of illustration only and should not be construed as limiting the scope of the present disclosure. Various modifications and changes may be made to the principles and embodiments described herein without departing from the scope of the present disclosure and without departing from the appended claims.

Claims

1. A method for video processing, comprising: receiving video captured using a wide-angle lens at a conference endpoint; selecting a viewing area in a frame of the video; Selecting one of a deformation correction technique and a distortion correction technique based on a size of the viewing area to obtain a selected correction technique; applying the selected correction technique to the view area to generate a corrected video frame; as well as transmitting the corrected video frame to a remote endpoint; Wherein selectively applying the deformation correction or distortion correction based on the size of the view area includes applying the deformation correction to the view area in response to determining that the size of the view area is less than a threshold size. The method of claim 1 , wherein the threshold size corresponds to half the size of a frame of the video.

3. The method of claim 1, wherein selecting the view area in the frame of the video comprises detecting an active participant in a conference at the conference endpoint and cropping the frame to an area around the active participant.

4. The method of claim 1 , wherein the deformation correction corresponds to a first table mapping indicating one or more first transformations to be applied to the view area to correct deformation effects in the view area, and the distortion correction corresponds to a second table mapping indicating one or more second transformations to be applied to the view area to correct distortion effects in the view area.

5. The method of claim 1 , wherein selectively applying the deformation correction or the distortion correction based on the size of the view area further comprises: identifying a location in the frame of a face depicted in the view area, and; The deformation correction is applied in response to determining that the position of the face is within a threshold range of a boundary of the frame and the size of the view area is less than a threshold size. 6 . The method of claim 1 , wherein selectively applying the deformation correction or the distortion correction based on the size of the view area is further based on whether the deformation correction or the distortion correction was applied in a previous frame.

7. A computer-readable storage device storing instructions executable by one or more processors to: receiving video captured using a wide-angle lens; selecting a viewing area in a frame of the video; selectively applying a warping correction or a distortion correction to the view area based on a size of the view area to generate a corrected video frame; as well as transmitting the corrected video frame to a remote endpoint; Wherein selectively applying the deformation correction or distortion correction based on the size of the view area includes applying the deformation correction to the view area in response to determining that the size of the view area is less than a threshold size.

8. The computer-readable storage device of claim 7, wherein the threshold size corresponds to half the size of a frame of the video.

9. The computer readable storage device of claim 7, wherein selecting the view area in the frame of the video comprises detecting an active participant in a conference and cropping the frame to an area around the active participant.

10. The computer-readable storage device of claim 7, wherein the deformation correction corresponds to a first table mapping indicating one or more first transformations to be applied to the view area to correct deformation effects in the view area, and the distortion correction corresponds to a second table mapping indicating one or more second transformations to be applied to the view area to correct distortion effects in the view area.

11. The computer readable storage device of claim 7, wherein selectively applying the deformation correction or the distortion correction based on a size of the view area further comprises: identifying a location in the frame of a face depicted in the view area, and; The deformation correction is applied in response to determining that the position of the face is within a threshold range of a boundary of the frame and the size of the view area is less than a threshold size.

12. The computer readable storage device of claim 7, wherein based on the size of the view area, selectively applying the warping correction or the distortion correction is further based on whether the warping correction or the distortion correction was applied to a previous frame.

13. A device for video processing, comprising: Camera, including wide-angle lens; one or more processors; and a memory storing instructions executable by the one or more processors to: receiving video captured by the camera; selecting a viewing area in a frame of the video; selectively applying a warping correction or a distortion correction to the view area based on a size of the view area to generate a corrected video frame; as well as transmitting the corrected video frame to a remote endpoint; Wherein selectively applying the deformation correction or distortion correction based on the size of the view area includes applying the deformation correction to the view area in response to determining that the size of the view area is less than a threshold size. The device of claim 13 , wherein the threshold size corresponds to half the size of a frame of the video.

15. The apparatus of claim 13, wherein selecting the view area in the frame of the video comprises detecting an active participant in a conference and cropping the frame to an area around the active participant.

16. The apparatus of claim 13 , wherein the deformation correction corresponds to a first table mapping indicating one or more first transforms to be applied to the view area to correct deformation effects in the view area, and the distortion correction corresponds to a second table mapping indicating one or more second transforms to be applied to the view area to correct distortion effects in the view area.

17. The apparatus of claim 13, wherein selectively applying the deformation correction or the distortion correction based on the size of the view area further comprises: identifying a location in the frame of a face depicted in the view area, and; The deformation correction is applied in response to determining that the position of the face is within a threshold range of a boundary of the frame and the size of the view area is less than a threshold size.

Images