Photography device, photography method, and program

Abstract

To reduce discomfort in an image caused by distortion correction.SOLUTION: The present disclosure is a photographing apparatus (600) that outputs an image for video distribution, and the photographing apparatus has: panoramic image creation means (21) that creates a panoramic image from a fisheye image obtained by performing imaging; position specification means (23) that detects the position of a predetermined part of a person in the panoramic image to specify the position of the person; and layout means (25) that lays out, in the image for video distribution, a post-conversion image using a predetermined conversion map, of a plurality of types of conversion maps for converting the coordinates of pixels of an original image into a predetermined position, according to the position of the predetermined part of the person.SELECTED DRAWING: Figure 5

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an imaging apparatus, an imaging method, and a program.

BACKGROUND ART

[0002] In imaging apparatuses used for video distribution such as video conferences, a technique is known for developing an ultra-wide-angle (fisheye) image captured using an ultra-wide-angle (fisheye) lens into a rectangular panoramic (360° panoramic for fisheye) image. In the case of a video conference, panoramic video allows all participants to be captured and grasped at the same time, and furthermore, a technique for displaying a close-up of a speaker from within the panoramic video using audio beamforming and person (face) detection means is already known.

[0003] Further, Patent Document 1 discloses control means for suppressing unnatural feeling after correction by setting the center position of a subject as a reference position for panoramic development, in relation to an image processing method for fisheye video (images). The invention disclosed in Patent Document 1 can suppress unnaturalness in the corrected image of a subject by providing a correction method that accounts for the difference in distortion degree from the center of the fisheye video toward its end (peripheral portion).

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0004] However, for application to video distribution such as video conferences with a plurality of participants, it is necessary to perform unnatural feeling suppression processing on a plurality of subjects while displaying a panorama. In the invention disclosed in Patent Document 1, since the reference position for panoramic development is fixed, a problem arises in that unnaturalness in the image due to distortion correction (Affine transformation) occurs when there are a plurality of subjects.

[0005] The present invention has been made in view of the above-described problems, and aims to suppress unnaturalness in images caused by distortion correction.

MEANS FOR SOLVING THE PROBLEMS

[0006] The invention according to claim 1 is a shooting device that outputs an image for video distribution, comprising: a panoramic image generation means for generating a panoramic image from a fisheye image obtained by capturing an image; a position identification means for identifying the position in the height direction of a predetermined part of a person by detecting the position in the height direction of a predetermined part of a person in the panoramic image; and a layout means for laying out the image for video distribution using a predetermined transformation map from among a plurality of transformation maps for transforming the coordinates of each pixel of the fisheye image to a predetermined position, according to the position in the height direction of the predetermined part of the person, wherein the position identification means detects the area in which the person is moving from the panoramic image, Based on the detection of the position of a predetermined part of the person in the panoramic image and the beamforming information obtained by sound collection, This is a photographic device characterized by its ability to identify the location of the aforementioned person. [Effects of the Invention]

[0007] As explained above, the present invention has the effect of suppressing the unnatural appearance of images caused by distortion correction. [Brief explanation of the drawing]

[0008] [Figure 1] This is an overall configuration diagram of the video distribution system according to the embodiment. [Figure 2] This is a diagram illustrating the use of the imaging device at any given location. [Figure 3] (a) is a plan view of a fisheye lens, and (b) is a conceptual diagram of a fisheye image. [Figure 4] This diagram shows the shooting range (vertical direction) of the imaging device. [Figure 5] This is a hardware configuration diagram of the imaging device 600 and the communication terminal 200. [Figure 6] This diagram shows the processing procedure for panoramic and close-up video output performed by a typical video processing unit 220 in block diagram format. [Figure 7] This diagram shows the processing procedure for panoramic and close-up video output performed by the video processing unit 20 of this embodiment in block diagram format. [Figure 8]The diagrams below show a conceptual view of the transformation map; (a) is a diagram showing an example of the input to the transformation map, and (b) is a diagram showing an example of the output to the transformation map. [Figure 9] This diagram shows the coordinate information in the transformation map. [Figure 10] (a) is a diagram showing an example of an input transformation map represented by coordinate information, and (b) is a diagram showing an example of an output transformation map represented by coordinate information. [Figure 11] This is a flowchart illustrating the processing procedure of the video processing unit in this embodiment. [Figure 12] This figure shows the display image shown on the display device 590A at the first location. [Figure 13] (a) is a conceptual diagram showing the process of generating a display image using multiple types of transformation maps of this embodiment for a panoramic image, (b) is a conceptual diagram showing the process of image transformation using one transformation map, and (b) is a conceptual diagram showing the process of image transformation using other transformation maps. [Figure 14] This is a conceptual diagram illustrating the process of generating display images from panoramic video using a common method. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings.

[0010] [Overall structure] Figure 1 is an overall diagram of the video distribution system according to the embodiment.

[0011] As shown in Figure 1, the communication system 1 of the present embodiment is constructed by a communication terminal 200A, a display device 590A, and an imaging device 600A installed at a first site (site α), a communication terminal 200B, a display device 590B, and an imaging device 600B installed at a second site (site β), and a communication management server 300 on the cloud. The communication terminal 200A is a PC, a video conference terminal, or the like, and can communicate with the communication management server 300 via a communication network 400 such as the Internet. Similarly, the communication terminal 200B is a PC, a video conference terminal, or the like, and can communicate with the communication management server 300 via the communication network 400. Thereby, the communication terminal 200A and the communication terminal 200B can communicate with each other via the communication management server.

[0012] Hereinafter, the communication terminal 200A and the communication terminal 200B are collectively referred to as "communication terminal 200". The display device 590A and the display device 590B are collectively referred to as "display device 590". The imaging device 600A and the imaging device 600B are collectively referred to as "imaging device 600".

[0013] The display device 590 is a display, an electronic whiteboard, or the like. The display device 590 is an example of a display unit. The display unit includes not only the display device 590 but also a screen projected by a projector (including a display target for projection mapping).

[0014] Although two sites are shown in Figure 1, communication at three or more sites is also possible. In this case, the number of communication terminals 200, display devices 590, and imaging devices 600 increases according to the number of sites.

[0015] The imaging device 600A at site α transmits video data obtained by imaging the surroundings and audio data obtained by collecting surrounding sounds to the communication terminal 200A. This transmission method may be wired or wireless. Similarly, the imaging device 600B at site β transmits video data obtained by imaging the surroundings and audio data obtained by collecting surrounding sounds to the communication terminal 200B.

[0016] Further, the communication terminal 200A at site α acquires video data and audio data output from the imaging device 600A, and transmits the acquired data to the communication management server 300, and the communication management server 300 transfers the video data and the audio data to the communication terminal 200B at site β which is the counterpart. Similarly, the communication terminal 200B at site α acquires video data and audio data output from the imaging device 600B, and transmits the acquired data to the communication management server 300, and the communication management server 300 transfers the video data and the audio data to the communication terminal 200A at site α which is the counterpart.

[0017] Further, at site α, the communication terminal 200A transmits the video data and audio data transmitted from the communication terminal 200B at site β to the display device 590A, and the display device 590A outputs the video and audio on the site β side. Similarly, at site β, the communication terminal 200B transmits the video data and audio data transmitted from the communication terminal 200A at site α to the display device 590B, and the display device 590A outputs the video and audio on the site α side.

[0018] [Usage Image] Figure 2 is a diagram showing a usage image of the imaging device at an arbitrary site.

[0019] As shown in Figure 2, for example, four people participate in a meeting at an arbitrary site, and the imaging device 600 is installed on a desk 250. The imaging device 600 is provided with a fisheye lens FL to be described later, and is capable of 360-degree surrounding imaging. The display device 590 outputs video and audio of other sites.

[0020] [Imaging Target Range of Imaging Device] Next, the imaging target range of the imaging device 600 will be described with reference to Figure 3 and Figure 4. In Figure 3, (a) is a plan view of the fisheye lens, and (b) is a conceptual diagram of a fisheye image. Figure 4 is a diagram showing the imaging target range (vertical direction) of the imaging device.

[0021] As shown in Figure 3(a), the lens of the imaging device 600 is provided with a fisheye lens FL having an angle of view of 180° or more. Further, as shown in Figure 3(b), the center of the fisheye image (video) is defined as an invalid region, and the donut-shaped region is defined as an effective region.

[0022] The imaging range of the donut-shaped area, when viewed from the side, is as shown in Figure 4. The image in the direction of the ceiling is cut off, and the image in the direction of the floor is outside the imaging range. Nevertheless, the area where people are present can be covered, so the head (or face) and chest of participants who are using a table for a meeting can be captured.

[0023] [Hardware configuration] Next, the hardware configuration of the communication terminal 200 and the imaging device 600 will be explained using Figure 5. Figure 5 is a hardware configuration diagram of the imaging device 600 and the communication terminal 200. Note that Figure 5 shows the imaging device 600, the terminal device 210, and the touch panel section 41 of the display device 590, while the other configurations are those of the communication terminal 200.

[0024] As shown in Figure 5, the communication terminal 200 includes a camera Md (module) 10, a video processing unit 20, a video CODEC unit 30, a video output processing unit 40, a microphone array 50, an audio output unit 60, an audio processing unit 70, a network processing unit 80, an overall processing unit 90, an operation unit 100, a RAM (Random Access Memory) 26, a ROM (Read Only Memory) (or SSD (Solid State Drive)) 22, a recording processing unit 120, and a Capture processing unit 150.

[0025] The Camera MD10 can also be configured to connect to an external general-purpose camera.

[0026] Camera Md10 is an example of an "imaging device". Camera Md10 captures images of meeting scenes, etc. Camera Md10 has a lens 11, an imaging unit 12 (image sensor), and a DSP (Digital Signal Processor) 13. The imaging unit 12 generates video data (RAW data) by converting the image collected through the lens 11 into an electrical signal. The DSP 13 generates video data (YUV (luminance signal (Y), difference between luminance signal and blue component (U), difference between luminance signal and red component (V))) by performing known camera video processing such as Bayer conversion and 3A control on the video data (RAW data) output from the imaging unit 12. Regarding lens 11, the lens used in this invention is assumed to be an ultra-wide-angle (fisheye) lens.

[0027] Details regarding the video processing unit 20, its associated RAM 26, and ROM (SSD) 27 will be described later.

[0028] The video CODEC unit 30 is an example of an "encoding unit". The video CODEC unit 30 encodes and decodes video data (video stream data) transmitted to and from other communication terminals 200. For example, the video CODEC unit 30 encodes the video data output from the video processing unit 20 using a video Endocer, and transmits the encoded video data to other communication terminals 200 via the network processing unit 80.

[0029] Alternatively, the video output processing unit 40 encodes the video data that has been processed for layout, and transmits the encoded video data to another communication terminal 200 via the network processing unit 80.

[0030] Furthermore, for example, the video CODEC unit 30 acquires video data transmitted from another communication terminal 200 (video data encoded by the other communication terminal 200) via the network processing unit 80 and decodes the video data using a video dedocer. The video CODEC unit 30 then outputs the decoded video data to the video output processing unit 40. The video CODEC unit 30 is composed of a CODEC circuit or software using a compression standard such as H.264 / 265.

[0031] The video output processing unit 40 displays the video based on the video data on the display provided by the touch panel unit 41. The display may be an external, general-purpose monitor. (a) For example, the video output processing unit 40 displays video (i.e., video from another location) based on the video data decoded by the video CODEC unit 30 on the display provided by the touch panel unit 41. There may be multiple video from other locations. (b) For example, the video output processing unit 40 displays video based on the video data output from the camera Md10 (i.e., video of the local location) on the display provided by the touch panel unit 41. The video from the camera Md10 may also be video that has been processed by the video processing unit 20 using speaker CloseUP. (c) For example, the video output processing unit 40 displays video based on video data output from the Capture processing unit 150 (i.e., video such as a document screen displayed on an external Host PC) on the display provided by the touch panel unit 41.

[0032] As described above (a), (b), and (c), the types of video to be displayed are diverse, but since it is necessary to fit all of these video conference-related videos into a single frame, layout processing is performed within the video output processing unit 40. Sometimes the individual display videos are simply arranged side by side, or sometimes shared documents are displayed prominently and the participant videos from each location are displayed in Picture-in-Picture format. The layout is changed as needed, depending on the characteristics of the displayed video and the current layout.

[0033] The microphone array 50 includes a microphone array 51 and an A / D converter 52. The microphone array 51 collects the voices of video conference participants and outputs an audio signal (analog signal). The A / D converter 52 converts the audio signal (analog signal) output from the microphone array 51 into a digital signal and outputs the converted audio signal (digital signal) to the audio processing unit 70.

[0034] The audio output unit 60 includes a D / A converter 62 and a speaker 61. The D / A converter 62 converts an audio signal (digital signal) transmitted from another communication terminal 200 into an analog signal. The speaker 61 receives the converted audio signal (analog signal) from the D / A converter 62 and outputs the audio of video conference participants collected at another location.

[0035] The audio processing unit 70 performs predetermined audio processing (e.g., codec processing, noise cancellation (NC), etc.) on the audio data that constitutes the video data received from other communication terminals 200. The audio processing unit 70 then outputs the processed audio data to the audio output unit 60. Simultaneously, while keeping track of the audio data to be output to the audio output unit 60, the audio processing unit 70 performs echo cancellation (EC) processing on the audio data that is routed back to the microphone array 50. The audio processing unit 70 also performs predetermined audio processing (e.g., codec processing, noise cancellation (NC), etc.) on the audio data that constitutes the video data transmitted to other communication terminals 200. The audio processing unit 70 then outputs the processed audio data to the network processing unit 80.

[0036] Furthermore, the audio processing unit 70 identifies the direction of sound using beamforming technology, and the video processing unit 20 generates a speaker close-up video based on that information.

[0037] The network processing unit 80 transmits the encoded video data output from the video CODEC unit (encoder) 30 to another communication terminal 200 via the communication network 400. The network processing unit 80 also receives encoded video data transmitted from the other communication terminal 200 via the communication network 400. The network processing unit 80 then outputs this video data to the video CODEC unit (encoder) 30. The network processing unit 80 also has a function (network status detection unit) to monitor the network bandwidth in order to determine encoding parameters (QP value, etc.). The network processing unit 80 also has a function (recipient station function determination unit) to acquire information about the functions and performance of the receiving station in order to optimize the settings of encoding parameters (QP value, etc.) and transmission mode.

[0038] The central processing unit 90 controls the entire communication terminal 200. The central processing unit 90 is comprised of a CPU (Central Processing Unit), ROM, RAM, SSD, etc. For example, the central processing unit 90 performs mode settings and status management for each module and block according to the operator's instructions. The central processing unit 90 also has functions for arbitrating the usage rights of system memory (RAM) and access rights to the system bus.

[0039] Furthermore, the overall processing unit 90 sets the shooting mode of the camera Md10. The shooting mode settings of the camera Md10 may include automatic setting items that are set automatically according to the environment (for example, metering conditions, etc.) and manual setting items that are set manually by operator input.

[0040] Furthermore, the overall processing unit 90 configures settings related to the layout processing performed by the video output processing unit 40. It selects and sets predetermined display formats, such as using Picture-in-Picture display to prioritize the display of shared documents, or displaying a specific location in a larger size. These settings are made by the operator using the control unit 100 and stored in the memory (RAM) of the communication terminal 200. These settings are then used by the video processing unit 20.

[0041] The control unit 100 is equipped with various input devices (e.g., touch panel, operation buttons, remote control, etc.). The control unit 100 receives various inputs (e.g., various settings, calling meeting participants, etc.) by an operator operating the various input devices.

[0042] The recording processing unit 120 combines the audio data output from the audio processing unit 70 and the video data generated by the video output processing unit 40 to construct the recording data, which is then stored in the SSD. The SSD can be an SSD built into the recording processing unit, or an external device (SD card or external SSD). If using an external device, the device must be equipped with the necessary SDIO (Secure Digital Input / Output) or USB (Universal Serial Bus) interface.

[0043] The Capture Processing Unit 150 captures video input from an external PC and transfers it to a subsequent unit (in this case, the Video Output Processing Unit 40). Simultaneously, if it is also a host site for document sharing, it transfers the video to the Video CODEC Unit 30. There, the still image Endocer encodes the video data, and the encoded video data is transmitted to other communication terminals 200 via the Network Processing Unit 80. • Video streamed from a PC is treated here as an image for sharing documents (screen), and not as a video like camera MD footage. Therefore, an upper limit is set on the frame rate (i.e., a certain interval is ensured for detecting page breaks (updates)). • If you want to transfer a video being played on your PC as a regular video without frame rate restrictions, you can disable the video input from the camera's Md. This setting should be input to the Capture control unit when specifying the system's operating mode.

[0044] Next, the video processing unit 20 will be described. The video processing unit 20 performs various video processing on the video data (YUV data) output from the camera Md10 according to the purpose. For this purpose, the video processing unit 20 has an Affine conversion (Dewarp / distortion correction) unit 21, a Cropping processing + scaling processing unit 22, a face detection / person detection unit 23, a motion / non-motion determination unit 24, and an Overlay unit 25. Note that each unit 21 to 25 can be implemented not only by hardware but also by software.

[0045] The Affine conversion unit 21 performs a predetermined distortion correction process on the ultra-wide-angle (fisheye) image transferred from the camera Md (module) 10. The correction is performed according to the correction map read from the RAM 26 to generate the output image.

[0046] The Cropping + Scaling Processing Unit 22 crops a specific area from the frame images that make up the video data output from the camera Md10, or from the panoramic video generated by the Affine Conversion Unit 21, and scales the cropped area to match the output format. However, in the processing procedure to which this embodiment is applied, the Cropping + Scaling processing is not performed in this module. This is because the Affine Conversion Unit 21 can perform processing equivalent to the Cropping + Scaling processing by providing a conversion map that enables equivalent processing.

[0047] The face detection / person detection unit 23 detects a region (area) where a face (or head), which is an example of a predetermined part of a person, exists, from the frame images that make up the video data output from the camera Md10, or from the panoramic video (image) generated by the Affine conversion unit 21. From the beamforming information (sound data with enhanced directionality) transferred from the audio processing unit 70 and the results of person detection, the unit identifies the person who is speaking, at least from the results of person detection.

[0048] The motion / non-motion determination unit 24 detects areas where a person is moving from the frame image (in this embodiment, the panoramic image output by the Affine conversion unit 21) that constitutes the video data output from the camera Md10. This can serve as auxiliary information to improve the detection accuracy of face detection / person detection.

[0049] The Overlay unit 25 lays out the multiple images generated by the Affine conversion unit into a (frame) image for video distribution.

[0050] The video processing unit 20 uses RAM 26 as a buffer (work memory) when performing various video processing operations. ROM 27 pre-stores multiple types of conversion maps prepared for Affine processing. Only the necessary portion of the conversion map is loaded into RAM 26 each time it is used.

[0051] [Processing in a typical video processing unit] Next, we will explain the general processing of the video processing unit 220 using Figure 6. Figure 6 is a block diagram showing the processing procedure for panoramic and close-up video output performed by a general video processing unit 220.

[0052] The processing of the video processing unit 220 is described below. (1) First, the video processing unit 220 acquires fisheye video from the camera Md10 via the input interface and stores it in the RAM 26. (2) The fisheye video data acquired in (1) is transferred from the input I / F to the Dewarp processing unit (Affine conversion processing unit). Simultaneously, the data from the conversion map already deployed in RAM26 is transferred to the Dewarp processing unit. The Dewarp processing unit transforms the fisheye image into a panoramic rectangle according to this transformation map. The conversion map is created in advance according to the lens projection scheme in camera Md and the resolution of the input and output video of the Dewarp processing unit. (3) The dewarped panoramic video data is stored again in RAM26. (4) The panoramic video data acquired in (3) above is transferred to the cropping processing unit (+ scaling processing unit). The panoramic video is reduced to the minimum resolution necessary for person detection before being transferred to the person detection unit. The person detection unit identifies the area where a person is located from the data acquired above, and determines the speaker's position using an AND condition with the beamforming information. The person detection unit returns the area information (coordinate information) where the person and speaker are located, obtained as described above, to the cropping processing unit. The cropping process involves cropping and scaling the relevant area from the panoramic image according to the coordinate information obtained above. The reasons for performing the scaling process here are as follows:

[0053] The first reason is that scaling is required depending on the size of the person detected and the layout and resolution of the final output video. The size of the detected person varies depending on the distance between the camera Md and the person.

[0054] The second reason is that panoramic images also need to be scaled down according to the layout and resolution of the final output image. The OverLay section handles the layout processing of the cropped human footage and panoramic footage. (5) The OverLay section lays out the panoramic and close-up images within the frame, and then returns to RMA26. (6) The final display video data acquired in (5) above is transferred to the video CODEC unit 30 via the output I / F. • Simultaneously, the data is transferred to the video output processing unit 40.

[0055] The above describes the general processing of the video processing unit 220, but the following problems arise.

[0056] First, in the case of a typical camera Md, no image processing as described above is performed, so the video output by the camera Md is transferred directly to the video CODEC unit 30. Therefore, the processing shown in (2) to (5) of Figure 6 is unnecessary, and video transfer delay is hardly a concern. Conversely, panoramic CloseUP video will have a larger delay time.

[0057] Furthermore, because the circumference of the original fisheye image differs between the upper and lower parts of the panoramic image, the stretching ratio is greater towards the top of the panorama relative to the original image. This difference in stretching ratio can cause unnatural appearances, such as different aspect ratios depending on the position of a person's face.

[0058] [Processing in the video processing unit of this embodiment] Next, the processing of the video processing unit 20 of this embodiment will be explained using Figures 7 to 11. Figure 7 is a block diagram showing the processing procedure for panoramic and close-up video output performed by the video processing unit 20 of this embodiment. Figure 11 is a flowchart showing the processing procedure of the video processing unit of this embodiment. Figure 12 is a diagram showing the display screen displayed on the display device 590A at the first base.

[0059] The processing of the video processing unit 20 is described below. The processing shown in Figure 11 will be explained simultaneously with the processing shown in the block format in Figure 7. S0: After the initialization process of communication terminal 200 is completed, video streaming (video call) will start. S1: (1) First, the video processing unit 20 acquires a fisheye image from the camera Md10 via the input I / F and stores it in the RAM 26. S2:(2) The Affine conversion processing (Dewarp / distortion correction) unit 21 acquires fisheye image data from RAM 26. Simultaneously, the Affine conversion processing unit 21 retrieves the conversion Map data that has been expanded into RAM 26. It is assumed that immediately after the imaging device 600 is started up, the conversion Map is expanded according to the default layout.

[0060] <Explanation of the Conversion Map> Here, the transformation map of this embodiment will be explained using Figures 8 to 10. Figure 8 shows a conceptual diagram of the transformation map, where (a) is an example of an input to the transformation map and (b) is an example of an output to the transformation map. Figure 9 shows coordinate information in the transformation map. In Figure 10, (a) is an example of an input to the transformation map represented by coordinate information and (b) is an example of an output to the transformation map represented by coordinate information.

[0061] The pixel (0,0) in the output image of Figure 8(b) is located at the pixel coordinates (a0,b0) in the original image of Figure 8(a) before distortion correction. The pixel (1,0) in the output image of Figure 8(b) is located at the pixel coordinates (a1,b0) in the original image of Figure 8(a) before distortion correction. In this manner, a list of the coordinate information of the original image before transformation for all pixels of the output image is called a "transformation map". In other words, a "transformation map" is coordinate information for transforming the coordinates of each pixel in the original image before transformation to a predetermined position in the transformed image. Multiple types of transformation maps have different reference positions for aspect ratio optimization.

[0062] Furthermore, as shown in Figure 9, if the number of pixels (resolution) of the output video is mxn pixels, mxn coordinate information is set. The pixel at coordinate (0,0) in the output image is generated by referencing the pixel value at coordinate (a0,b0) in the original image before conversion. The pixel at coordinate (1,0) in the output image is generated by referencing the pixel value at coordinate (a1,b0) in the original image before conversion. The process is repeated in the same manner thereafter. Finally, the pixel at coordinate (m,n) in the output image will have its output pixel value generated by referencing the pixel value at coordinate (an,bn) in the original image before conversion.

[0063] The coordinate information of the transformation map is generated in advance according to the transformation shape of the original image and parameters indicating the lens characteristics of the imaging device 600. These parameters can be stored in RAM 26 or ROM 27, etc., within the imaging device 600. Furthermore, since the above coordinate information is not necessarily integer coordinate values, as shown in Figure 10, interpolation is performed using surrounding pixels as reference pixels to generate output pixel values.

[0064] For example, as shown in Figure 10, the data for pixel A1 is generated by calculating the values ​​of the four pixels surrounding pixel a1', weighted according to their distance. Similarly, the data for pixel A2 is generated by calculating the values ​​of the four pixels surrounding pixel a2', weighted according to their distance. This process is then repeated for all pixels. Note that this interpolation method is merely one example.

[0065] Next, returning to Figure 7, multiple types of conversion maps are transferred to match the layout of the final output video.

[0066] If the final output video layout is as shown in Figure 7, the following multiple types of conversion maps will be transferred. →Transformation Mapa: Transformation Map for generating a close-up video of person A →Transformation Map b: Transformation Map for generating a close-up video of person B →Transformation Mapc: Transformation Map for generating a close-up video of person C →Transformation Mapd: Transformation Map for generating panoramic video of the final output video. →Conversion Mapx: A conversion map (pre-installed in the Affine conversion section) for generating panoramic video with reduced resolution to the minimum necessary for person detection. The Affine conversion processing unit 21 transforms the fisheye image into a panoramic rectangle according to these conversion maps. Simultaneously, it transforms it into a CloseUp rectangle (since the Affine conversion process using the conversion map also serves as the cropping process, the processing time can be shortened compared to a typical video processing unit 220). The conversion map is created in advance according to the lens projection method in camera Md10 and the resolution of the input / output video (image) of the Affine conversion processing unit 21. Furthermore, the transformation map will differ depending on the area of ​​the subject being zoomed in on. S3: The Affine conversion processing unit processes the fisheye image acquired in (2) according to the conversion map acquired in (2'). →Generate a close-up video of person A according to the conversion map. →Generate a close-up video of person B according to the conversion map. →Generate a close-up video of person C according to the conversion map. →Converting according to Mapa, generating a panoramic image of the final output video. This leads to steps S21 and S4, described later. Both are processed in parallel. S4: If the status is "Updating Conversion Map," do nothing and exit. Otherwise, proceed to S5. S5: Change status to "Map Update in Progress". S6:(3) The panoramic video generated by the conversion Mapx is transferred to the face detection / person detection unit 23. S7: The face detection / person detection unit 23 identifies a region where a predetermined part of a person exists from the video data acquired in step S6, and here, the speaker's position is determined by an AND condition with beamforming information. S8: The face detection / person detection unit 23 selects the optimal predetermined conversion map from the conversion maps in ROM 27 based on the area information (coordinate information) of the person and speaker obtained above (feedback). Note that the conversion map does not have to be stored in the shooting device 600. For example, it may be stored in the memory of the communication terminal 200 or the memory of the communication management server 300. Note that ROM 27, the memory of the communication terminal 200, and the communication management server 300 are examples of storage means. In this case, one of the various conversion maps deployed in ROM27 is selected, and only the necessary area (matching the size of the output video display area (Window)) is extracted from the selected predetermined conversion map. S9:(4) The predetermined conversion Map selected in step S8 above is transferred to RAM26. The next Affine conversion process is executed according to this updated conversion Map. However, if the predetermined conversion Map remains the same as the previous one, process (4) is omitted. S10: The status is that the "Updating the conversion map" state is canceled, and the conversion map update process is completed.

[0067] Note that the processing in steps S5-S9 (conversion map update) does not need to be performed every frame, but can be repeated after a certain time interval. S21:(5) The images generated by the transformations Mapa, Mapb, Mapc, and Mapd described in (2') above are returned to RAM26 as the image of the entire frame. S22:(6) The video processing unit 20 transfers the final display video data acquired in step S21 to the video CODEC unit 30 via the output I / F. At the same time, the video processing unit 20 transfers the final display video data acquired in step S21 to the video output processing unit 40. S23: If a video call such as a conference is ongoing, return to step S1.

[0068] As described above, the processes in steps S1-S3 and S21-S23 are executed every frame.

[0069] Furthermore, the image data of the entire frame generated by the process shown in Figure 11 is transmitted to a communication terminal 600 at another location, and the display image 410 shown in Figure 12 is displayed on the display device 590 at the other location. The upper display area 420 of the display image 410 displays a panoramic 360° video, and the lower display areas 431, 432, and 433 of the display image 410 display, from left to right, a close-up video of the person currently speaking, a close-up video of the person who spoke last time, and a close-up video of the person who spoke two sessions ago.

[0070] <How to use different types of conversion maps> Here, we will use Figure 13 to explain the necessity of using multiple types of transformation maps individually. In Figure 13, (a) is a conceptual diagram showing the process of generating a display image using multiple types of transformation maps in this embodiment, (b) is a conceptual diagram showing the process of image transformation using one transformation map, and (b) is a conceptual diagram showing the process of image transformation using other transformation maps.

[0071] Since close-up images do not need to maintain continuity or connection with the entire panorama, even if the magnification is the same, it is better to use them individually based on conditions other than magnification, such as the area position of the subject, as shown in the example below, to reduce the sense of incongruity in the shape after conversion. • By using a transformation map that optimizes the aspect ratio of the upper part (closer to the center of the circle) even for the same donut area, the image of person B in Figure 13 is generated. • By using a transformation map that optimizes the aspect ratio of the central part (neither towards the center nor towards the outer edge) for the same donut area, the image of person A in Figure 13 is generated. • By using a transformation map that optimizes the aspect ratio of the lower part (closer to the outer edge of the circle) even for the same donut area, the image of person D in Figure 7 is generated. For example, as shown in Figure 13(b), when the height of the face (head) is located in the upper region, using the B-type transformation map optimizes the aspect ratio of the upper region. However, the lower region becomes thinner. Also, for example, as shown in Figure 13(c), when the height of the face (head) is located in the central region, using the D-type transformation map optimizes the aspect ratio of the central region. However, the upper and lower regions become thicker.

[0072] In this way, the video processing unit 20 detects the position of the face (head) and selects a predetermined transformation map that optimizes the aspect ratio of the face (head) position even with the same magnification.

[0073] [Main effects of the embodiment] As described above, this embodiment has the effect of reducing the time required to generate the output image while suppressing the unnatural appearance of the image due to distortion correction.

[0074] The following effects will be explained in detail. (1) Time reduction (reduction of delay time) By preparing multiple conversion maps optimized for each requirement to match the requirements of the final output video, the time required for conversion processing can be reduced, resulting in a reduction in the delay time until the panoramic + close-up video is output.

[0075] A typical panoramic close-up processing method generates the final display image by performing the following two main steps (see Figures 6 and 14). Figure 14 is a conceptual diagram illustrating the process of generating a display image from panoramic video using a common method. S101: Converts fisheye image to panoramic image (this process itself involves careful image compression). S102: Extract and resize the relevant area from the panoramic image. In contrast, in this embodiment, as a preliminary step, since the scaling factor of the image changes dynamically depending on the CloseUP region (area) and CloseUP display layout, multiple types of conversion maps are prepared for each scaling factor. Then, in a single step, the video processing unit 20 selects the appropriate predetermined conversion map according to the required scaling factor of the CloseUP image, and then expands only the target area from that conversion map and lays it out on the display image. That is, the video processing unit 20 prepares multiple types of conversion maps specifically for CloseUP processing, enabling the generation of a CloseUP region directly from ultra-wide-angle video (fisheye video), thereby enabling the processing of steps S101 and S102 above to be performed simultaneously.

[0076] Furthermore, the amount of computation required is basically proportional to the number of pixels in the displayed image. Also, because intermediate images are omitted, the amount of computation and the delay time until video output can be halved. In addition, since the interpolation processing between pixels can be halved, image quality degradation can be reduced. (2) Avoiding image inconsistencies through distortion correction (Affine conversion) (especially in close-up images) In addition to the conversion map used for generating panoramic images, multiple types of conversion maps optimized for each area of ​​the subject being zoomed in on are used to perform distortion correction and scaling. This effectively avoids the unnatural appearance of images caused by distortion correction (Affine conversion).

[0077] The image processing unit 20 of this embodiment prepares a distortion correction conversion map each time, tailored to the area of ​​the subject to be Close-Up. Here, "area" refers to where the subject is positioned within the lens. For example, the degree of distortion will differ between an image projected in the central area of ​​a fisheye lens and an image projected near the circumference, so the parameters of the distortion correction conversion map will also differ. Furthermore, this conversion map is different from the conversion map used for generating panoramic images; unlike the overall optimized panoramic conversion map, it is a conversion map optimized for a specific area.

[0078] In other words, the above embodiment is an example in which, when expanding an ultra-wide-angle image (or a donut area in the case of a fisheye image) into an overall optimized panoramic image, a conversion map is prepared for each magnification because the magnification of each CloseUP image is different. However, since the CloseUP image does not need to maintain connection or continuity with the entire panorama, even if the magnification is the same, a distortion correction conversion map is prepared each time to match the area of ​​the subject to be close-up (here, "area" refers to where it is positioned on the lens. For example, the degree of distortion will be different for an image captured in the central area of ​​a fisheye lens and an image captured near the circumference, so the parameters of the distortion correction conversion map will also be different). Furthermore, this conversion map is different from the conversion map used for generating the panoramic image, and unlike the overall optimized panoramic conversion map, it is a conversion map optimized for a specific area.

[0079] [Other application examples] Although embodiments of the present invention have been described above, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0080] Furthermore, although the above embodiment has been explained using fisheye images as an example, the same concept can be applied to distortion correction using ultra-wide-angle cameras other than fisheyes, as well as to panoramic images (not limited to 360° panoramas).

[0081] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0082] Furthermore, each of the parts 21-25 of the video processing unit 20 can be implemented by a computer through the execution of a program. This program can also be recorded on a (non-temporary) recording medium and provided. [Explanation of symbols]

[0083] 1. Communication System 20 Video Processing Section 21 Affine conversion (Dewarp / distortion correction) unit (an example of a panoramic image generation means) 22 Croping processing + scaling processing 23 Face detection / person detection unit (an example of a location determination means) 24. Motion / Non-Motion Determination Unit (Example of Position Identification Means) 25 Overlay section (an example of layout method) 26 RAM 27 ROM (An example of a storage device) 200 communication terminals 300 Communication Management Server 590 Display device 600 Imaging device (example of display unit) [Prior art documents] [Patent Documents]

[0084] [Patent Document 1] Japanese Patent Publication No. 2018-164143

Claims

1. A shooting device that outputs images for video distribution, A panoramic image generation means for generating a panoramic image from a fisheye image obtained by imaging, A position identification means that detects the position in the height direction of a predetermined part of a person in the panoramic image, thereby identifying the position in the height direction of a predetermined part of a person, A layout means for arranging the converted image, which is obtained by using a predetermined conversion map according to the height position of a predetermined part of the person, from among multiple types of conversion maps for converting the coordinates of each pixel of the fisheye image to a predetermined position, into the image for video distribution. It has, The position determination means is characterized by detecting the area in which the person is moving from the panoramic image, thereby determining the position of the person based on the result of detecting the position of a predetermined part of the person in the panoramic image and beamforming information obtained by sound collection.

2. The imaging apparatus according to claim 1, A photographing device characterized by having a storage means for storing the aforementioned multiple types of conversion maps.

3. The photographic device according to claim 1 or 2, characterized in that the predetermined part of the person is the face or head of the person.

4. The shooting apparatus according to any one of claims 1 to 3, characterized in that the conversion map is created by the lens projection method of the camera that captures the fisheye image and the resolution of the input and output images of the panoramic image generation means.

5. The imaging apparatus according to any one of claims 1 to 4, characterized in that the multiple types of conversion maps have different reference positions for aspect ratio optimization.

6. The shooting apparatus according to any one of claims 1 to 5, characterized in that the layout means lays out the panoramic image and the converted image on the image for video distribution.

7. A shooting method performed by a shooting device that outputs images for video distribution, The aforementioned imaging device is A panoramic image is generated from the fisheye image obtained by imaging. By detecting the height position of a predetermined part of a person in the panoramic image, the height position of the predetermined part of the person is determined. Among multiple types of transformation maps for transforming the coordinates of each pixel in the fisheye image to a predetermined position, the transformed image using a predetermined transformation map according to the height position of a predetermined part of the person is laid out on the image for video distribution. A shooting method characterized by detecting the area in which the person is moving from the panoramic image, thereby detecting the position of a predetermined part of the person in the panoramic image, and determining the position of the person based on the result of detecting the position of a predetermined part of the person in the panoramic image and beamforming information obtained by sound collection.

8. To the computer that outputs images for video distribution, A panoramic image is generated from the fisheye image obtained by capturing data. By detecting the height position of a predetermined part of a person in the panoramic image, the height position of the predetermined part of the person is determined. Among multiple types of transformation maps for transforming the coordinates of each pixel in the fisheye image to a predetermined position, the transformed image using a predetermined transformation map according to the height position of a predetermined part of the person is laid out on the image for video distribution. A program characterized by detecting the area in which the person is moving from the panoramic image, thereby detecting the position of a predetermined part of the person in the panoramic image, and determining the position of the person based on the result of detecting the position of a predetermined part of the person in the panoramic image and beamforming information obtained by sound collection.

Images