Imaging device, image processing method, communication system, and program

Abstract

To reduce work of setting an area of interest in a wide-field image such as a panoramic image in a fixed display area 440 by solving a problem in which, in a conventional technology, a participant (user) at a video distribution source sets the area of interest in a panoramic image in the fixed display area 440, which made it difficult to prepare for video distribution such as a video conference.SOLUTION: A photography device 6 automatically detects a rectangular display 280 of a display device 2, which is mainly treated as an area of interest to enlarge the area of interest within a wide-field image and set the area of interest as a fixed display area 440. This has an effect in which work of a user of a video distribution source can be reduced.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] The present disclosure relates to a photographing apparatus, an image processing method, a communication system, and a program.

Background Art

[0002] In a photographing apparatus used for video distribution such as a video conference, a technique of expanding a wide-angle (fisheye) image photographed using an ultra-wide-angle (fisheye) lens into a rectangular panorama (360° panorama in the case of fisheye) image is known. In the case of a video conference, by making a panorama image, participants at the video distribution destination can simultaneously grasp all participants at the video distribution source (see the display area 420 in FIG. 13).

[0003] Further, since a panorama image is an elongated image, a conventional apparatus can also generate a composite image in which an image of an area to be noted in the panorama image is enlarged and fitted into a fixed display area on the screen (see the fixed display area 440 in FIG. 13) and distribute it to the video distribution destination (see Non-Patent Document 1). As a result, it becomes easier for participants at the video distribution destination to grasp an image to be noted such as a document.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, since an operation for a participant (user) at the video distribution source to set an area to be noted in the panorama image in the fixed display area has occurred, there has been a problem that it takes time to prepare for video distribution such as a video conference.

[0005] The present invention has been made in view of the above problems, and an object thereof is to reduce the operation of setting an area to be noted in a wide-view image such as a panorama image in a fixed display area.

Means for Solving the Problems

[0006] The invention according to claim 1 is a shooting device that outputs an image for video distribution, comprising: a wide-field image generation unit that generates a wide-field image from a fisheye image obtained by capturing an image; a code information detection unit that detects code information contained in the wide-field image; an instruction information detection unit that, when the code information is detected by the code information detection unit, detects instruction information indicating that a rectangular region should be detected from the code information; a rectangular region detection unit that, when the instruction information is detected by the instruction information detection unit, detects a predetermined rectangular region outside the code information within the wide-field image; a composite image generation unit that, when the predetermined rectangular region is detected by the rectangular region detection unit, generates a predetermined composite image by combining a predetermined rectangular region image showing the predetermined rectangular region within the wide-field image with the wide-field image; and a transmission unit that transmits the predetermined composite image generated by the composite image generation unit to the outside. [Effects of the Invention]

[0007] As described above, the present invention has the effect of reducing the effort required to set a region of interest within a wide-field image as a fixed display area. [Brief explanation of the drawing]

[0008] [Figure 1] This is an overall configuration diagram of the communication system according to the embodiment. [Figure 2] This is a diagram illustrating the use of the imaging device at any given location. [Figure 3] (a) is a plan view of a fisheye lens, and (b) is a conceptual diagram of a fisheye image. [Figure 4] This diagram shows the shooting range (vertical direction) of the imaging device. [Figure 5] This is an electrical hardware configuration diagram of a display device. [Figure 6] This is an electrical hardware configuration diagram of a communication terminal and server. [Figure 7] This is an electrical hardware configuration diagram of the imaging device. [Figure 8]This is a functional configuration diagram of a display device, a communication terminal, and a camera. [Figure 9] This flowchart shows the process of specifying a fixed display area using a camera. [Figure 10] This flowchart shows the process of transmitting predetermined information from a camera to a communication terminal. [Figure 11] This is a diagram showing a 2D code displayed on the screen of a display device. [Figure 12] This is a diagram showing a 2D code displayed on the screen of a display device. [Figure 13] This diagram shows the screen displayed by communication terminal 5 at base α. [Figure 14] This figure shows an example of a 2D code displayed by a display device. [Figure 15] This figure shows an example of how a 2D code displayed by a display device can be shown. [Figure 16] This figure shows an example of how a 2D code displayed by a display device can be shown. [Figure 17] This figure shows an example of how a 2D code displayed by a display device can be shown. [Figure 18] This figure shows an example of how a 2D code displayed by a display device can be shown. [Figure 19] This figure shows an example of how a 2D code displayed by a display device can be shown. [Figure 20] This figure shows an example of how a 2D code displayed by a display device can be shown. [Figure 21] This figure shows an example of how a 2D code displayed by a display device can be shown when the camera in the imaging device is a hemispherical camera. [Figure 22] This figure shows an example of how a display device can show a barcode. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings.

[0010] 〔Overall Configuration〕 FIG. 1 is an overall configuration diagram of a communication system according to an embodiment.

[0011] As shown in FIG. 1, the communication system 1 of this embodiment is constructed by a display device 2, a communication terminal 5, and a photographing device 6 installed at the first base (base α), a communication control server 7 on the cloud, and a communication terminal 10 installed at the second base (base β). Further, the communication terminal 5 is a PC (Personal Computer), a video conference terminal, etc., and can communicate with the communication control server 7 via a communication network 100 such as the Internet. Similarly, the communication terminal 10 is a PC, a video conference terminal, etc., and can communicate with the communication control server 7 via the communication network 100. Thereby, the communication terminal 5 and the communication terminal 10 can communicate via the communication control server.

[0012] Note that on the second base β side, the display device 2 and the photographing device 6 may be provided in the same manner as on the first base α side.

[0013] The display device 2 is an electronic blackboard, but may be merely a display.

[0014] Also, in FIG. 1, two bases are shown, but it is also possible to communicate with three or more bases. In this case, the number of communication terminals increases according to the number of bases.

[0015] The photographing device 6 transmits video data obtained by photographing the surroundings and audio data obtained by collecting the surrounding sounds to the communication terminal 5. This transmission method may be wired or wireless.

[0016] Also, at base α, the communication terminal 5 acquires the video data and audio data output from the photographing device 6 and transmits them to the communication control server 7, and the communication control server 7 transfers the video data and audio data to the communication terminal 10 at the destination base β. Similarly, at base β, the communication terminal 10 acquires video data and audio data from the surroundings and transmits them to the communication control server 7, and the communication control server 7 transfers the video data and audio data to the communication terminal 5 at the destination base α.

[0017] [Usage Image] Figure 2 is an illustrative diagram showing the use of the imaging device at an arbitrary location.

[0018] As shown in Figure 2, for example, four people are participating in a video broadcast of a meeting at base α, and a camera 6 is set up on the desk 110. The camera 6 is equipped with a fisheye lens FL, which will be described later, and is capable of capturing a 360° view of the surroundings. Documents (shared screen) are displayed on the display device 2. In this case, the communication terminal 5 may be the PC of one of the participants shown in Figure 2 (for example, the organizer).

[0019] [Shooting range of the imaging device] Next, the shooting range of the imaging device 6 will be explained using Figures 3 and 4. In Figure 3, (a) is a plan view of the fisheye lens, and (b) is a conceptual diagram of a fisheye image. Figure 4 shows the shooting range (vertical direction) of the imaging device.

[0020] As shown in Figure 3(a), the lens of the imaging device 6 is a fisheye lens FL with a field of view of 180° or more. Also, as shown in Figure 3(b), the center of the fisheye image (video) is considered an invalid region, and the donut-shaped region is considered an effective region.

[0021] The target area of ​​the donut-shaped region, when viewed from the side, is as shown in Figure 4. The image in the direction of the ceiling is cut off, and the image in the direction of the floor is outside the imaging range. Nevertheless, since the area where people are present can be covered, the head (or face) and chest of participants who are using a table for a meeting can be captured.

[0022] [Hardware configuration] Next, the electrical hardware configurations of the display device 2, communication terminals 5 and 10, imaging device 6, and communication control server 7 will be described using Figures 5 to 7.

[0023] <Display device hardware configuration> First, we will explain the electrical hardware configuration of the display device 2 using Figure 5. Figure 5 is an electrical hardware configuration diagram of the display device.

[0024] As shown in Figure 5, Figure 2 is a hardware configuration diagram of the display device. As shown in Figure 2, the display device 2 is equipped with a CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 203, SSD (Solid State Drive) 204, network I / F 205, and external device connection I / F (Interface) 206.

[0025] Of these, the CPU 201 controls the operation of the entire display device 2. The ROM 202 stores programs used to drive the CPU 201, such as the CPU 201 and the IPL (Initial Program Loader). The RAM 203 is used as the work area for the CPU 201. The SSD 204 stores various data, such as programs for the display device. The network controller 205 controls communication with the communication network 100. The external device connection I / F 206 is an interface for connecting various external devices. In this case, external devices include, for example, a USB (Universal Serial Bus) memory 230 and external devices (microphone 240, speaker 250, camera 260).

[0026] The display device 2 also includes a capture device 211, a GPU 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a short-range communication circuit 219, an antenna 219a for the short-range communication circuit 219, a power switch 222, selector switches 223, and a display 280. Note that the display 280 is an example of a display unit. The display unit includes a screen projected by a projector (including the object to be displayed for projection mapping).

[0027] Of these, the capture device 211 displays video information on the display 280 as a still image or video. The GPU (Graphics Processing Unit) 212 is a semiconductor chip specializing in graphics. The display controller 213 controls and manages screen display in order to output the output image from the GPU 212 to the display 280, etc. The contact sensor 214 detects when an electronic pen 290 or the user's hand H, etc., touches the display 280. The sensor controller 215 controls the processing of the contact sensor 214. The contact sensor 214 performs coordinate input and coordinate detection using an infrared blocking method. This method of coordinate input and coordinate detection involves two light-emitting and receiving devices installed at both upper ends of the display 280 emitting multiple infrared rays parallel to the display 280, and receiving the light that is reflected by reflective members provided around the display 280 and returns along the same optical path as the light emitted by the light-receiving element. The contact sensor 214 outputs the infrared ID emitted by two light-receiving devices that are blocked by an object to the sensor controller 215, which then identifies the coordinate position of the object's contact. The electronic pen controller 216 communicates with the electronic pen 290 to determine whether the pen tip or the pen end has touched the display 280. The near-field communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for turning the power of the display device 2 ON / OFF. The selection switches 223 are a group of switches for adjusting, for example, the brightness and color of the display 280.

[0028] Furthermore, the display device 2 is equipped with a bus line 210. The bus line 210 is an address bus, data bus, etc., for electrically connecting each component, such as the CPU 201 shown in Figure 2.

[0029] Furthermore, the contact sensor 214 is not limited to the infrared blocking method. Various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting changes in capacitance, a resistive touch panel that identifies the contact position by voltage changes between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by contact between an object and the display. In addition, the electronic pen controller 216 may be configured to determine whether or not there is touch not only at the tip and end of the electronic pen 290, but also at the part of the electronic pen 290 held by the user or other parts of the electronic pen.

[0030] <Hardware configuration of communication terminals and communication control servers> Next, the electrical hardware configuration of the communication terminal 5 will be explained using Figure 6. Figure 5 is an electrical hardware configuration diagram of the display device. As shown in Figure 5, the communication terminal 5 is built by a computer and includes a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0031] Of these, the CPU 501 controls the operation of the entire communication terminal 5. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, the display device 2, a display device, a USB (Universal Serial Bus) memory, and a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 6.

[0032] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium 513 may be a CD, DVD, Blu-ray (registered trademark), etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0033] Since the communication control server 7 and the communication terminal 10 have the same configuration as the communication terminal 5, their descriptions will be omitted.

[0034] <Hardware configuration of the imaging device> Next, we will explain the electrical hardware configuration of the imaging device using Figure 7. Figure 7 is a diagram of the electrical hardware configuration of the imaging device.

[0035] As shown in Figure 7, in the following, the shooting device 6 is a device that uses an image sensor to capture 360° video of the surroundings at a predetermined height, but there may be one or more image sensors. Furthermore, it does not necessarily have to be a dedicated device; a 360° video capture unit can be attached to a PC, digital camera, smartphone, etc., to have substantially the same function.

[0036] As shown in Figure 7, the imaging device 6 includes an imaging unit 601, an image processing unit 604, an imaging control unit 605, a microphone 608, a sound processing unit 609, a CPU (Central Processing Unit) 611, a ROM (Read Only Memory) 612, an SRAM (Static Random Access Memory) 613, a DRAM (Dynamic Random Access Memory) 614, an operation unit 615, an external device connection interface 616, a communication unit 617, an antenna 617a, a sound sensor 618, and a speaker 619.

[0037] Of these, the imaging unit 601 includes a wide-angle lens (a so-called fisheye lens) 602 with a 360° field of view for forming a hemispherical image, and an image sensor 603 provided corresponding to each wide-angle lens. The image sensor 603 has an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts the optical image from the fisheye lens 602 into electrical signal image data and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for this image sensor, and a group of registers in which various commands and parameters necessary for the operation of this image sensor are set. Note that the imaging unit 601 may also be a 360° camera, and is an example of an imaging means capable of imaging 360° around the imaging device 6.

[0038] Each image sensor 603 of the imaging unit 601 is connected to the image processing unit 604 via a parallel I / F bus. On the other hand, the image sensor 603 of the imaging unit 601 is connected to the imaging control unit 605 via a serial I / F bus (such as an I2C bus). The image processing unit 604, the imaging control unit 605, and the sound processing unit 609 are connected to the CPU 611 via bus 610. Furthermore, ROM 612, SRAM 613, DRAM 614, operation unit 615, external device connection I / F 616, communication unit 617, sound sensor 618, and speaker 619 are also connected to bus 610.

[0039] The image processing unit 604 receives image data output from the image sensor 603 via a parallel I / F bus, performs predetermined processing on the image data, and creates wide-field images and speaker images from the fisheye image. A "wide-field image" is an image with a wider field of view than a typical image obtained by taking a picture with a single-lens reflex camera, etc., and is an image with a field of view of 180° or more. Wide-field images include omnidirectional images, hemispherical images, 3D panoramic images, 2D panoramic images, VR images, etc.

[0040] Furthermore, the image processing unit 604 combines the wide-field image and the speaker image, etc., to output a single video.

[0041] The imaging control unit 605 generally uses the I2C bus to set commands and other information in the registers of the image sensor 603, with the imaging control unit 605 acting as the master device and the image sensor 603 as the slave device. It receives necessary commands and other information from the CPU 611. The imaging control unit 605 also uses the I2C bus to acquire status data and other information from the registers of the image sensor 603 and send it to the CPU 611.

[0042] Furthermore, the imaging control unit 605 instructs the image sensor 603 to output image data when the imaging start button on the operation unit 615 is pressed or when an imaging start instruction is received from the PC. Depending on the imaging device 6, there may also be functions to support preview display on a display (for example, the display of a PC or smartphone) or video display. In this case, the image data output from the image sensor 603 is performed continuously at a predetermined frame rate (frames / second).

[0043] Furthermore, as will be described later, the imaging control unit 605 also functions as a synchronization control means that works in cooperation with the CPU 611 to synchronize the output timing of image data from the image sensor 603. In this embodiment, the imaging device 6 is not provided with a display, but a display unit may be provided.

[0044] Microphone 608 converts sound into sound (signal) data. The sound processing unit 609 takes in the audio data output from microphones 608a, 608b, and 608c through the I / F bus, mixes this audio data, and performs predetermined processing. The sound processing unit 609 also determines the direction of the sound source (speaker) from the audio level (volume) input from microphones 608a, 608b, and 608c.

[0045] Speaker 619 converts the input audio data into sound. Microphone 608 and speaker 619 are examples of input / output means. The input / output means do not need to be combined into a single configuration; the input means and output means may be configured separately.

[0046] The CPU 611 controls the overall operation of the imaging device 6 and executes necessary processing. The ROM 612 stores various programs for operating the imaging device 6. The SRAM 613 and DRAM 614 are work memories that store programs executed by the CPU 611 and data in progress. In particular, the DRAM 614 stores image data in progress and processed equirectangular projection image data from the image processing unit 604.

[0047] The control unit 615 is a collective term for the control buttons, such as the image capture start button 615a. By operating the control unit 615, the user can start imaging and recording, turn the power on / off, establish a communication connection, and input various settings such as imaging modes and imaging conditions.

[0048] The external device connection interface 616 is an interface for connecting various external devices. In this case, external devices include, for example, the display device 2, a PC (Personal Computer), a display, a projector, etc. The external device connection interface 616 may also be equipped with, for example, a USB terminal, an HDMI® terminal, etc. Video data and image data stored in the DRAM 614 are transmitted to an external terminal via this external device connection interface 616 or recorded on external media. Alternatively, multiple external device connection interfaces 616 may be used to transmit image information captured by the imaging device 6 to a PC via USB for recording, while simultaneously acquiring video (for example, image information to be displayed in a remote conferencing application) from the PC to the imaging device 6, and then transmitting it from the imaging device 6 to other external devices (display device 2, display, projector, etc.) via HDMI for display.

[0049] The communication unit 617 may communicate with a cloud server via the internet using wireless communication technology such as Wi-Fi through the antenna 617a provided on the imaging device 6, and transmit the stored video data and image data to the cloud server. The communication unit 617 may also be able to communicate with nearby devices using short-range wireless communication technology such as BLE (Bluetooth Low Energy; registered trademark) or NFC.

[0050] The sound sensor 618 is a sensor that acquires 360° sound information in order to determine from which direction loud sound is input within the 360° surrounding (horizontal plane) of the imaging device 6. Based on the input 360° sound parameters, the sound processing unit 609 identifies the direction with the strongest sound and outputs the sound input direction within the 360°.

[0051] Furthermore, other sensors (such as compass / accelerometers or GPS) can be used to calculate orientation, position, angle, acceleration, etc., and these can be used for image correction or to add location information.

[0052] The image processing unit 604 also creates wide-field images in the following way: The CPU 611 performs predetermined camera image processing, such as Bayer conversion (RGB interpolation processing), on the RAW data input from the image sensor that inputs spherical images to create a fisheye image (curved image). Furthermore, it performs DeWarp processing (distortion correction processing) on ​​the created fisheye image (curved image) to create a wide-field image (planar image) that captures 360° around the shooting device 6.

[0053] The CPU 611 creates speaker images in the following way: The CPU 611 creates speaker images by extracting the speaker from a wide-field image (a flat image) that captures the surrounding 360°. The CPU 611 uses the direction of the audio input, identified from the 360° output by the audio sensor 618 and the sound processing unit 609, as the direction of the speaker, and extracts the speaker image from the wide-field image. In this case, the method for extracting a person's image from the direction of the audio input involves cutting out a 30° section centered on the audio direction identified from the 360° section, and then performing face detection within that section to extract the image. The CPU 611 further identifies speaker images for a specific number of people (e.g., 3 people) who have spoken most recently from among the extracted speaker images.

[0054] [Functional configuration of the communication system] Next, using Figure 8, we will explain the functional configuration of the main devices (terminals) in communication system 1: the display device 2, the communication terminal 5, and the imaging device 6. Figure 8 is a functional configuration diagram of the display device, the communication terminal, and the imaging device.

[0055] <Functional Configuration of Display Device> As shown in Figure 8, the display device 2 has a transmitting / receiving unit 21, a receiving unit 22, and a display control unit 24. Each of these units performs functions implemented by instructions from the CPU 201 and / or GPU 212 according to a program stored in the RAM 203, etc. The display device 2 also has a storage unit 29 constructed from the RAM 203 and / or SSD 204.

[0056] Of these, the transmitting / receiving unit 21 transmits and receives various types of data (information) with the communication terminal 5. The transmitting / receiving unit 21 also functions as a code information receiving unit because it receives a matrix-type two-dimensional code (hereinafter referred to as "two-dimensional code") from the communication terminal 5.

[0057] The reception unit 22 accepts various selections or inputs from participants (users).

[0058] The display control unit 24 causes the display unit, the display 280, to display various images or information. The display control unit 24 also includes a web browser function.

[0059] The memory unit 29 stores a two-dimensional code. This two-dimensional code is identification information for identifying the display device 2 and is an example of code information. The code information also includes barcodes. Furthermore, the code information also includes specific shapes, specific icons, specific characters, etc. In addition, this identification information includes predetermined content information indicating predetermined content (for example, the name of the destination display device). The two-dimensional code also includes instruction information indicating that a rectangular area should be detected.

[0060] It should be noted that the storage unit 29 does not necessarily have to store the two-dimensional code. For example, if the manufacturer or seller of the display device 2 and the imaging device 6 are the same, it is possible to store the two-dimensional code in the storage unit 29 in advance. However, if they are different, the storage unit 29 may not store the two-dimensional code.

[0061] <Functional Configuration of Communication Terminals> As shown in Figure 8, the communication terminal 5 has a transmitting / receiving unit 51, a receiving unit 52, and a display control unit 54. Each of these units performs functions that are implemented by instructions from the CPU 501 according to a program stored in the RAM 503 or the like.

[0062] Of these, the transmitting / receiving unit 51 transmits and receives various types of data (information) with the display device 2 or the imaging device 6. The transmitting / receiving unit 51 also acts as a code information transmission unit to transmit a two-dimensional code to the display device 2. Furthermore, the transmitting / receiving unit 51 transmits and receives data (information) with the communication terminal 10, which is the recipient of the video distribution, via the communication network 100 and the communication control server 7.

[0063] The reception unit 52 accepts various selections or inputs from participants (users).

[0064] The display control unit 54 causes the display 506 to display various images or information. The display control unit 54 also includes a web browser function.

[0065] The memory unit 59 may store a two-dimensional code. This two-dimensional code is identification information for identifying the destination display device 2. The two-dimensional code stored in the memory unit 59 is generated by a dedicated application installed on the communication terminal 5. This two-dimensional code is used when the memory unit 59 of the display device 2 does not store a two-dimensional code, so the information contained in both two-dimensional codes (the predetermined content information and instruction information mentioned above) is basically the same. If the memory unit 29 of the display device 2 does not store a two-dimensional code, the transmitting / receiving unit 51 of the communication terminal 5 can transmit the two-dimensional code stored in the memory unit 59 to the transmitting / receiving unit 21.

[0066] Alternatively, instead of using the display device 2, a projector can be connected to the communication terminal 5, and a 2D code can be projected and displayed on a screen or the like. In this case, video distribution of meetings, etc., will be carried out using materials displayed on the screen or the like.

[0067] <Functional configuration of the imaging device> As shown in Figure 8, the imaging device 6 includes a transmission unit 61, a reception unit 62, an imaging unit 63, a wide-field image generation unit 64, a code information detection unit 65, an instruction information detection unit 66, a rectangular area detection unit 67, a composite image generation unit 68, and a predetermined content information detection unit 69. Each of these units is a function realized by instructions from the CPU 611 according to a program stored in the SRAM 613 or the like.

[0068] Of these, the transmission unit 61 transmits various data (information) to the display device 2. For example, the transmission unit 61 transmits a predetermined composite image (or specific composite image) generated by the composite image generation unit 68 to the communication terminal 5 as an external device. The predetermined composite image and specific composite image will be described later.

[0069] The reception unit 62 receives various selections or inputs from the user.

[0070] The imaging unit 63 captures images of subjects, landscapes, etc., to obtain fisheye image data.

[0071] The wide-field image generation unit 64 generates a wide-field image from the fisheye image captured by the imaging unit 63.

[0072] The code information detection unit 65 detects two-dimensional codes contained in the wide-field image.

[0073] When the code information detection unit 65 detects a two-dimensional code, the instruction information detection unit 66 detects instruction information indicating that a rectangular area should be detected from the two-dimensional code.

[0074] When instruction information is detected by the instruction information detection unit 66, the rectangular area detection unit 67 detects a predetermined rectangular area outside the 2D code 8 within the wide-field image. Since the frame of the display 280 is rectangular, the rectangular area detection unit 67 effectively detects the display 280. Note that "outside the 2D code 8" refers to the area outside the 2D code 8, including the 2D code 8, as shown in Figure 11.

[0075] When the rectangular region detection unit 67 detects a predetermined rectangular region, the composite image generation unit 68 generates a predetermined composite image by combining a predetermined rectangular region image, which represents the predetermined rectangular region within the wide-field image, with the wide-field image. The composite image generation unit 68 also fits the predetermined rectangular region image into a predetermined fixed display area 440 (see Figure 13) within the predetermined composite image. Even if the code information detection unit 65 no longer detects a 2D code after the composite image generation unit 68 has generated the predetermined composite image, the composite image generation unit 68 continues to generate the predetermined composite image by combining the predetermined rectangular region image and the wide-field image.

[0076] On the other hand, if instruction information is detected by the instruction information detection unit 66, but a predetermined rectangular area is not detected by the rectangular area detection unit 67, the composite image generation unit 68 generates a specific composite image by combining a specific rectangular area image, which indicates a specific rectangular area calculated based on the position and size of the 2D code in the wide-field image, with the wide-field image. In this case as well, the composite image generation unit 68 fits the specific rectangular area image into a predetermined fixed display area 440 (see Figure 13) in the specific composite image. Furthermore, even if the 2D code is no longer detected by the code information detection unit 65 after the composite image generation unit 68 has generated the specific composite image, the composite image generation unit 68 continues to generate a specific composite image by combining the specific rectangular area image and the wide-field image.

[0077] When a two-dimensional code is detected by the code information detection unit 65, the predetermined content information detection unit 69 detects predetermined content information from the two-dimensional code. The predetermined content information includes display device identification information for identifying the display device 2 used in conjunction with the imaging device 6. The display device identification information is, for example, information indicating "Electronic Whiteboard No. 1" displayed in the display area 453 of Figure 13. When the predetermined content information is detected by the predetermined content information detection unit 69, the transmission unit 61 transmits the predetermined content information to the communication terminal 5 as an external device.

[0078] [Communication system processing] Next, the processing of the communication system 1, including the image processing method by the imaging device 6, will be explained using Figures 9 to 13. Figures 9 and 10 are flowcharts showing the process of specifying a fixed display area by the imaging device. Figures 11 and 12 show a 2D code displayed on the display of the display device.

[0079] First, assuming that video distribution will be performed from base α to base β as shown in Figure 2, the display device 2 at base α will display a 2D code 8 on the display 280, as shown in Figure 11.

[0080] There are two ways to display the 2D code 8. In the first display method, the display control unit 24 of the display device 2 reads the 2D code data stored in the memory unit 29 and displays the 2D code 8 on the display 280. In the second display method, if the display device 2 does not store the 2D code in its own device, the transmitting / receiving unit 51 of the communication terminal 5 reads the 2D code data from the memory unit 59 and transmits it to the display device 2, and the display control unit 24 of the display device 2 displays the 2D code 8 on the display 280 based on the 2D code data received by the transmitting / receiving unit 21. Note that if neither the display device 2 nor the communication terminal 5 stores the 2D code data, the display device 2 cannot display the 2D code 8.

[0081] S11: In Figure 9, first, the wide-field image generation unit 64 generates a wide-field image from the fisheye image captured by the imaging unit 63.

[0082] S12: The code information detection unit 65 detects a two-dimensional code contained in the wide-field image. If a two-dimensional code is present, the process proceeds to S13; otherwise, the process shown in Figure 9 ends.

[0083] S13: If a two-dimensional code is detected by the code information detection unit 65 (S12; YES), the instruction information detection unit 66 detects instruction information indicating that a rectangular area should be detected from the two-dimensional code. If the two-dimensional code contains instruction information, the process proceeds to S14; if the two-dimensional code does not contain instruction information, the process shown in Figure 9 ends.

[0084] S14: If instruction information is detected by the instruction information detection unit 66 (S13; YES), the rectangular area detection unit 67 detects a predetermined rectangular area outside the 2D code within the wide-field image. If a rectangular area is found, proceed to S15; otherwise, proceed to S16.

[0085] S15: If a predetermined rectangular region is detected by the rectangular region detection unit 67 (S14; YES), the composite image generation unit 68 generates a predetermined composite image by combining at least a predetermined rectangular region image showing the predetermined rectangular region in the wide-field image with the wide-field image. In this case, the composite image generation unit 68 fits the predetermined rectangular region image into the fixed display area 440 (see Figure 13).

[0086] Figure 13 shows the screen displayed by the communication terminal 5 at base α. As described above, the communication terminal 5 may be any of the participants' PCs shown in Figure 2, or it may be a dedicated PC. The predetermined composite image generated by the composite image generation unit 68 is displayed in the display area 410 of the screen 400 displayed by the communication terminal 5, which contains the image to be transmitted to the recipient of the video distribution from base β. This allows the participants at base α to understand the content of the image transmitted to base β.

[0087] Furthermore, the display area 410 includes a wide-field image display area 420, display areas 431-433 for people in the wide-field image displayed in the order of each utterance, and a fixed display area 440 for a predetermined rectangular area image. In display areas 431-433, the most recent speaker is displayed in display area 431, the previous speaker in display area 432, and the speaker two utterances prior in display area 433. Each time a new speaker appears, the displayed person moves to the display area one position lower. The other icons 451 and display areas 452 and 453 of screen 400 will be described later.

[0088] Furthermore, even if the code information detection unit 65 no longer detects a 2D code after the composite image generation unit 68 has generated a predetermined composite image, the composite image generation unit 68 continues to generate the predetermined composite image by combining the predetermined rectangular area image and the wide-field image. Also, if the code information detection unit 65 detects a 2D code again after it has stopped detecting one, the processing from S12 onwards is restarted. This allows for readjustment of the predetermined rectangular area image.

[0089] S16: If the rectangular area detection unit 67 does not detect a predetermined rectangular area (S14; NO), the composite image generation unit 68 generates a specific composite image by combining a specific rectangular area image, which shows a specific rectangular area calculated based on the position and size of the 2D code within the wide-field image, with the wide-field image. For example, as shown in Figure 12, there are cases where the rectangular area detection unit 67 cannot detect the frame (rectangular area) of the display 280 of the display device 2, such as when a participant displays a 2D code at a predetermined position on the screen 11 from a projector connected to the communication terminal 5 without using the display device 2. In such cases, the composite image generation unit 68 calculates a specific rectangular area 12 related to the specific rectangular area image to be displayed in the fixed display area 440 in Figure 13, based on the position of the displayed 2D code and a predetermined size of the 2D code. For example, the composite image generation unit 68 calculates a specific rectangular area 12 with a width six times the width of the 2D code 8 and a height 3.5 times the height of the 2D code 8. As a result, the composite image generation unit 68 can estimate a specific rectangular region 12 related to a specific rectangular region image to be displayed in the fixed display area 440, and thus fit the specific rectangular region image into the fixed display area 440. Note that the aspect ratio of width to height can be 4:3, 16:9, etc., in addition to 6:3.5.

[0090] Furthermore, even if the code information detection unit 65 no longer detects a 2D code after the composite image generation unit 68 has generated a specific composite image, the composite image generation unit 68 continues to generate a specific composite image by combining a specific rectangular region image and a wide-field image. Also, if the code information detection unit 65 detects a 2D code again after it has stopped detecting one, the processing from S12 onwards is restarted. This allows for readjustment of the specific rectangular region image.

[0091] Furthermore, the imaging device 6 performs the process shown in Figure 10 in parallel with the process shown in Figure 9.

[0092] S21: In the process of S12 in Figure 9, if a two-dimensional code is detected by the code information detection unit 65 (S12; YES), predetermined content information indicating predetermined content is detected from the two-dimensional code. The display device identification information is, as described above, information indicating, for example, "Electronic Whiteboard No. 1" displayed in the display area 453 of Figure 13. If the predetermined content information is included in the two-dimensional code, the process proceeds to S22; if the predetermined content information is not included in the two-dimensional code, the process in Figure 10 ends.

[0093] S22: If the predetermined content information is detected by the predetermined content information detection unit 69 (S21; YES), the transmission unit 61 transmits the predetermined content information to the communication terminal 5 as an external device. The screen 400 displayed by the communication terminal 5 includes, as shown in Figure 13, an icon 451 for reading meeting information from the scheduler, indicating the start date and time, end date and time, and location of the meeting, a display area 452 for displaying the read meeting information, and the aforementioned display area 453, in addition to the display area 410. As a result, participants at site α can grasp almost all of the information related to the video distribution of the meeting by looking at the screen 400 in Figure 13. Furthermore, the screen 400 shown in Figure 13 is recorded by the communication terminal 5 and also serves as meeting minutes.

[0094] This concludes the explanation of the communication system 1, including the image processing method by the imaging device 6.

[0095] [Other Embodiments] The following describes a modified example in which the display device 2 displays the two-dimensional code 8.

[0096] Figure 14 shows an example of a two-dimensional code 8 displayed by the display device 2. In Figure 14, one large two-dimensional code 8 is displayed. A larger two-dimensional code 8 makes it easier for the imaging device 6 to detect it. The method of displaying the two-dimensional code 8 will be explained below.

[0097] Figure 15 shows an example of the position of the 2D code 8 displayed by the display device 2. As described above, the 2D code 8 contains identification information, etc. On the desk 110 on which the imaging device 6 is placed, there is an obstacle 3 between the imaging device 6 and the display device 2. Note that the imaging device 6 does not need to be a 360-degree camera for the detection of the 2D code 8. The display device 2 displays the identification information above the center line 320 of the display screen of the display device 2 in the vertical direction. This makes it easier for the imaging device 6 to capture the identification information even if there is an obstacle 3 between the imaging device 6 and the display device 2.

[0098] As shown in Figure 16, the display device 2 may also display the center of the 2D code 8 above the center line 320 of the display screen of the display device 2 in the vertical direction.

[0099] Furthermore, as shown in Figure 17, the display device 2 may move the 2D code 8 over time. In Figure 17, the 2D code 8 moves from left to right. The display device 2 may move while displaying the 2D code 8, or it may display, erase, change location, and redisplay. This makes it easier for the imaging device 6 to capture equipment identification information even if the position of the obstacle 4 is uncertain. Also, even if the display device 2 displays the 2D code 8 in a smaller size to reduce the feeling of pressure on the user, it still makes it easier for the imaging device 6 to capture the image.

[0100] Furthermore, the display device 2 may change the size of the two-dimensional code 8 while moving it.

[0101] Furthermore, as shown in Figure 18, the display device 2 may display multiple 2D codes 8 simultaneously. This makes it easier for the imaging device 6 to capture the remaining 2D codes 8 even if some of them are hidden by obstacles 3 and 4. Note that the multiple 2D codes 8 may all contain the same information, or they may contain different information.

[0102] Furthermore, as shown in Figure 19, the display device 2 may also display a two-dimensional code 8 next to (in close proximity to) the menu 71. The menu 71 is positioned vertically along the right edge, and is displayed above the center line 320 of the display screen of the display device 2 in the vertical direction, as in Figure 15.

[0103] By displaying QR code 8 near menu 71, users are less likely to feel confused. Users can also utilize the screen more effectively.

[0104] Furthermore, as shown in Figure 20, the display device 2 may display a two-dimensional code 8 within the menu 71. Compared to Figure 19, this further reduces the feeling of being cramped and allows the user to use the screen more effectively.

[0105] Figure 21 illustrates the case where the camera in the imaging device 6 is a hemispherical camera. By imaging with a hemispherical camera that has a wide horizontal field of view, the detection of the 2D code 8 can be made easier.

[0106] Note that barcodes, rather than 2D codes (8), can be displayed in the same way as in Figures 14 to 21. Figure 22 shows an example of a barcode (9).

[0107] Barcode 9 is less robust to tilt and other factors than 2D code 8. Therefore, the code information detection unit 65 cuts out the black and white pattern and corrects the skew angle and pitch angle. The code information detection unit 65 enhances the edges of the black bars. The code information detection unit 65 performs pattern matching on the cut-out image with the pattern from the start character to the stop character at the right end that is registered as the pattern of barcode 9, and detects the barcode 9 displayed on the display device 2.

[0108] Furthermore, while Figures 14 to 22 illustrate an example where a two-dimensional code 8 or barcode 9 is displayed, the display device 2 may display identification information such as letters or numbers, and the code information detection unit 65 may recognize the identification information using OCR (Optical Character Recognition / Reader) processing.

[0109] [Main effects of the embodiment] As described above, according to this embodiment, the imaging device 6 automatically detects that the display 280 of the display device 2, which is mainly treated as an area of ​​interest, is rectangular, and enlarges the area of ​​interest within the wide-field image to set it as a fixed display area 440. This has the effect of reducing the workload of the user at the video distribution source.

[0110] [Other Embodiments] In the above embodiment, when the instruction information detection unit 66 detects instruction information indicating the detection of a rectangular area, the rectangular area detection unit 67 detects a predetermined rectangular area outside the code information within the wide-field image. However, it is not limited to this. For example, the code information does not include "instruction information indicating the detection of a rectangular area," and the code information indicates specific information predetermined (such as the name of the manufacturing or sales company, or the product name). When the code information detection unit 65 detects the code information, the rectangular area detection unit 67 may detect a rectangular area around the code information within the wide-field image.

[0111] Furthermore, the rectangular region detection unit 67 may not only detect rectangular regions but also generate rectangular region images based on the detected rectangular regions. In this case, the rectangular region detection unit 67 also functions as a rectangular region generation unit.

[0112] Furthermore, other embodiments can be summarized as follows.

[0113] <Additional notes> A photographic device that outputs an image, A wide-field image generation unit generates a wide-field image from a fisheye image obtained by imaging, A code information detection unit for detecting code information contained in the wide-field image, When the code information is detected by the code information detection unit, the rectangular region generation unit detects a rectangular region outside the code information within the wide-field image and generates a rectangular region image. A composite image generation unit generates a predetermined composite image by combining the rectangular region image and the wide-field image, A transmission unit that transmits the predetermined composite image generated by the composite image generation unit to an external source, A photographic device characterized by having the following features.

[0114] [Other application examples] Although embodiments of the present invention have been described above, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0115] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0116] Furthermore, each of the parts 61-63 and 65-68 of the imaging device 6 can also be implemented by a computer through the execution of a program. This program can also be recorded on a (non-temporary) recording medium and provided to the computer. [Explanation of Symbols]

[0117] 1. Communication System 2 Display device 5. Communication terminals 6. Imaging device 21 Transmitting / receiving unit (an example of a code information receiving unit) 22 Reception Department 24 Display Control Unit 29 Memory section 51 Transmitting and Receiving Unit (An example of a code information transmission unit) 52 Reception Department 54 Display Control Unit 59 Memory section 61 Transmitter 62 Reception Department 63 Imaging Unit 64 Wide-field image generation unit 65 Code Information Detection Unit 66 Instruction Information Detection Unit 67 Rectangular area detection unit 68 Composite Image Generation Unit 69 Predetermined Content Information Detection Unit [Prior art documents] [Non-patent literature]

[0118] [Non-Patent Document 1] Meeting Owl Pro User Manual<https: / / www.sourcenext.com / produce / app / manual / meetingowl / JP / index.html#! / 06_02>

Claims

1. A shooting device that outputs images for video distribution, A wide-field image generation unit generates a wide-field image from a fisheye image obtained by imaging, A code information detection unit for detecting code information contained in the wide-field image, When the code information is detected by the code information detection unit, the instruction information detection unit detects instruction information indicating that a rectangular area should be detected from the code information. When the instruction information is detected by the instruction information detection unit, a rectangular area detection unit detects a predetermined rectangular area outside the code information within the wide-field image, When the rectangular region detection unit detects the predetermined rectangular region, the composite image generation unit generates a predetermined composite image by combining the predetermined rectangular region image showing the predetermined rectangular region in the wide-field image with the wide-field image. A transmission unit that transmits the predetermined composite image generated by the composite image generation unit to an external source, A photographic device characterized by having the following features.

2. The imaging apparatus according to claim 1, characterized in that the composite image generation unit fits the predetermined rectangular area image into a predetermined fixed display area within the predetermined composite image.

3. The imaging apparatus according to claim 1, characterized in that even if the code information is no longer detected by the code information detection unit after the composite image generation unit has generated the predetermined composite image, the composite image generation unit continues to generate the predetermined composite image obtained by combining the predetermined rectangular area image and the wide field image.

4. A photographic apparatus according to any one of claims 1 to 3, When the code information is detected by the code information detection unit, the unit has a predetermined content information detection unit that detects predetermined content information from the code information, The imaging device is characterized in that, when the predetermined content information is detected by the predetermined content information detection unit, the transmission unit transmits the predetermined content information to an external source.

5. The imaging device according to claim 4, characterized in that the predetermined content information includes display device identification information for identifying a display device used in conjunction with the imaging device.

6. The imaging apparatus according to claim 1, wherein if the instruction information is detected by the instruction information detection unit but the predetermined rectangular area is not detected by the rectangular area detection unit, the composite image generation unit generates a specific composite image by combining the wide-field image with a specific rectangular area image that indicates a specific rectangular area calculated based on the position and size of the code information within the wide-field image.

7. The imaging apparatus according to claim 6, characterized in that the composite image generation unit fits the specific rectangular area image into a predetermined fixed display area within the specific composite image.

8. The imaging apparatus according to claim 6, characterized in that even if the code information is no longer detected by the code information detection unit after the composite image generation unit has generated the specific composite image, the composite image generation unit continues to generate the specific composite image obtained by combining the specific rectangular region image and the wide field image.

9. The imaging device according to claim 1 or 6, A rectangular display device, A communication system having, The aforementioned display device is A storage unit for storing the aforementioned code information, A display control unit that displays the code information on the display unit, A communication system characterized by having the following features.

10. The imaging device according to claim 1 or 6, Communication terminal and A rectangular display device, A communication system having, The aforementioned communication terminal is A storage unit for storing the aforementioned code information, A code information transmission unit that transmits the code information to the display device, It has, The aforementioned display device is A code information receiving unit that receives the aforementioned code information, A display control unit that displays the code information on the display unit, A communication system characterized by having the following features.

11. An image processing method performed by a shooting device that outputs images for video distribution, A wide-field image generation process that generates a wide-field image from a fisheye image obtained by imaging, A code information detection process for detecting code information contained in the wide-field image, An instruction information detection process that detects instruction information indicating the detection of a rectangular region from the aforementioned code information, Based on the instruction information detection process, a rectangular region detection process is performed to detect a predetermined rectangular region outside the code information within the wide-field image, A composite image generation process that generates a predetermined composite image by combining a predetermined rectangular region image showing the predetermined rectangular region within the wide-field image with the wide-field image, A transmission process for transmitting the predetermined composite image generated by the composite image generation process to an external source, An image processing method characterized by performing the following.

12. A program that causes a computer to perform the method described in claim 11.

Images