Image synthesis device, image synthesis method, and computer program

The image synthesis device addresses the challenge of combining images from multiple sources by using template-defined regions and PTZ control, ensuring natural appearances of multiple people in the composite data, thus overcoming the limitations of conventional methods.

WO2024241720A1PCT designated stage expired Publication Date: 2024-11-28PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/013818
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-04-03
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Conventional image compositing methods fail to naturally combine images of multiple people captured by different imaging devices, resulting in unnatural appearances due to pose and angle discrepancies.

Method used

An image synthesis device connected to multiple imaging systems uses template data to define regions for each person, with an arithmetic circuit assigning each imaging system to a specific region and performing PTZ control to capture images that match the template, ensuring each person appears naturally in the composite data.

Benefits of technology

The solution generates composite data where multiple people appear naturally, maintaining the subtleties of their original poses and angles, creating a unified and realistic image from disparate sources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024013818_28112024_PF_FP_ABST
    Figure JP2024013818_28112024_PF_FP_ABST
Patent Text Reader

Abstract

This image synthesis device comprises an arithmetic circuit and a communication circuit, and generates synthetic data by synthesizing a plurality of pieces of image data pertaining to a plurality of people, which have been captured by a plurality of connected imaging systems. A plurality of areas in which a plurality of people are respectively placed in the synthetic data are defined in advance as templates, and the respective imaging systems are uniquely corresponded in advance to the respective areas defined as the templates. The calculation circuit makes the respective imaging systems image corresponding image data in which corresponding people appear in the corresponding areas defined as the templates, obtains the corresponding image data from the respective imaging systems via the communication circuit, and generates synthetic data by assigning a plurality of extracted images corresponding to the plurality of people respectively extracted from the plurality of pieces of image data to the plurality of areas of the template, respectively and synthesizing the plurality of extracted images.
Need to check novelty before this filing date? Find Prior Art

Description

Image synthesis device, image synthesis method, and computer program

[0001] The present disclosure relates to an image synthesis device, an image synthesis method, and a computer program that generate synthetic data by synthesizing multiple images of different people obtained by multiple image capture devices.

[0002] Images of multiple people taken separately by different image capture devices can be combined into a single image data. By viewing the combined image data, it is possible to imagine that multiple people are in the same location. Specifically, there is a method in which multiple users in remote locations can easily capture images of themselves in real time (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-123987

[0004] However, while image data obtained using conventional synthesis methods can take into account the poses in which each person is photographed, the synthesized data may sometimes appear unnatural because each person is photographed separately.

[0005] The present disclosure provides an image synthesis device, an image synthesis method, and a computer program that generate synthetic data that combines images of multiple different people obtained by multiple image capturing devices in such a way that multiple people appear naturally.

[0006] The image composition device of the present disclosure is connected to a plurality of photographing systems, includes an arithmetic circuit and a communication circuit, and generates composite data by combining a plurality of image data each of which depicts a plurality of different people, each obtained by the plurality of photographing systems. A plurality of regions in which the plurality of people are to be arranged in the composite data are defined in advance as templates, and each photographing system is uniquely associated in advance with each region defined in the template. The arithmetic circuit causes each photographing system to capture corresponding image data from a plurality of image data in which a corresponding person among the plurality of people appears in a corresponding region among the plurality of regions defined in the template, obtains the corresponding image data from each photographing system via the communication circuit, and generates composite data by assigning a plurality of extracted images corresponding to the plurality of people extracted from the plurality of image data to a plurality of regions of the template, respectively.

[0007] Such general and specific aspects may be realized by a system, a method, and a computer program, as well as a combination thereof.

[0008] The image synthesis device, image synthesis method, and computer program disclosed herein can generate synthetic data that combines images of multiple different people obtained by multiple photographing devices in such a way that the multiple people appear natural.

[0009] 1 is a block diagram showing the configuration of an image synthesis system according to a first embodiment. FIG. 2 is a block diagram showing the configuration of an imaging system that captures images with the image synthesis system of FIG. 1. FIG. 3 shows an example of template data used in the image synthesis system of FIG. 1. FIG. 4 shows an example of area data transmitted to the imaging system 3A of FIG. 1. FIG. 5 shows an example of area data transmitted to the imaging system 3B of FIG. 1. FIG. 6 shows an example of area data transmitted to the imaging system 3C of FIG. 1. FIG. 7 shows an example of image data obtained by the imaging system 3A of FIG. 1. FIG. 8 shows an example of image data obtained by the imaging system 3B of FIG. 1. FIG. 9 shows an example of image data obtained by the imaging system 3C of FIG. 1. FIG. 10 shows an example of an extracted image obtained from the image data of FIG. 5A. FIG. 11 shows an example of an extracted image obtained from the image data of FIG. 5B. FIG. 12 shows an example of an extracted image obtained from the image data of FIG. 5C. FIG. 13 shows an example of synthesized data obtained using the image data of FIGS. 6A to 6C. FIG. 14 is an example illustrating PTZ control performed by the imaging system 3A of FIG. 1. FIG. 15 is an example illustrating PTZ control following FIG. 8A. FIG. 16 is an example illustrating PTZ control following FIG. 8B. FIG. 17 is a flowchart showing an example of image synthesis processing performed in the image synthesis device of FIG. 1. FIG. 18 is a flowchart showing an example of imaging processing performed in the imaging system of FIG. 2. Fig. 13 is a block diagram showing the configuration of an image synthesis system according to a second embodiment. Fig. 14 is a block diagram showing the configuration of an imaging system that captures an image in the image synthesis system of Fig. 11. Fig. 15 is a flowchart showing an example of image synthesis processing executed by the image synthesis device of Fig. 11. Fig. 16 is a flowchart showing an example of imaging processing executed in the imaging system of Fig. 12. Fig. 17 is a block diagram showing the configuration of an image synthesis system according to a first modified example.

[0010] [Embodiments] Hereinafter, embodiments of the present disclosure will be described using the drawings, with appropriate reference to the drawings. However, in the detailed description, unnecessary parts of the description of the prior art and substantially identical configurations may be omitted. This is for the sake of simplicity. Furthermore, the following description and the accompanying drawings are disclosed to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

[0011] The image synthesis device, image synthesis method, and computer program according to the present disclosure generate synthetic data by synthesizing a plurality of images of different people obtained by a plurality of imaging systems.

[0012] [First Embodiment] <Image Synthesis System> As shown in FIG. 1, an image synthesis system 1 according to an embodiment includes an image synthesis device 2 and multiple image capture systems 3A-3C. The image synthesis device 2 and the multiple image capture systems 3A-3C are connected via a network 4. The multiple image capture systems 3A-3C are located in different locations and capture different people. The image synthesis device 2 acquires image data captured by each of the image capture systems 3A-3C, extracts person regions from each image data, and combines the images of the people captured by the different image capture systems 3A-3C to generate synthetic data. Note that the image data captured by each of the image capture systems 3A-3C may be, for example, video data capturing a situation in which people captured by the multiple image capture systems 3A-3C are having a conversation in real time via the network 4. By using the image synthesis system 1 to generate synthetic data, video data of people in different spaces can be edited into video data that appears as if multiple people are having a conversation in the same space. In this case, the image synthesis system 1 does not simply photograph each person using multiple imaging systems 3A-3C, but rather adjusts the image data so that each person appears naturally in the generated composite data. Specifically, each imaging system 3A-3C photographs each person so that they appear in the area in which they will be placed in the composite data. By photographing people in this manner using each imaging system 3A-3C, subtle distortions and angles of view that may occur in the captured images of the people are adapted to the placement of the people in the composite data. This allows the generated composite data to reflect the effects of subtle distortions and other issues that may occur when the people are actually positioned. For example, when generating composite data showing three people lined up horizontally with a gap between them, extracting the person on the left and the person on the right from the image data in the center will result in unnatural composite data. Therefore, the people are positioned in the positions in which they will appear in the composite data, and images are captured before extraction. As a result, in composite data using image data obtained by different imaging systems 3A-3C, each person appears natural and without any sense of incongruity.

[0013] Therefore, in the image synthesis system 1, template data D1 is prepared in advance. The template data D1 is data in which the area in which each person is placed in the synthesized image is specified in advance. Each of the image capture systems 3A-3C is uniquely associated in advance with a plurality of areas defined in the template data. In the image synthesis system 1, the image synthesis device 2 transmits area data indicating the area defined for each of the image capture systems 3A-3C in the template data D1 to each of the image capture systems 3A-3C. Furthermore, each of the image capture systems 3A-3C performs PTZ control in accordance with the area data indicating the area received from the image synthesis device 2, and acquires image data in which a person is captured within the area. Note that, here, the zoom control and pan / tilt control performed by the image capture systems 3A-3C are collectively referred to as PTZ control. In the image synthesis system 1 shown in FIG. 1, three image capture systems are connected to the image synthesis device 2 via the network 4, but the number of image capture systems connected to the image synthesis device 2 is not limited.

[0014] 1, the image synthesis device 2 is an information processing device including an arithmetic circuit 20, an input device 21, an output device 22, a communication circuit 23, and a storage device 24. The image synthesis device 2 acquires image data from a plurality of imaging systems 3A to 3C, and can extract areas of people from each image data to generate synthetic data.

[0015] The arithmetic circuit 20 is a controller that controls the entire image synthesis device 2. For example, the arithmetic circuit 20 reads and executes the image synthesis program P1 stored in the storage device 24 to realize various processes related to the control of image synthesis using image data acquired from each of the imaging systems 3A to 3C. The arithmetic circuit 20 may be various processors such as a CPU, an MPU, a GPU, an FPGA, a DSP, an ASIC, or a dedicated hardware circuit.

[0016] The input device 21 is input means such as operation buttons, a keyboard, a mouse, a touch panel, a microphone, etc., which are used for operations and data input. The output device 22 is output means such as a display, a speaker, etc., which are used for outputting processing results and data.

[0017] The communication circuit 23 performs data communication with the imaging systems 3A to 3C and external devices. The data communication is performed wired and / or wirelessly, for example, in accordance with known communication standards. For example, wired data communication may be performed by using a communication controller of a semiconductor integrated circuit that operates in accordance with the Ethernet (registered trademark) standard and / or the USB (registered trademark) standard as the communication circuit 33. Wireless data communication may be performed by using a communication controller of a semiconductor integrated circuit that operates in accordance with the IEEE 802.11 standard for wireless LANs (Local Area Networks) and / or the fourth-, fifth-, and sixth-generation mobile communication systems, known as 4G, 5G, and 6G, for mobile communications, as the communication circuit 23.

[0018] The storage device 24 is a recording medium for recording various information. The storage device 24 is realized by, for example, a RAM, a ROM, a flash memory, an SSD (Solid State Drive), a hard disk drive, or other storage devices, or an appropriate combination thereof. The storage device 24 stores an image synthesis program P1, which is a computer program executed by the arithmetic circuit 20, and various data used in image synthesis, such as template data D1, background image data D2, image data D3, and synthesis data D4.

[0019] Template data D1 is data that indicates areas that specify the positions of people photographed by each of the photography systems 3A to 3C in composite data generated by combining image data obtained from each of the photography systems 3A to 3C. Template data D1 shown in Figure 3 is an example that indicates three areas F1 to F3 in which three people are respectively positioned in the composite data. In template data D1 shown in Figure 3, for example, area F1 corresponds to the area of ​​the person photographed by photography system 3A, area F2 corresponds to the area of ​​the person photographed by photography system 3B, and area F3 corresponds to the area of ​​the person photographed by photography system 3C.

[0020] The storage device 24 may store multiple template data D1 corresponding to different numbers of people and / or layouts. For example, "Layout 1" could be "a layout in which three people seated on chairs are lined up horizontally, showing their entire bodies from head to toe." Furthermore, "Layout 2" could be "a layout in which three people seated on chairs are lined up horizontally, showing only their upper bodies." Furthermore, "Layout 3" could be a layout in which four people standing on the floor are lined up horizontally, showing their entire bodies from head to toe. In this way, template data D1 indicating areas in which users of various poses and numbers are placed can be stored in the storage device 24. Note that in the template data D1 shown in FIG. 3, rectangular areas F1 to F3 indicate the positions in which people are placed, but the format of the areas indicating the positions in which users are placed in template data D1 is not limited to this.

[0021] Furthermore, the multiple people represented by one template data D1 may have the same posture or different postures. For example, template data D1 may be one in which all people are seated. Alternatively, template data D1 may be one in which one person acting as the moderator is standing and the remaining people are seated.

[0022] The background image data D2 is data that serves as the background for multiple people when generating composite data. This background image data D2 may be associated with template data. For example, the background image data D2 corresponding to the above-mentioned "Layout 1" may be an image of a space including a floor with three chairs lined up horizontally. For example, the background image data D2 corresponding to the above-mentioned "Layout 2" may be an image of a space without a floor or chairs for people to sit on. Furthermore, the background image data D2 corresponding to the above-mentioned "Layout 3" may be an image of a space including a floor but without chairs.

[0023] The image data D3 is image data captured by the imaging systems 3A to 3C. The storage device 24 can store multiple sets of image data D3, each containing a different person, acquired by each of the imaging systems 3A to 3C. The person included in each set of image data D3 is included in a different position identified by the region of the template data D1 for each of the imaging systems 3A to 3C. For example, the person may be included to the left in the image data D3 captured by the imaging system 3A, in the center in the image data D3 captured by the imaging system 3B, and to the right in the image data D3 captured by the imaging system 3C.

[0024] The composite data D4 is data generated by the image synthesis device 2 using the background image data D2 and the image data D3. Specifically, the composite data D4 is generated by combining the background image data D2 with an extracted image of a person extracted from the image data D3 captured by each of the imaging systems 3A to 3C.

[0025] The processing executed by the arithmetic circuit 20 will now be described in detail. When each of the image capture systems 3A to 3C captures an image, the arithmetic circuit 20 reads template data D1 from the storage device 24, which indicates the area in which a person is located in the composite image. The arithmetic circuit 20 then transmits area data indicating the area assigned to each of the image capture systems 3A to 3C based on the read template data D1. For example, when the arithmetic circuit 20 reads template data D1 indicating areas F1 to F3 described above with reference to FIG. 3, it generates area data indicating only area F1 as shown in FIG. 4A and transmits this data to the image capture system 3A. It also generates area data indicating only area F2 as shown in FIG. 4B and transmits this data to the image capture system 3B. It also generates area data indicating only area F3 as shown in FIG. 4C and transmits this data to the image capture system 3C.

[0026] When a plurality of different template data D1 are stored in storage device 24, arithmetic circuit 20 reads out one selected template data D1. Specifically, arithmetic circuit 20 reads out one template data D1 selected in accordance with the number of people and / or layout to be photographed by a plurality of photography systems 3A to 3C in response to an operation by an operator via input device 21. Furthermore, arithmetic circuit 20 generates area data corresponding to each photography system 3A to 3C from the read-out template data D1 and transmits the area data to each photography system 3A to 3C via communication circuit 23.

[0027] After transmitting the region data, the arithmetic circuit 20 accepts image data D3 acquired by the multiple image capture systems 3A to 3C, which is received by the communication circuit 23. The arithmetic circuit 20 also stores the image data D3 accepted from each of the image capture systems 3A to 3C in the storage device 24. For example, the arithmetic circuit 20 accepts image data D3 shown in FIGS. 5A to 5C from the image capture systems 3A to 3C, respectively. Each piece of image data D3 is data obtained by photographing a person so that they appear in the positions of the regions F1 to F3 corresponding to each of the image capture systems 3A to 3C, as shown in FIGS. 4A to 4C.

[0028] The arithmetic circuit 20 detects the position coordinates of a target person from each of the image data D3 acquired by the multiple imaging systems 3A to 3C, extracts a region using the detection results, and creates an extracted image. A general pose estimation technique can be used to detect the position coordinates of the person's skeleton, such as the position coordinates of the ankles, knees, wrists, and elbows, using the pose estimation technique. An example of a pose estimation technique is OpenPose, which can detect the positions of a person's joints. Furthermore, a general matting technique can be used to extract the person's region. An example of a matting technique is Robust Video Matting (RVM). For example, the arithmetic circuit 20 can extract the person's region from the image data D3 shown in FIGS. 5A to 5C, as shown in FIGS. 6A to 6C, respectively. In FIGS. 6A to 6C, the extracted image represents the person and the region of the chair on which the person is sitting, and maintains the coordinate information of the person in the image data D3.

[0029] The arithmetic circuit 20 generates composite data D4 by combining extracted images of people extracted from each of the multiple image data D3 with background image data D2. At this time, the arithmetic circuit 20 generates composite data D4 by allocating, to the position of an area defined in the template data D1, extracted images extracted from the image data D3 obtained by the imaging systems 3A-3C associated with the area. For example, when the arithmetic circuit 20 extracts the extracted images shown in FIGS. 6A-6C, it can generate composite data such as that shown in FIG. 7. While the above example describes an example in which a person's area is extracted using matting technology, a person's area may also be extracted using chromakey technology and combined with the background image data D2.

[0030] The arithmetic circuit 20 may, for example, output the generated composite data D4 to an external device (not shown) used by an operator or the like via the communication circuit 23 and display the composite data D4 on the external device. Alternatively, the arithmetic circuit 20 may display the composite data D4 on a display, which is the output device 22.

[0031] <Photography System> Although the multiple photographing systems 3A to 3C are located in different locations, they preferably have similar performance so that the balance of each person included in the composite data generated by the image synthesis device 2 is consistent. An example of the photographing system 3A is described below using FIG. 2 , but because photographing systems 3B and 3C are similar to the photographing system 3A, they are not shown or described here. For example, as shown in FIG. 2 , the photographing system 3A includes an arithmetic circuit 30, an input device 31, an output device 32, a communication circuit 33, a storage device 34, a photographing device 35, and a drive mechanism 36. The photographing system 3A may be, for example, a camera-like photographing device 35, in which the arithmetic circuit 30, the input device 31, the output device 32, the communication circuit 33, and the storage device 34 are built in. Alternatively, the photographing system 3A may be a combination of an information processing device including the arithmetic circuit 30, the input device 31, the output device 32, the communication circuit 33, and the storage device 34 with the photographing device 35, such as a camera.

[0032] The arithmetic circuit 30 is a controller that controls the entire photographing system 3A. For example, the arithmetic circuit 30 reads and executes a photographing control program P2 stored in the storage device 34 to realize various processes when photographing an image with the photographing system 3A. The arithmetic circuit 30 may also be various processors such as a CPU, an MPU, a GPU, an FPGA, a DSP, an ASIC, or a dedicated hardware circuit.

[0033] The input device 31 is input means such as operation buttons, a touch panel, a microphone, etc., used for inputting operations and data. The output device 32 is output means such as a display, a speaker, etc., used for outputting processing results and data.

[0034] The communication circuit 33 is a communication means that enables data communication with an external device such as the image synthesis device 2. The data communication may be wired and / or wireless, and may be performed in accordance with known communication standards.

[0035] The storage device 34 may be implemented by a RAM, a ROM, a flash memory, a solid-state drive (SSD), a hard disk drive, or other storage devices, or by an appropriate combination thereof. The storage device 34 stores the imaging control program P2, which is a computer program executed by the arithmetic circuit 30, and various data used or obtained in the imaging process, such as the region data D6, test image data D7, and image data D3.

[0036] The photographing device 35 includes a series of mechanisms related to the acquisition of image data, and acquires image data under control of the arithmetic circuit 30. Specifically, the photographing device 35 is a camera including an image sensor, an optical element, an A / D converter, etc. When photographing an image, the photographing device 35 can perform focus control, white balance control, etc. in response to control signals from the arithmetic circuit 30. The photographing device 35 may also be capable of controlling the zoom magnification through digital zoom control.

[0037] The driving mechanism 36 is a mechanism that drives the image capturing device 35 to control the pan angle and tilt angle. The driving mechanism 36 can control the optical zoom and the pan / tilt in response to a control signal output from the arithmetic circuit 30.

[0038] The arithmetic circuit 30 receives area data, which is generated based on the template data D1 and transmitted from the image synthesis device 2 via the communication circuit 33, and indicates the area in which the person is located when the image data is acquired.

[0039] The arithmetic circuit 30 acquires test image data D7 via the photographing device 35.

[0040] The arithmetic circuit 30 controls the image so that the target person is captured within the area indicated by the area data received from the image synthesis device 2. Specifically, the arithmetic circuit 30 detects the position coordinates of the person from the test image data D7 using a person detection technique. Depending on the detected position, the arithmetic circuit 30 calculates and controls at least one control signal for pan control, tilt control, or zoom control so that the person is captured within the area.

[0041] Specifically, when the data received from the image synthesis device 2 is associated with the above-mentioned "Layout 1 (three people sitting on chairs lined up in a row, showing their entire bodies from head to toe)," the arithmetic circuit 30 generates a control signal for performing PTZ control so that, for example, the tops of the people's heads contact the top edge of the area and the soles of the people's feet contact the bottom edge of the area. Also, when the data received from the image synthesis device 2 is associated with the above-mentioned "Layout 2 (three people sitting on chairs lined up in a row, showing only their upper bodies)," the arithmetic circuit 30 generates a control signal for performing PTZ control so that, for example, the tops of the people's heads contact the top edge of the area and the height of the people's elbows contact the bottom edge of the area. Also, when the data received from the image synthesis device 2 is associated with the above-mentioned "Layout 3 (four people standing on the floor lined up in a row, showing their entire bodies from head to toe)," the arithmetic circuit 30 generates a control signal for performing PTZ control so that, for example, the tops of the people's heads contact the top edge of the area and the soles of the people's feet contact the bottom edge of the area. The aforementioned "Layout 1" and "Layout 2" have in common the fact that the postures of the people in the images are seated in chairs when capturing the images, but the ranges used for compositing differ. On the other hand, "Layout 3" differs in that the people are standing. Therefore, a control signal is generated according to the posture of the people defined in each layout. Here, the example uses the height of the person's head, soles, and elbows as reference points, but this is not limited thereto. The reference positions can be determined depending on the purpose of generating the composite image and the layout. For example, the arithmetic circuit 30 may acquire the position coordinates of each bone structure of the person and control the position of each bone structure to fit within a target position. Specifically, when placing a person within a specified frame, the vertical position of the head or eyes may be located 5% of the frame size from the top of the frame, and the vertical position of the ankles may be located 5% of the frame size from the bottom of the frame. In this way, the arithmetic circuit 30 can use a specific position of the person, such as the ankles, as a reference point to include the entire person in the capture range without any interruptions.

[0042] For example, the arithmetic circuit 30 can use the above-mentioned posture estimation technique that is also used in the image synthesis device 2 to detect a person.

[0043] Here, an example of the arithmetic circuit 30 when performing PTZ control will be described. For example, as shown in FIG. 8A , the arithmetic circuit 30 determines the center Pt1 of the person T1 obtained by region extraction from the test image data D7. The center Pt1 is, for example, a point determined by the center of the width and the center of the height of the person T1. Alternatively, the center Pt1 may be determined by, for example, averaging multiple position coordinates detected using posture estimation technology. The arithmetic circuit 30 determines the difference between the person's center Pt1 and the center Pf1 of the region F1 indicated in the data received from the image synthesis device 2, and determines a pan / tilt control signal that aligns the center Pt1 with the center Pf1, and outputs the signal to the drive mechanism 36. At this time, each center coordinate is set based on the same standard. For example, if the center coordinate is a rectangular region, the aspect ratio is set based on the same standard. Specifically, as shown in FIG. 8A , if the person T1 is a template that targets the entire body, the center Pt1 of the person T1 is also set based on the entire body. Furthermore, in the case of a template for which the upper body of a person is targeted, the coordinates of the center of the person are also set from the upper body. For example, the arithmetic circuit 30 calculates the difference between the coordinates of the center of the person extracted from the test image data D7 and the coordinates of the center of the area included in the area data. Using the calculated difference, the arithmetic circuit 30 calculates the horizontal and vertical movement distances required to move the coordinates of the center of the person extracted from the test image data D7 to the coordinates of the center of the area indicated by the area data.

[0044] In addition, in PTZ control, the arithmetic circuit 30 calculates a zoom control signal for aligning the person T1 extracted from the test image data D7 with the frame of the region F1 indicated by the data received from the image synthesis device 2, and transmits the signal to the image capture device 35. Specifically, as shown in FIG. 8B , the arithmetic circuit 30 calculates the scaling ratio for aligning the size of the person T1 extracted from the test image data D7 with the region F1 indicated by the region data. The arithmetic circuit 30 then generates a control signal for executing PTZ control using the distance by which the coordinates of the center of the person are moved to the coordinates of the center of the region indicated by the region data, and the scaling ratio related to the size of the person. For example, simply aligning the center Pt1 of the person T1 with the center Pf1 of the region F1 will result in the person T1 being significantly outside the region F1, as shown in FIG. 8B , and will not be balanced with other people in the composite data D4. Therefore, as shown in FIG. 8C , the person T1 is photographed so that it fits within the region F1. For example, the arithmetic circuit 30 controls the zoom magnification so that the person T1 fits within the region F1. Specifically, the arithmetic circuit 30 refers to a table, formula, etc., prepared in advance, which defines the relationship between the enlargement / reduction ratio and the amount of movement of the zoom lens, derives the amount of movement of the zoom lens from the enlargement / reduction ratio, and generates a control signal for zoom control to move the zoom lens by that amount. As a result, when the image synthesis device 2 generates the synthesis data D4, it can be balanced with people photographed by other imaging systems, resulting in natural image data.

[0045] Furthermore, for example, when the region in which a person is to be placed in the template is divided into a plurality of areas at equal intervals in advance, the arithmetic circuit 30 determines an area to correspond to each of the position coordinates of the person's skeleton detected by the posture estimation technology. The arithmetic circuit 30 may then calculate a control signal such that each of the position coordinates of the person's skeleton is positioned in the determined area within the region.

[0046] 9, the image synthesis process executed by the arithmetic circuit 20 of the image synthesis device 2 will be described. Since the specific processing of each step in the flowchart has been described above, each step will be described in a simplified manner.

[0047] At the start of the image synthesis process, the arithmetic circuit 20 accepts, via the input device 21, a selection of template data D1 to be used in the image synthesis (S001). When accepting the selection of template data D1, conditions including the "number of people" and "composition" may be accepted. The composition may include not only the arrangement of people but also the state of the people as they appear in the final image data, such as whether they are standing or sitting, or whether they are shown as a bust shot or a close-up of their faces. Furthermore, when selecting these, a simple illustration showing the placement and final size of the people as they appear in the final image data may be provided. Furthermore, the size of the people in the template data D1 may be changeable via the input device 21.

[0048] The arithmetic circuit 20 reads out the template data D1 selected in step S001 from the storage device 24, and generates area data indicating the area of ​​the person's position defined by the read template data D1 (S002).

[0049] The arithmetic circuit 20 transmits the area data generated in step S002 to each of the imaging systems 3A to 3C via the communication circuit 23 (S003).

[0050] Thereafter, the arithmetic circuit 20 acquires image data D3 from each of the image capture systems 3A to 3C (S004). In the image data D3 acquired from each of the image capture systems 3A to 3C, a person is located in the area indicated by the area data transmitted in step S003.

[0051] The arithmetic circuit 20 acquires the position coordinates of the person from each image data D3 acquired in step S004 (S005).

[0052] The arithmetic circuit 20 extracts the area of ​​the person from each image data D3 acquired in step S004 using the position coordinates acquired in step S005, and sets it as an extracted image (S006).

[0053] The arithmetic circuit 20 combines the background image data D2 with the plurality of extracted images obtained in step S006 to generate composite data D4 (S007).

[0054] The arithmetic circuit 20 may display the composite data D4 generated in step S007 on a display, which is the output device 22 (S008). Alternatively, the arithmetic circuit 20 may output the composite data D4 to an external device (not shown) via the communication circuit 23 and display it thereon.

[0055] <Image Acquisition Processing in the Photography System> The image acquisition processing in the photography systems 3A to 3C will be described using the flowchart shown in Fig. 10. Since the specific processing of each step in the flowchart has been described above, each step will be described in a simplified manner.

[0056] The arithmetic circuit 30 receives area data indicating an area defining the position of a person at the time of photographing from the image composition device 2 via the communication circuit 33 (S101). The area data is the data transmitted from the image composition device 2 in step S003 of the flowchart in Fig. 9 described above. Since the position of the associated area differs for each of the photographing systems 3A to 3C, the data received in step S101 differs for each of the photographing systems 3A to 3C.

[0057] The arithmetic circuit 30 acquires test image data by capturing an image with the image capturing device 35 (S102).

[0058] The arithmetic circuit 30 acquires the position coordinates of the person from the test image data acquired in step S102 (S103).

[0059] The arithmetic circuit 30 generates a control signal for controlling the person whose position coordinates were acquired in step S103 to appear within the area indicated by the area data received in step S101 (S104).

[0060] The arithmetic circuit 30 performs PTZ control using the control signal generated in step S104 (S105).

[0061] The arithmetic circuit 30 acquires image data D3 via the photographing device 35 under the conditions controlled in step S105 (S106).

[0062] The arithmetic circuit 30 transmits the image data D3 acquired in step S107 to the image synthesis device 2 via the communication circuit 23 (S107).

[0063] The arithmetic circuit 30 returns to step S106 and continues the process of acquiring the image data D3 until acquisition of the image data D3 is completed (YES in S108). Note that the flowchart shown in FIG. 10 illustrates an example in which the arithmetic circuit 30 transmits the image data D3 to the image synthesis device 2 before acquisition of the series of image data is completed, but this is not limiting. For example, after acquisition of the series of image data D3 is completed, the arithmetic circuit 30 may transmit the series of continuously acquired image data, i.e., video data, to the image synthesis device 2.

[0064] In the image synthesis system 1 according to the first embodiment, the image synthesis device 2 uses the template data D1 as described above to generate area data indicating the position of a person to be photographed, and transmits the area data to each of the image capture systems 3A to 3C. When acquiring image data, each of the image capture systems 3A to 3C performs PTZ control to acquire the image data so that the person fits within the area indicated by the area data received from the image synthesis device 2. Furthermore, the image synthesis device 2 can use an extracted image of the person extracted at the position indicated by the area data, thereby generating synthesis data in which multiple people are positioned naturally and without any awkwardness.

[0065] [Second Embodiment] <Image Synthesis System> As shown in FIG. 11 , an image synthesis system 1 according to the second embodiment is similar to the image synthesis system 1 described above with reference to FIG. 1 , in which an image synthesis device 2 and multiple image capture systems 3A-3C are connected. In the image synthesis system 1 according to the first embodiment, each image capture system 3A-3C generates a control signal within its own system based on area data received from the image synthesis device 2 to perform PTZ control. In contrast, in the image synthesis system 1 according to the second embodiment, each image capture system 3A-3C does not generate a control signal within its own system. Each image capture system 3A-3C transmits test image data D7 to the image synthesis device 2 in advance. Furthermore, upon receiving a control signal generated by the image synthesis device 2 in accordance with the test image data D7 of each image capture system 3A-3C, the image synthesis device 2 performs PTZ control based on the received control signal. In other words, the image capture systems 3A-3C of the image synthesis system 1 according to the first embodiment may require complex processing to generate a control signal for PTZ control. In contrast, the photographing systems 3A to 3C of the image compositing system 1 according to the second embodiment transmit test image data D7 in advance to the image compositing device 2. As a result, the image compositing system 1 according to the second embodiment only needs to perform PTZ control in accordance with the control signal received from the image compositing device 2 in response to the test image data D7, and does not require complex processing.

[0066] <Image Synthesis Device> As shown in FIG. 11 , the image synthesis device 2 of the second embodiment differs from the configuration described above using FIG. 1 in the first embodiment 1 in that test image data D7 is stored in the storage device 24, but the other configurations are similar.

[0067] The arithmetic circuit 20 receives test image data D7 obtained in advance from each of the plurality of imaging systems 3A to 3C via the communication circuit 23. The arithmetic circuit 20 stores the plurality of test image data D7 received from each of the imaging systems 3A to 3C in the storage device 24.

[0068] The arithmetic circuit 20 acquires the position coordinates of the person from the plurality of test image data D7. To acquire the position coordinates of the person, the arithmetic circuit 20 can use the posture estimation technique described above in the first embodiment.

[0069] The arithmetic circuit 20 uses a person detection technique to detect the position of a person in the test image data D7 received from the image capture systems 3A to 3C, and calculates a control signal based on the detected position. Specifically, the arithmetic circuit 20 generates a control signal including at least one of pan control, tilt control, and zoom control control signals that control the person captured by each image capture system 3A to 3C so that the person is captured in the template area assigned to each image capture system 3A to 3C. Here, the control signal for PTZ control (pan control, tilt control, and zoom control) is generated using the same process as the process executed by the arithmetic circuit 30 described above with reference to Figures 8A to 8C in the first embodiment.

[0070] The arithmetic circuit 20 transmits the generated control signals to the corresponding imaging systems 3A to 3C via the communication circuit 23.

[0071] The arithmetic circuit 20 acquires, via the communication circuit 23, image data obtained by PTZ control according to the transmitted control signal.

[0072] <Photography System> As shown in FIG. 12, the photographing systems 3A to 3C of the second embodiment differ in that the storage device 34 does not store the area data D6, but the other configurations are similar.

[0073] The arithmetic circuit 30 acquires test image data D7 using the photographing device 35. This test image data D7 is data that includes a target person but is not subject to PTZ control.

[0074] The arithmetic circuit 30 transmits the acquired test image data D7 to the image synthesis device 2 via the communication circuit 33.

[0075] The arithmetic circuit 30 acquires a control signal for PTZ control via the communication circuit 33 in response to the transmitted test image data D7.

[0076] The arithmetic circuit 30 outputs the acquired control signal to the photographing device 35, and executes PTZ control to acquire image data D3.

[0077] The arithmetic circuit 30 transmits the image data D3 obtained by the photographing device 35 to the image synthesis device 2 via the communication circuit 33.

[0078] <Image Synthesis Processing in Image Synthesis Device> The image synthesis processing executed by the arithmetic circuit 20 of the image synthesis device 2 will be described using the flowchart shown in Fig. 13. Note that in Fig. 13, processing that differs from the processing described above using Fig. 9 is distinguished by underlining the numbers. Since the specific processing of each step in the flowchart has been described above, each step will be described in a simplified manner.

[0079] At the start of the image synthesis process, the arithmetic circuit 20 receives, via the input device 21, the selection of template data D1 to be used in the image synthesis (S001).

[0080] The arithmetic circuit 20 requests each of the imaging systems 3A to 3C to transmit test image data via the communication circuit 23 (S201).

[0081] The arithmetic circuit 20 acquires the test image data D7 from each of the imaging systems 3A to 3C via the communication circuit 23 (S202).

[0082] The arithmetic circuit 20 acquires the position coordinates of the person from the test image data D7 acquired in step S202 (S203).

[0083] The arithmetic circuit 30 generates a control signal for each of the photographing systems 3A to 3C to control the photographing systems 3A to 3C so that the person extracted in step S203 appears within the area indicated by the template data D1 selected in step S101 (S204).

[0084] The arithmetic circuit 30 transmits the control signal generated in step S204 to each of the imaging systems 3A to 3C (S205).

[0085] The arithmetic circuit 20 acquires image data D3 from each of the image capture systems 3A to 3C (S004). In the image data D3 acquired from each of the image capture systems 3A to 3C, PTZ control has been performed in accordance with the control signal transmitted in step S205, and as a result, the person is positioned in the area corresponding to each of the image capture systems 3A to 3C shown in the template data D1 received in step S001.

[0086] The arithmetic circuit 20 acquires the position coordinates of the person from each image data D3 acquired in step S004 (S005).

[0087] The arithmetic circuit 20 extracts the area of ​​the person from each image data D3 acquired in step S004 using the position coordinates acquired in step S005, and sets it as an extracted image (S006).

[0088] The arithmetic circuit 20 combines the background image data D2 with the plurality of extracted images obtained in step S006 to generate composite data D4 (S007).

[0089] The arithmetic circuit 20 may display the composite data D4 generated in step S007 on a display, which is the output device 22 (S008). Alternatively, the arithmetic circuit 20 may output the composite data D4 to an external device (not shown) via the communication circuit 23 and display it thereon.

[0090] <Image Acquisition Processing in the Photography System> The image acquisition processing in the photography systems 3A to 3C will be described using the flowchart shown in Fig. 14. Note that in Fig. 14, processing that differs from the processing described above using Fig. 10 is distinguished by underlining the numbers. Since the specific processing of each step in the flowchart has been described above, each step will be described in a simplified manner.

[0091] The arithmetic circuit 30 receives a request for test image data from the image synthesis device 2 via the communication circuit 33 (S301). The request received in step S301 is the data transmitted from the image synthesis device 2 in step S201 of the flowchart in FIG. 13 described above.

[0092] The arithmetic circuit 30 acquires test image data D7 by capturing an image with the photographing device 35 (S302).

[0093] The arithmetic circuit 30 transmits the test image data D7 acquired in step S302 to the image synthesis device 2 via the communication circuit 33 (S303). The test image data D7 transmitted in step S303 is acquired by the image synthesis device 2 in step S202 of the flowchart in FIG. 13 described above.

[0094] The arithmetic circuit 30 receives the control signal generated based on the test image data D7 transmitted in step S303 (S304). The control signal received in step S304 is the control signal transmitted from the image synthesis device 2 in step S205 of the flowchart in FIG. 13 described above.

[0095] The arithmetic circuit 30 performs PTZ control using the control signal received in step S304 (S305).

[0096] The arithmetic circuit 30 acquires image data D3 via the photographing device 35 under the conditions controlled in step S305 (S106).

[0097] The arithmetic circuit 30 transmits the image data D3 acquired in step S107 to the image synthesis device 2 via the communication circuit 23 (S107).

[0098] The arithmetic circuit 30 returns to step S106 and continues the process of acquiring the image data D3 until acquisition of the image data D3 is completed (YES in S108).

[0099] In the image synthesis system 1 according to the second embodiment, the image synthesis device 2 generates a control signal using template data D1 in accordance with test image data D7 acquired in advance from each of the imaging systems 3A to 3C, as described above. This control signal is a signal for performing PTZ control to capture an image in which a person is positioned within a desired area. Each of the imaging systems 3A to 3C acquires image data D3 under control of the control signal generated by the image synthesis device 2. This enables the image synthesis device 2 to use an extracted image of a person extracted from the position indicated by the template data D1, making it possible to generate synthesis data in which multiple people are positioned naturally and without any awkwardness.

[0100] <Variation 1> In the image synthesis system 1 according to the first embodiment described above with reference to FIG. 1 and the image synthesis system 1 according to the second embodiment described above with reference to FIG. 11, background image data D2 stored in advance in the storage device 24 is synthesized with multiple extracted images extracted from multiple image data D3 obtained by the image capture systems 3A to 3C to generate synthesized data D4. In contrast, as shown in FIG. 15, the image synthesis system 1 according to Variation 1 does not store background image data D2 in the storage device 24. The image synthesis system 1 according to Variation 1 differs in that image data D3 captured by one of the image capture systems is used as background image data. Specifically, in the image synthesis system 1 according to Variation 1, one of the multiple image capture systems 3A to 3C is designated as the main image capture system, and the remaining systems are designated as sub-image capture systems. The image synthesis system 1 according to Variation 1 then uses image data D3 captured by the main image capture system as background image data. In the following description, the image capture system 3A is designated as the main image capture system, and the image capture systems 3B and 3C are designated as sub-image capture systems.

[0101] In the image synthesis device 2 according to the first modification, the arithmetic circuit 20 transmits area data to all of the main and sub-image capture systems 3A to 3C via the communication circuit 23, as in the case described above. Furthermore, in all of the image capture systems 3A to 3C, the arithmetic circuit 20 executes PTZ control based on the area data transmitted from the image synthesis device 2 and acquires image data via the image capture device 35. Furthermore, each arithmetic circuit 30 transmits the acquired image data to the image synthesis device 2 via the communication circuit 23. The arithmetic circuit 20 of the image synthesis device 2 then extracts a person area from only the image data acquired from the sub-image capture systems 3B and 3C to generate an extracted image. The arithmetic circuit 20 then uses the image data acquired from the main image capture system 3A as background image data and synthesizes each of the extracted images to generate synthesized data.

[0102] In this way, by using the image data D3 captured by the main imaging system 3A as background image data, there is no need to prepare background image data in advance. Furthermore, in the image compositing system 1 according to the first modification, extraction processing and compositing processing with background image data are not required for the image data of the main imaging system 3A, and the processing of the image compositing system 1 as a whole can be simplified.

[0103] 15 has been described as an example corresponding to the image synthesis system 1 according to the first embodiment, but is not limited thereto. For example, in the case of the image synthesis system 1 according to the second embodiment, one of the systems may be set as the main imaging system 3A and the remaining systems may be set as sub imaging systems 3B and 3C. In the image synthesis system 1 according to the second embodiment, the image data D3 captured by the main imaging system 3A is used as background image data, thereby simplifying the processing of the image synthesis system 1 as a whole.

[0104] <Variation 2> The image synthesis system 1 according to Variation 2 has the same configuration as the image synthesis system 1 described above with reference to FIG. 1 or FIG. 11 , and therefore additional illustrations will be omitted and reference will be made to FIG. 1 and FIG. 11 . As described above in Variation 1, when one of the multiple image capture systems 3A-3C is set as the main image capture system and the remaining systems are set as sub image capture systems, the arithmetic circuits 30 of the sub image capture systems 3B and 3C may be controlled in accordance with the image capture conditions of the main image capture system 3A. For example, the sub image capture systems 3B and 3C may acquire image data D3 by setting the height of the image capture device 35 from the floor, which is a capture condition, to the same as that of the main image capture system 3A. This allows the sub image capture systems 3B and 3C to acquire image data of a person by setting the elevation angle or depression angle, which is a capture condition, to the same as that of the main image capture system 3A. Furthermore, the sub image capture systems 3B and 3C may acquire image data D3 by setting the angle of view, which is a capture condition, to the same as that of the main image capture system 3A.

[0105] When the shooting conditions of the sub-photography systems 3B and 3C are adjusted to the shooting conditions of the main photography system 3A, all of the photography systems 3A to 3C are placed in different locations with the same conditions relative to the subject of photography. Specifically, each photography system 3A to 3C is placed with similar conditions, such as the height from the floor where the subject of photography is located and the distance from the subject. As a result, the elevation angle or depression angle, which is the shooting condition of each photography system 3A to 3C, is also similar, and it can be considered that multiple people have been virtually photographed by the same photography systems 3A to 3C. Furthermore, each photography system 3A to 3C is placed at a similar distance from the subject of photography.

[0106] When adjusting the photographing conditions of the sub photographing systems 3B and 3C to match the photographing conditions of the main photographing system 3A, the image composition device 2 may acquire the photographing conditions of the main photographing system 3A and transmit them to the sub photographing systems 3B and 3C. Alternatively, the image composition device 2 may present to the sub photographing systems 3B and 3C, in numerical form, changes necessary to adjust the photographing conditions to those of the main photographing system 3A. This allows the operators of the sub photographing systems 3B and 3C to easily make various changes to adjust the photographing conditions to the acquired photographing conditions.

[0107] <Variation 3> The image synthesis system 1 according to Variation 3 has the same configuration as the image synthesis system 1 described above with reference to FIG. 1 or FIG. 11 , and therefore will be omitted from the illustrations. In the image synthesis system 1 according to the first embodiment described above with reference to FIG. 1 and the image synthesis system 1 according to the second embodiment described above with reference to FIG. 11 , PTZ control is performed based on the same criteria for all people photographed by the image capture systems 3A to 3C. In this case, the person's physique may not be taken into consideration. For example, regardless of the person's height or sitting height, the composite data D4 may represent the person as having the same height or sitting height, resulting in unnatural image data. In the image synthesis system 1 according to Variation 3, the image synthesis device 2 can, after generating the composite data D4, receive a signal from the operator via the input device 21 to adjust the size of the person and adjust the size of the person. For example, when the image synthesis device 2 receives a signal to adjust the size of the person, it generates area data or a control signal corresponding to the received signal and transmits it to each of the image capture systems 3A to 3C. Each of the photographing systems 3A to 3C executes PTZ control according to the physique of the target person in response to the received area data or control signal, and can acquire image data D3. Therefore, in the composite data D4, the size of the person does not become unnaturally uniform, and natural image data can be obtained.

[0108] For example, if the ground surface of multiple people is the same in the composite data D4, the image data will appear natural. Therefore, when adjusting the size of a person, the image composition system 1 may perform PTZ control so that the person's feet are aligned with the floor surface portion of the background image data D2.

[0109] <Modification 4> The image synthesis system 1 according to Modification 4 has the same configuration as the image synthesis system 1 described above with reference to FIG. 1 or FIG. 11 , and therefore, additional illustrations will be omitted, with reference to FIG. 1 and FIG. 11 . In the image synthesis system 1 according to Modification 4, a plurality of background image data D2 captured under different conditions is stored in the storage device 24. For example, in the image synthesis system 1 according to Modification 4, the image data D3 are captured under the same conditions by the plurality of imaging systems 3A-3C. Furthermore, the image synthesis device 2 selects, from the plurality of background image data D2, background image data D2 obtained under conditions equivalent to the conditions under which the images were captured by the imaging systems 3A-3C. For example, the image synthesis device 2 selects background image data D2 captured by a camera with the same depression or elevation angle, angle of view, and height as the camera that captured the image data D3. Even if there is no background image data D2 captured under exactly the same conditions as the conditions under which the image data D3 to be synthesized was captured by the imaging systems 3A-3C, the image synthesis device 2 selects background image data D2 captured under the closest conditions. As a result, the image data D3 from which the person is extracted and the background image data D2 from which the extracted image is synthesized are photographed under the same conditions, so that the synthesized data D4 is natural image data.

[0110] Overview of the Embodiments (1) An image synthesis device according to the present disclosure is an image synthesis device that is connected to a plurality of photographing systems, includes an arithmetic circuit and a communication circuit, and generates composite data by synthesizing images of a plurality of different people obtained by the plurality of photographing systems, wherein an area in which each person is placed in the composite data is defined in advance as a template, and each of the photographing systems and each area defined in the template are uniquely associated in advance, and the arithmetic circuit causes each of the photographing systems to capture image data in which the person to be photographed appears within the area defined in the template, obtains the plurality of image data respectively from each of the photographing systems via the communication circuit, and generates composite data by assigning extracted images of the people extracted from the plurality of image data to positions in the areas of the template.

[0111] This makes it possible to generate natural composite data from image data captured separately by different image capture devices.

[0112] (2) In the image synthesis device of (1), the arithmetic circuit may generate the synthesis data by synthesizing an extracted image obtained from the image data received from the corresponding photographing system in an area defined by the template of predetermined background image data.

[0113] This allows images captured by each imaging system to be combined with a desired background.

[0114] (3) In the image synthesis device of (1) to (2), one of the multiple shooting systems is set as a main shooting system and the other shooting systems are set as sub-shooting systems, and the arithmetic circuit may use image data acquired by the main shooting system as background image data and synthesize the image data acquired by the main shooting system with the extracted image obtained from the image data acquired by the sub-shooting system to generate the synthesized data.

[0115] This allows a plurality of imaging systems to be used to obtain image data with a consistent look.

[0116] (4) In the image synthesis device of (1) to (3), the arithmetic circuit may be able to access a storage device, and the storage device may store a plurality of templates corresponding to different numbers of people and / or layouts. The arithmetic circuit may read out one template selected from the plurality of templates stored in the storage device in accordance with the total number of people to be photographed by the plurality of photographing systems, and allocate the template to the position of the area of ​​the read template to generate composite data.

[0117] This allows the images to be combined in a desired layout.

[0118] (5) The image synthesis method of the present disclosure is an image synthesis method in which an image synthesis device having an arithmetic circuit and a communication circuit is connected to a plurality of photographing systems, and generates synthetic data by synthesizing images of a plurality of different people obtained by the plurality of photographing systems, wherein an area in which each person is placed in the synthetic data is defined in advance as a template, and each of the photographing systems and each area defined in the template are uniquely associated in advance, and the arithmetic circuit causes each of the photographing systems to capture image data in which the person to be photographed appears within the area defined in the template, obtains the plurality of image data respectively from each of the photographing systems via the communication circuit, and generates synthesized data by assigning extracted images of the people extracted from the plurality of image data to the positions of the areas of the template.

[0119] This makes it possible to generate natural composite data from image data captured separately by different image capture devices.

[0120] (6) In the image synthesis method of (5), the arithmetic circuit may transmit area data indicating the area of ​​the template assigned to each of the photographing systems via the communication circuit, and each of the photographing systems may be controlled so that the target person appears within the area indicated by the received area data when acquiring image data.

[0121] This allows each imaging system to use the area data to capture image data that is used to generate natural composite data.

[0122] (7) The image synthesis method of (6) may involve, in each of the photographing systems, obtaining at least one control signal of pan control, tilt control, or zoom control to acquire image data in which the person appears within the area indicated by the area data received from the image synthesis device, and performing control using the obtained control signal.

[0123] This allows each imaging system to use the area data to perform PTZ control for capturing image data that is used to generate natural composite data.

[0124] (8) The image synthesis method of (5) may include the following steps: in the arithmetic circuit, receiving test image data in advance from each of the photographing systems via the communication circuit; extracting an area of ​​the target person from the plurality of test image data; generating a plurality of control signals according to the extracted area of ​​the person so that the person appears in an area assigned to each of the photographing systems in the template; transmitting each of the control signals to the corresponding photographing system via the communication circuit; and controlling the photographing system in accordance with the control signal received from the image synthesis device.

[0125] This allows the image synthesis device to transmit to each imaging system a control signal for capturing image data that is used to generate natural synthesized data.

[0126] (9) The image synthesis method of (8) may further comprise the steps of: in the arithmetic circuit, determining at least one control signal for pan control, tilt control, or zoom control for each of the photographing systems in order to obtain image data in which the person is captured within the area indicated by the template; and transmitting the control signal to each of the photographing systems via the arithmetic circuit.

[0127] This allows the image synthesis device to transmit to each imaging system a control signal for PTZ control that captures image data used to generate natural synthesized data.

[0128] (10) In the image synthesis methods of (5) to (9), the plurality of image capturing systems may be arranged under the same conditions relative to the subject to be captured.

[0129] This allows each imaging system to acquire image data with a consistent look.

[0130] (11) In the image synthesis method of (10), the image capturing systems may be arranged at the same height from a floor where the target person is located.

[0131] This allows each imaging system to acquire image data with a consistent look.

[0132] (12) In the image synthesis method of (10) to (11), the plurality of image capturing systems may have the same angle of view, which is a condition for capturing images.

[0133] This allows each imaging system to acquire image data with a consistent look.

[0134] (13) In the image synthesis methods of (10) to (12), the plurality of image capture systems may acquire image data of a person by setting the same elevation angle or depression angle, which is a capture condition.

[0135] This allows each imaging system to acquire image data with a consistent look.

[0136] (14) In the image synthesis methods of (10) to (13), one of the plurality of photographing systems may be set as a main photographing system and the other photographing systems as sub photographing systems, and the arithmetic circuit may generate a control signal for controlling the photographing conditions of the sub photographing system to match the photographing conditions of the main photographing system.

[0137] This allows each imaging system to acquire image data with a consistent look.

[0138] (15) A computer program of the present disclosure is a computer program that causes an image synthesis device connected to a plurality of photographing systems and equipped with an arithmetic circuit and a communication circuit to generate composite data that synthesizes images of a plurality of different people obtained by the plurality of photographing systems, wherein an area in which each person is placed in the composite data is defined in advance as a template, and each of the photographing systems and each area defined in the template are uniquely associated in advance, and when the computer program is loaded into the image synthesis device and executed, the arithmetic circuit may function to: cause each of the photographing systems to photograph image data in which the person to be photographed appears within the area defined in the template; obtain the plurality of image data captured from each of the photographing systems via the communication circuit; and generate composite data by assigning the extracted images of the people extracted from the plurality of image data to the positions of the areas of the template.

[0139] This makes it possible to generate natural composite data from image data captured separately by different image capture devices.

[0140] The image synthesis device, image synthesis method, and computer program according to all claims of the present disclosure are realized by the cooperation of hardware resources, such as a processor, a memory, and a computer program.

[0141] The image synthesis system, image synthesis device, and computer program of the present disclosure are useful for generating synthetic image data from images obtained at different locations.

[0142] REFERENCE SIGNS LIST 1 Image synthesis system 2 Image synthesis device 20, 30 Arithmetic circuit 21, 31 Input device 22, 32 Output device 23, 33 Communication circuit 24, 34 Storage device D1 Template data D2 Background image data D3 Image data D4 Synthesis data D6 Area data D7 Test image data P1 Image synthesis program 3A, 3B, 3C Photography system P2 Photography control program T1 Person F1, F2, F3 Area

Claims

1. An image synthesis device connected to a plurality of photographing systems, equipped with an arithmetic circuit and a communication circuit, for generating synthetic data by synthesizing a plurality of image data of a plurality of different people respectively obtained by the plurality of photographing systems, wherein a plurality of areas in which the plurality of people are respectively arranged in the synthetic data are defined in advance as a template, and each of the photographing systems and each of the areas defined in the template are uniquely associated in advance, the arithmetic circuit causes each of the photographing systems to photograph corresponding image data from the plurality of image data in which a corresponding person of the plurality of people appears in a corresponding area of ​​the plurality of areas defined in the template, obtains the corresponding image data from each of the photographing systems via the communication circuit, and generates the synthetic data by assigning a plurality of extracted images corresponding to the plurality of people respectively extracted from the plurality of image data to the multiple areas of the template, respectively.

2. The image synthesis device according to claim 1, wherein the template includes background image data.

3. The image synthesis device according to claim 2, wherein one of the plurality of shooting systems is set as a main shooting system and the other shooting systems are set as sub shooting systems, and the arithmetic circuit uses image data acquired by the main shooting system as the background image data, and synthesizes the image data acquired by the main shooting system with an extracted image obtained from image data acquired by the sub shooting system to generate the synthetic data.

4. The image synthesis device of claim 1, wherein the arithmetic circuit is capable of accessing a storage device, the storage device stores a plurality of templates corresponding to different numbers of people and / or layouts, the plurality of templates including a plurality of regions different from one another, and the arithmetic circuit reads out one template selected from the plurality of templates stored in the storage device in accordance with the total number of the plurality of people to be photographed by the plurality of photographing systems, and assigns the plurality of people to each of the plurality of regions of the one read template to generate composite data.

5. An image synthesis method in which an image synthesis device having an arithmetic circuit and a communication circuit is connected to a plurality of photographing systems, and which generates synthetic data by synthesizing a plurality of image data of a plurality of different people respectively obtained by the plurality of photographing systems, wherein a plurality of areas in which the plurality of people are respectively arranged in the synthetic data are defined in advance as templates, and each of the photographing systems and each of the areas defined in the template are uniquely associated in advance, and the arithmetic circuit causes each of the photographing systems to capture corresponding image data from the plurality of image data in which a corresponding person of the plurality of people appears in a corresponding area of ​​the plurality of areas defined in the template, obtains the corresponding image data from each of the photographing systems via the communication circuit, and generates the synthetic data by assigning a plurality of extracted images corresponding to the plurality of people respectively extracted from the plurality of image data to the multiple areas of the template, respectively.

6. The image synthesis method according to claim 5, wherein the arithmetic circuit transmits area data indicating the corresponding area of ​​the template assigned to each of the photographing systems via the communication circuit, and each of the photographing systems controls so that the corresponding person appears within the corresponding area indicated by the received area data when acquiring the corresponding image data.

7. The image synthesis method according to claim 6, wherein in each of the photographing systems, at least one control signal for pan control, tilt control, or zoom control is obtained to acquire the corresponding image data in which the corresponding person appears within the corresponding area indicated by the area data received from the image synthesis device, and control is performed using the at least one control signal obtained.

8. The image synthesis method according to claim 5, wherein the arithmetic circuit: receives a plurality of test image data in advance from the plurality of photographing systems via the communication circuit; extracts areas of a plurality of people appearing in the plurality of test image data from the plurality of test image data; generates a plurality of control signals for controlling the plurality of photographing systems according to the extracted areas of the plurality of people so that the plurality of people appear in the plurality of areas of the template; transmits the plurality of control signals to the plurality of photographing systems via the communication circuit; and controls the plurality of photographing systems in accordance with the plurality of control signals received from the image synthesis device.

9. The image synthesis method according to claim 8, wherein each of the control signals is at least one of pan control, tilt control, and zoom control in each of the image capture systems, and the at least one control signal is transmitted to each of the image capture systems via the arithmetic circuit.

10. The image synthesis method according to claim 5, wherein the plurality of photographing systems are arranged under the same conditions with respect to the subject to be photographed.

11. The image synthesis method according to claim 10, wherein the image capturing systems are disposed at the same height from a floor surface on which the corresponding persons are positioned.

12. The image synthesis method according to claim 10, wherein the plurality of image capture systems have the same angle of view as a shooting condition.

13. The image synthesis method according to claim 10, wherein the plurality of image capture systems acquire the corresponding image data of the corresponding persons by setting the same elevation angle or depression angle as the image capture condition.

14. The image synthesis method according to claim 10, wherein one of the plurality of shooting systems is set as a main shooting system and the other shooting systems are set as sub shooting systems, and the arithmetic circuit generates a control signal for controlling the shooting conditions of the sub shooting system to match those of the main shooting system.

15. A computer program for causing an image synthesis device connected to a plurality of photographing systems and equipped with an arithmetic circuit and a communication circuit to generate synthetic data by synthesizing a plurality of image data of a plurality of different people respectively obtained by the plurality of photographing systems, wherein a plurality of areas in which the plurality of people are respectively arranged in the synthetic data are defined in advance as templates, and each of the photographing systems and each of the areas defined in the template are uniquely associated in advance, the computer program being loaded into the image synthesis device and executed to cause the arithmetic circuit to: cause each of the photographing systems to photograph corresponding image data among the plurality of image data in which a corresponding person among the plurality of people appears in a corresponding area among the plurality of areas defined in the template; obtain the corresponding image data from each of the photographing systems via the communication circuit; and generate the synthetic data by assigning a plurality of extracted images corresponding to the plurality of people respectively extracted from the plurality of image data to the plurality of areas of the template, respectively.

Citation Information

Patent Citations

  • Video display system, and presence improving method in the system

    JP1998260808A

  • Image printer, image printing system and image printing method

    JP2016116117A

  • Photographing system

    JP2021048452A

  • Image control device, imaging apparatus, and image control method

    JP2021057659A

  • Information processing device, information processing method, and program

    JP2022123987A