Imaging apparatus, image processing apparatus, imaging system, imaging method, and program

Abstract

Provided are an imaging device, an image processing device, an imaging system, an imaging method, and a program that make it easy to realize augmented reality desired by a user in a captured image.SOLUTION: An imaging device includes an imaging unit that captures a subject image to generate original image data, and a control unit that recognizes a subject in an original image indicated by the original image data. The control unit generates composite image data by compositing a target image indicating a composition target with an original image so as to arrange the composition target different from a subject on the basis of a posture and a position of the subject in the original image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an imaging device, an image processing device, an imaging system, an imaging method, and a program. [Background technology]

[0002] Patent Document 1 discloses an image generation system that generates a composite image by performing a rearrangement process to prevent overlapping of a character with a subject image or a subject character connected to the subject image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6362634 [Patent Document 2] Japanese Patent Application Laid-Open No. 2015-22208 Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure provides an imaging device, an image processing device, an imaging system, an imaging method, and a program that make it easier to realize the augmented reality desired by a user in a captured image. [Means for solving the problem]

[0005] An imaging device according to one aspect of the present disclosure includes: an imaging unit that captures an image of a subject and generates original image data; a control unit that recognizes a subject in an original image represented by the original image data, The control unit generates composite image data by combining a target image indicating a composite target with the original image so as to position a composite target different from the subject based on the posture and position of the subject in the original image.

[0006] An image processing device according to one aspect of the present disclosure includes: An acquisition unit that acquires data and a control unit that processes the acquired data, The control unit Acquiring original image data generated by capturing an image of a subject via an acquisition unit; Recognizing a subject in an original image represented by the original image data; Based on the posture and position of the subject in the original image, a target image representing a synthesis target different from the subject is synthesized with the original image to generate synthetic image data.

[0007] An imaging system according to one aspect of the present disclosure includes: an imaging device including an imaging unit that captures an image of a subject and generates original image data; an image processing device including a control unit that receives original image data from an imaging device and recognizes a subject in an original image indicated by the original image data; The control unit generates composite image data by combining a target image indicating a composite target with the original image so as to position a composite target different from the subject based on the posture and position of the subject in the original image.

[0008] An imaging method according to one aspect of the present disclosure includes: capturing an image of a subject to generate original image data; A step of recognizing a subject in an original image represented by the original image data; generating composite image data by compositing a target image representing a composite target with the original image so as to position a composite target different from the subject based on the posture and position of the subject in the original image; Includes.

[0009] A program according to an aspect of the present disclosure causes a control unit to execute the imaging method. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to easily realize the augmented reality desired by the user in a captured image. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a digital camera according to a first embodiment. [Figure 2] Schematic diagram showing an example of 3D data [Figure 3] Schematic diagram showing an example of a person image [Figure 4] Schematic diagram showing an example of clothing data [Figure 5] 1 is a flowchart illustrating the operation of a digital camera according to a first embodiment; [Figure 6A] Schematic diagram illustrating an operation screen [Figure 6B] Schematic diagram illustrating an operation screen [Figure 6C] Schematic diagram illustrating an operation screen [Figure 7] A flowchart showing an example of the pose registration process of FIG. 5. [Figure 8] A flowchart showing an example of composite photography processing in FIG. [Figure 9] Schematic diagram illustrating a composite image [Figure 10] A flowchart showing an example of the one-shot pose photography process shown in FIG. [Figure 11] 10 is a flowchart illustrating a video shooting process performed by the digital camera 100 according to the second embodiment. [Figure 12] A flowchart showing an example of composite photography processing in FIG. 11. [Figure 13] 13 is a flowchart showing an example of the blur, shadow, and focus adjustment process of FIG. 12. [Figure 14] Schematic diagram for explaining focus adjustment processing [Figure 15A] Schematic diagram for explaining focus adjustment processing [Figure 15B] FIG. 10 is a schematic diagram illustrating a modified example of the focus adjustment process; [Figure 16] FIG. 10 is a schematic diagram illustrating an image processing system according to a third embodiment. [Figure 17] FIG. 10 is a block diagram showing a configuration example of an absentee terminal according to a third embodiment. [Figure 18]FIG. 10 is a sequence diagram illustrating the operation of the image processing system according to the third embodiment. [Figure 19] FIG. 10 is a block diagram showing a configuration example of an imaging system according to a second modification. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0013] 1. Embodiment 1 1-1.Configuration 1-1-1.Hardware configuration 1 is a block diagram showing an example configuration of a digital camera 100 according to a first embodiment of the present disclosure. The digital camera 100 captures an image of a subject and generates image data. The image data generated by the digital camera 100 includes video data and still image data.

[0014] Digital camera 100 generates image data by capturing an image of a subject formed via optical system 110 with image sensor 115. Image processing unit 120 generates image data by performing various processes on the image data generated by image sensor 115. Control unit 135 records the image data generated by image processing unit 120 in flash memory 145 or a memory card 142 inserted in card slot 141. Control unit 135 can also display (play) the image data recorded in flash memory 145 or memory card 142 on display 130 in accordance with a user's operation of operation unit 150.

[0015] The optical system 110 includes a focus lens, a zoom lens, an optical image stabilization lens (OIS), an aperture, a shutter, etc. The various lenses included in the optical system 110 may be configured with any number of lenses or any number of groups.

[0016] The image sensor 115 captures an image of a subject formed via the optical system 110 and generates image data. The image sensor 115 generates image data for a new frame at a predetermined frame rate (e.g., 30 frames / second). The timing of generating the image data and the electronic shutter operation in the image sensor 115 are controlled by the control unit 135.

[0017] The image sensor 115 performs operations such as capturing moving images and still images, and capturing through images. The through images are mainly moving images, and are displayed on the display 130 so that the user can decide on the composition. The image sensor 115 may be any of various image sensors, such as a CMOS image sensor, a CCD image sensor, or an NMOS image sensor. The image sensor 115 is an example of an imaging unit in this embodiment.

[0018] The image processing unit 120 generates image data by performing various processes on the image data output from the image sensor 115. The image processing unit 120 also performs various processes on the image data read from the memory card 142 to generate an image to be displayed on the display 130. The various processes include, but are not limited to, white balance correction, gamma correction, YC conversion processing, electronic zoom processing, compression processing, expansion processing, composite shooting processing (described later), and the like. The image processing unit 120 may be configured as a hardwired electronic circuit, or may be configured as a microcomputer using a program.

[0019] Display 130 is a display device capable of displaying information, such as a liquid crystal display or an organic EL display. For example, display 130 displays an image based on image data processed by image processing unit 120. Display 130 also displays a menu screen that allows the user to check the settings of digital camera 100.

[0020] The control unit 135 controls the overall operation of the digital camera 100. The control unit 135 may include a processor configured to execute programs to achieve predetermined functions. For example, the control unit 135 can be realized by various processors such as a CPU, MPU, GPU, DSU, FPGA, or ASIC. The control unit 135 may be composed of one or more processors. The control unit 135 may also be composed of a single semiconductor chip together with the image processing unit 120, etc. The control unit 135 may have various types of internal memory, and may include, for example, a built-in ROM. The ROM stores various programs executed by the control unit 135. The control unit 135 may also include a built-in RAM that functions as a work area for the CPU.

[0021] The buffer memory 125 is a recording medium that functions as a work memory for the image processing unit 120 and the control unit 135. The buffer memory 125 is realized by a DRAM (Dynamic Random Access Memory) or the like. The flash memory 145 is a non-volatile recording medium.

[0022] A memory card 142 is removably inserted into the card slot 141. The card slot 141 can electrically and mechanically connect to the memory card 142. The memory card 142 is an external memory that includes a recording element such as a flash memory.

[0023] The communication module 143 performs data communication in accordance with existing wired or wireless communication standards. For example, the digital camera 100 can communicate with a communication network such as the Internet and / or other devices equipped with a Wi-Fi module via the communication module 143. The digital camera 100 may communicate with other devices directly via the communication module 143, or may communicate via an access point.

[0024] For example, the digital camera 100 can communicate data with an external storage device 160 via the communication module 143. The storage device 160 may be implemented by, for example, a semiconductor storage device such as a flash memory or a solid-state drive (SSD), a magnetic storage device such as a hard disk drive (HDD), or other recording media, either singly or in combination. The storage device 160 may also be a cloud server configured by one or more server devices.

[0025] Operation unit 150 is a general term for user interfaces such as hard keys and software keys of digital camera 100, and accepts operations by the user. Operation unit 150 includes, for example, buttons, a mode dial, a touch panel, a switch, and a joystick. When operation unit 150 accepts an operation by the user, it transmits an operation signal corresponding to the user operation to control unit 135.

[0026] The operation unit 150 includes various buttons such as a back button, a decision button, a shutter button, and a cursor button. The cursor buttons include push-type buttons corresponding to the up, down, left, and right directions, and when pressed by the user, can move a selection area, a cursor, etc. displayed on the display 130. Operations using these various buttons may be configured to be executable not only by physical buttons but also by a touch panel.

[0027] 1-1-2. Data structure The digital camera 100 captures an image of a subject to generate original image data, and then composites the image of the absent person into the original image so that the image of the absent person is positioned according to the pose (posture) and position of the subject in the original image. For example, the digital camera 100 composites an image of the absent person posing similarly to the subject in the original image into the original image. To achieve this processing, the digital camera 100 stores three-dimensional (3D) data, images of people, clothing data, and the like in advance, for example, in the flash memory 145. These data may be stored in advance in a storage device 160 that can communicate with the digital camera 100.

[0028] FIG. 2 is a schematic diagram showing an example of 3D data 10. The 3D data 10 is data indicating, for example, skeleton information, bone structure information, etc. In the example shown in FIG. 2, information (three-dimensional coordinates) indicating the joint positions of a person is managed as the 3D data 10. There may be one piece of 3D data 10. Alternatively, the 3D data 10 may include bone structure information of the user and bone structure information of each of multiple people who are candidates for absent persons (composite subjects).

[0029] FIG. 3 is a schematic diagram showing an example of a person image 20. The person image 20 is, for example, a group of images obtained by photographing one or more people from various angles. The person image 20 may include depth information obtained when photographing the people. The digital camera 100 can generate images of people in various poses using the person image 20 and the 3D data 10 of FIG. 2. The person image 20 may be an image of the person's face as shown in FIG. 3, or may be an image showing the person's entire body.

[0030] FIG. 4 is a schematic diagram showing an example of clothing data 30. Clothing data 30 is a group of images obtained by photographing various types of clothing from various angles, or a 3D model of the clothing. Clothing data 30 includes classification information indicating the classification of the clothing. Classification information includes, for example, casual, formal, Western clothing, Japanese clothing, clothing color, pattern, etc. Clothing data 30 may also include depth information, three-dimensional position information, etc. obtained when photographing the clothing. Digital camera 100 can generate images of people wearing desired clothing and striking various poses using clothing data 30 and the 3D data 10 of FIG. 2.

[0031] 1-2.Operation 1-2-1. Overall operation As described above, for example, digital camera 100 composites an image of an absentee person who is posing similarly to the subject in the original image into the original image. This allows digital camera 100 to capture an image that makes it appear as if an absent person who was unable to be in the shooting environment is actually in the shooting environment (hereinafter, this may be referred to as "synchronous shooting"). The operation of digital camera 100 to achieve this effect will be described below.

[0032] 5 is a flowchart illustrating the operation of the digital camera 100 according to this embodiment. Each process shown in FIG.

[0033] The control unit 135 acquires the 3D data 10, the person image 20, and the clothing data 30 from, for example, the flash memory 145 (S1).

[0034] The control unit 135 receives, via the operation unit 150, an input of a user operation to select an image of a target person (e.g., an absent person) to be composited (i.e., composited with the original image) from among the person images 20 (S2). The control unit 135 also receives, via the operation unit 150, an input of a user operation to select clothing to be applied to the image of the target person and a position to place the target person (S3).

[0035] 6A to 6C are schematic diagrams illustrating operation screens for user operation. The operation screens are displayed on the display 130. FIG. 6A illustrates an example of a setting screen for synchronized photography in this embodiment. When the "Select Subject Data" menu is selected, the screen transitions to a subject selection screen, where the user selects a subject. The subject selection screen displays, for example, multiple person images or identification information indicating the names, IDs, etc. of these people, and the user selects a subject from among these.

[0036] When the "Select Clothes" menu is selected in Fig. 6A, the control unit 135 transitions the screen to a clothing selection screen in Fig. 6B. On the clothing selection screen, the user can select whether the desired clothing is casual or formal, whether Western or Japanese clothing, the color of the desired clothing, etc.

[0037] When the "Select placement" menu is selected in Fig. 6A, the control unit 135 transitions the screen to a placement selection screen in Fig. 6C. In the placement selection screen, the user can select whether to place the target person in contact with the subject (user) or at a distance from the subject. In addition, in the placement selection screen, the user can select whether to place the target person to the right or left of the subject in the image.

[0038] Returning to FIG. 5, the control unit 135 captures an image of the subject to generate original image data, and causes the display 130 to display a live view screen including a through image indicated by the original image data and icons for various operations (S4).

[0039] The control unit 135 acquires depth information of the subject together with the original image data. Alternatively, the control unit 135 acquires the depth information of the subject included in the original image data. The depth information of the subject represents, for example, the distance between the image sensor 115 and the subject. The depth information is acquired by various known methods. For example, the depth information is acquired by a distance sensor mounted on the digital camera 100. The depth information may be obtained by a method of estimating the depth information from an RGB image. Alternatively, or in addition to this, the depth information may be acquired by at least one of an image plane phase difference detection technique and a DFD (Depth From Defocus) technique.

[0040] Next, the control unit 135 determines whether or not an omission setting has been made to omit the pose registration process S7 (described later) (S5). The pose registration omission setting can be set in advance by a user operation via the operation unit 150.

[0041] If pose registration has not been set to be omitted (No in S5), the control unit 135 determines whether or not the pose registration process S7 has already been completed and the pose has been registered (S6).

[0042] If the pose has not been registered (No in S6), the control unit 135 executes a pose registration process S7. If the pose has been registered (Yes in S6), the control unit 135 executes a composite photographing process S8 in which an image of the absentee taking a pose similar to the registered pose is composited with the original image.

[0043] On the other hand, if pose registration is set to be omitted in step S5, the control unit 135 executes a one-shot pose composite photographing process S9 (hereinafter referred to as "one-shot pose photographing process") that can composite an image of an absentee person who has taken a pose similar to that of the subject into the original image without prior pose registration.

[0044] The pose registration process S7, composite photography process S8, and one-shot pose photography process S9 will be described in detail later.

[0045] 1-2-2. Pose registration process FIG. 7 is a flowchart showing an example of the pose registration process S7 of FIG.

[0046] In this embodiment, an example will be described in which the user himself / herself is the subject, and the user's pose is automatically photographed and registered. However, the present disclosure is not limited to this. For example, the user may photograph and register his / her own pose using timer photography. Alternatively, the user may operate the digital camera 100 to photograph the pose of a third party as the subject. Conversely, a third party may operate the digital camera 100 to photograph the user's pose as the subject.

[0047] In the pose registration process S7, first, the control unit 135 recognizes the subject in the original image indicated by the original image data (S11). For example, the control unit 135 applies a person recognition technique to the original image data and identifies the three-dimensional shape of the subject based on depth information. Then, for example, the control unit 135 estimates the position and motion vector of the subject using an optical flow method, thereby estimating the positions of the subject's joints. The control unit 135 determines 3D data (see FIG. 2) corresponding to the estimated joint positions and acquires information regarding the position and pose of the subject.

[0048] Next, the control unit 135 determines whether the pose of the subject has continued for a predetermined time (S12). Based on the information of the subject recognized in step S11, the control unit 135 determines whether the pose of the subject has continued for a predetermined time (e.g., several seconds). For example, if the pose of the subject has remained still for the predetermined time, the control unit 135 determines that the pose has continued for the predetermined time. Even if the subject has moved slightly, if the movement distance of the position of the joints of the subject is less than a predetermined threshold, the control unit 135 determines that the pose of the subject is still.

[0049] If it is determined that the pose of the subject has not continued for the predetermined time (No in S12), the control unit 135 ends the processing of FIG. 7 and returns to step S4 in FIG.

[0050] If it is determined that the pose of the subject has continued for a predetermined time (Yes in S12), the control unit 135 generates a subject image showing a composite subject taking a pose similar to that of the subject (S13). The control unit 135 places the generated subject image in the placement position selected in step S3 in the image. The control unit 135 displays the image in which the subject image has been placed on the live view screen.

[0051] While checking the live view screen, the user can adjust the pose and placement position of the subject image using the operation unit 150. The control unit 135 receives a user operation input for adjusting the pose and placement position of the subject image via the operation unit 150, and makes the adjustment in accordance with the user operation (S14). If there is no user operation input, step S14 may be omitted.

[0052] The control unit 135 registers the data of the image of the subject that has been adjusted in step S14 (S15). For example, the data of the image of the subject that has been adjusted is recorded in the flash memory 145.

[0053] 1-2-3. Shooting composite processing Fig. 8 is a flowchart showing an example of the composite photographing process S8 of Fig. 5. As shown in Fig. 5, the composite photographing process S8 is executed after the pose registration process S7. The composite photographing process S8 may be started when the control unit 135 receives a user operation to select a photographing mode such as the composite photographing mode.

[0054] In the composite photographing process S8, the user checks the image in which the person to be composited (absent person) is positioned on the live view screen, and then places the user as the subject in the field of view of the digital camera 100. The photographing may be performed automatically (without a photographer) or may be performed by a photographer other than the user. Alternatively, the user may act as the photographer and photograph a third party as the subject.

[0055] The control unit 135 recognizes a subject in the original image (S21). In step S21, for example, the same person recognition technology as in step S11 can be used.

[0056] The control unit 135 generates composite image data by superimposing the subject image registered in the pose registration process S7 on the original image (S22). In the composite image represented by the composite image data, the subject image assumes the pose registered in the pose registration process S7.

[0057] The control unit 135 adjusts blurring, shading, and overlapping of the composite image for the composite subject. In step S23, depth information obtained at the time of shooting is used. For example, the control unit 135 performs blurring processing on the subject image according to the focus position of the optical system and the position on the original image where the subject image is placed.

[0058] In step S23, control unit 135 shades the subject image and / or the composite image according to the position on the composite image where the subject image is placed. For example, control unit 135 shades based on the positional relationship between the subject and the subject in the composite image, and the positional relationship between other objects and the subject. When performing the shading process, control unit 135 may estimate the position of the light source based on the original image or the composite image, and shade based on the estimated position of the light source.

[0059] In step S23, control unit 135 adjusts the overlap between the subject image and the target image in the composite image based on the depth information. For example, when at least a part of the subject image and at least a part of the target image are present at the same position on the image and overlap, control unit 135 displays only the image with the shallower depth (closer to digital camera 100) in the overlapping area.

[0060] Control unit 135 displays the adjusted composite image on display 130 (S24). FIG. 9 is a schematic diagram illustrating a composite image displayed on display 130. The composite image in FIG. 9 displays subject image 40 and target person image 50 indicating a composite target (absent person). In the illustrated example, the arm of subject image 40 and the arm of target person image 50 overlap, but due to the overlap adjustment process included in step S23, the arm of target person image 50 appears to be hidden by the arm of subject image 40.

[0061] When the user wishes to correct the position, etc., of the subject image 50, the user can correct the position, etc. by performing a user operation to specify the position, etc. via the operation unit 150, such as a touch panel superimposed on the display 130 or buttons for operating a setting menu. Here, the "position, etc." includes at least one of the position where the subject image 50 is placed in the composite image, the size of the subject image, or the pose (posture) of the subject to be composited. Upon receiving input of such a user operation, the control unit 135 adjusts the position, etc., of the subject image 50 (S25).

[0062] When automatic photography is performed without a photographer, steps S24 and S25 may be omitted.

[0063] The control unit 135 determines whether or not a shooting instruction has been issued (S26). When shooting by a photographer, the control unit 135 determines that a shooting instruction has been issued when the photographer presses the shutter button, for example. When shooting automatically without a photographer, the control unit 135 determines that a shooting instruction has been issued when the subject maintains a pose for a predetermined period of time (for example, several seconds).

[0064] If it is determined that a shooting instruction has been issued (Yes in S26), the control unit 135 stores the composite image data in, for example, the flash memory 145. If a shooting instruction has not been issued (No in S26), the process returns to step S21.

[0065] 1-2-4. One-shot pose shooting processing FIG. 10 is a flowchart showing an example of the one-shot pose photography process S9 of FIG.

[0066] First, the control unit 135 recognizes a subject in an original image indicated by the original image data (S31). Next, the control unit 135 generates a subject image indicating a subject to be synthesized who is posing similarly to the pose of the subject, and generates synthetic image data by arranging the generated subject image in the original image at the arrangement position selected in step S3 (S32).

[0067] The control unit 135 adjusts blurring, shading, and overlapping of the subject in the composite image indicated by the composite image data (S33). The control unit 135 displays the adjusted composite image on the display 130 (S34). When automatic photography is performed without a photographer, step S34 may be omitted.

[0068] The control unit 135 determines whether or not a shooting instruction has been issued (S35). If it is determined that a shooting instruction has been issued (Yes in S35), the control unit 135 stores the composite image data in, for example, the flash memory 145 (S36). If a shooting instruction has not been issued (No in S35), the process returns to step S31.

[0069] 1-3. Summary As described above, the digital camera 100 according to this embodiment includes the image sensor 115, which is an example of an imaging unit that captures an image of a subject and generates original image data, and the control unit 135 that recognizes the subject in the original image indicated by the original image data. The control unit 135 generates composite image data by combining a subject image indicating a subject to be combined with the original image based on the posture and position of the subject in the original image, so as to position the subject to be combined differently from the subject.

[0070] The digital camera 100 according to this embodiment can easily realize the augmented reality desired by the user in a captured image. For example, the digital camera 100 can capture an image in which an absent person, who is an example of a composite subject who was unable to visit the shooting environment, appears as if they were in the shooting environment.

[0071] In the digital camera 100 according to this embodiment, the control unit 135 may set the posture of the person to be synthesized in the synthesized image shown by the synthesized image data to a posture corresponding to the posture of the subject in the original image, based on the 3D data 10 that specifies the posture of the person to be synthesized.

[0072] According to this configuration, the posture of the person to be synthesized in the synthesized image can be set to a posture desired by the user.

[0073] In the digital camera 100 according to this embodiment, the control unit 135 may generate composite image data by combining an image of the subject with the original image so that the image of the subject appears together with an image showing the subject in the composite image indicated by the composite image data.

[0074] This configuration makes it possible to capture an image in which the subject of synthesis appears as if they were in the shooting environment.

[0075] In the digital camera 100 according to this embodiment, the control unit 135 may acquire depth information indicating the distance between the image sensor 115 and the subject, and adjust the overlap between the image showing the subject and the image of the subject in the composite image based on the depth information.

[0076] In the digital camera 100 according to this embodiment, the control unit 135 may apply a blurring process to the image of the subject depending on the focus position of the optical system 110 and the position on the original image where the image of the subject is placed, and combine the image of the subject with the original image.

[0077] In the digital camera 100 according to this embodiment, the control unit 135 may perform shading processing to add shading to the subject image depending on the position on the original image where the subject image is placed, and then combine the subject image with the original image.

[0078] These configurations make it possible to display the image showing the subject and the image of the subject in the composite image more naturally, and to capture an image in which the subject of the composite image appears as if they were in the photographing environment.

[0079] The digital camera 100 according to this embodiment may further include an operation unit 150, which is an example of a setting unit. The operation unit 150 accepts a user operation to specify at least one of the position where the subject image is to be placed, the size of the subject image, and the posture of the subject to be synthesized in the composite image indicated by the composite image data.

[0080] This configuration makes it easier to realize the augmented reality desired by the user in the captured image.

[0081] In the digital camera 100 according to this embodiment, the operation unit 150 may accept a user operation that specifies the clothing of a person to be composited in the composite image represented by the composite image data. In this case, the control unit 135 may change the clothing of the person to be composited in the composite image in response to the user operation.

[0082] This configuration makes it easier to realize the augmented reality desired by the user in the captured image.

[0083] 2. Embodiment 2 In the first embodiment, an example in which an image of a compositing subject is mainly combined with a still image has been described, but in the second embodiment, an example in which an image of a compositing subject is combined with a moving image will be described. In the following description of this embodiment, the description of the same configuration and operation as in the first embodiment will be omitted as appropriate.

[0084] FIG. 11 is a flowchart illustrating the operation of the digital camera 100 according to the second embodiment (hereinafter, sometimes referred to as "moving image shooting processing").

[0085] In this embodiment, the control unit 135 executes steps S1 to S4 similar to those in the first embodiment, and then determines whether or not a video recording start operation by the user has been accepted (S201). For example, the user may accept a video recording start operation via the operation unit 150. If it is determined that a video recording start operation has not been accepted, the process returns to step S4.

[0086] When it is determined that the moving image shooting start operation has been accepted (Yes in S201), the control unit 135 executes the composite shooting process S202. The control unit 135 repeats the composite shooting process S202 until the control unit 135 accepts the moving image shooting end operation from the user (S203). The composite shooting process S202 will be described in detail below.

[0087] FIG. 12 is a flowchart showing an example of the composite shooting process S202 in FIG.

[0088] First, the control unit 135 recognizes a subject in the original image indicated by the original image data (S41).

[0089] Next, the control unit 135 determines whether or not the target person image can be placed in the original image (S42). For example, the control unit 135 uses depth information obtained at the time of shooting to determine whether or not there is enough space in the space captured in the original image to place the target person image. For this determination, known person recognition technology or structure recognition technology may be used.

[0090] If it is determined that the subject image can be placed in the original image (Yes in S42), the control unit 135 detects the pose of a predetermined subject in the original image (S43). The predetermined subject may be set in advance by a user operation, or may be set to the subject closest to the placement position of the subject image. In step S43, for example, the control unit 135 determines that the predetermined subject is not posed when a person who is the predetermined subject is not shown in the original image, or when a person's face is shown but not their body is shown, etc.

[0091] If it is determined that the predetermined subject pose exists (Yes in S43), the control unit 135 generates a subject image showing a subject to be synthesized who has taken a pose similar to the pose of the predetermined subject. The control unit 135 generates synthetic image data by arranging the generated subject image in the arrangement position identified in step S42 in the original image (S44).

[0092] Here, a pose similar to the pose of the subject may be a pose that is completely identical to the pose of the subject, or a pose that is shifted temporally and / or spatially from the pose of the subject. For example, if it is unnatural for the subject and the target to move in a completely identical manner in a composite video, a temporal and / or spatial shift is imparted to the pose of the target from the pose of the subject by presetting or user operation. For example, the control unit 135 imparts random movements to the movement of the target to impart a temporal and / or spatial shift from the pose of the subject.

[0093] If it is determined that the predetermined pose of the subject is not present (No in S43), the control unit 135 generates composite image data by arranging an image of the subject performing a pre-registered action on the original image (S45).

[0094] After executing step S44 or S45, the control unit 135 adjusts the overlap between the subject image and the target person image in the composite image based on the depth information (S46).

[0095] The control unit 135 performs adjustment processing for blurring, shading, and focus for the composite subject in the composite image indicated by the composite image data (S47). Details of step S47 will be described later. The control unit 135 displays the adjusted composite image on the display 130 (S48), and stores it in a recording medium such as flash memory 145 or memory card 142 (S49).

[0096] FIG. 13 is a flowchart showing an example of the blur, shadow, and focus adjustment process S47 in FIG.

[0097] 13, the control unit 135 first performs AF control based on a preset autofocus (AF) mode (S52). The AF mode includes, for example, an automatic recognition mode, a tracking mode, a full area mode, a zone mode, a single-point focus mode, a combination of the above modes, etc., with respect to the focus area.

[0098] The focus area indicates an image area within a range to be focused on in a captured image such as an original image. The focus area is set using, for example, a touch panel superimposed on display 130. For example, when a subject on a composite image displayed on display 130 is touched, control unit 135 sets the area occupied by the touched subject as the focus area.

[0099] In the automatic recognition mode, the control unit 135 recognizes people or animals and automatically adjusts the focus area to them. In the tracking mode, the control unit 135 tracks the movement of a subject and continues to adjust the focus area to the tracked subject. In the full area mode, the control unit 135 selects the optimal focus area from the entire captured image. In the zone mode, the control unit 135 adjusts the focus area to a partial zone of the captured image.

[0100] Next, control unit 135 adjusts the blur and shading of the subject to be composited in the composite image (S53). Step S53 may be the same process as step S23 in FIG. 8. For example, control unit 135 adjusts the degree of blur (amount of blur) of the subject image in the composite image using preset relative placement setting information of the subject with respect to the subject (see, for example, FIG. 6C). For example, if the subject is set to be positioned 1 m behind the subject, control unit 135 calculates the relative degree of blur at a position 1 m behind the subject using the degree of blur at the position of the subject as a reference, and adjusts the image of the subject by adding the calculated relative degree of blur to the degree of blur of the subject.

[0101] The control unit 135 determines whether or not the screen has been touched using the touch panel (S54). If it is determined that the screen has not been touched (No in S54), the control unit 135 ends the flow of Fig. 13. If it is determined that the screen has been touched (Yes in S54), the control unit 135 determines whether or not the image at the touched position on the composite image is a target person image (S55).

[0102] If it is determined that the image at the touched position is a subject image (Yes in S55), the control unit 135 adjusts the position of the focus lens so as to focus on a position corresponding to the subject image (S56). Because there is no subject in the shooting environment and the subject is not captured in the original image, in step S56 the control unit 135 adjusts the focus so as to focus on the "position corresponding to the subject image" rather than on the subject.

[0103] For example, when the user selects a subject image by moving a joystick or touching the subject image displayed on the display 130, the control unit 135 adjusts the position of the focus lens so as to focus on a position corresponding to the subject image. The operation of selecting a subject image may be performed by eye-gaze input as disclosed in Patent Document 2, for example. Patent Document 2 discloses a technology for detecting the photographer's eye-gaze direction, detecting which area in the viewfinder field the photographer is observing, and controlling shooting functions such as automatic focus adjustment.

[0104] Next, the control unit 135 adjusts the blur for the composite subject in the composite image (S57). Since the position of the focus lens has already been adjusted in step S56 to focus on a position corresponding to the subject image, in step S57 the control unit 135 reduces the degree of blur for the subject image so that the subject image appears to be in focus. For images of other subjects, the control unit 135 may generate a composite image by generating images with a changed degree of focus using relative distance information from the position of the subject to the subject.

[0105] In step S55, if it is determined that the image at the touched position is not the target person image (No in S55), control unit 135 adjusts the position of the focus lens so as to focus on the touched subject (S58).

[0106] Next, control unit 135 adjusts the blur for the composite subject (S59). The position of the focus lens has already been adjusted in step S58 to focus on the touched subject. Therefore, in step S59, control unit 135 generates an image of the subject with a changed focus level using information about the relative distance between the in-focus position of the focus lens and the subject, and composites the generated image of the subject with the original image.

[0107] Steps S54 to S59 relating to focus adjustment will be further described with reference to Figures 14, 15A, and 15B. Figures 14, 15A, and 15B are schematic diagrams illustrating live view screens according to this embodiment, displayed on the display 130. In this embodiment, the live view screen displays a composite image in which a subject image 50 is composited with a through image. That is, since a subject does not exist in the shooting environment, the subject image 50 is not included in the through image, but the subject image 50 appears on the live view screens shown in Figures 14, 15A, and 15B.

[0108] 14 shows a situation in which the user touches the subject image 41. In this case, the screen has been touched (Yes in S54 of FIG. 13), but the target person image 50 is not located at the touched position (No in S55), so the control unit 135 adjusts the position of the focus lens so as to focus on the flower indicated by the touched subject image 41 (S58).

[0109] Subject image 40 showing a person who exists at a different depth (distance from digital camera 100 or image sensor 115) in the shooting environment than the flower shown in subject image 41 is blurred in the through image because the position of the focus lens is adjusted in step S58 to focus on the flower.

[0110] On the other hand, since there is no subject in the shooting environment and the through image does not include the subject image 50, the control unit 135 adjusts the blur of the subject image 50 in the composite image (S59). For example, the control unit 135 adjusts the degree of blur of the subject image 50 in the composite image using relative distance information between the in-focus position of the focus lens and the position of the subject image 50. The adjusted composite image is displayed on the live view screen. As a result, as shown in FIG. 14, the subject image 41 is in focus and a composite image in which the subject image 40 and the subject image 50 are blurred is displayed on the live view screen.

[0111] 15A shows a situation in which the user touches the target person image 50. In this case, the screen is touched (Yes in S54), and the target person image 50 is located at the touched position (Yes in S55), so the control unit 135 adjusts the position of the focus lens so as to focus on the position corresponding to the target person image 50 (S56).

[0112] In step S56, the position of the focus lens is adjusted to focus on a position corresponding to subject image 50, so that a real subject present in the shooting environment will appear blurred or in focus depending on its physical distance from digital camera 100. In the example of Fig. 15A, the person shown in subject image 40 is at a similar distance to the position of subject image 50 as viewed from digital camera 100, and therefore is in focus as a result of step S56. In contrast, the flower shown in subject image 41 is at a different distance from the position of subject image 50 as viewed from digital camera 100, and therefore appears blurred in the through-the-lens image.

[0113] Alternatively, or in addition, when a composite subject is specified as the focus target by user operation (Yes in S55), the control unit 135 may apply a blurring process to the subject image showing the subject according to depth information indicating the depth of the subject and the position on the original image where the subject image is placed.

[0114] FIG. 15B shows an example in which such blurring processing has been performed on subject image 40. The depths (distances from digital camera 100) of subjects 40 and 41 in the shooting environment are the same in FIGS. 15A and 15B. The person shown in subject image 40 is located slightly closer to digital camera 100 than the position of subject image 50. In such a case, control unit 135 may perform blurring processing on subject image 40. Control unit 135 may also perform blurring processing on subject image 41. Alternatively, control unit 135 may perform blurring processing on the subject image regardless of depth information indicating the depth of the subject or the position on the original image where the subject image is located.

[0115] In this way, by performing a blurring process on at least one of the subject images 40, 41 showing a subject that actually exists in the shooting environment, it is possible to achieve shooting that looks as if a lens with a shallower depth of field (for example, a bright telephoto lens) was used than the lens actually used.

[0116] As described above, a subject that actually exists in the shooting environment appears naturally blurred or in focus in the through image depending on the position of the focus lens adjusted in step S56. On the other hand, since there is no subject in the shooting environment and the through image does not include subject image 50, control unit 135 adjusts the blur of subject image 50 in the composite image so that it appears in focus (S57). Furthermore, control unit 135 may apply blurring to at least one of subject images 40 and 41, as shown in FIG. 15B. The adjusted composite image is displayed on the live view screen. As a result, a composite image in which subject image 50 is in focus is displayed on the live view screen, as shown in FIGS. 15A and 15B.

[0117] As described above, in digital camera 100 according to this embodiment, operation unit 150, which is an example of a setting unit, accepts a user operation that specifies a target to be focused on by optical system 110 (for example, a focus lens) in a composite image indicated by composite image data. When a composite target person is specified as a target to be focused on by a user operation, control unit 135 adjusts the focus position of the optical system so that the target person image is positioned in the composite image.

[0118] The digital camera 100 according to this embodiment can composite an image of a subject into a moving image.

[0119] Furthermore, the digital camera 100 according to this embodiment can generate an image that appears to be in focus on a subject that does not exist in the shooting environment, making it easier to realize the augmented reality desired by the user in the captured image.

[0120] 3. Embodiment 3 In the second embodiment, a video shooting process was described in which the digital camera 100 generates a composite image as a video (simply referred to as a "video" in this embodiment) (see FIG. 11). In the third embodiment, an example will be described in which an absentee, who is an example of a subject of composition, receives a video from the digital camera 100 via an absentee terminal and records his or her own voice along with the received video.

[0121] 16 is a schematic diagram illustrating an overview of an image processing system 1 according to embodiment 3. The image processing system 1 includes a digital camera 100 and an absentee terminal 200 communicably connected to the digital camera 100 via a network 60. The absentee terminal 200 is an information processing device such as a smartphone, a tablet terminal, or a personal computer.

[0122] The digital camera 100 performs lip-sync processing, moving the mouth of the image of the absent person in the video in sync with the recorded data, to generate a lip-synced video (lip-sync video). The lip-sync video is transmitted to the absent person terminal 200, and the absent person 70 can view the lip-sync video displayed on the internal or external display of the absent person terminal 200. This allows the absent person 70 to feel as if they are actually in the recording environment. For example, even if a woman (absent person 70) lying in a hospital bed cannot attend a relative's wedding, she can virtually participate in the wedding by viewing the absent person image in the video. In addition, the absent person 70 can feel even more realistic about attending the wedding by viewing a lip-sync image in which her own utterance of "congratulations" is reflected in the absent person image in the video.

[0123] The configuration and operation of the image processing system 1 according to this embodiment will be described in further detail below.

[0124] 17 is a block diagram showing an example of the configuration of an absentee terminal 200 according to embodiment 3. The absentee terminal 200 includes a control unit 201, a storage unit 202, an input / output interface (I / F) 203, and a communication interface 204.

[0125] The control unit 201 performs information processing to realize the functions of the absentee terminal 200. Such information processing is realized, for example, by the control unit 201 operating in accordance with a program stored in the storage unit 202. The control unit 201 includes an arithmetic circuit that performs calculations for information processing. For example, the control unit 201 is configured with circuits such as a CPU, an MPU, and an FPGA.

[0126] The storage unit 202 is a recording medium that stores various information including programs necessary to realize the functions of the absentee terminal 200. The storage unit 202 is realized, for example, by a semiconductor storage device such as a flash memory, an SSD, a magnetic storage device such as an HDD, or other recording media, either alone or in combination. The storage unit 202 may include memories such as SRAM and DRAM.

[0127] The input / output interface 203 is an example of an input unit that connects the absentee terminal 200 to an input device such as the voice input device 210 in order to input information from the input device to the absentee terminal 200. The input / output interface 203 is also an example of an output unit that connects the absentee terminal 200 to an output device so that the absentee terminal 200 can output a signal to an output device such as the display 220, a voice output device, or a printer.

[0128] The communication interface 240 performs data communication in accordance with existing wired or wireless communication standards.

[0129] 18 is a sequence diagram for explaining the operation of the image processing system 1 according to this embodiment. The operation of FIG. 18 is executed by the control unit 135 of the digital camera 100 and the control unit 201 of the absentee terminal 200.

[0130] First, the digital camera 100 executes a video shooting process S301. An example of the video shooting process S301 is the video shooting process shown in Fig. 11. In the video shooting process S301, video data is generated that shows a video in which an image of a subject to be combined is combined with an original image. In this embodiment, the video data may be simply referred to as a "video."

[0131] The digital camera 100 transmits the video to the absentee terminal 200 (S302). The absentee terminal receives the video (S303) and displays the received video on the display 220 (S304). This allows the absent person 70 to view the video combined with an absent person image representing the absent person 70.

[0132] The absent person 70 can record his / her own voice into the video by speaking into the audio input device 210 such as a microphone in time with the video. The absent person terminal 200 acquires the audio of the absent person 70 via the audio input device 210 (S305) and transmits the audio data to the digital camera 100 (S306).

[0133] The digital camera 100 receives the audio data (S307) and executes lip-sync processing to control the speech state of the target person image in the video in accordance with the received audio data (S308). For example, the control unit 135 of the digital camera 100 controls the speech state of the target person image by synchronizing the movement of the mouth, etc. of the target person image with the audio data. For the lip-sync processing, a known technique such as that described in Japanese Patent No. 7314387 is used.

[0134] The digital camera 100 stores the lip-sync processed video (lip-sync video) in a recording medium such as the flash memory 145 or the memory card 142 (S309).

[0135] The digital camera 100 transmits the lip-sync video to the absentee terminal 200 (S310). The absentee terminal receives the lip-sync video (S311) and displays the received lip-sync video on the display 220 (S312). This allows the absentee 70 to view the lip-sync video and feel as if he or she were in the shooting environment.

[0136] If the shooting environment is an event venue such as a wedding ceremony, the lip-sync video may be displayed on a display device such as a display or projector installed at the event venue.

[0137] Furthermore, the audio represented by the audio data transmitted from the absentee terminal 200 in step S306 may be output from an audio output device such as a speaker installed at the event venue. In this case, the digital camera 100 may acquire the audio output from the audio output device using an audio input device such as a microphone, and store the lip-sync video together with the audio data represented by the acquired audio on a recording medium.

[0138] As described above, in the digital camera 100 according to this embodiment, the control unit 135 acquires voice data indicating the speech of the person to be synthesized (S307), and controls the speech state of the person to be synthesized indicated by the subject image based on the voice data (S308).

[0139] A control signal for controlling the movement of the subject image in the video may be transmitted to the digital camera 100, and the digital camera 100 may control the movement state of the subject image in the video based on the received control signal.

[0140] For example, the absent person 70 performs a control operation using an input device such as a game controller, and the absent person terminal 200, upon receiving the control operation, transmits a control signal corresponding to the control operation to the digital camera 100. As a result, for example, the absent person 70 can make the absent person image representing the absent person 70 in the video perform actions such as raising his / her hands, clapping, or winking. The input device is not limited to a game controller, but may be a keyboard, mouse, touch panel, or a user interface using motion capture. By using motion capture, for example, the absent person 70 can reflect movements similar to those he / she made in his / her hospital room in the absent person image in the video.

[0141] 4. Other Embodiments As described above, the embodiments have been described as examples of the technology in the present disclosure. However, the technology in the present disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. Therefore, below, modified examples as other embodiments are shown.

[0142] 4-1. First modified example In the above embodiments, an example has been described in which synthetic image data is generated by combining an image of a subject with an original image so that the image of the subject appears together with an image showing the subject in the synthetic image shown by the synthetic image data (see, for example, S22 in FIG. 8, S32 in FIG. 9, and S32 in FIG. 10, etc.). However, it is not necessary for both the subject and the subject to appear in the synthetic image, and the control unit 135 may place the subject in place of the subject. For example, the control unit 135 generates synthetic image data by combining an image of the subject in place of the image showing the subject with a background image showing the background of the subject in the original image.

[0143] 4-2. Second modified example In the above embodiments, examples have been described in which the control unit 135 of the digital camera 100 composites the image of the subject to be composited with the original image, but the composite may also be performed by an external device. Fig. 19 is a block diagram showing an example of the configuration of an imaging system 2 according to such a second modified example.

[0144] The imaging system 2 includes a digital camera 100 and an image processing device 300 communicably connected to the digital camera 100 via a network 60. The image processing device 300 is an information processing device such as a smartphone, a tablet terminal, or a personal computer. The image processing device 300 may also be an administration server that manages application software.

[0145] The image processing device 300 includes a control unit 301 , a storage unit 302 , an input / output interface 303 , and a communication interface 304 .

[0146] The control unit 301 performs information processing to realize the functions of the image processing device 300. Such information processing is realized, for example, by the control unit 301 operating in accordance with a program stored in the storage unit 302. The control unit 301 includes an arithmetic circuit that performs calculations for information processing. For example, the control unit 301 is configured with circuits such as a CPU, an MPU, and an FPGA.

[0147] The storage unit 302 is a recording medium that records various information including programs necessary to realize the functions of the image processing device 300. The storage unit 302 is realized, for example, by a semiconductor storage device such as a flash memory, an SSD, a magnetic storage device such as an HDD, or other recording media, either alone or in combination. The storage unit 302 may include memories such as SRAM and DRAM.

[0148] The input / output interface 303 connects the image processing device 300 to an input device. The input / output interface 303 also connects the image processing device 300 to an output device. The communication interface 240 is an example of an acquisition unit that acquires data, and performs data communication in accordance with an existing wired communication standard or wireless communication standard.

[0149] In this modification, digital camera 100 captures an image of a subject and generates original image data. Control unit 301 of image processing device 300 receives the original image data from digital camera 100 and recognizes the subject in the original image indicated by the original image data. Based on the posture and position of the subject in the original image, control unit 301 generates composite image data by combining a subject image indicating a subject to be combined with the original image so as to position a subject to be combined that is different from the subject.

[0150] 4-3.Third modified example In the above embodiments, an example has been described in which a subject image representing a subject to be synthesized is synthesized with an original image. The subject image may be an avatar image as long as it can represent the subject to be synthesized. The subject image is not limited to a person, and may be an animal such as a dog or cat, and the subject image may appear together with an image representing an animal subject such as a dog.

[0151] 4-4.Fourth Modification In the above embodiments, a digital camera has been described as an example of an imaging device, but the imaging device is not limited to this. The imaging device of the present disclosure may be an electronic device having an image capturing function (for example, a video camera, a smartphone, a tablet terminal, etc.).

[0152] 5.Examples The following describes exemplary aspects of the present disclosure.

[0153] <Aspect 1> an imaging unit that captures an image of a subject and generates original image data; a control unit for recognizing a subject in an original image represented by the original image data, the control unit generates composite image data by compositing a target image indicating the composite target with the original image so as to arrange a composite target different from the subject based on the posture and position of the subject in the original image. Imaging device.

[0154] <Aspect 2> the control unit sets the orientation of the synthesis target in the synthetic image represented by the synthetic image data to an orientation corresponding to the orientation of the subject in the original image, based on three-dimensional data defining the orientation of the synthesis target. 2. The imaging device according to claim 1.

[0155] <Aspect 3> The imaging device according to aspect 1 or 2, wherein the control unit generates the composite image data by combining the target image with a background image showing the background of the subject in the original image in place of an image showing the subject.

[0156] <Aspect 4> The imaging device according to any one of aspects 1 to 3, wherein the control unit generates composite image data by combining the target image with the original image so that the target image appears together with an image showing the subject in the composite image represented by the composite image data.

[0157] <Aspect 5> The control unit acquiring depth information indicating a distance between the imaging unit and a subject; adjusting an overlap between the image showing the subject and the target image in the composite image based on the depth information; 5. The imaging device according to embodiment 4.

[0158] <Aspect 6> An imaging device described in any one of aspects 1 to 5, wherein the control unit applies a blurring process to the target image and combines the target image with the original image depending on the focus position of the optical system and the position on the original image where the target image is placed.

[0159] <Aspect 7> The imaging device according to any one of aspects 1 to 6, wherein the control unit applies shading to the target image depending on the position on the original image where the target image is placed, and then combines the target image with the original image.

[0160] <Aspect 8> a setting unit configured to accept a user operation to specify a focus target of the optical system in the composite image represented by the composite image data; When the synthesis target is designated as the focusing target by the user operation, the control unit adjusts the focus position of the optical system so that the focus position of the optical system becomes a position where the target image is arranged in the synthetic image. The imaging device according to any one of aspects 1 to 7.

[0161] <Aspect 9> The control unit acquiring depth information indicating a distance between the imaging unit and a subject; When the synthesis target is designated as the focusing target by the user operation, blurring is performed on the subject image showing the subject according to the depth information and the position on the original image where the target image is to be placed. An imaging device according to embodiment 8.

[0162] <Aspect 10> The imaging device according to any one of aspects 1 to 9, further comprising a setting unit that accepts a user operation that specifies at least one of the position where the target image is to be placed, the size of the target image, or the orientation of the target to be synthesized in the synthetic image represented by the synthetic image data.

[0163] <Aspect 11> a setting unit configured to accept a user operation to specify clothing to be combined in a composite image represented by the composite image data; 11. The imaging device according to any one of aspects 1 to 10, wherein the control unit changes the clothing of the object to be synthesized in the synthesized image in response to the user operation.

[0164] <Aspect 12> the composite object is a person, the control unit acquires voice data indicating the speech of the person, and controls the speech state of the synthesis target indicated by the target image based on the voice data. The imaging device according to any one of aspects 1 to 11.

[0165] <Aspect 13> An acquisition unit that acquires data and a control unit that processes the acquired data, The control unit Acquiring original image data generated by capturing an image of a subject via the acquisition unit; Recognizing a subject in an original image represented by the original image data; generating composite image data by compositing a target image representing the composite target with the original image so as to arrange a composite target different from the subject based on the posture and position of the subject in the original image; Image processing device.

[0166] <Aspect 14> an imaging device including an imaging unit that captures an image of a subject and generates original image data; an image processing device including a control unit that receives the original image data from the imaging device and recognizes a subject in an original image indicated by the original image data, the control unit generates composite image data by compositing a target image indicating the composite target with the original image so as to arrange a composite target different from the subject based on the posture and position of the subject in the original image. Imaging system.

[0167] <Aspect 15> capturing an image of a subject to generate original image data; recognizing a subject in an original image represented by the original image data; generating composite image data by compositing a target image representing the composite target with the original image so as to position a composite target different from the subject based on the posture and position of the subject in the original image; An imaging method comprising:

[0168] <Aspect 16> A program for causing a control unit to execute the method according to aspect 15. [Industrial Applicability]

[0169] The present disclosure is applicable to an imaging device or an imaging system. [Explanation of symbols]

[0170] 1. Image processing system 2. Imaging system 10 3D data 20 Portraits 30 Clothing Data 40 Subject images 50 Subject Images 60 Network 70 Absentee 100 digital cameras 110 Optical system 115 Image Sensor 120 Image processing unit 125 buffer memory 130 Display 135 Control Unit 141 card slots 142 memory card 143 Communication Module 145 Flash Memory 150 Operation section 200 Absentee Terminal 201 Control Unit 202 Storage section 203 Input / Output Interface 204 Communication Interface 210 Voice input device 220 Display 240 Communication Interface 300 Image processing device 301 Control Unit 302 Storage section 303 Input / Output Interface 304 Communication Interface

Claims

1. an imaging unit that captures an image of a subject and generates original image data; a control unit for recognizing a subject in an original image represented by the original image data, the control unit generates composite image data by compositing a target image indicating the composite target with the original image so as to arrange a composite target different from the subject based on the posture and position of the subject in the original image. Imaging device.

2. the control unit sets the posture of the synthesis target in the synthetic image represented by the synthetic image data to a posture corresponding to the posture of the subject in the original image, based on three-dimensional data defining the posture of the synthesis target; The imaging device according to claim 1 .

3. The imaging device according to claim 1 , wherein the control unit generates the composite image data by combining the target image with a background image showing a background of the subject in the original image in place of an image showing the subject.

4. The imaging device according to claim 1 , wherein the control unit generates composite image data by combining the target image with the original image so that the target image appears together with an image showing the subject in the composite image indicated by the composite image data.

5. The control unit acquiring depth information indicating a distance between the imaging unit and a subject; adjusting an overlap between the image showing the subject and the target image in the composite image based on the depth information; The imaging device according to claim 4 .

6. 2. The imaging device according to claim 1, wherein the control unit performs a blurring process on the target image in accordance with a focus position of an optical system and a position on the original image where the target image is placed, and combines the target image with the original image.

7. The imaging device according to claim 1 , wherein the control unit applies shading to the target image according to a position on the original image where the target image is to be placed, and then combines the target image with the original image.

8. a setting unit configured to accept a user operation to specify a focus target of the optical system in the composite image represented by the composite image data; When the synthesis target is designated as the focusing target by the user operation, the control unit adjusts the focus position of the optical system so that the focus position of the optical system becomes a position where the target image is arranged in the synthetic image. The imaging device according to claim 1 .

9. The control unit acquiring depth information indicating a distance between the imaging unit and a subject; When the synthesis target is designated as the focusing target by the user operation, blurring is performed on the subject image showing the subject in accordance with the depth information and the position on the original image where the target image is to be placed. The imaging device according to claim 8 .

10. The imaging device according to claim 1 , further comprising a setting unit that accepts a user operation that specifies at least one of a position where the target image is to be placed, a size of the target image, or an orientation of the target to be synthesized in the synthesized image indicated by the synthetic image data.

11. a setting unit configured to accept a user operation to specify clothing to be combined in a composite image represented by the composite image data; The imaging device according to claim 1 , wherein the control unit changes clothing of the object to be combined in the combined image in response to the user operation.

12. the composite object is a person, the control unit acquires voice data indicating the speech of the person, and controls the speech state of the synthesis target indicated by the target image based on the voice data. The imaging device according to claim 1 .

13. An acquisition unit that acquires data and a control unit that processes the acquired data, The control unit Acquiring original image data generated by capturing an image of a subject via the acquisition unit; Recognizing a subject in an original image represented by the original image data; generating composite image data by compositing a target image representing the composite target with the original image so as to arrange a composite target different from the subject based on the posture and position of the subject in the original image; Image processing device.

14. an imaging device including an imaging unit that captures an image of a subject and generates original image data; an image processing device including a control unit that receives the original image data from the imaging device and recognizes a subject in an original image indicated by the original image data, the control unit generates composite image data by compositing a target image indicating the composite target with the original image so as to arrange a composite target different from the subject based on the posture and position of the subject in the original image. Imaging system.

15. capturing an image of a subject to generate original image data; recognizing a subject in an original image represented by the original image data; generating composite image data by compositing a target image representing the composite target with the original image so as to position a composite target different from the subject based on the posture and position of the subject in the original image; An imaging method comprising:

16. A program for causing a control unit to execute the imaging method according to claim 15.

Images