Image processing apparatus, image pickup apparatus, image processing method, and storage medium

Abstract

To provide an image processing apparatus, an imaging apparatus, an image processing method, and a program capable of providing an image useful for three dimensional decoration to a user.SOLUTION: The image processing apparatus includes a processor. The processor acquires distance information regarding a distance from the imaging device to a subject. The processor outputs a composite image in which a live view image obtained by imaging a subject by an imaging device and an object defined three dimensionally on the basis of at least distance information are combined.SELECTED DRAWING: Figure 24

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an imaging device, an image processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a virtual studio system that creates an image by combining a real subject with a background in a three-dimensional virtual space. The invention described in Patent Document 1 includes a camera that captures an image of the real subject, a camera tracker that is fixed to the camera and detects the position and attitude of the camera and outputs a camera movement signal, a rendering unit that depicts an image of the three-dimensional virtual space, a virtual camera that is placed in the three-dimensional virtual space and has parameters of position, attitude, and angle of view, and whose own position and attitude parameters are manipulated in accordance with the position and attitude of the camera based on the camera movement signal output by the camera tracker, and which specifies a projection range based on the parameters of its own position, attitude, and angle of view, and a composition unit that generates a composite image by combining an image of the subject captured by the camera with an image of the projection range in the three-dimensional virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-102923 Summary of the Invention

[0004] An embodiment of the present disclosure provides an image processing device, an imaging device, an image processing method, and a program that can provide a user with an image that is useful for three-dimensional decoration. [Means for solving the problem]

[0005] A first aspect of the present disclosure is an image processing device that includes a processor, the processor acquiring distance information regarding the distance from an imaging device to a subject, and outputting a composite image that combines a live view image obtained by capturing an image of the subject by the imaging device with an object defined in three dimensions based on at least the distance information.

[0006] A second aspect of the present disclosure is the image processing device according to the first aspect, in which the object includes a photo booth.

[0007] A third aspect of the present disclosure is an image processing device according to the first or second aspect, in which the live view image shows a reference plane included in the subject, and the composite image is an image in which an object is positioned on a mounting plane determined based on the reference plane.

[0008] A fourth aspect of the present disclosure is the image processing device according to the third aspect, in which the installation location of the object is determined in accordance with the accepted operation.

[0009] A fifth aspect of the present disclosure is an image processing device according to the third or fourth aspect, in which the installation surface is a divided surface selected in accordance with given instructions from among multiple divided surfaces obtained by dividing the reference surface.

[0010] A sixth aspect of the present disclosure is the image processing device according to any one of the third to fifth aspects, in which the reference surface is recognized by performing object recognition processing on a live view image.

[0011] A seventh aspect of the present disclosure is the image processing device according to the sixth aspect, in which the reference surface is a surface having a feature that is recognized by performing object recognition processing on a live view image.

[0012] An eighth aspect of the present disclosure is an image processing device according to any one of the first to seventh aspects, in which an object in a composite image is changed according to a first condition.

[0013] A ninth aspect according to the present disclosure is the image processing device according to the eighth aspect, in which the first condition includes a first change instruction that is an instruction to change the object.

[0014] A tenth aspect of the present disclosure is the image processing device according to the eighth or ninth aspect, in which the first condition includes a state of a subject captured in a live view image.

[0015] An eleventh aspect of the present disclosure is an image processing device according to any one of the first to tenth aspects, in which a processor outputs an augmented reality image, the augmented reality image being an image synthesized from a live view image and at least one virtual space defined based on the geometric characteristics of an object, and the virtual space includes a virtual three-dimensional object.

[0016] A twelfth aspect of the present disclosure is the image processing device according to the eleventh aspect, in which the three-dimensional object is changed in accordance with a second condition.

[0017] A thirteenth aspect of the present disclosure is the image processing device according to the twelfth aspect, in which the second condition includes a second modification instruction that is an instruction to modify the three-dimensional object.

[0018] A fourteenth aspect of the present disclosure is the image processing device according to the twelfth or thirteenth aspect, in which the second condition includes a state of a subject captured in a live view image.

[0019] A fifteenth aspect of the present disclosure is an image processing device according to any one of the eleventh to fourteenth aspects, in which the augmented reality image includes, as virtual spaces, one or more background virtual spaces capable of representing the background of an object shown in the live view image and one or more foreground virtual spaces capable of representing the foreground of an object shown in the live view image, the one or more background virtual spaces including background three-dimensional objects as three-dimensional objects, and the one or more foreground virtual spaces including foreground three-dimensional objects as three-dimensional objects.

[0020] A sixteenth aspect of the present disclosure is an image processing device according to the fifteenth aspect, in which the background three-dimensional object and the foreground three-dimensional object are expressed with pseudo-optical properties that influence each other between the background three-dimensional object and the foreground three-dimensional object.

[0021] A seventeenth aspect of the present disclosure is the image processing device according to any one of the eleventh to sixteenth aspects, in which the three-dimensional object includes a dynamic three-dimensional object that is dynamically expressed.

[0022] An 18th aspect of the present disclosure is an image processing device according to any one of the 11th to 17th aspects, in which objects and three-dimensional objects appearing in a live view image are represented by occlusion based on distance information.

[0023] A 19th aspect of the present disclosure is an image processing device according to any one of the 11th to 18th aspects, in which processing is performed on a virtual space and / or a three-dimensional object in accordance with processing execution instructions given from each of a plurality of terminal devices.

[0024] A twentieth aspect of the present disclosure is the image processing device according to any one of the first to nineteenth aspects, in which the object is updated in accordance with when a live view image is obtained.

[0025] A 21st aspect of the present disclosure is an image processing device according to any one of the first to 20th aspects, in which output of the composite image is realized by displaying the composite image on a screen.

[0026] A 22nd aspect of the present disclosure is an image processing device according to any one of the 1st to 21st aspects, in which playback information for playing an image including an object is stored in a storage medium, and when playback conditions are satisfied, the image including the object is played based on the playback information stored in the storage medium.

[0027] A 23rd aspect of the present disclosure is the image processing device according to any one of the first to 22nd aspects, in which the synthetic image is an image realized by augmented reality.

[0028] A 24th aspect of the present disclosure is an image processing device according to any one of the first to 23rd aspects, in which distance information is obtained by performing image analysis on an image obtained by capturing an image of a subject using an imaging device.

[0029] A 25th aspect of the present disclosure is the image processing device according to any one of the first to 24th aspects, wherein the imaging device is provided with a distance measuring sensor that measures distance.

[0030] A 26th aspect of the present disclosure is an imaging device including the image processing device according to any one of the first to 25th aspects and an image sensor that captures an image of a subject.

[0031] A 27th aspect of the present disclosure is an image processing method that includes acquiring distance information regarding a distance from an imaging device to a subject, and outputting a composite image that combines a live view image obtained by imaging the subject with an object defined in three dimensions based on at least the distance information.

[0032] A 28th aspect of the present disclosure is a program for causing a computer to execute processing including acquiring distance information regarding the distance from an imaging device to a subject, and outputting a composite image that combines a live view image obtained by imaging the subject with an object defined in three dimensions based on at least the distance information. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 is a schematic perspective view showing an example of how a smart device is used. [Figure 2] FIG. 2 is a rear perspective view showing an example of the appearance of the rear side of the smart device. [Figure 3] FIG. 1 is a schematic perspective view showing an example of the structure of a photoelectric conversion element included in a smart device. [Figure 4] 3 is a front perspective view showing an example of the appearance of the front side of the smart device shown in FIG. 2. FIG. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of electrical hardware of a smart device. [Figure 6] 2 is a block diagram showing an example of the contents stored in a storage device and an example of the main functions of a processor included in the smart device. FIG. [Figure 7] FIG. 10 is a conceptual diagram showing an example of an aspect in which an imaging area is imaged by a distance measuring imaging device included in a smart device, and a first live view image is obtained. [Figure 8] FIG. 10 is a conceptual diagram showing an example of an aspect in which a distance measuring device included in a smart device measures the distance to a subject and obtains a distance image based on the distance measurement result. [Figure 9] FIG. 10 is a conceptual diagram showing an example of processing details in which a second live view image is generated by the control unit and a flat surface shown in the second live view image is recognized by the recognition unit. [Figure 10] 10 is a conceptual diagram showing an example of processing content in which a flat surface is divided by a control unit to obtain a plurality of divided surfaces. FIG. [Figure 11] FIG. 10 is a conceptual diagram showing an example of how a second live view image and a plurality of divided planes are displayed on a screen. [Figure 12] FIG. 10 is a conceptual diagram showing an example of a manner in which one divided surface is selected by a user from a plurality of divided surfaces. [Figure 13] 10 is a conceptual diagram showing an example of a manner in which a composite image is generated by a control unit and displayed on a screen. FIG. [Figure 14] FIG. 10 is a conceptual diagram showing an example in which the geometric characteristics of a photo booth are changed in response to a booth change instruction. [Figure 15] FIG. 10 is a conceptual diagram showing an example of a mode in which a photo booth is confirmed in response to a confirmation instruction, and reproduction information is generated and stored in storage on the condition that the photo booth has been confirmed. [Figure 16] FIG. 10 is a conceptual diagram showing an example of how a photo booth is reproduced based on reproduction information. [Figure 17] FIG. 10 is a conceptual diagram showing an example of a situation in which a person is positioned inside a photo booth in a composite image. [Figure 18] 10 is a conceptual diagram showing an example of how an augmented reality image including a foreground virtual space and a background virtual space defined based on a photo booth is generated and displayed on a screen. FIG. [Figure 19] FIG. 10 is a conceptual diagram showing an example of the positional relationship between a photo booth, a foreground virtual space, and a background virtual space. [Figure 20] FIG. 10 is a conceptual diagram showing an example of a state in which a foreground three-dimensional object is arranged in a foreground virtual space. [Figure 21] FIG. 10 is a conceptual diagram showing an example of how a foreground three-dimensional object arranged in a foreground virtual space is changed. [Figure 22] FIG. 10 is a conceptual diagram showing an example of a state in which a background three-dimensional object is arranged in a background virtual space. [Figure 23] FIG. 10 is a conceptual diagram showing an example of how a background three-dimensional object arranged in a background virtual space is changed. [Figure 24] This is a conceptual diagram showing an example of an embodiment in which an augmented reality image is generated by combining a foreground three-dimensional object placed in a foreground virtual space, a second live view image, and a background three-dimensional object placed in a background virtual space, and the augmented reality image is displayed on the screen. [Figure 25] FIG. 10 is a conceptual diagram showing an example of the contents of occlusion processing. [Figure 26] This is a conceptual diagram showing an example of how an augmented reality image is generated by combining a foreground three-dimensional object placed in a foreground virtual space, a main exposure image, and a background three-dimensional object placed in a background virtual space, and the augmented reality image is output to a default output destination. [Figure 27] 10 is a flowchart illustrating an example of the flow of an imaging control process. [Figure 28]FIG. 28 is a conceptual diagram showing an example of how the geometric characteristics (size in the example shown in FIG. 28) of the photo booth are changed depending on the state of the subject (a person in the example shown in FIG. 28). [Figure 29] FIG. 10 is a conceptual diagram showing an example of a manner in which an augmented reality image is generated using a three-dimensional background object according to a person's facial expression. [Figure 30] FIG. 1 is a conceptual diagram showing an example of how pseudo-optical properties (e.g., specular reflection and / or projection) that influence each other between a background three-dimensional object and a foreground three-dimensional object are expressed in the background three-dimensional object and the foreground three-dimensional object. [Figure 31] FIG. 10 is a conceptual diagram showing an example of the positional relationship between a photo booth, a foreground virtual space, a foreground virtual space, and a background virtual space. [Figure 32] This is a conceptual diagram showing an example of an embodiment in which an augmented reality image is generated by combining a dynamic three-dimensional object placed in a foreground virtual space, a foreground three-dimensional object placed in a foreground virtual space, a second live view image, and a background three-dimensional object placed in a background virtual space, and the augmented reality image is displayed on the screen. [Figure 33] FIG. 1 is a conceptual diagram showing an example of how a distance image is generated by AI-based image analysis. [Figure 34] FIG. 10 is a conceptual diagram showing an example of how a foreground three-dimensional object and a background three-dimensional object are edited in accordance with editing instructions given to each of a plurality of smart devices. [Figure 35] FIG. 1 is a schematic diagram illustrating an example of the configuration of an imaging system. DETAILED DESCRIPTION OF THE INVENTION

[0034] Hereinafter, exemplary embodiments of an image processing device, an imaging device, an image processing method, and a program according to the present disclosure will be described with reference to the accompanying drawings. Note that the present disclosure can also be applied to a program and a computer program product.

[0035] First, the terms used in the following description will be explained.

[0036] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". GPGPU is an abbreviation for "General-Purpose computing on Graphics Processing Units". APU is an abbreviation for "Accelerated Processing Unit". TPU is an abbreviation for "Tensor Processing Unit". RAM is an abbreviation for "Random Access Memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". LD is an abbreviation for "Laser Diode". EL is an abbreviation for "Electro-Luminescence". UI is an abbreviation for "User Interface". I / F is an abbreviation for "Interface". TOF is an abbreviation for "Time of Flight". AI is an abbreviation for "Artificial Intelligence". CG is an abbreviation for "Computer Graphics". LAN is an abbreviation for "Local Area Network". WAN is an abbreviation for "Wide Area Network". 5G is an abbreviation for "5th Generation Mobile Communication System".

[0037] In the following description, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU, a GPU, a GPGPU, an APU, and a TPU.

[0038] In the following description, a signed memory is a memory such as a RAM in which information is temporarily stored, and is used as a work memory by a processor.

[0039] In the following description, the term "storage" refers to one or more nonvolatile storage devices that store various programs, various parameters, etc. Examples of nonvolatile storage devices include flash memory, magnetic disks, and magnetic tapes. Another example of storage is cloud storage.

[0040] In the following embodiments, the external I / F with a symbol controls the exchange of various information between multiple devices connected to each other. An example of the external I / F is a USB interface. A communication I / F including a communication processor, an antenna, etc. may be applied to the external I / F. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G, Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0041] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0042] As an example, as shown in FIG. 1, in response to instructions given by a user 12, a smart device 10 performs an imaging operation (hereinafter also simply referred to as an “imaging operation”) of imaging a subject 14 that fits within an angle of view θ1, and a ranging operation (hereinafter also simply referred to as a “ranging operation”) of measuring the distance by irradiating laser light onto the subject 14 and receiving the reflected light of the laser light from the subject 14.

[0043] In this embodiment, "distance measurement" refers to a process of measuring the distance from the smart device 10 to the subject 14. In the example shown in FIG. 1, the subject 14 includes a flat surface 14A and a person 14B standing on the flat surface 14A. Examples of the smart device 10 include a smartphone, a smartwatch, smart glasses, and a tablet terminal. In this embodiment, the smart device 10 is an example of an "imaging device" according to the present disclosure. Also, in this embodiment, the subject 14 is an example of a "subject" according to the present disclosure.

[0044] In the present embodiment, a smart device 10 is illustrated, but this is merely an example, and the present disclosure can be applied to an instant camera, a compact camera, a mirrorless single-lens camera, a digital single-lens reflex camera, or the like. Furthermore, the present disclosure can be applied to a device that integrates an imaging function and a printing function instead of the smart device 10. An example of a device that integrates an imaging function and a printing function is a hybrid instant camera (for example, a camera equipped with multiple functions, such as displaying a captured image on a screen, recording a captured image on a recording medium such as a memory card, editing and / or processing an image displayed on a screen in response to a user's instruction, and printing an image specified by the user).

[0045] As an example, as shown in FIG. 2, the smart device 10 includes a housing 16. The housing 16 houses a distance measuring image sensor 18. The distance measuring image sensor 18 includes a light irradiator 20 and a light receiver 22. The light irradiator 20 includes an LD 24, and the light receiver 22 includes a photoelectric conversion element 26. The imaging operation and distance measuring operation of the smart device 10 are realized using the distance measuring image sensor 18.

[0046] Command keys 28 are arranged on the side of the smart device 10. The command keys 28 accept various commands. Here, "various commands" refers to, for example, a command to display a menu screen from which various menus can be selected, a command to select one or more menus, a command to confirm the selection, a command to delete the selection, etc.

[0047] When the smart device 10 is placed vertically, translucent windows 30 and 32 are provided at the top of the rear surface 16A of the housing 16. The translucent windows 30 and 32 are translucent optical elements (e.g., lenses) arranged at predetermined intervals (e.g., intervals of several millimeters) along the horizontal direction and exposed from the rear surface 16A. The light irradiator 20 irradiates the subject 14 (see FIG. 1 ) with laser light emitted from the LD 24 through the translucent windows 30. In this embodiment, laser light in the infrared wavelength range is used. However, the wavelength range of the laser light is not limited to this, and laser light in other wavelength ranges may also be used.

[0048] The light receiver 22 captures reflected IR light through the light-transmitting window 32. The reflected IR light refers to light reflected by the laser light irradiated onto the subject 14 by the light irradiator 20. The light receiver 22 also captures reflected visible light through the light-transmitting window 32. The reflected visible light refers to light reflected by the visible light irradiated onto the subject 14. The photoelectric conversion element 26 receives the reflected IR light captured by the light receiver 22 through the light-transmitting window 32 and outputs an electrical signal corresponding to the amount of reflected IR light received. The photoelectric conversion element 26 also receives the reflected visible light captured by the light receiver 22 through the light-transmitting window 32 and outputs an electrical signal corresponding to the amount of reflected visible light received. For ease of explanation, hereinafter, when there is no need to distinguish between reflected IR light and reflected visible light, they will simply be referred to as "reflected light."

[0049] 3, the photoelectric conversion element 26 has a plurality of photodiodes arranged in a matrix. An example of the plurality of photodiodes is photodiodes for "4896 x 3265" pixels.

[0050] A color filter is disposed on each photodiode included in the photoelectric conversion element 26. The color filters include a G filter corresponding to the G (green) wavelength range that contributes most to obtaining a luminance signal, an R filter corresponding to the R (red) wavelength range, a B filter corresponding to the B (blue) wavelength range, and an IR filter corresponding to the IR (infrared) wavelength range. In this embodiment, the G filter, R filter, and B filter also function as infrared light cut filters that cut infrared light.

[0051] The photoelectric conversion element 26 has R pixels, G pixels, B pixels, and IR pixels. The R pixels correspond to photodiodes on which R filters are arranged, the G pixels correspond to photodiodes on which G filters are arranged, the B pixels correspond to photodiodes on which B filters are arranged, and the IR pixels correspond to photodiodes on which IR filters are arranged. The R pixels, G pixels, B pixels, and IR pixels are arranged with a predetermined periodicity in the row direction (horizontal direction) and column direction (vertical direction). In this embodiment, the arrangement of the R pixels, G pixels, B pixels, and IR pixels is an arrangement obtained by replacing some of the G pixels in an X-Trans (registered trademark) arrangement with IR pixels. The IR pixels are arranged with a specific periodicity in the row direction and column direction.

[0052] Note that, here, an arrangement based on an X-Trans arrangement is exemplified as the arrangement of the R pixels, G pixels, B pixels, and IR pixels, but the present disclosure is not limited to this, and the arrangement of the R pixels, G pixels, B pixels, and IR pixels may be an arrangement based on other arrangements such as a Bayer arrangement or a Honeycomb (registered trademark) arrangement.

[0053] Furthermore, although an arrangement obtained by replacing some G pixels with IR pixels in an arrangement commonly known as an arrangement of R pixels, G pixels, and B pixels is exemplified here as an arrangement of R pixels, G pixels, B pixels, and IR pixels, the present disclosure is not limited to this. For example, each color filter corresponding to each of the R pixels, G pixels, and B pixels (hereinafter also referred to as "visible light pixels") may be a color filter that also transmits infrared light, and a pair of photodiodes, one for the visible light pixels and one for the IR pixels (e.g., InGaAs APD), may be arranged per color filter.

[0054] In this embodiment, the photoelectric conversion element 26 is divided into two regions. That is, the photoelectric conversion element 26 has a visible light image division region 26N1 and a distance measurement division region 26N2. The visible light image division region 26N1 is a visible light pixel group consisting of a plurality of visible light pixels and is used to generate a visible light image. The distance measurement division region 26N2 is an IR pixel group consisting of a plurality of IR pixels and is used for distance measurement. The visible light image division region 26N1 receives reflected visible light and outputs an electrical signal according to the amount of received light. The distance measurement division region 26N2 receives reflected IR light and outputs an electrical signal according to the amount of received light.

[0055] 4, a touch panel display 34 is provided on the front surface 16B of the housing 16. The touch panel display 34 includes a display 36 and a touch panel 38. An example of the display 36 is an EL display. The display 36 may not be an EL display, but may be another type of display such as a liquid crystal display.

[0056] Images (e.g., live view images, actual exposure images, and playback images), text information, etc. are displayed on the screen 36A of the display 36. The touch panel 38 is a transmissive touch panel that is overlaid on the surface of the display area of ​​the display 36. The touch panel 38 receives instructions from the user 12 (see FIG. 1) by detecting contact with a pointing device such as a finger or a stylus pen. Note that, although an out-cell type touch panel display is given here as an example of the touch panel display 34, this is merely one example. For example, an on-cell type or in-cell type touch panel display can also be used as the touch panel display 34.

[0057] 5, the smart device 10 includes a computer 40, an input / output interface 42, an image memory 44, a UI device 46, and an external I / F 48 in addition to a light irradiator 20 and a light receiver 22. In this embodiment, the computer 40 is an example of the "image processing device" and "computer" according to the present disclosure.

[0058] The computer 40 includes a processor 40A, a storage 40B, and a memory 40C. In this embodiment, the processor 40A is an example of a "processor" according to the present disclosure. The processor 40A, the storage 40B, and the memory 40C are connected via a bus 50, which is connected to an input / output interface 42. Note that in the example shown in FIG. 9, for convenience of illustration, one bus is shown as the bus 50, but multiple buses may be used. The bus 50 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0059] Various programs are stored in the storage 40B. The processor 40A reads out the necessary programs from the storage 40B and executes the read programs on the memory 40C. The processor 40A controls the entire smart device 10 in accordance with the programs executed on the memory 40C.

[0060] A plurality of devices are connected to the input / output interface 42, and the input / output interface 42 controls the exchange of various information between the plurality of devices. In the example shown in Fig. 5, the plurality of devices connected to the input / output interface 42 include a computer 40, a light irradiator 20, a light receiver 22, an image memory 44, a UI device 46, and an external I / F 48.

[0061] The external I / F 48 controls the exchange of various information with devices external to the smart device 10 (hereinafter also referred to as "external devices"). An example of the external I / F 48 is a USB interface. External devices (not shown), such as smart devices, personal computers, servers, USB memory sticks, memory cards, and / or printers, can be directly or indirectly connected to the USB interface.

[0062] The UI-based device 46 includes a display 36, and the processor 40A causes the display 36 to display various pieces of information. The UI-based device 46 also includes a reception device 52. The reception device 52 includes a touch panel 38 and a hard key unit 54. The hard key unit 54 is at least one hard key including the instruction keys 28 (see FIG. 2). The processor 40A operates in response to various instructions received by the touch panel 38. Note that, although the hard key unit 54 is included in the UI-based device 46 here, the present disclosure is not limited to this, and for example, the hard key unit 54 may be connected to the external I / F 48.

[0063] The light irradiator 20 includes a light-transmitting window 30, a beam expander 56, a collimating lens 58, an LD 24, and an LD driver 60, and the light-transmitting window 30, the beam expander 56, and the collimating lens 58 are arranged in this order along the optical axis L1 from the subject 14 side (object side) to the LD 24. The LD driver 60 is connected to the LD 24 and the input / output interface 42, and drives the LD 24 to emit laser light in response to instructions from the processor 40A.

[0064] The laser light emitted from the LD 24 is converted into parallel light by a collimator lens 58, and then the beam diameter is expanded by a beam expander 56, and the laser light is irradiated onto the subject 14 through the light-transmitting window 30.

[0065] The light receiver 22 includes a light-transmitting window 32, an objective lens 61A, a focus lens 61B, an aperture 61C, a photoelectric conversion element 26, a photoelectric conversion element driver 62, and a signal processing circuit 72. In the light receiver 22, the light-transmitting window 32, the objective lens 61A, the focus lens 61B, and the aperture 61C are arranged in this order along the optical axis L2 from the subject 14 side (object side) to the photoelectric conversion element 26. The photoelectric conversion element driver 62 is connected to the photoelectric conversion element 26 and the input / output interface 42 and drives the photoelectric conversion element 26 in response to instructions from the processor 40A. For example, under the control of the processor 40A, the photoelectric conversion element driver 62 supplies the photoelectric conversion element 26 with an imaging timing signal that specifies the timing of imaging performed by the photoelectric conversion element 26. The photoelectric conversion element 26 performs resetting, exposure, and output of an electrical signal in accordance with the imaging timing signal supplied from the photoelectric conversion element driver 62. The imaging timing signal includes, for example, a vertical synchronization signal and a horizontal synchronization signal.

[0066] The optical receiver 22 includes a focusing control mechanism 64. The focusing control mechanism 64 includes a focus lens 61B, a moving mechanism 66, a motor 68, and a motor driver 70. The focus lens 61B is supported by the moving mechanism 66 so as to be slidable along the optical axis L2. The motor 68 is connected to the moving mechanism 66 and the motor driver 70. The motor driver 70 is connected to the input / output interface 42 and drives the motor 68 in response to instructions from the processor 40A. The moving mechanism 66 is connected to a drive shaft (not shown) of the motor 68 and receives power from the motor 68 to selectively move the focus lens 61B along the optical axis L2 toward the object side or the image side. That is, the processor 40A adjusts the focusing position by controlling the driving of the motor 68 via the motor driver 70. Here, the "focusing position" refers to the position of the focus lens 61B on the optical axis L2 when the image is in focus (for example, when the contrast of the visible light image is maximized or when a predetermined depth of field is achieved).

[0067] The aperture 61C is a fixed aperture whose opening does not change. In the case of a fixed aperture, exposure adjustment is performed by an electronic shutter of the photoelectric conversion element 26. The aperture 61C may not be a fixed aperture, but may be a variable aperture. Note that the objective lens 61A, focus lens 61B, and aperture 61C included in the light receiver 22 are merely examples, and the present disclosure will be valid even if the lens configuration and / or the position of the aperture 61C are changed.

[0068] The reflected light is incident on the light receiver 22 through the light-transmitting window 32. The reflected light incident on the light-transmitting window 32 is imaged on the photoelectric conversion element 26 via the objective lens 61A, the focus lens 61B, and the diaphragm 61C.

[0069] The photoelectric conversion element 26 is connected to the signal processing circuit 72, and outputs pixel data indicating pixel values ​​for each of the visible light pixels and IR pixels to the signal processing circuit 72. The signal processing circuit 72 digitizes the pixel data input from the photoelectric conversion element 26 by performing A / D conversion, and performs various signal processing operations on the digitized pixel data.

[0070] The signal processing circuit 72 includes a visible light pixel data processing circuit 72A and a distance image generation circuit 72B. The visible light pixel data processing circuit 72A generates a visible light image 74 by performing known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction on the pixel data for the visible light pixels. The visible light pixel data processing circuit 72A then stores the visible light image 74 in the image memory 44. Note that the visible light image 74 in the image memory 44 is updated by overwriting and saving one frame of the visible light image 74 in the image memory 44.

[0071] The distance measuring image sensor 18 includes a TOF camera 76. The TOF camera 76 includes a light irradiator 20, a distance measurement section 26N2, and a distance image generation circuit 72B. The distance image generation circuit 72B acquires an emission timing signal indicating the timing at which laser light is emitted from the LD 24 (hereinafter also referred to as "emission timing") from the processor 40A. The distance image generation circuit 72B measures the distance from the smart device 10 to the subject 14 (see FIG. 1) for each IR pixel based on the emission timing indicated by the emission timing signal and the timing at which reflected IR light is received by each IR pixel (hereinafter also referred to as "light reception timing"). In this embodiment, the TOF camera 76 is an example of a "distance measuring sensor" according to the present disclosure.

[0072] Based on the measurement results for each IR pixel, distance image generation circuit 72B generates distance image 78 relating to the distance from smart device 10 to subject 14 (see FIG. 1 ) and stores the generated distance image 78 in image memory 44. Distance image 78 in image memory 44 is updated by overwriting one frame of distance image 78 in image memory 44. In this embodiment, distance image 78 is an example of "distance information" according to the present disclosure.

[0073] 6, the storage 40B stores an imaging control program 80. In this embodiment, the imaging control program 80 is an example of the "program" according to the present disclosure.

[0074] The processor 40A performs imaging control processing by reading the imaging control program 80 from the storage 40B and executing the read imaging control program 80 on the memory 40C. The imaging control processing is realized by the processor 40A operating as a control unit 40A1 and a recognition unit 40A2 in accordance with the imaging control program 80 executed on the memory 40C.

[0075] Storage 40B stores a flat surface recognition model 82 and a person recognition model 84. As will be described in detail later, flat surface recognition model 82 and person recognition model 84 are trained models used in AI-based processing. Flat surface recognition model 82 is used by recognition unit 40A2, and person recognition model 84 is used by control unit 40A1.

[0076] 7, when an instruction to start capturing an image is received via the touch panel 38 of the smart device 10, the photoreceiver 22 captures an image of the subject 14 within the angle of view θ1. That is, the photoreceiver 22 receives reflected visible light and generates a first live view image 74A, which is a live view image corresponding to the received reflected visible light. The first live view image 74A is a type of visible light image 74.

[0077] The first live view image 74A is stored in the image memory 44 and acquired by the control unit 40A1. The control unit 40A1 performs processing using the first live view image 74A (for example, displaying the first live view image 74A on the screen 36A, etc.).

[0078] As an example, as shown in FIG. 8, when an instruction to start capturing images is received via the touch panel 38, the smart device 10 performs distance measurement in units of a predetermined number of frames of the first live view image 74A (here, for example, in units of one frame). When the timing to start distance measurement arrives, the light irradiator 20 emits laser light. The angle at which the laser light is emitted (hereinafter also referred to as the "irradiation angle") is θ2. The irradiation angle θ2 is an angle of width that includes the angle of view θ1. When the angle of view θ1 is changed in accordance with the instruction received via the touch panel 38, the irradiation angle θ2 is also changed in conjunction with the change in the angle of view θ1.

[0079] In the smart device 10, the distance from the smart device 10 to the subject 14 (see FIG. 1) is measured based on the time required from when the laser light is emitted by the light irradiator 20 until the reflected IR light is received by the distance measurement segment 26N2 (see FIGS. 3 and 5) of the light receiver 22, and the speed of light. For example, if the distance to the subject 14 is "L," the speed of light is "c," and the time required from when the laser light is emitted by the light irradiator 20 until the reflected IR light is received by the distance measurement segment 26N2 is "t," the distance L is calculated according to the formula "L=c×t×0.5."

[0080] In the smart device 10, distance measurement is performed for each of the multiple IR pixels included in the distance measurement section area 26N2 by receiving reflected IR light. The distance measurement results for each IR pixel are then generated as a distance image 78, which is stored in the image memory 44. The distance image 78 in the image memory 44 is acquired and used by the control unit 40A1. Here, the distance image 78 refers to an image that represents the distance to the subject 14 measured for each IR pixel using different colors and / or shades of gray.

[0081] 9, controller 40A1 generates second live view image 74B by mapping the distance obtained from distance image 78 onto first live view image 74A. That is, second live view image 74B is generated by mapping, to each pixel included in first live view image 74A, the distance indicated by distance image 78 or a distance calculated by interpolation using multiple distances indicated by distance image 78. Second live view image 74B includes first live view image 74A and three-dimensional coordinates 86 assigned to each pixel included in first live view image 74A. Three-dimensional coordinates 86 are defined by two coordinates that define the position of each pixel included in first live view image 74A within first live view image 74A (i.e., its two-dimensional position), and coordinates that indicate the distance obtained from distance image 78.

[0082] The recognition unit 40A2 acquires the second live view image 74B generated by the control unit 40A1 from the control unit 40A1. Then, the recognition unit 40A2 executes flat surface recognition processing. The flat surface recognition processing refers to processing in which the recognition unit 40A2 recognizes the flat surface 14A appearing in the second live view image 74B by using the second live view image 74B and a flat surface recognition model 82. In this embodiment, the flat surface recognition processing is an example of the "object recognition processing" according to the present disclosure.

[0083] The flat surface recognition model 82 is a trained model (e.g., U-Net or Mask R-CNN) for object recognition using an AI segmentation method, and is obtained by performing machine learning on a neural network.

[0084] The flat surface recognition model 82 is a trained model that has been optimized by machine learning using first training data, which is a data set including multiple data (i.e., multiple frames of data) in which first example data and first correct answer data are associated with each other.

[0085] The first example data is an image (for example, a sample image simulating the second live view image 74B) showing a surface with predetermined characteristics. Examples of surfaces with predetermined characteristics include flat surfaces on which visual textures (for example, patterns, bumps, etc.) are not visually perceptible (for example, a commonly known flat surface on which a person stands in a portrait shooting scene, such as a floor or road surface).

[0086] The first supervised answer data is supervised answer data (i.e., annotation) for the first example data. That is, the first supervised answer data is information that can identify a surface having predetermined characteristics that appears in the image used as the first example data. An example of the first supervised answer data is an annotation (e.g., three-dimensional coordinates) that identifies the geometric characteristics (e.g., position, size, and shape) of the surface having the predetermined characteristics.

[0087] The recognition unit 40A2 inputs the second live view image 74B into the flat surface recognition model 82, causing the flat surface recognition model 82 to generate and output a segmentation map 90. The coordinate system of the segmentation map 90 is the same as the coordinate system applied to the second live view image 74B. The segmentation map 90 includes a segmentation mask 90A, which is information that can identify the flat surface 14A. The recognition unit 40A2 recognizes the flat surface 14A that appears in the second live view image 74B from the segmentation mask 90A in the segmentation map 90.

[0088] In this embodiment, the second live view image 74B is an example of a "live view image" according to the present disclosure. Also, in this embodiment, the flat surface 14A is an example of a "reference surface" and a "surface having characteristics recognized by performing object recognition processing on the live view image" according to the present disclosure.

[0089] 10 , the recognition unit 40A2 generates position identification information 92 based on the segmentation mask 90A. The position identification information 92 is information that can identify the position of the flat surface 14A in the second live view image 74B (for example, three-dimensional coordinates defined in the same coordinate system as the three-dimensional coordinates 86). The control unit 40A1 acquires the position identification information 92 from the recognition unit 40A2 and identifies the position of the flat surface 14A in the second live view image 74B based on the position identification information 92.

[0090] The control unit 40A1 divides the flat surface 14A in the second live view image 74B. Dividing the flat surface 14A results in a plurality of divided surfaces 94. The control unit 40A1 causes the display 36 to display the second live view image 74B and the plurality of divided surfaces 94. In this case, as shown in FIG. 11 as an example, the screen 36A displays the second live view image 74B, which shows the subject 14 including the flat surface 14A, and also displays the plurality of divided surfaces 94 superimposed on the flat surface 14A shown in the second live view image 74B. A logo mark may be attached to the plurality of divided surfaces 94.

[0091] 12, the user 12 selects one of a plurality of divided surfaces 94. The selection of the divided surface 94 by the user 12 is realized by receiving an instruction from the user 12 via the touch panel 38.

[0092] 13, the control unit 40A1 generates a composite image 96 using the second live view image 74B. The composite image 96 is an image realized by augmented reality. For example, the composite image 96 is an image obtained by combining the second live view image 74B with a photo booth 98, which is a virtual three-dimensional object generated by CG.

[0093] The control unit 40A1 generates a photo booth 98 in the second live view image 74B. In the second live view image 74B, the photo booth 98 is an object defined in three dimensions based on at least the distance image 78. Here, an object defined in three dimensions refers to, for example, an object defined by three-dimensional coordinates 86 based on the distance image 78. In the second live view image 74B, the photo booth 98 is set on an installation surface 100 that is determined based on the flat surface 14A. The installation surface 100 is the division surface 94 selected by the user 12 (see FIG. 12).

[0094] In the composite image 96, the photo booth 98 is a semi-transparent object located on the installation surface 100. The photo booth 98 has a flat floor surface 98A located on the installation surface 100, and a flat back surface 98B that rises vertically from one end of the floor surface 98A on the far side. The photo booth 98 is updated whenever the second live view image 74B is obtained (for example, at a timing determined by the frame rate of the second live view image 74B). Here, "updating" refers to the regeneration of the photo booth 98.

[0095] When the composite image 96 is generated as described above, the control unit 40A1 outputs the composite image 96 to the display 36. The composite image 96 is displayed on the screen 36A of the display 36.

[0096] In this embodiment, the composite image 96 is an example of a "composite image" according to the present disclosure. Also, in this embodiment, the photo booth 98 is an example of a "three-dimensionally defined object" and a "photo booth" according to the present disclosure.

[0097] 14, the control unit 40A1 changes the geometric characteristics of the photo booth 98 in accordance with change conditions, which are conditions for changing the geometric characteristics of the photo booth 98. The change conditions include a booth change instruction 102, which is an instruction to change the geometric characteristics of the photo booth 98. In this embodiment, the change conditions are an example of a "first condition" according to the present disclosure, and the booth change instruction 102 is an example of a "first change instruction" according to the present disclosure.

[0098] When the booth change instruction 102 is received by the reception device 52, the control unit 40A1 changes the geometric characteristics of the photo booth 98 (in the example shown in FIG. 14, the position of the photo booth 98 in the composite image 96) in accordance with the booth change instruction 102. The geometric characteristics of the photo booth 98 are changed in accordance with the timing at which the second live view image 74B is updated (i.e., the timing determined according to the frame rate of the second live view image 74B). When the geometric characteristics of the photo booth 98 are changed, the control unit 40A1 displays the composite image 96 including the photo booth 98 whose geometric characteristics have been changed on the screen 36A.

[0099] 15, when a confirmation instruction 104, which is an instruction to confirm the installation location of the photo booth 98 within the composite image 96, is received by the reception device 52, the control unit 40A1 confirms the installation position of the photo booth 98 within the composite image 96 in accordance with the confirmation instruction 104. When the installation position of the photo booth 98 within the composite image 96 is confirmed, the control unit 40A1 generates playback information 106 for reproducing the photo booth 98 and stores it in the storage 40B. The playback information 106 includes three-dimensional coordinates that can identify the geometric characteristics (e.g., position, size, and shape) of the photo booth 98 within the composite image 96. In this embodiment, the confirmation instruction 104 is an example of an "accepted operation" according to the present disclosure.

[0100] 16, when a playback condition for playing back the photo booth 98 is satisfied, the control unit 40A1 acquires a second live view image 74B and acquires playback information 106 from the storage 40B. An example of the playback condition is when a playback instruction 108 for playing back the photo booth 98 is accepted by the acceptance device 52.

[0101] The control unit 40A1 generates a composite image 96 by reproducing the photo booth 98 within the second live view image 74B in accordance with the reproduction information 106 obtained from the storage 40B. Then, the control unit 40A1 displays the generated composite image 96 on the screen 36A.

[0102] As an example, as shown in FIG. 17, once the installation position of the photo booth 98 within the composite image 96 has been determined and the composite image 96 is displayed on the screen 36A, the user 12 guides the person 14B into the photo booth 98 so that the person 14B fits within the photo booth 98 on the composite image 96.

[0103] As an example, as shown in FIG. 18, when a person 14B is inside a photo booth 98 on a composite image 96 and a decoration start instruction 110, which is an instruction to start decorating the second live view image 74B, is received by the reception device 52, the control unit 40A1 generates an augmented reality image 112.

[0104] The augmented reality image 112 is an image obtained by combining the second live view image 74B with at least one virtual space defined based on the geometric characteristics (i.e., position, size, and shape) of the photo booth 98. Virtual three-dimensional objects generated by computer graphics are placed in the virtual space. In this embodiment, an image obtained by combining the second live view image 74B with multiple virtual spaces is used as the augmented reality image 112. Furthermore, in this embodiment, a foreground virtual space 114 and a background virtual space 116 are used as the multiple virtual spaces. Here, one foreground virtual space 114 and one background virtual space 116 are illustrated as examples, but there may be multiple foreground virtual spaces 114 and / or multiple background virtual spaces 116.

[0105] The foreground virtual space 114 is a virtual space capable of representing the foreground of the person 14B inside the photo booth 98, and the background virtual space 116 is a virtual space capable of representing the background of the person 14B inside the photo booth 98. The foreground virtual space 114 and the background virtual space 116 are defined in the same coordinate system as the second live view image 74B.

[0106] The control unit 40A1 displays the augmented reality image 112 on the screen 36A. The photo booth 98 and the multiple virtual spaces (here, as an example, the foreground virtual space 114 and the background virtual space 116) are not visualized in the augmented reality image 112 displayed on the screen 36A. Here, "not visualized" not only means not displayed, but also means displayed with a display intensity that is not visually perceptible. Note that, although an example is given here in which the photo booth 98 and the multiple virtual spaces are not visualized, at least one of the photo booth 98 and the multiple virtual spaces may be visualized.

[0107] As an example, as shown in FIGS. 18 and 19, the foreground virtual space 114 is a virtual space located in front of the photo booth 98. The geometric characteristics (i.e., position, size, and shape) of the foreground virtual space 114 are determined based on the geometric characteristics of the photo booth 98. For example, they are calculated from an arithmetic expression in which the geometric characteristics of the photo booth 98 are independent variables and the geometric characteristics of the foreground virtual space 114 are dependent variables. Furthermore, the background virtual space 116 is a virtual space located behind the photo booth 98. The geometric characteristics (i.e., position, size, and shape) of the background virtual space 116 are also determined based on the geometric characteristics of the photo booth 98. For example, they are calculated from an arithmetic expression in which the geometric characteristics of the photo booth 98 are independent variables and the geometric characteristics of the background virtual space 116 are dependent variables.

[0108] In this embodiment, the augmented reality image 112 is an example of an "augmented reality image" according to the present disclosure. Also, in this embodiment, the foreground virtual space 114 is an example of a "virtual space" and a "foreground virtual space" according to the present disclosure. Also, in this embodiment, the background virtual space 116 is an example of a "virtual space" and a "background virtual space" according to the present disclosure. Also, in this embodiment, the person 14B is an example of an "object appearing in a live view image" according to the present disclosure.

[0109] As an example, as shown in FIG. 20, when a foreground decoration instruction 118, which is an instruction to decorate a foreground virtual space 114, is received by the receiving device 52, the control unit 40A1 places a foreground three-dimensional object 120 in the foreground virtual space 114 in accordance with the foreground decoration instruction 118.

[0110] The foreground three-dimensional object 120 is a virtual three-dimensional object generated by CG. The foreground three-dimensional object 120 may be one or more three-dimensional objects selected from multiple existing three-dimensional objects in accordance with the foreground decoration instruction 118, or may be one or more three-dimensional objects newly drawn in accordance with the foreground decoration instruction 118.

[0111] Controller 40A1 displays second live view image 74B included in augmented reality image 112 on screen 36A, and also displays foreground three-dimensional object 120 located in foreground virtual space 114 included in augmented reality image 112 on screen 36A. On screen 36A, foreground three-dimensional object 120 is displayed in front of person 14B appearing in second live view image 74B. In augmented reality image 112, person 14B is partially occluded by foreground three-dimensional object 120.

[0112] As an example, as shown in FIG. 21, control unit 40A1 changes foreground three-dimensional object 120 (see FIG. 20) in foreground virtual space 114 to foreground three-dimensional object 122 in accordance with a foreground change condition, which is a condition for changing the foreground decoration. The foreground change condition includes a foreground change instruction 124 for changing the foreground decoration. In this embodiment, the foreground change condition is an example of a "second condition" according to the present disclosure, and foreground change instruction 124 is an example of a "second change instruction" according to the present disclosure.

[0113] When the foreground change instruction 124 is received by the receiving device 52, the control unit 40A1 changes the foreground three-dimensional object 120 (see FIG. 20 ) in the foreground virtual space 114 to the foreground three-dimensional object 122 in response to the foreground change instruction 124. The foreground three-dimensional object 122 may be one or more 3D objects selected from a plurality of existing 3D objects in accordance with the foreground change instruction 124, or may be one or more 3D objects generated by partially modifying the foreground three-dimensional object 120 in accordance with the foreground change instruction 124. When the foreground three-dimensional object 120 is changed to the foreground three-dimensional object 122 in this manner, the foreground three-dimensional object 122 is displayed on the screen 36A in front of the person 14B appearing in the second live view image 74B. In the augmented reality image 112, the person 14B is partially occluded by the foreground three-dimensional object 122.

[0114] As an example, as shown in FIG. 22, when a background decoration instruction 126, which is an instruction to decorate the background virtual space 116, is received by the receiving device 52, the control unit 40A1 places a background three-dimensional object 128 within the background virtual space 116 in accordance with the background decoration instruction 126.

[0115] The background three-dimensional object 128 is a virtual three-dimensional object generated by CG. The background three-dimensional object 128 may be one or more three-dimensional objects selected from multiple existing three-dimensional objects in accordance with the background decoration instruction 126, or may be one or more three-dimensional objects newly drawn in accordance with the background decoration instruction 126.

[0116] The control unit 40A1 displays on the screen 36A the second live view image 74B included in the augmented reality image 112, and also displays on the screen 36A a background three-dimensional object 128 that is positioned in the background virtual space 116 included in the augmented reality image 112. On the screen 36A, the background three-dimensional object 128 is displayed behind the person 14B who appears in the second live view image 74B. In the augmented reality image 112, the background three-dimensional object 128 is partially occluded by the person 14B.

[0117] As an example, as shown in FIG. 23, the control unit 40A1 changes a background three-dimensional object 128 (see FIG. 22) in the background virtual space 116 to a background three-dimensional object 130 in accordance with a background change condition, which is a condition for changing the decoration of the background. The background change condition includes a background change instruction 132 for changing the decoration of the background. In this embodiment, the background change condition is an example of a "second condition" according to the present disclosure, and the background change instruction 132 is an example of a "second change instruction" according to the present disclosure.

[0118] When the background change instruction 132 is received by the receiving device 52, the control unit 40A1 changes the background three-dimensional object 128 (see FIG. 22 ) in the background virtual space 116 to the background three-dimensional object 130 in response to the background change instruction 132. The background three-dimensional object 130 may be one or more 3-dimensional objects selected from multiple existing three-dimensional objects in accordance with the background change instruction 132, or may be one or more 3-dimensional objects generated by partially modifying the background three-dimensional object 128 in accordance with the background change instruction 132. When the background three-dimensional object 128 is changed to the background three-dimensional object 130 in this manner, the background three-dimensional object 130 is displayed on the screen 36A behind the person 14B appearing in the second live view image 74B. In the augmented reality image 112, the background three-dimensional object 130 is partially occluded by the person 14B.

[0119] 24 , the control unit 40A1 generates an augmented reality image 112 by combining a second live view image 74B, a foreground virtual space 114 including a foreground three-dimensional object 120, and a background virtual space 116 including a background three-dimensional object 128. The control unit 40A1 then outputs the augmented reality image 112 to the display 36. The augmented reality image 112 is displayed on a screen 36A of the display 36.

[0120] As an example, as shown in FIG. 25, the control unit 40A1 performs occlusion processing based on three-dimensional coordinates 86 on a person 14B and a foreground three-dimensional object 120 appearing in an augmented reality image 112 in accordance with an occlusion condition, which is a condition for performing the occlusion processing. The occlusion processing refers to a process for realizing occlusion. Occlusion refers to a phenomenon in which an object is partially or completely hidden by another object. The occlusion condition includes an occlusion instruction 134, which is an instruction to perform occlusion.

[0121] When the occlusion instruction 134 is accepted by the accepting device 52, the control unit 40A1 executes person recognition processing on the second live view image 74B. The person recognition processing refers to processing in which the control unit 40A1 recognizes the person 14B appearing in the second live view image 74B by using the second live view image 74B and the person recognition model 84.

[0122] The person recognition model 84 is a trained model (e.g., U-Net or Mask R-CNN) for object recognition using an AI segmentation method, and is obtained by performing machine learning on a neural network.

[0123] The person recognition model 84 is optimized by machine learning using second training data, which is a data set including multiple data (i.e., multiple frames of data) in which second example data and second correct answer data are associated with each other.

[0124] The second example data is an image in which a person appears (for example, a sample image assuming the second live view image 74B). The second supervised answer data is supervised answer data (i.e., annotation) for the second example data. That is, the second supervised answer data is information that can identify a person appearing in the image used as the second example data. An example of the second supervised answer data is an annotation (e.g., three-dimensional coordinates) that identifies the geometric characteristics (e.g., position, size, and shape) of a person.

[0125] The control unit 40A1 inputs the second live view image 74B into the person recognition model 84, causing the person recognition model 84 to generate and output a segmentation map 136. The coordinate system of the segmentation map 136 is the same as that of the second live view image 74B. The segmentation map 136 includes a segmentation mask 136A that is information that can identify the person 14B. The control unit 40A1 recognizes the person 14B appearing in the second live view image 74B from the segmentation mask 136A in the segmentation map 136.

[0126] The control unit 40A1 identifies an overlapping area between the person 14B and the foreground three-dimensional object 120. The overlapping area between the person 14B and the foreground three-dimensional object 120 is identified based on the segmentation mask 136A and the foreground three-dimensional object 120. The control unit 40A1 then calculates information (e.g., three-dimensional coordinates) that can identify the overlapping area between the segmentation mask 136A and the foreground three-dimensional object 120. The overlapping area identification information 138 is calculated based on the three-dimensional coordinates that can identify the geometric characteristics of the segmentation mask 136A and the three-dimensional coordinates that can identify the geometric characteristics of the foreground three-dimensional object 120.

[0127] Control unit 40A1 cuts out an image area corresponding to the overlap area identified from overlap area identification information 138 from the person image of second live view image 74B included in augmented reality image 112 (i.e., an image showing person 14B), erases the image area from foreground three-dimensional object 120 that corresponds to the overlap area identified from overlap area identification information 138, and superimposes the image area cut out from the person image on the erased area. Augmented reality image 112 obtained in this manner is displayed on screen 36A by control unit 40A1. That is, augmented reality image 112 is displayed on screen 36A in a state in which the area of ​​foreground three-dimensional object 120 that overlaps with person 14B is hidden by person 14B.

[0128] 26 , when the augmented reality image 112 is displayed on the screen 36A, if a main exposure instruction 140, which is an instruction to start the main exposure, is received by the receiving device 52, the main exposure is performed by the light receiver 22, thereby generating a main exposure image 74C, and the main exposure image 74C is stored in the image memory 44. The main exposure image 74C is a type of visible light image 74.

[0129] The control unit 40A1 generates the augmented reality image 144 by replacing the second live view image 74B included in the augmented reality image 112 with the actual exposure image 74C. The augmented reality image 144 differs from the augmented reality image 112 in that the second live view image 74B is replaced with the actual exposure image 74C. The control unit 40A1 outputs the augmented reality image 144 to a default output destination. A first example of the default output destination is the storage 40B or a storage medium such as a memory card connected to the external I / F 48. A second example of the default output destination is the display 36. In this embodiment, the augmented reality image 144 is stored in the storage 40B, and is displayed on the screen 36A.

[0130] Next, a portion of smart device 10 related to the present disclosure will be described with reference to Fig. 27. The imaging control process shown in Fig. 27 is an example of an "image processing method" related to the present disclosure. For ease of explanation, the following description will be based on the assumption that a first live view image 74A showing flat surface 14A and a main exposure image 74C showing flat surface 14A and person 14B are selectively stored in image memory 44, and that a distance image 78 generated in synchronization with first live view image 74A is stored in image memory 44.

[0131] 27, first, in step ST10, the control unit 40A1 acquires the first live view image 74A and the distance image 78 from the image memory 44 (see FIGS. 7 and 8). After the processing of step ST10 is executed, the imaging control processing proceeds to step ST12.

[0132] In step ST12, the control unit 40A1 generates a second live view image 74B based on the first live view image 74A and the distance image 78 (see FIG. 9). After the processing of step ST12 is executed, the imaging control processing proceeds to step ST14.

[0133] In step ST14, the recognition unit 40A2 recognizes the flat surface 14A appearing in the second live view image 74B by using the second live view image 74B and the flat surface recognition model 82 (see FIG. 9). In step ST14, the second live view image 74B is input to the flat surface recognition model 82, and a segmentation map 90 is generated by the flat surface recognition model 82 (see FIG. 9). After the processing of step ST14 is executed, the imaging control processing proceeds to step ST16.

[0134] In step ST16, the recognition unit 40A2 generates position identification information 92 based on the segmentation map 90 (see FIG. 10). After the processing of step ST16 is executed, the imaging control processing proceeds to step ST18.

[0135] In step ST18, the control unit 40A1 identifies the position of the flat surface 14A in the second live view image 74B based on the position identification information 92, and divides the flat surface 14A in the second live view image 74B (see FIG. 10). By dividing the flat surface 14A, a plurality of divided surfaces 94 are obtained (see FIG. 10). After the processing of step ST18 is executed, the imaging control processing proceeds to step ST20.

[0136] In step ST20, the control section 40A1 displays the second live view image 74B and the plurality of divided planes 94 on the screen 36A (see FIG. 11). After the processing of step ST20 is executed, the imaging control processing proceeds to step ST22.

[0137] In step ST22, the control unit 40A1 determines the divided surface 94 selected by the user 12 from the plurality of divided surfaces 94 as the installation surface 100 (see FIGS. 12 and 13). After the processing of step ST22 is executed, the imaging control processing proceeds to step ST24.

[0138] In step ST24, the control unit 40A1 generates a composite image 96 by placing a photo booth 98 on an installation surface 100 in the second live view image 74B, and displays the composite image 96 on the screen 36A (see FIG. 13). The installation position of the photo booth 98 can be changed in accordance with a booth change instruction 102 received by the reception device 52 (see FIG. 14). The photo booth 98 whose installation position has been changed is displayed on the screen 36A (see FIG. 14). The geometric characteristics, transparency, color, and / or pattern of the photo booth 98 in the composite image 96 are changed in accordance with instructions given by the user 12 or the like. An example of the timing of the change is a timing determined according to the frame rate of the second live view image 74B. In this case, the change content for changing the geometric characteristics, transparency, color, and / or pattern of the photo booth 98 is reflected in the composite image 96 at a timing determined according to the frame rate of the second live view image 74B. After the processing of step ST24 is executed, the imaging control processing proceeds to step ST26.

[0139] In step ST26, the control unit 40A1 determines the installation position of the photo booth 98 in the composite image 96 in accordance with the determination instruction 104 received by the reception device 52 (see FIG. 15). After the processing of step ST26 is executed, the imaging control processing proceeds to step ST28.

[0140] In step ST28, the control unit 40A1 generates the playback information 106 and stores it in the storage 40B (see FIG. 15). After the process of step ST28 is executed, the imaging control process proceeds to step ST30.

[0141] The reproduction information 106 stored in the storage 40B is acquired from the storage 40B by the control unit 40A1 in accordance with a reproduction instruction 108 accepted by the acceptance device 52, and is used to reproduce the photo booth 98 (see FIG. 16).

[0142] In step ST30, the control unit 40A1 generates an augmented reality image 112 on the condition that the reception device 52 accepts a decoration start instruction 110 while the person 14B is positioned within the photo booth 98 shown in the composite image 96 (see FIG. 19 ). The control unit 40A1 displays the augmented reality image 112 on the screen 36A. The augmented reality image 112 is an image obtained by combining the second live view image 74B with a foreground virtual space 114 and a background virtual space 116 that are defined based on the geometric characteristics of the photo booth 98. After the processing of step ST30 is executed, the imaging control processing proceeds to step ST32.

[0143] In step ST32, the control unit 40A1 places the foreground three-dimensional object 120 in the foreground virtual space 114 and the background three-dimensional object 128 in the background virtual space 116, thereby displaying the foreground three-dimensional object 120 and the background three-dimensional object 128 on the screen 36A (see FIG. 24 ). The geometric characteristics, transparency, color, and / or pattern, etc. of the augmented reality image 112 (e.g., the geometric characteristics, transparency, color, and / or pattern, etc., of the foreground virtual space 114, the background virtual space 116, the foreground three-dimensional object 120, and / or the background three-dimensional object 128) are changed in accordance with the content of an instruction given by the user 12 or the like. An example of the timing of the change is a timing determined according to the frame rate of the second live view image 74B. An example of an instruction given by the user 12 or the like to change the geometric characteristics, transparency, color, and / or pattern, etc. of the augmented reality image 112 is a foreground change instruction 124 and a background change instruction 132. For example, the changes made in the foreground change instruction 124 and the background change instruction 132 are reflected in the augmented reality image 112 (e.g., the foreground three-dimensional object 120 and the background change instruction 132) at a timing determined according to the frame rate of the second live view image 74B. After the processing of step ST32 is executed, the imaging control processing proceeds to step ST34.

[0144] In step ST34, the control unit 40A1 causes the light receiver 22 to perform the main exposure, on the condition that the main exposure instruction 140 has been accepted by the accepting device 52 while the augmented reality image 112 is being displayed on the screen 36A. Then, the control unit 40A1 generates the augmented reality image 144 by replacing the second live view image 74B included in the augmented reality image 112 with the main exposure image 74C, and outputs the augmented reality image 144 to the default output destination (see FIG. 26). After the processing of step ST34 has been executed, the imaging control processing ends.

[0145] As described above, in this embodiment, a composite image 96 is displayed on the screen 36A, which is a composite of the second live view image 74B, whose geometric characteristics are defined by the three-dimensional coordinates 86 including the distance measured by the TOF camera 76, and the photo booth 98, whose geometric characteristics are defined in three dimensions based on the three-dimensional coordinates 86. The composite image 96 is an image realized by augmented reality. Because the composite image 96 includes the photo booth 98 defined in three dimensions, the user 12 can guide the person 14B into the photo booth 98. Fitting the person 14B within the photo booth 98 defined in three dimensions makes it easier to three-dimensionally decorate the foreground, background, etc. of the person 14B in the second live view image 74B. In this way, this embodiment can provide the user 12 with an image useful for three-dimensional decoration.

[0146] Furthermore, in this embodiment, the photo booth 98 is updated in accordance with when the second live view image 74B is obtained. Therefore, compared to when the photo booth 98 is always positioned in the same place in the composite image 96, the photo booth 98 can be repositioned in an appropriate position in the composite image 96 in accordance with when the second live view image 74B is obtained.

[0147] Furthermore, in this embodiment, the photo booth 98 is positioned on an installation surface 100 within the composite image 96 that is determined based on the flat surface 14A shown in the second live view image 74B. Therefore, the photo booth 98 can be easily installed within the composite image 96.

[0148] Furthermore, in this embodiment, the installation position of the photo booth 98 in the composite image 96 is confirmed on the condition that the confirmation instruction 104 is accepted by the acceptance device 52. Therefore, the installation position of the photo booth 98 in the composite image 96 can be confirmed at the timing intended by the user 12.

[0149] Furthermore, in this embodiment, when the installation position of the photo booth 98 within the composite image 96 is determined, the reproduction information 106 is stored in the storage 40B, and the photo booth 98 is reproduced based on the reproduction information 106 stored in the storage 40B in response to the reproduction instruction 108. Therefore, the photo booth 98 obtained in the past can be reused.

[0150] Furthermore, in this embodiment, of the multiple divided surfaces 94 obtained by dividing the flat surface 14A, the divided surface 94 selected in response to an instruction from the user 12 is set as the installation surface 100, and the photo booth 98 is installed on the installation surface 100. Therefore, the photo booth 98 can be installed in the position intended by the user 12.

[0151] Furthermore, in this embodiment, the flat surface 14A is recognized by performing flat surface recognition processing on the second live view image 74B using the flat surface recognition model 82. Therefore, the flat surface 14A can be more easily identified than when the user 12 visually identifies the flat surface 14A from the second live view image 74B.

[0152] Furthermore, in this embodiment, the geometric characteristics of the photo booth 98 in the composite image 96 are changed by the booth change instruction 102. Therefore, compared to when the geometric characteristics of the photo booth 98 are always the same, a photo booth 98 having geometric characteristics closer to those intended by the user 12 can be installed in the composite image 96.

[0153] Furthermore, in this embodiment, an augmented reality image 112, which is a composite of a second live view image 74B defined in three dimensions, a foreground virtual space 114, and a background virtual space 116, is displayed on the screen 36A. A foreground three-dimensional object 120 is placed in the foreground virtual space 114. As a result, the foreground three-dimensional object 120 is displayed on the screen 36A. Furthermore, a background three-dimensional object 128 is placed in the background virtual space 116. As a result, the background three-dimensional object 128 is displayed on the screen 36A. Therefore, it is possible to provide the user 12 with an image in which the foreground and background of the person 14B are three-dimensionally decorated.

[0154] Furthermore, in this embodiment, the foreground three-dimensional object 120 is changed in response to the foreground change instruction 124, and the background three-dimensional object 128 is changed in response to the background change instruction 132. Therefore, the foreground decoration of the person 14B and the background decoration of the person 14B can be changed to the decoration intended by the user 12.

[0155] Furthermore, in this embodiment, when the occlusion instruction 134 is received by the receiving device 52, the person 14B and the foreground three-dimensional object 120 in the second live view image 74B are expressed by occlusion based on the three-dimensional coordinates 86. This makes it possible to impart a visual sense of reality to the relationship between the person 14B and the foreground three-dimensional object 120 that exist in real space.

[0156] In the above embodiment, an example was given in which the geometric characteristics of the photo booth 98 are changed in response to the booth change instruction 102 received by the reception device 52. However, this is merely an example, and the geometric characteristics of the photo booth 98 may be changed in response to the state of the subject 14. For example, as shown in FIG. 28 , when a person 14B that fits in the photo booth 98 is replaced by a second person whose body size is larger than that of the first person, the geometric characteristics of the photo booth 98 (size in the example shown in FIG. 28 ) may be changed to match the body size of the second person. Furthermore, the geometric characteristics of the photo booth 98 may be changed in response to the pose of the person 14B. For example, the size of the photo booth 98 may be changed depending on whether the person 14B is sitting or standing. For example, the size of the photo booth 98 may be larger when the person 14B is standing than when the person 14B is sitting. Furthermore, the photo booth 98 may follow the person 14B as the person 14B moves. Furthermore, the brightness, color, color density, and / or transparency of the photo booth 98 may be changed according to the brightness of the subject 14. In this way, a composite image 96 including the photo booth 98 according to the state of the subject 14 can be provided to the user 12.

[0157] In the above embodiment, the person 14B is exemplified as the main subject, but a subject other than the person 14B may be the main subject. In this case, it is sufficient that the main subject fits within the photo booth 98. Also, in this case, instead of the person recognition process described above, an object recognition process that recognizes a main subject other than a person may be executed.

[0158] In the above embodiment, an example has been given in which the foreground three-dimensional object 120 is changed in response to the foreground change instruction 124 and the background three-dimensional object 128 is changed in response to the background change instruction 132. However, this is merely an example, and the foreground three-dimensional object 120 and / or the background three-dimensional object 128 may be changed depending on the state of the subject 14. For example, as shown in FIG. 29 , the background three-dimensional object 128 may be changed depending on the facial expression of the person 14B. In this case, the control unit 40A1 performs facial expression recognition processing using the facial expression recognition model 146, which is a trained model obtained by, for example, having a neural network learn various facial expressions of the person 14B by machine learning. In the facial expression recognition processing, the control unit 40A1 inputs the second live view image 74B to the facial expression recognition model 146, thereby causing the facial expression recognition model 146 to recognize the facial expression of the person 14B appearing in the second live view image 74B. Then, control unit 40A1 places a background three-dimensional object in background virtual space 116 according to the facial expression recognized by facial expression recognition model 146.

[0159] For example, storage 40B stores a plurality of background three-dimensional objects including background three-dimensional objects 128 and 130, and control unit 40A1 acquires a background three-dimensional object corresponding to the facial expression of person 14B from storage 40B and places the background three-dimensional object in background virtual space 116. In the example shown in FIG. 29, when the facial expression of person 14B is other than a smile, control unit 40A1 acquires background three-dimensional object 128 from storage 40B and places background three-dimensional object 128 in background virtual space 116. In addition, in the example shown in FIG. 29, when the facial expression of person 14 is a smile, control unit 40A1 acquires background three-dimensional object 130 from storage 40B and places background three-dimensional object 130 in background virtual space 116.

[0160] Furthermore, when the person 14B that fits in the photo booth 98 is replaced by a second person whose body size is larger than that of the first person, the geometric characteristics of the background three-dimensional object 128 may be changed to match the body size of the second person. Furthermore, the geometric characteristics of the background three-dimensional object 128 may be changed depending on the pose of the person 14B. For example, the size, position, and / or shape of the background three-dimensional object 128 may be changed between when the person 14B is sitting and when the person 14B is standing. Furthermore, the background three-dimensional object 128 may be configured to follow the person 14B as the person 14B moves. Furthermore, the brightness, color, color density, and / or transparency of the background three-dimensional object 128 may be changed depending on the brightness of the subject 14.

[0161] In this way, it is possible to provide the user 12 with an augmented reality image 112 that includes a background three-dimensional object that corresponds to the state of the subject 14. Note that while an example in which the background three-dimensional object 128 changes depending on the state of the subject 14 has been given here, the same can be said for the foreground three-dimensional object.

[0162] Although the above embodiment does not mention pseudo-optical characteristics that influence each other between the background three-dimensional object 128 and the foreground three-dimensional object 120, the background three-dimensional object 128 and the foreground three-dimensional object 120 may be configured to express pseudo-optical characteristics (e.g., specular reflection and / or projection) that influence each other between the background three-dimensional object 128 and the foreground three-dimensional object 120. For example, as shown in FIG. 30 , a pseudo-gloss 148A of the background three-dimensional object 128 may be projected as pseudo-gloss 148B on the foreground three-dimensional object 120. Such pseudo-optical characteristics are realized by computer graphics. In this way, the pseudo-optical characteristics that influence each other between the background three-dimensional object 128 and the foreground three-dimensional object 120 are expressed in the background three-dimensional object 128 and the foreground three-dimensional object 120, thereby imparting optical realism to the three-dimensional objects displayed in the foreground and background of the person 14B.

[0163] In the above embodiment, an example was given in which the foreground virtual space 114 is placed in front of the photo booth 98 and the background virtual space 116 is placed behind the photo booth 98, but this is merely one example. For example, as shown in Fig. 31, a foreground virtual space 150 may be placed in front of the foreground virtual space 114. Like the foreground virtual space 114 and the background virtual space 116, the foreground virtual space 150 is also a virtual space defined based on the geometric characteristics of the photo booth 98 and is defined in the same coordinate system as the photo booth 98.

[0164] As an example, as shown in FIGS. 31 and 32 , a dynamic three-dimensional object 152, which is a dynamically rendered three-dimensional object, is placed in a foreground virtual space 150. The dynamic three-dimensional object 152 is realized by CG and moves within the foreground virtual space 150. The control unit 40A1 generates an augmented reality image 154 by combining the dynamic three-dimensional object 152 placed in the foreground virtual space 150, the foreground three-dimensional object 120 placed in the foreground virtual space 114, the second live view image 74B, and the background three-dimensional object 128 placed in the background virtual space 116. The control unit 40A1 displays the augmented reality image 154 on the screen 36A. This allows the user 12 to visually recognize the augmented reality image 154 in which the foreground of the person 14B is decorated with the dynamically rendered three-dimensional object.

[0165] 31 and 32 show an example in which the dynamic three-dimensional object 152 is placed in the foreground virtual space 150, but this is merely an example, and the dynamic three-dimensional object 152 may be placed in a virtual space surrounding the foreground virtual space 114, the person 14B, and the background virtual space 116. In this case, occlusion (e.g., occlusion between the dynamic three-dimensional object 152 and the foreground three-dimensional object 120, occlusion between the dynamic three-dimensional object 152 and the person 14B, and occlusion between the dynamic three-dimensional object 152 and the background three-dimensional object 128) may be realized in a manner similar to the occlusion processing described above, depending on the positional relationships between the dynamic three-dimensional object 152, the foreground three-dimensional object 120, the person 14B, and the background three-dimensional object 128.

[0166] In the above embodiment, an example has been given in which the distance image 78 is generated based on the distance measurement results obtained by the TOF camera 76. However, this is merely an example, and the distance image 78 may also be generated by performing image analysis on a plurality of first live view images 74A. For example, as shown in FIG. 33 , the control unit 40A1 may cause the distance image generation model 156 to generate the distance image 78. The distance image generation model 156 is a trained generation model obtained by performing machine learning on a neural network using training data in which a plurality of images captured from a plurality of positions are used as example data and a distance image showing the distribution of distances from the image capture positions (e.g., one of the plurality of positions) to the subject is used as correct answer data. The control unit 40A1 inputs the plurality of first live view images 74A captured from a plurality of positions (two frames of images in the example shown in FIG. 33 ) to the distance image generation model 156, thereby causing the distance image generation model 156 to generate the distance image 78.

[0167] 33 shows an example in which distance image 78 is generated by a generation AI, but this is merely one example, and distance image 78 may also be generated by a non-AI method. In this case, for example, the distance from the imaging position to subject 14 may be measured by stereo matching using a plurality of first live view images 74A (e.g., two frames of images) obtained by capturing images at a plurality of positions, and distance image 78 may be generated based on the measurement results. Alternatively, distance image 78 may be generated based on the distance measurement results obtained by performing phase difference distance measurement using phase difference pixels.

[0168] In the above embodiment, an example has been given in which the foreground three-dimensional object 120 and the background three-dimensional object 128 are changed in response to an instruction received by the receiving device 52 of the smart device 10, but this is merely an example. For example, as shown in Fig. 34, the foreground three-dimensional object 120 and the background three-dimensional object 128 may be changed in response to an instruction received by smart devices 158A and 158B communicatively connected to the smart device 10. In the example shown in Fig. 34, the smart devices 158A and 158B are an example of "multiple terminal devices" according to the present disclosure.

[0169] In the example shown in FIG. 34, smart device 158A includes a touch panel display 158A1, and smart device 158B includes a touch panel display 158B1. An augmented reality image 112 is displayed on touch panel displays 158A1 and 158B1. A user of smart device 158A provides an editing instruction, which is an example of a processing execution instruction according to the technology of the present disclosure, to smart device 158A via touch panel display 158A1 while observing the augmented reality image 112 displayed on touch panel display 158A1. Control unit 40A1 edits the foreground three-dimensional object 120 in accordance with the editing instruction provided to smart device 158A. Meanwhile, a user of smart device 158B provides an editing instruction, which is an example of a processing execution instruction according to the technology of the present disclosure, to smart device 158B via touch panel display 158B1 while observing the augmented reality image 112 displayed on touch panel display 158B1. The control unit 40A1 edits the background three-dimensional object 128 in accordance with an editing instruction given to the smart device 158B. In this way, the users of the smart devices 158A and 158B can simultaneously edit the augmented reality image 112.

[0170] Although an example in which foreground three-dimensional object 120 and background three-dimensional object 128 are edited has been given here, the geometric characteristics of foreground virtual space 114 and / or the geometric characteristics of background virtual space 116 may be changed in response to an instruction given to smart device 158A and / or 158B. Also, the geometric characteristics of foreground virtual space 150 and / or dynamic three-dimensional object 152 may be changed in response to an instruction given to smart device 158A, smart device 158B, or a smart device other than these.

[0171] Furthermore, when there are multiple foreground virtual spaces 114, one smart device may be associated with each foreground virtual space 114, and processing (e.g., editing) may be performed on the corresponding foreground virtual space 114 and the foreground three-dimensional object 120 in the corresponding foreground virtual space 114 in accordance with instructions received by each smart device. The same can be said for when there are multiple background virtual spaces 116.

[0172] In addition, in the example shown in Figure 34, an example is given in which both smart device 158A and smart device 158B are connected to smart device 10 so as to be able to communicate with each other, but smart device 158A or 158B may also be connected to smart device 10 so as to be able to communicate with each other.

[0173] With the augmented reality image 112 displayed on the touch panel display 158A1 of the smart device 158A, the processor of the smart device 158A may cause the smart device 10 to perform a main exposure for capturing an image for recording, on the condition that an image capturing instruction from a user of the smart device 158A or the like (for example, a subject whose image is being captured by the smart device 10 in a live view mode) is received by the touch panel display 158A1. This can also be achieved by a terminal other than the smart device 158A (for example, a terminal such as the smart device 158B that is communicatively connected to the smart device 10 and has an image capturing function, a display function, and a reception function). Here, the concept of a "terminal" also includes a printer that has an image capturing function, a display function, and a reception function.

[0174] Furthermore, when an editing instruction is received from a user of the smart device 158A (e.g., a subject whose live-view image is being captured by the smart device 10) while the augmented reality image 112 is being displayed on the touch panel display 158A1 of the smart device 158A, the processor of the smart device 158A may control the smart device 10 so that the decoration of the augmented reality image 112 displayed on the touch panel display 158A1 is edited in accordance with the editing instruction. This can also be achieved by a terminal other than the smart device 158A (e.g., a terminal such as the smart device 158B that is communicatively connected to the smart device 10 and has an imaging function, a display function, and a reception function). Here, the concept of a "terminal" also includes a printer that has an imaging function, a display function, and a reception function.

[0175] In the above embodiment, an example in which the imaging control process is performed by the computer 40 has been described, but the present disclosure is not limited to this, and at least a part of the processing included in the imaging control process may be performed by a device provided outside the computer 40. An example of this case will be described below with reference to FIG.

[0176] Fig. 35 is a conceptual diagram showing an example of the configuration of an imaging system 160. In the example shown in Fig. 35, the imaging system 160 is an example of the "imaging device" according to the present disclosure.

[0177] The imaging system 160 includes a computer 40 and an external device 162. For example, the external device 162 is a server, and is communicatively connected to the computer 40 via a network 164 (for example, a WAN and / or a LAN, etc.). Although a server is exemplified here, at least one personal computer or the like may be used as the external device 162 instead of a server.

[0178] An example of the external device 162 is at least one server that directly or indirectly transmits and receives data to and from the computer 40 via the network 164. The external device 162 receives a processing execution instruction provided from the processor 40A of the computer 40 via the network 164. The external device 162 then executes processing in accordance with the received processing execution instruction and transmits the processing result to the computer 40 via the network 164. In the computer 40, the processor 40A receives the processing result transmitted from the external device 162 via the network 164 and executes processing using the received processing result.

[0179] An example of the processing execution instruction is an instruction to cause the external device 162 to execute at least a part of the imaging control processing. A first example of at least a part of the imaging control processing (i.e., a processing to be executed by the external device 162) is a flat surface recognition processing. In this case, the external device 162 executes the flat surface recognition processing in response to the processing execution instruction provided from the processor 40A via the network 164, and transmits a first processing result, which is the processing result of the flat surface recognition processing, to the computer 40 via the network 164. In the computer 40, the processor 40A receives the first processing result and executes the same processing as in the above embodiment using the received first processing result.

[0180] A second example of at least a part of the imaging control process (i.e., a process to be executed by the external device 162) is a process by the control unit 40A1. In this case, the external device 162 executes the process by the control unit 40A1 in response to a process execution instruction provided from the processor 40A via the network 164, and transmits a second processing result (e.g., the composite image 96, the processing result of the person recognition process, the augmented reality image 112, the augmented reality image 144, etc.) to the computer 40 via the network 164. In the computer 40, the processor 40A receives the second processing result and executes a process using the received second processing result.

[0181] The external device 162 may be realized by cloud computing. Cloud computing is merely an example, and the external device 162 may be realized by network computing such as fog computing, edge computing, or grid computing.

[0182] In the above embodiment, an example in which the imaging control program 80 is stored in the storage 40B has been described, but the present disclosure is not limited to this. For example, the imaging control program 80 may be stored in a portable, computer-readable, non-transitory storage medium such as an SSD or a USB flash drive. The imaging control program 80 stored in the non-transitory storage medium is installed in the computer 40 of the smart device 10. The processor 40A executes imaging control processing in accordance with the imaging control program 80.

[0183] In addition, the imaging control program 80 may be stored in a storage device such as another computer or server connected to the smart device 10 via a network, and the imaging control program 80 may be downloaded and installed on the computer 40 in response to a request from the smart device 10.

[0184] It is not necessary to store the entire imaging control program 80 in a storage device such as another computer or server device connected to the smart device 10, or to store the entire imaging control program 80 in the storage 40B; only a portion of the imaging control program 80 may be stored.

[0185] The hardware resources for executing the imaging control process can be various processors, as listed below. Examples of processors include a CPU, which is a general-purpose processor that functions as a hardware resource for executing the imaging control process by executing software, i.e., a program. Examples of processors include dedicated electrical circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute the imaging control process.

[0186] The hardware resource that executes the imaging control process may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the imaging control process may be a single processor.

[0187] As an example of a system configured with one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes the imaging control process. Second, there is a system that uses a processor that realizes the functions of the entire system, including multiple hardware resources that execute the imaging control process, on a single IC chip, as typified by SoCs. In this way, the imaging control process is realized using one or more of the various processors described above as hardware resources.

[0188] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The above-described image capture control process is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the process.

[0189] The above-described description and illustrations are a detailed explanation of the parts related to the present disclosure and are merely an example of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or elements may be replaced with other parts from the above-described description and illustrations, as long as they do not deviate from the gist of the present disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the parts related to the present disclosure, the above-described description and illustrations omit explanations of common general technical knowledge that do not require particular explanation to enable the implementation of the present disclosure.

[0190] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0191] The following additional notes are provided regarding the above-described embodiments.

[0192] (Appendix 1) A communication device capable of communicating with an imaging device that generates a live view image by capturing an image of a subject and decorates the live view image using augmented reality, a receiving unit that receives, from the imaging device by communicating with the imaging device, a decorated live view image obtained by applying the decoration to the live view image by the imaging device; a screen for displaying the decorated live view image received by the receiving unit. Communication equipment.

[0193] (Appendix 2) a first reception unit that receives an imaging instruction; an imaging control unit that causes the imaging device to capture an image to obtain a still image corresponding to the decoration-added live-view image displayed on the screen, on condition that the imaging instruction is accepted by the first accepting unit while the decoration-added live-view image is being displayed on the screen. 2. The communication device of claim 1.

[0194] (Appendix 3) a second reception unit that receives an editing instruction; an ornament control unit that, when the editing instruction is accepted by the second accepting unit while the ornamented live view image is being displayed on the screen, controls the imaging device so that the ornament of the ornamented live view image being displayed on the screen is edited in accordance with the editing instruction. 10. The communication device according to claim 1 or 2.

[0195] (Appendix 4) The communication device is used by the subject. 4. A communication device according to any one of claims 1 to 3.

[0196] (Appendix 5) The communication device is a smart device or a printer. 5. A communication device according to any one of claims 1 to 4. [Explanation of symbols]

[0197] 10,158A,158B Smart Devices 12 users 14 Subject 14A flat surface 14B People 16 Case 16A back 16B Front 18 Distance imager 20 Light irradiator 22 Receiver 24LD 26 Photoelectric conversion element 26N1 Segmented area for visible light images 26N2 Distance measurement segment area 28 Instruction keys 30,32 Translucent window 34,158A1,158B1 Touch panel display 36 Display 36A screen 38 Touch Panel 40 Computer 40A processor 40A1 Control unit 40A2 recognition section 40B Storage 40C Memory 42 Input / Output Interface 44 Image Memory 46 UI devices 48 External I / F 50 Bus 52 Reception Device 54 Hard key section 56 Beam Expander 58 Collimating Lens 60 LD driver 61A Objective Lens 61B Focus Lens 61C Aperture 62 Photoelectric conversion element driver 64 Focusing control mechanism 66 Moving mechanism 68 Motor 70 Motor Driver 72 Signal Processing Circuit 72A Visible light pixel data processing circuit 72B Distance image generation circuit 74 visible light images 74A First Live View Image 74B Second Live View image 74C Original exposure image 76 TOF camera 78 Range Images 80 Imaging control program 82 Flat surface recognition model 84 Person Recognition Model 86 3D coordinates 90,136 segmentation maps 90A,136A Segmentation Mask 92 Location information 94 Split plane 96 composite images 98 Photo Booth 98A Floor 98B Back 100 Installation surface 102 Booth change instructions 104 Confirmation instruction 106 Playback information 108 Play instructions 110 Decoration start instruction 112,144,154 Augmented Reality Images 114 Foreground Virtual Space 116 Background Virtual Space 118 Foreground Decoration Instructions 120,122 Foreground 3D objects 124 Foreground change instruction 126 Background decoration instructions 128,130 3D background objects 132 Background change instructions 134 Occlusion Instructions 138 Overlapping area identification information 140 main exposure instructions 146 Facial Expression Recognition Model 148A, 148B Glossy 150 Foreground Virtual Space 152 Dynamic 3D Objects 156 Range Image Generation Model 160 Imaging System 162 External device 164 Network L1,L2 optical axis θ1 angle of view θ2 Irradiation angle

Claims

1. a processor; The processor: Acquire distance information relating to the distance from the imaging device to the subject; and outputting a composite image obtained by combining a live view image obtained by capturing an image of the subject with the imaging device and an object defined three-dimensionally based on at least the distance information. Image processing device.

2. The object includes a photo booth. The image processing device according to claim 1 .

3. the live view image captures a reference plane included in the subject, The composite image is an image in which the object is positioned on a placement plane determined based on the reference plane. The image processing device according to claim 1 .

4. The installation location of the object is determined according to the accepted operation. The image processing device according to claim 3 .

5. The installation surface is a divided surface selected in accordance with a given instruction from among a plurality of divided surfaces obtained by dividing the reference surface. The image processing device according to claim 3 .

6. The reference surface is recognized by performing an object recognition process on the live view image. The image processing device according to claim 3 .

7. The reference surface is a surface having a feature that is recognized by performing the object recognition process on the live view image. The image processing device according to claim 6 .

8. The object in the composite image is changed according to a first condition. The image processing device according to any one of claims 1 to 7.

9. The first condition includes a first change instruction that is an instruction to change the object. The image processing device according to claim 8 .

10. The first condition includes a state of the subject captured in the live view image. The image processing device according to claim 8 .

11. The processor outputs an augmented reality image; the augmented reality image is an image obtained by combining the live view image with at least one virtual space defined based on geometric characteristics of the object; The virtual space includes a virtual three-dimensional object. The image processing device according to claim 1 .

12. The three-dimensional object is changed according to a second condition. The image processing device according to claim 11 .

13. The second condition includes a second change instruction that is an instruction to change the three-dimensional object. The image processing device according to claim 12.

14. The second condition includes a state of the subject captured in the live view image. The image processing device according to claim 12.

15. the augmented reality image includes, as the virtual spaces, one or more background virtual spaces capable of representing a background of an object shown in the live view image, and one or more foreground virtual spaces capable of representing a foreground of an object shown in the live view image; the one or more background virtual spaces include a background three-dimensional object as the three-dimensional object, The one or more foreground virtual spaces include a foreground three-dimensional object as the three-dimensional object. The image processing device according to claim 11 .

16. The background three-dimensional object and the foreground three-dimensional object are expressed with pseudo-optical properties that influence each other between the background three-dimensional object and the foreground three-dimensional object. The image processing device according to claim 15.

17. The three-dimensional object includes a dynamic three-dimensional object that is dynamically expressed. The image processing device according to claim 11 .

18. The object in the live view image and the three-dimensional object are expressed by occlusion based on the distance information. The image processing device according to claim 11 .

19. Processing is executed for the virtual space and / or the three-dimensional object in accordance with processing execution instructions given from each of a plurality of terminal devices. The image processing device according to claim 11 .

20. The object is updated as the live view image is obtained. The image processing device according to claim 1 .

21. The output of the composite image is realized by displaying the composite image on a screen. The image processing device according to claim 1 .

22. reproduction information for reproducing an image including the object is stored in a storage medium; When the reproduction condition is satisfied, an image including the object is reproduced based on the reproduction information stored in the storage medium. The image processing device according to claim 1 .

23. The synthetic image is an image realized by augmented reality. The image processing device according to claim 1 .

24. The distance information is obtained by performing image analysis on an image obtained by capturing an image of the subject with the imaging device. The image processing device according to claim 1 .

25. The imaging device is provided with a distance measuring sensor for measuring the distance. The image processing device according to claim 1 .

26. An image processing device according to any one of claims 1 to 25; an image sensor that captures an image of the subject; Imaging device.

27. Obtaining distance information relating to the distance from the imaging device to the subject; and and outputting a composite image obtained by combining a live view image obtained by capturing an image of the subject with the imaging device and an object defined in three dimensions based on at least the distance information. Image processing methods.

28. Obtaining distance information relating to the distance from the imaging device to the subject; and A program for causing a computer to execute processing including outputting a composite image obtained by combining a live view image obtained by capturing an image of the subject using the imaging device with an object defined in three dimensions based on at least the distance information.

Images