Information Processing Method and Information Processing Apparatus

Abstract

To provide an information processing method and an information processing device in which distortion of a subject at an image combining portion can be lessened.SOLUTION: An information processing method includes: detecting the position of a feature point of a subject in an overlap region 32 in each of a plurality of images 30A and 30B having the overlap region 32, respectively obtained by photographing the subject by means of a plurality of photographing units 14A and 14B of which imaging ranges partially overlap each other; trimming, at the position of a virtual line 34 that is based on the positions of the feature point in the respective images, an image in which the virtual line 34 is included in the overlap region 32; and creating one combined image 40 by combining the plurality of images.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing method and an information processing apparatus.

Background Art

[0002] An apparatus is known that synthesizes two images based on the result of detecting an overlapping region of two images obtained by photographing (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When combining two images, in the combined portion of the images, the subject may be distorted due to differences in the photographing position or photographing direction of each image.

[0005] In view of such circumstances, an object of the present disclosure is to reduce distortion of a subject in a combined portion of images.

Means for Solving the Problems

[0006] An information processing method according to an embodiment of the present disclosure includes, for each of a plurality of images having an overlapping region obtained by photographing a subject with each of a plurality of photographing units whose photographing ranges partially overlap each other, detecting positions of feature points of the subject in the overlapping region; trimming an image included in the overlapping region at positions of the virtual lines with respect to an image in which the virtual lines based on the positions of the feature points in each image are included in the overlapping region; and combining the plurality of images to generate one composite image.

[0007] An information processing apparatus according to an embodiment of the present disclosure includes a plurality of imaging units and a control unit. Each of the plurality of imaging units captures an image having an overlapping region with each other. The control unit detects the positions of feature points of a subject in the overlapping region for the images captured by the respective imaging units. The control unit performs trimming at the position of the virtual line for the images in which the virtual line based on the positions of the feature points in each image is included in the overlapping region, and combines the plurality of images to generate one composite image.

Effects of the Invention

[0008] According to the information processing method and the information processing apparatus according to an embodiment of the present disclosure, distortion of a subject in a combined portion of images can be reduced.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 4

Modes for Carrying Out the Invention

[0010] (Configuration Example of Information Processing Apparatus 10) As shown in FIG. 1, an information processing apparatus 10 according to an embodiment includes a control unit 11, a storage unit 12, an interface 13, and an imaging unit 14. The information processing apparatus 10 is connected to a display device 20 although not essential. The information processing apparatus 10 may include the display device 20.

[0011] The information processing apparatus 10 may be configured to include a PC such as a notebook PC (Personal Computer), a desktop PC, or a tablet PC. The information processing apparatus 10 may also be configured to include a mobile terminal such as a smartphone or a tablet possessed by a user. The information processing apparatus 10 is not limited to these examples and may be configured to include various devices. Hereinafter, each component of the information processing apparatus 10 will be described.

[0012] <Control unit 11> The control unit 11 controls at least one component of the information processing apparatus 10. The control units 11 and 21 may be configured identically or similarly. The control unit 11 may be configured to include one or more processors. In the present embodiment, the "processor" may be a general-purpose processor, a dedicated processor specialized for a specific process, etc., but is not limited thereto. The control unit 11 may also be configured to include one or more dedicated circuits. The dedicated circuit may include, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 11 may be configured to include a dedicated circuit instead of a processor, or may be configured to include a dedicated circuit together with a processor.

[0013] <Storage unit 12> The storage unit 12 stores any information used for the operation of the information processing apparatus 10. The storage units 12 or 22 may be configured identically or similarly. The storage unit 12 may store a system program, an application program, etc. The storage unit 12 may be configured to include, for example, a semiconductor memory, a magnetic memory, or an optical memory, etc., but is not limited thereto and may be configured to include various other memories. The storage unit 12 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 may be configured to include an electromagnetic storage medium such as a magnetic disk. The storage unit 12 may be configured to include a non-transitory computer-readable medium. The storage unit 12 may be included in the control unit 11.

[0014] <Interface 13> Interface 13 outputs information, data, etc. from the control unit 11 or inputs information, data, etc. to the control unit 11 in the information processing apparatus 10. Interface 13 may include a communication module configured to be communicable with other devices such as other information processing apparatuses 10 via a network or without using a network. The communication module may conform to a mobile communication standard such as 4G (4th Generation) or 5G (5th Generation), for example. The communication module may conform to a communication standard such as LAN (Local Area Network). The communication module may conform to a wired or wireless communication standard. The communication module is not limited thereto and may conform to various communication standards. Interface 13 may be configured to be connectable to an external communication module.

[0015] The network may be configured to include the Internet, at least one WAN (Wide Area Network), at least one MAN (Metropolitan Area Network), or any combination thereof. The network may be configured to include at least one wireless network, at least one optical network, or any combination thereof. The wireless network may be configured to include, for example, an ad hoc network, a cellular network, a wireless LAN (Local Area Network), a satellite communication network, or a terrestrial microwave network.

[0016] Interface 13 may be configured to conform to a standard such as USB (Universal Serial Bus) or Bluetooth (registered trademark).

[0017] The interface 13 may be configured to include an input device that receives input such as information or data from a user. The input device may be configured to include, for example, a touch panel or touch sensor, or a pointing device such as a mouse. The input device may be configured to include physical keys. The input device may be configured to include a voice input device such as a microphone. The interface 13 may be configured to be connectable to an external input device.

[0018] The interface 13 may be configured to include an output device that outputs information or data etc. to the user. The output device may be configured to output information or data to be displayed on the display device 20 described later to the display device 20. The output device may include, for example, an audio output device such as a speaker that outputs auditory information such as sound. The output device may include, for example, a vibration device such as a vibrator that outputs tactile information such as vibration. The output device is not limited to these examples and may include various other devices. The interface 13 may be configured to be connectable to an external output device.

[0019] <Imaging unit 14> The imaging unit 14 may include an imaging device such as a camera that captures RGB images. The imaging unit 14 may include a ranging device such as a depth sensor or a stereo camera that acquires depth images. The information processing apparatus 10 may acquire an image or ranging data (depth data) from an external imaging device or ranging device via the interface 13. The functions of the imaging device or ranging device of the imaging unit 14 may be included in the functions of the interface 13.

[0020] As shown in FIG. 2, the imaging unit 14 includes a first imaging unit 14A and a second imaging unit 14B. The first imaging unit 14A and the second imaging unit 14B are installed above and below the frame of the display device 20, but are not limited thereto and may be installed in various manners. For example, the first imaging unit 14A and the second imaging unit 14B may be installed on the left and right of the display device 20, or may be installed separately from the display device 20. The number of imaging units 14 is not limited to two and may be three or more.

[0021] <Display device 20> The display device 20 is configured to display visual information such as images, characters, or graphics. The display device 20 may include, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, an inorganic EL display, or a PDP (Plasma Display Panel). The display device 20 is not limited to these displays and may include various other types of displays. The display device 20 may include a light-emitting device such as an LED (Light Emitting Diode) or an LD (Laser Diode). The display device 20 may include various other devices.

[0022] (Operation example of the information processing apparatus 10) The information processing apparatus 10 captures a plurality of images with a plurality of parts of the user as separate subjects, and generates an image of a wide range of the user by combining the plurality of images. By doing so, the degree of freedom of the user's position is increased so that the entire user to be photographed as a subject is not out of the shooting range of the photographing unit 14. The information processing apparatus 10 may operate to photograph the entire face of the user as a subject, or may operate to photograph the upper body or the whole body of the user as a subject. Hereinafter, specific operation examples of the information processing apparatus 10 will be described.

[0023] <Shooting by the photographing unit 14> The information processing apparatus 10 includes a plurality of photographing units 14. As shown in FIG. 2, it is assumed that the photographing unit 14 includes a first photographing unit 14A and a second photographing unit 14B. In this operation example, it is assumed that the number of the photographing units 14 is two.

[0024] The first imaging unit 14A captures a partial range of the entire range to be imaged with the user as the subject, and generates a first user image 30A illustrated in FIG. 3A. The second imaging unit 14B captures another partial range of the entire range to be imaged with the user as the subject, and generates a second user image 30B illustrated in FIG. 3B. In the example of FIG. 3A, the upper part of the user's face is shown in the first user image 30A. In the example of FIG. 3B, the lower part of the user's face is shown in the second user image 30B. In this operation example, the information processing apparatus 10 operates so as to image the entire face of the user as the subject.

[0025] The imaging ranges of the first imaging unit 14A and the second imaging unit 14B are set such that a partial part of the user as the subject is included overlappingly in a part of the imaging range. The part included overlappingly in the imaging range is also referred to as an overlapping part. In the first user image 30A and the second user image 30B, the area capturing the overlapping part of the user is shown as an overlapping area 32 surrounded by a broken-line frame. The part of the user other than the overlapping part is also referred to as a non-overlapping part. The area capturing the non-overlapping part of the user corresponds to the area other than the overlapping area 32 in the first user image 30A and the second user image 30B.

[0026] The imaging ranges of the first imaging unit 14A and the second imaging unit 14B are set such that different parts of the non-overlapping parts are included in the imaging range. For example, when the imaging range of the first imaging unit 14A is set to include the overlapping part of the user and a first non-overlapping part, the imaging range of the second imaging unit 14B is set to include the overlapping part of the user and a second non-overlapping part that does not overlap with the first non-overlapping part. In the first user image 30A of FIG. 3A and the second user image 30B of FIG. 3B, the area capturing the overlapping part of the user corresponds to the overlapping area 32. In the first user image 30A of FIG. 3A, the area capturing the first non-overlapping part of the user corresponds to the area above the eyebrows that is not included in the overlapping area 32. In the second user image 30B of FIG. 3B, the area capturing the second non-overlapping part of the user corresponds to the area below the chin that is not included in the overlapping area 32.

[0027] <Synthesis of Images> The control unit 11 of the information processing apparatus 10 analyzes each of the first user image 30A and the second user image 30B. For example, the control unit 11 may extract characteristic parts such as eyes, nose, mouth, ears, eyebrows, or hair from the face of the user depicted in each image. For example, the control unit 11 may extract characteristic parts such as hands, feet, head, or torso from the body of the user depicted in each image. The control unit 11 may detect the position or orientation of each extracted part. The control unit 11 may detect each part of the user's face or body as skeletal information including bones or joints.

[0028] The control unit 11 detects an overlapping region 32 as a region that appears overlapping in the first user image 30A and the second user image 30B. The control unit 11 may detect the overlapping region 32 between the first user image 30A and the second user image 30B based on the extraction results of the characteristic parts of the user. The control unit 11 may also detect the overlapping region 32 by partially comparing the first user image 30A and the second user image 30B.

[0029] The control unit 11 generates an image including the entire range in which the user is to be photographed by synthesizing the overlapping regions 32 of the first user image 30A and the second user image 30B so as to overlap each other. Specifically, the control unit 11 trims one or both of the first user image 30A and the second user image 30B along a line located within the overlapping region 32. By trimming, the control unit 11 removes from the second user image 30B the portion of the overlapping region 32 that remains in the first user image 30A, and removes from the first user image 30A the portion of the overlapping region 32 that remains in the second user image 30B. The control unit 11 generates a single image by combining the trimmed first user image 30A and second user image 30B, thereby synthesizing the first user image 30A and the second user image 30B.

[0030] <Determination of the Line for Trimming the Image> The control unit 11 determines a line for trimming the image based on the analysis result of the image. The virtual line determined by the control unit 11 as the line for trimming the image is exemplified as a virtual line 34 of a two-dot chain line in FIGS. 3A and 3B.

[0031] The control unit 11 determines the virtual line 34 so that the distortion at the combined portion of the first user image 30A and the second user image 30B is reduced. For example, when the virtual line 34 is positioned so as to divide the user's nose vertically, the distortion of the nose may look large in the image obtained by synthesizing the first user image 30A and the second user image 30B. Therefore, the control unit 11 may determine the virtual line 34 so as not to divide a coherent part.

[0032] The control unit 11 may determine the virtual line 34 based on various features such as the position or shape of each part of the user extracted from the user's image. In other words, the control unit 11 may detect the position of the feature points of the user as the subject in the overlapping region 32, and determine the virtual line 34 based on the position of the feature points.

[0033] The control unit 11 may determine, for example, a line passing through the user's jaw or a portion below the jaw as the virtual line 34. The control unit 11 may determine, for example, a line passing through the region between the bottom of the user's nose and the mouth, or a line passing through a position a predetermined distance below the bottom of the user's nose as the virtual line 34. The control unit 11 may determine, for example, a line passing through the region between the user's eyebrows and hair, that is, the forehead, or a line passing through a position a predetermined distance above the user's eyebrows as the virtual line 34. The control unit 11 may determine, for example, a line passing through at least a part of the upper end or the lower end of the mask when the user is wearing a mask as the virtual line 34. The control unit 11 may determine the virtual line 34 over the entire user image, or may determine the virtual line 34 only within the overlapping region 32.

[0034] In FIGS. 3A and 3B, the virtual line 34 is exemplified as a straight line, but it may be configured to include a curve, or may be configured as a broken line including a plurality of line segments.

[0035] When the entire determined virtual line 34 is located within the overlapping region 32, the control unit 11 trims the first user image 30A or the second user image 30B with the virtual line 34. The control unit 11 combines the first user image 30A and the second user image 30B trimmed with the virtual line 34 with the trimmed line to generate a composite image 40 as shown in FIG. 3C. In the composite image 40 illustrated in FIG. 3C, the overlapping region 32 and the virtual line 34 are shown. The virtual line 34 illustrated in FIGS. 3A, 3B, and 3C is determined as a line passing through the region between the lower part of the user's nose and the mouth. By determining the virtual line 34 so as not to divide a coherent part such as the nose or the mouth, the distortion of the user's face in the composite image 40 can be reduced.

[0036] When only a part of the determined virtual line 34 is located within the overlapping region 32, the control unit 11 may trim the image along a line connecting the virtual line 34 located within the overlapping region 32 and the contour line of the overlapping region 32. When the entire determined virtual line 34 is not located within the overlapping region 32, the control unit 11 may trim the image along the contour line of the overlapping region 32. That is, when at least a part of the determined virtual line 34 is not located within the overlapping region 32, the control unit 11 may trim the image along a line including the contour line of the overlapping region 32.

[0037] When the virtual line 34 does not exist within the overlapping region 32 and exists only outside the overlapping region 32, the control unit 11 may propose a movement to the user so that the virtual line 34 enters the overlapping region 32. The amount of movement required in this case can be reduced compared to the case where the entire range in which the user is to be photographed is contained within the photographing range of one photographing unit 14. That is, the degree of freedom of the user's position when generating an image obtained by photographing the entire range in which the user is to be photographed can be increased.

[0038] The control unit 11 may re-determine the virtual line 34 every time the user moves. For example, the control unit 11 may re-determine the virtual line 34 for each one or more frames of the video capturing the user. That is, the control unit 11 may dynamically change the virtual line 34. By doing so, even when the subject moves, the distortion of the subject can be reduced.

[0039] Even when synthesizing three or more images, the control unit 11 may detect the overlapping regions of each image, determine the virtual line 34 for each overlapping region, and trim the images at the positions of the virtual lines 34 and then synthesize them into one image.

[0040] <Example of the procedure of the information processing method> As described above, the information processing apparatus 10 according to the present embodiment synthesizes and displays a plurality of images captured with the user as a subject. The control unit 11 of the information processing apparatus 10 may execute an information processing method including the procedure of the flowchart illustrated in FIG. 4, for example. The information processing method may be realized as an information processing program to be executed by the control unit 11. The information processing program may be stored in a non-transitory computer-readable medium.

[0041] The control unit 11 acquires the first user image 30A and the second user image 30B (step S1). The control unit 11 analyzes the overlapping region 32 and the virtual line 34 in the first user image 30A and the second user image 30B (step S2).

[0042] The control unit 11 determines whether the virtual line 34 is included in the overlapping region 32 (step S3). When the virtual line 34 is included in the overlapping region 32 (step S3: YES), the control unit 11 trims the first user image 30A and the second user image 30B at the position of the virtual line 34 (step S4). When the virtual line 34 is not included in the overlapping region 32 (step S3: NO), the control unit 11 trims the first user image 30A and the second user image 30B at a predetermined position appropriately set within the overlapping region 32 (step S5).

[0043] After executing the trimming procedure in step S4 or S5, the control unit 11 synthesizes the trimmed first user image 30A and second user image 30B to generate a composite image 40 (step S6). After executing the procedure in step S6, the control unit 11 ends the execution of the flowchart procedure in FIG. 4.

[0044] <parentheses> As described above, the information processing apparatus 10 according to the present embodiment combines a plurality of images with the virtual line 34 determined in the overlapping region 32 to generate a single composite image 40. By doing so, the distortion of the subject's face or body in the composite image 40 can be reduced. That is, the distortion of the subject at the combined portion of the composite image 40 can be reduced. Further, the information processing apparatus 10 can determine the virtual line 34 based on the feature points of the subject. That is, the information processing apparatus 10 can dynamically change the virtual line 34 used for trimming. By doing so, the distortion of the subject can be reduced even when the subject moves.

[0045] (Other Embodiments) Hereinafter, a configuration example of the information processing apparatus 10 according to another embodiment will be described.

[0046] <Display of 3D Image> The information processing apparatus 10 may capture a user's RGB image and depth image with the imaging unit 14 and display a three-dimensional image of the user, that is, a 3D image. Hereinafter, specific operations for displaying a three-dimensional image will be described.

[0047] The imaging unit 14 of the information processing apparatus 10 captures an RGB image of at least a part of the user's body and measures the distance to at least a part of the user's body to generate a depth image. The depth image may be represented as a distance map having a plurality of pixels. In the distance map, the measurement result of the distance to the part corresponding to each pixel is associated with each pixel. The number and arrangement of the pixels of the depth image may be the same as or different from the number and arrangement of the pixels of the RGB image.

[0048] The control unit 11 of the information processing apparatus 10 generates a three-dimensional image of the user based on the depth image. The control unit 11 may generate the three-dimensional polygon data of the user based on the depth image, and generate a three-dimensional image by rendering the RGB image on each polygon constituting the three-dimensional polygon data. The control unit 11 may generate the voxel data of the user based on the depth image, and generate a three-dimensional image by assigning each pixel of the RGB image to each voxel. That is, the control unit 11 may generate a three-dimensional image of the user by mapping the RGB image to the three-dimensional data of the user. The control unit 11 may generate a three-dimensional image of the user in various modes not limited to this.

[0049] The control unit 11 may use the first user image 30A and the second user image 30B as the RGB images to be rendered on the three-dimensional polygon data of the user. In this case, the control unit 11 may trim the first user image 30A and the second user image 30B to match the joints of some polygons of the three-dimensional polygon data. Specifically, the control unit 11 may determine the virtual line 34 when trimming the first user image 30A and the second user image 30B as a line along the contour of the polygon on which the image is rendered.

[0050] The control unit 11 may use the first user image 30A and the second user image 30B as the RGB images to be assigned to the voxel data of the user. In this case, the control unit 11 may trim the first user image 30A and the second user image 30B so as to connect them with the portion where the radius of curvature of the surface of the voxel data of the user is small (the portion where it bends sharply) as the boundary. Specifically, the control unit 11 may calculate the radius of curvature of each part of the surface averaged in voxel units of the voxel data of the user, and trim the image so as to combine a plurality of images at positions where the radius of curvature is less than a predetermined value. That is, the control unit 11 may determine the virtual line 34 when trimming the first user image 30A and the second user image 30B as a line including positions where the radius of curvature is less than a predetermined value.

[0051] As described above, the control unit 11 may generate a three-dimensional image of the user by mapping the composite image 40 onto the three-dimensional data of the user as the subject.

[0052] Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each means or each step, etc. can be rearranged so as not to be logically contradictory, and a plurality of means or steps, etc. can be combined into one or divided.

Description of Reference Numerals

[0053] 10 Information processing apparatus (11: Control unit, 12: Storage unit, 13: Interface, 14: Imaging unit, 14A: First imaging unit, 14B: Second imaging unit) 20 Display device 30A, 30B First user image, Second user image 32 Overlapping area 34 Virtual line 40 Composite image

Claims

1. For each of a plurality of images having an overlapping region, which are obtained by simultaneously photographing a partial range of the entire range in which each of a plurality of photographing units with partially overlapping photographing ranges attempts to photograph a user as a subject, detecting the positions of the feature points of the user in the overlapping region; determining a virtual line so as to pass through the portion where the user is shown, based on the positions of the feature points of the user in each of the images; proposing a movement to the user so that the virtual line enters the overlapping region; performing trimming at the position of the virtual line on the images included in the overlapping region; combining the plurality of images to generate one composite image An information processing method comprising the above.

2. Obtaining a depth image of the user; Generating three-dimensional data of the user based on the depth image; Mapping the composite image onto the three-dimensional data of the user The information processing method according to claim 1, further comprising the above.

3. Comprising a plurality of photographing units and a control unit, Each of the plurality of photographing units simultaneously photographs an image of a partial range of the entire range in which it attempts to photograph a user as a subject, such that the images have an overlapping region with each other, The control unit Detects the positions of the feature points of the user in the overlapping region for the images photographed by each of the photographing units, Determines a virtual line so as to pass through the portion where the user is shown, based on the positions of the feature points of the user in each of the images, Proposes a movement to the user so that the virtual line enters the overlapping region, Performs trimming at the position of the virtual line on the images included in the overlapping region, Combines the images to generate one composite image, An information processing apparatus.

Images