Image processing apparatus, server apparatus, and image processing method

Abstract

An image processing device includes: a partial image extraction unit that extracts partial images of a plurality of persons to be photographed included in each of a plurality of photographed images having different photographing times, the plurality of persons to be photographed including the plurality of persons to be photographed; a voting unit that receives a plurality of votes of a plurality of voters from a partial image of a subject person, which is one of the subject persons among the plurality of partial images extracted by the partial image extraction unit; a partial image determination unit that determines one partial image on the basis of the voting results of the partial images received by the voting unit; and an image processing unit that generates a partial image of the subject person on the basis of the partial image determined by the partial image determination unit.

Description

Technical Field

[0001] The present disclosure relates to an image processing device, a server device, and an image processing method. Background Art

[0002] The following technology is known: when taking a group photo including a plurality of subject persons, a group photo in which all subject persons are satisfied is generated. For example, Patent Document 1 describes the following technology: a group photo is generated by synthesizing a partial image selected by each subject person in a continuously taken group photo with a composite image serving as a reference.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2010-16678. Summary of the invention

[0006] In the technology described in Patent Document 1, among the continuous photos taken by the digital camera, the control unit of the digital camera selects a partial image to be synthesized with the synthesized image by performing facial evaluation. In this case, when the group photo is viewed as a whole, it is possible that a group photo that satisfies all the subjects may not be generated.

[0007] An object of the present invention is to provide an image processing device, a server device, and an image processing method that can generate a photo that satisfies all subject persons.

[0008] The image processing device involved in the present disclosure includes: a partial image extraction unit, which extracts a plurality of partial images of the photographed subject persons contained in each of a plurality of captured images containing the plurality of photographed subject persons and taken at different times; a voting unit, which accepts a plurality of votes from a plurality of voters from a partial image of a subject person who is one of the photographed subject persons among the plurality of partial images extracted by the partial image extraction unit; a partial image determination unit, which determines a partial image based on the voting results of the partial images received by the voting unit; and an image processing unit, which generates a partial image of the subject person based on the partial image determined by the partial image determination unit.

[0009] The server device disclosed in the present invention includes: a partial image extraction unit, which extracts a plurality of partial images of a plurality of photographed subject persons contained in each of a plurality of photographed images having different shooting times; a partial image determination unit, which determines a partial image based on a plurality of votes performed by a plurality of terminal devices from a partial image of a subject person who is one of the plurality of partial images extracted by the partial image extraction unit; and an image processing unit, which generates a partial image of the subject person based on the partial image determined by the partial image determination unit.

[0010] The image processing method disclosed in the present invention comprises the following steps: extracting a plurality of partial images of the photographed subject persons contained in each of a plurality of captured images containing the plurality of photographed subject persons at different shooting times; determining a partial image based on a plurality of votes from a plurality of voters from a partial image of a subject person who is one of the photographed subject persons among the plurality of extracted partial images; and generating a partial image of the subject person based on the determined partial image.

[0011] According to the present disclosure, it is possible to generate a photo that satisfies all the subject persons. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing a configuration example of an image processing device according to the first embodiment.

[0013] Figure 2 This is a flowchart showing the flow of the partial image extraction process according to the first embodiment.

[0014] Figure 3 It is a diagram for explaining the method of extracting a partial image according to the first embodiment.

[0015] Figure 4 This is a diagram for explaining a method of collecting partial images for each person to be photographed according to the first embodiment.

[0016] Figure 5 This is a flowchart showing the flow of image processing in the first embodiment.

[0017] Figure 6 It is a diagram for explaining the method of displaying a partial image according to the first embodiment.

[0018] Figure 7 This is a flowchart showing the flow of image processing according to the second embodiment.

[0019] Figure 8 This is a block diagram showing a configuration example of an image processing system according to a third embodiment. DETAILED DESCRIPTION

[0020] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In addition, the present disclosure is not limited to the embodiment, and in the following embodiments, the same reference numerals are given to the same parts to omit repeated descriptions.

[0021] [First embodiment]

[0022] (Image processing device)

[0023] use Figure 1 A configuration example of the image processing device according to the first embodiment will be described. Figure 1 is a block diagram showing a configuration example of an image processing device according to the first embodiment.

[0024] like Figure 1 As shown, the image processing device 10 includes a shooting unit 20, an input unit 22, a display unit 24, a storage unit 26, and a control unit 28. The image processing device 10 is a device that, when taking a group photo including a plurality of shooting subjects, receives votes from the plurality of shooting subjects and generates a group photo that all the shooting subjects are satisfied with. Specifically, the image processing device 10 is a personal computer (PC), a smart phone, a photo camera installed in a game facility, etc. In addition, the image processing device 10 does not necessarily need to include the shooting unit 20.

[0025] The imaging unit 20 is a camera that captures an image of an object within the imaging range. For example, the imaging unit 20 captures a group photo of a plurality of objects within the imaging range.

[0026] The input unit 22 receives various input operations to the image processing device 10. The input unit 22 outputs an input signal corresponding to the received input operation to the control unit 28. The input unit 22 includes, for example, a touch panel, a button, a switch, etc. When a touch panel is used as the input unit 22, the input unit 22 is arranged on the display unit 24.

[0027] The display unit 24 displays various images. For example, the display unit 24 displays a group photo of a plurality of subject persons captured by the capturing unit 20. For example, the display unit 24 displays a partial image extracted by the partial image extracting unit 32 of the control unit 28. For example, the display unit 24 includes a liquid crystal display (LCD), an organic EL (Electro-Luminescence), or the like.

[0028] The storage unit 26 stores various information. The storage unit 26 stores information such as the calculation contents and programs of the control unit 28. The storage unit 26 includes, for example, at least one of a main storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory) or an external storage device such as a HDD (Hard Disk Drive).

[0029] The storage unit 26 may also store, for example, a learning model for determining the subject person included in the image captured by the imaging unit 20. The learning model is, for example, an AI (Artificial Intelligence) model for determining the subject person based on the facial image of the subject person. The learning model is, for example, configured to output information for determining the subject person by inputting the facial image of the subject person. The learning model may be, for example, a learned model using a CNN (Convolutional Neural Network).

[0030] The control unit 28 controls each unit of the image processing device 10. The control unit 28 includes, for example, an information processing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) and a storage device such as a RAM or a ROM. The control unit 28 may also be implemented by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 28 may also be implemented by a combination of hardware and software.

[0031] The control unit 28 includes a photographing control unit 30 , a partial image extraction unit 32 , a person identification unit 34 , a voting unit 36 ​​, a partial image determination unit 38 , and an image processing unit 40 .

[0032] The photographing control unit 30 controls the photographing unit 20 to shoot an image. For example, the photographing control unit 30 controls the photographing unit 20 to shoot a plurality of group photos including a plurality of photographing subject persons at different shooting times. The photographing control unit 30 acquires the image shot by the photographing unit 20. When the image processing device 10 does not include the photographing unit 20, the photographing control unit 30 may also be excluded.

[0033] The partial image extraction unit 32 extracts a specified area from the image captured by the imaging unit 20 as a partial image. For example, when the imaging unit 20 obtains a group photo including a plurality of subject persons, the partial image extraction unit 32 extracts the specified area of ​​each of the plurality of subject persons. The specified area of ​​the subject person is, for example, a facial area, but is not limited thereto. The specified area of ​​the subject person may also include a body area other than the facial area. When the specified area is a facial image, the partial image extraction unit 32 extracts the facial area of ​​each of the plurality of subject persons as a partial image. For example, the partial image extraction unit 32 extracts the facial area of ​​each of the plurality of subject persons from a plurality of group photos captured continuously. The partial image extraction unit 32 stores the extracted partial images in the storage unit 26. In addition, when the image processing device 10 does not include the imaging unit 20, the partial image extraction unit 32 may also receive an image captured by another imaging device such as a digital camera not shown in the figure and extract the partial image.

[0034] The personal identification unit 34 identifies whether the target person is a person who can vote for the partial image. The details of the personal identification unit 34 will be described later.

[0035] The voting unit 36 ​​receives a plurality of votes from a plurality of voters from the partial images of the voting target person among the plurality of partial images extracted by the partial image extraction unit 32. The voting unit 36 ​​will be described in detail later.

[0036] The partial image determination unit 38 determines a partial image to be combined with the reference image based on the voting result of the partial images. The partial image determination unit 38 will be described in detail later. Here, the reference image refers to any one of the plurality of set images captured by the imaging unit 20 .

[0037] The image processing unit 40 performs image processing on the corresponding area of ​​the reference image based on the partial image determined by the partial image determination unit 38. The image processing unit 40 performs, for example, a synthesis process of synthesizing the partial image with the reference image or a deformation process of deforming the reference image. The details of the image processing unit 40 will be described later.

[0038] (Partial image extraction processing)

[0039] use Figure 2 The partial image extraction process according to the first embodiment will be described. Figure 2 This is a flowchart showing the flow of the partial image extraction process according to the first embodiment.

[0040] The shooting control unit 30 obtains an image from the shooting unit 20 (step S10). Specifically, the shooting control unit 30 obtains continuous photos related to group photos of multiple shooting subject persons that are continuously shot from the shooting unit 20. Here, the continuous photos are, for example, a series of photos shot at regular intervals using the continuous shooting function of the shooting control unit 30. Here, the regular period is, for example, 100 msec. In addition, it is also possible not to use the continuous shooting function, and to use multiple photos shot with the same composition and a difference of several seconds in shooting time as continuous photos. Then, enter step S12.

[0041] The partial image extraction unit 32 extracts a partial image from the image acquired by the imaging control unit 30 (step S12 ). Figure 3 It is a diagram for explaining the method of extracting a partial image according to the first embodiment. Figure 3 An image IM1 is shown, which is one of the continuous photos including group photos of user U1, user U2, user U3, user U4, user U5, user U6 and user U7. User U1 to user U7 are examples of persons to be photographed. The partial image extraction unit 32 performs image recognition processing on the image IM1 and extracts the facial regions of user U1 to user U7. Figure 3 In the example shown, the partial image extraction unit 32 extracts the areas A1 to A7 as the face areas of the users U1 to U7 . Then, the process proceeds to step S14 .

[0042] The partial image extraction unit 32 determines whether a partial image has been extracted from all the images acquired by the shooting control unit 30 (step S14). Specifically, the partial image extraction unit 32 determines whether a partial image has been extracted from all the consecutive photos acquired in step S10. If it is determined that a partial image has been extracted from all the images acquired by the shooting control unit 30 (step S14; yes), the process proceeds to step S16. If it is not determined that a partial image has been extracted from all the images acquired by the shooting control unit 30 (step S14; no), the process proceeds to step S12. That is, the partial image extraction unit 32 repeats the processes of step S12 to step S14 until partial images are extracted from all the consecutive photos.

[0043] When it is determined to be YES in step S14 , the partial image extraction unit 32 collects partial images for each imaging target person (step S16 ). Figure 4 FIG. 1 is a diagram for explaining a method for collecting partial images for each person to be photographed according to the first embodiment. Figure 4IM1, IM2, and IM3 are shown in FIG. IM1 to IM3 are consecutive photos including group photos of users U1 to U7 taken consecutively. In step S12, the partial image extraction unit 32 performs image recognition processing on image IM1, and extracts regions A1 to A7 as facial regions of users U1 to U7 respectively. In step S12, the partial image extraction unit 32 performs image recognition processing on image IM2, and extracts regions B1 to B7 as facial regions of users U1 to U7 respectively. In step S12, the partial image extraction unit 32 performs image recognition processing on image IM3, and extracts regions C1 to C7 as facial regions of users U1 to U7 respectively. The specific processing of step S16 is described below. The partial image extraction unit 32 determines the same person included in images IM1 to IM3. The partial image extraction unit 32 determines the same person included in images IM1 to IM3, for example, using a learning model that determines a person by inputting a facial image and outputting it. Figure 4 For example, the partial image extraction unit 32 inputs partial images of regions A1 to A7, regions B1 to B7, and regions C1 to C7 into the learning model to determine the same person included in images IM1 to IM3. The partial image extraction unit 32 may also determine the same person based on the position of the person of the photographed subject included in images IM1 to IM3. The partial image extraction unit 32, for example, determines the user U1 included in images IM1 to IM3, and collects regions A1, B1, and C1 included in images IM1 to IM3 where the user U1 is photographed. That is, the partial image extraction unit 32 determines that the same person is photographed in regions A1, B1, and C1. The partial image extraction output unit 32 performs the same processing on users U2 to U7. Then, the process proceeds to step S18.

[0044] Here, a continuous photo consisting of a plurality of images may be managed as a block. For example, images IM1 to IM3 are set as a continuous photo G1. In this case, image IM1 can be represented as image G1-1, image IM2 can be represented as image G1-2, and image IM3 can be represented as image G1-3.

[0045] The partial image extraction unit 32 adds label information to each partial image of the subject person (step S18). Label information is information used to associate the subject person with the partial image between multiple images. For example, the partial image extraction unit 32 adds the same label information to the partial images of the same person in multiple images. Figure 4For example, the partial image extraction unit 32 adds the same label information T1 to the region A1 of the image IM1, the region B1 of the image IM2, and the region C1 of the image IM3. The partial image extraction unit 32 adds the same label information T2 to the region A2 of the image IM1, the region B2 of the image IM2, and the region C2 of the image IM3. The partial image extraction unit 32 adds the same label information T3 to the region A3 of the image IM1, the region B3 of the image IM2, and the region C3 of the image IM3. The partial image extraction unit 32 adds the same label information T4 to the region A4 of the image IM1, the region B4 of the image IM2, and the region C4 of the image IM3. The partial image extraction unit 32 adds the same label information T5 to the region A5 of the image IM1, the region B5 of the image IM2, and the region C5 of the image IM3. The partial image extraction unit 32 adds the same label information T6 to the region A6 of the image IM1, the region B6 of the image IM2, and the region C6 of the image IM3. The partial image extracting unit 32 adds the same label information T7 to, for example, the region A7 of the image IM1 , the region B7 of the image IM2 , and the region C7 of the image IM3 . Then, the process proceeds to step S20 .

[0046] The partial image extracting unit 32 stores the partial image to which the label information is added in the storage unit 26 (step S20 ). Then, the process proceeds to step S22 .

[0047] The voting unit 36 ​​sets the range of persons who can vote (step S22). For example, the voting unit 36 ​​sets only the multiple persons who are photographed in the group photo as the range of persons who can vote. Figure 3 In the example shown, the voting unit 36 ​​sets users U1 to U7 as the range of people who can vote. For example, the voting unit 36 ​​sets the members of the group to which the subject person in the group photo belongs as the range of people who can vote. The group to which they belong is, for example, a school, a grade in a school, a department activity, and the like. In a case where the subject person is a young child and cannot vote, the guardian of the subject person may be set as the range of people who can vote. For example, the voting unit 36 ​​sets all people as the range of people who can vote. In the case where the voting unit 36 ​​sets all people as the range of people who can vote, it may also set voting end conditions such as the number of people who can vote. Then, the voting ends. Figure 2 processing.

[0048] (Image Processing)

[0049] use Figure 5 The image processing according to the first embodiment will be described. Figure 5This is a flowchart showing the flow of image processing in the first embodiment.

[0050] exist Figure 5 In the process shown, the subject persons included in the group photo are set as subject persons who can vote.

[0051] The control unit 28 determines the photos of the processing object (step S30). Specifically, in step S30, for example, the control unit 28 causes the display unit 24 to display the photos of the continuous photos that are being voted on and are the processing object for each continuous photo. That is, the continuous photos can be multiple. The control unit 28, for example, displays the first photo taken in the continuous photos as a representative photo. Here, the photo of the continuous photos refers to the photo of the voting that has not been completed. The control unit 28 determines the continuous photos selected by the user from the continuous photos of ...

[0052] The personal identification unit 34 determines whether the user who selected the photo is a user who can vote (step S32). Specifically, when the user who selected the photo in step S30 is included in the processing target photo, the personal identification unit 34 determines that the user who selected the photo is a user who can vote. For example, the personal identification unit 34 inputs the facial image of the user who selected the photo and uses a learning model that outputs information indicating whether the user is included in the photo to select whether the user who selected the photo is included in the processing target photo. For example, when the user who selected the photo is user U1 and the processing target photo is image IM1, it is determined that user U1 is a user who can vote. Such a learning model can be used, for example, at the beginning Figure 5. In this case, when the user selects a photo in step S30, the shooting control unit 30 can control the shooting unit 20 to shoot an image containing the face of the user who selected the photo. Thus, the personal identification unit 34 can use the image captured when the user selects the photo in step S30 to determine whether the user who selected the photo is a user who can vote. For example, the personal identification unit 34 can also determine whether the user who selected the photo is a user who can vote based on whether the user who selected the photo is a pre-registered user. In this case, the personal identification unit 34 can determine whether the user who selected the photo is a user who can vote, for example, by password authentication, biometric authentication, etc. In another example, the personal identification unit 34 can issue an authentication code for voting for each consecutive photo, notify the user who can vote of the authentication code, and can determine whether the user is a user who can vote by entering the authentication code. In the case where it is determined that the user who selected the photo is a user who can vote (step S32; yes), enter step S34. In the following, the user who can vote determined in step S30 is sometimes referred to as a voter. In the case where it is not determined that the user who selected the photo is a user who can vote (step S32; no), end Figure 5 processing.

[0053] The voting unit 36 ​​displays a partial image of the voting target person who is one of the photographed target persons on the display unit 24 (step S34 ). Figure 6 FIG. 2 is a diagram for explaining a method for displaying a partial image according to the first embodiment. Figure 6 As shown, for Figure 4 For the user U1 shown in FIG. 1 , the voting unit 36 ​​causes the display unit 24 to display a list of images IM10 in which the area A1 of the image IM1, the area B1 of the image IM2, and the area C1 of the image IM3 are displayed in a list. The voting unit 36 ​​can cause the display unit 24 to display Figure 4 The overview images of each user are shown. For example, when the input unit 22 includes a button for switching the area images, the voting unit 36 ​​may switch the images of the displayed area A1, area B1, and area C1 each time the button for switching the area images of the input unit 22 is pressed. The voting unit 36 ​​may, for example, display the entirety of the image IM1, the image IM2, and the image IM3. In addition, the voting unit 36 ​​may also display the number of votes cast so far. For example, when one vote has been cast for the area A1 of the image IM1, three votes have been cast for the area B1 of the image IM2, and zero votes have been cast for the area C1 of the image IM3, the number of votes for each image is displayed on the display unit 24. Then, the process proceeds to step S36.

[0054] The voting unit 36 ​​accepts the votes (step S36). Figure 6In the example shown, the voting unit 36 ​​receives as a voting result a partial image of the region A1, region B1, and region C1 that is desired to be the composite image selected by the input unit 22. When the input unit 22 is a touch panel, the voting unit 36 ​​receives as a voting result an image displayed in the region pressed by the user among the plurality of images displayed on the display unit 24. Then, the process proceeds to step S38.

[0055] The voting unit 36 ​​determines whether the voter has voted for all the persons to be photographed (step S38). Figure 4 In the example shown, the voting unit 36 ​​determines whether the user U1 as the voter has voted for each of the users U1 to U7. If it is determined that the voter has voted for all the persons to be photographed (step S38; yes), the process proceeds to step S40. If it is not determined that the voter has voted for all the persons to be photographed (step S38; no), the process proceeds to step S34. That is, the voting unit 36 ​​repeatedly performs the processes of step S34 to step S38 until the voter has finished voting for all the persons to be photographed.

[0056] The voting unit 36 ​​determines whether all the voting subjects who can vote for the photo have voted (step S40). Figure 4 In the example shown, the voting unit 36 ​​determines whether each of the users U1 to U7 has voted. If it is determined that all the voting objects have voted (step S40; yes), the process proceeds to step S42. If all the voting objects have not voted (step S40; no), the process proceeds to step S34. That is, the voting unit 36 ​​repeatedly performs the process from step S34 to step S40 until all the voting objects have finished voting.

[0057] When the judgment is "yes" in step S40, the partial image determination unit 38 determines the partial image to be synthesized with the reference image (step S42). Specifically, the partial image determination unit 38 determines the partial image of each subject person based on the voting results of all voting subject persons. For example, the partial image determination unit 38 adds one point for each subject person, and determines the partial image with the largest number of points as the partial image to be synthesized with the reference image. The partial image determination unit 38 may also assign weights to the points according to the voting subject person.

[0058] For example, the partial image determination unit 38 may also assign weights to the points in a manner that respects the intention of the person. Here, the person refers to the user himself who is photographed in the partial image that is the object of voting, and the intention of the person refers to the voting object person voting for the partial image of the user himself. In this case, the partial image determination unit 38 assigns a weight greater than 1, such as 1.5 times, to the points for voting for the person, and assigns a weight less than 1, such as 0.7 times, to the points for voting for people other than the person. For example, when a total of 4 people vote, including the person and 3 people other than the person, (1.5×1)+(0.7×3)=3.6 points.

[0059] For example, the partial image determination unit 38 may also assign weights to the points in a manner that suppresses the influence of malicious votes from others. In this case, the control unit 28 preferably includes, for example, an expression analysis unit (not shown) that can analyze the expressions of the subject persons included in the group photo. The partial image determination unit 38, for example, assigns a weight of less than 1, for example, 0.5 times, to the points of the votes for the partial image determined by the expression analysis unit to be closed eyes. The partial image determination unit 38, for example, assigns a weight of greater than 1, for example, 1.5 times, to the points of the votes for the partial image determined by the expression analysis unit to be a smiling face.

[0060] For example, for a user who voted for a partial image of an arbitrary subject person with closed eyes in the processing target photo, the partial image determination unit 38 may assign a weight smaller than 1 to the points of the votes of other subject persons, even if the vote was for a partial image with open eyes. For example, when the processing target photo is a group photo of a school, etc., the partial image determination unit 38 may set a person with a neutral position such as a teacher or a photographer as a manager, and assign a weight larger than 1 to the points of the manager's vote.

[0061] The image processing unit 40 determines whether the partial image determined by the partial image determination unit 38 can be combined with the reference image that is the basis of the group photo (step S44). The image processing unit 40 determines that the partial image can be combined when the degree of consistency between the orientation of the face included in the partial image and the orientation of the face included in the reference image is greater than a predetermined threshold, and determines that the partial image cannot be combined when the degree of consistency is less than the predetermined threshold. For example, the image processing unit 40 determines that the partial image cannot be combined when the partial image is a face facing sideways and the reference image is a face facing frontally. The image processing unit 40 may also determine that the partial image can be combined when the degree of consistency between the size of the face included in the partial image and the size of the face included in the reference image is greater than a predetermined threshold. When it is determined that the partial image can be combined with the reference image that is the basis of the group photo (step S44; yes), the process proceeds to step S46. When it is determined that the partial image cannot be combined with the reference image that is the basis of the group photo (step S44; no), the process proceeds to step S48.

[0062] If the determination result in step S44 is "yes", the image processing unit 40 performs the synthesis process of the partial image determined by the partial image determination unit 38 (step S46). Specifically, the image processing unit 40 performs the synthesis process by replacing the image of the corresponding part of the reference image that will be the basis of the group photo with the partial image determined by the partial image determination unit 38. In addition, the method of selecting the reference image is arbitrary, for example, the photo with the least replacement of the partial image may be automatically determined, or it may be determined by voting of the persons to be photographed. For the synthesis process, the existing image processing technology can be used.

[0063] When it is determined as "No" in step S44, the image processing unit 40 performs deformation processing on the corresponding part of the reference image that is the basis of the group photo (step S48). Specifically, the image processing unit 40 performs deformation processing on the corresponding part of the reference image based on the partial image determined by the partial image determination unit 38. For example, the image processing unit 40 performs deformation processing on the corresponding part of the reference image to approximate the partial image determined by the partial image determination unit 38. For example, when the partial image is a smiling face facing sideways, the image processing unit 40 deforms the corresponding face of the reference image into a smiling face facing sideways and then facing frontward. For example, when the size of the face included in the partial image is smaller than the corresponding part of the reference image, the image processing unit 40 enlarges and deforms the partial image. For example, the image processing unit 40 may also deform the corresponding face of the reference image into a smiling face by a machine learning method. In this case, the image processing unit 40 may input part of the image into a learning model that has been learned by outputting image data of a face facing forward when inputting image data of a face facing sideways, and deform the corresponding face of the reference image based on the output image data of a face facing forward. In this case, the learning model may be stored in the storage unit 26. In step S48, the image processing unit 40 may perform synthesis processing on a part and deformation processing on a part.

[0064] exist Figure 5 In the process shown, step S44 may be omitted. That is, step S46 may be executed after the process of step S42, or step S48 may be executed after the process of step S42.

[0065] As described above, the first embodiment determines a partial image to be synthesized into the group photo based on the voting results of all the multiple subject persons included in the group photo. In the first embodiment, when it is determined that the partial image can be synthesized with the group photo, a synthesis process of synthesizing the partial image with the group photo is performed, and when it is determined that it cannot be synthesized with the group photo, a deformation process of deforming the group photo based on the partial image is performed. As a result, in the first embodiment, a group photo that satisfies all the subject persons can be generated.

[0066] [Second embodiment]

[0067] (Image Processing)

[0068] use Figure 7 Image processing according to the second embodiment will be described. Figure 7 This is a flowchart showing the flow of image processing according to the second embodiment.

[0069] exist Figure 7 In the process shown, the range of the target persons who can vote is set, and the voting end condition is set.

[0070] The processing of step S50 is Figure 5 The processing of step S30 shown is the same, so the description is omitted. In addition, before step S50, a step of setting a voting end condition may also be included. The voting end condition refers to, for example, the number of people who can vote, and only the set number of people can vote. The voting end condition refers to, for example, the period during which voting can be performed, and if the set period is exceeded, voting cannot be performed.

[0071] The personal identification unit 34 identifies the user who selected the photo (step S52). For example, the personal identification unit 34 identifies the user by the user logging in for voting. The login can be performed by a known method such as password authentication, biometric authentication, etc. Then, the process proceeds to step S54.

[0072] The personal identification unit 34 determines whether the photo selected by the user is a photo that can be voted for (step S54). Specifically, if the personal identification unit 34 determines that the user who selected the photo, i.e., the voter, is a user who is eligible to vote for the photo, the photo selected by the user is a photo that can be voted for. If the personal identification unit 34 determines that the voter is not a user who is eligible to vote for the photo, the photo selected by the user is not a photo that can be voted for. If it is determined that the photo selected by the user is a photo that can be voted for (step S54; yes), the process proceeds to step S56. If it is determined that the photo selected by the user is not a photo that can be voted for (step S54; no), the process ends. Figure 7 processing.

[0073] The processing from step S56 to step S60 is the same as Figure 5 The processes from step S34 to step S38 shown are the same, so the description is omitted.

[0074] If the determination result is "yes" in step S60, the voting unit 36 ​​determines whether the processing target photo satisfies the voting end condition (step S62). For example, if the voting end condition is that 10 people can vote, and the number of people who voted at the time of step S62 is 10, it is determined that the voting end condition is satisfied. If it is determined that the voting end condition is satisfied (step S62; yes), the process proceeds to step S64. If it is not determined that the voting end condition is satisfied (step S62; no), the process proceeds to step S56. That is, the voting unit 36 ​​repeatedly performs the process from step S56 to step S62 until the voting end condition is met.

[0075] The processing of step S64 and step S70 is the same as Figure 5 The processes of step S42 and step S48 shown are the same, so the description is omitted.

[0076] As described above, the second embodiment determines the partial image to be synthesized with the group photo based on the voting result for which the range of the subject persons who can vote and the voting end condition are set. In the second embodiment, when it is determined that the partial image can be synthesized with the group photo, a synthesis process of synthesizing the partial image with the group photo is performed, and when it is determined that it cannot be synthesized with the group photo, a deformation process of deforming the group photo based on the partial image is performed. Thus, in the second embodiment, a group photo that satisfies all the subject persons can be generated.

[0077] [Third Embodiment]

[0078] (Image Processing System)

[0079] Will refer to Figure 8 A configuration example of an image processing system according to the third embodiment will be described. Figure 8 It is a block diagram showing a configuration example of an image processing system according to a third embodiment.

[0080] like Figure 8 As shown in FIG. 1 , the image processing system 1 includes a terminal device 12 and a server device 14. The terminal device 12 and the server device 14 are communicably connected via a network N. Figure 8In the example shown, only one terminal device 12 is shown, but in reality, a plurality of terminal devices 12 and a server device 14 are connected to each other in a communicative manner via a network N. The network N is, for example, the Internet, but is not limited thereto. The image processing system 1 is a system in which, when a group photo including a plurality of subject persons is taken, the server device 14 receives votes from the terminal devices 12 of the plurality of subject persons, and generates a group photo that all the subject persons are satisfied with.

[0081] The terminal device 12 is a terminal device owned by a user who is a person to be photographed. The terminal device 12 is, for example, a smartphone, a digital camera, a PC, etc., but is not limited thereto. A plurality of terminal devices 12 can share image data photographed by each terminal device 12 via, for example, a server device 14.

[0082] The server device 14 is, for example, a cloud server. The server device 14, for example, collects image data related to the group photo from the terminal device 12. For example, the server device 14 receives votes from the terminal device 12 and generates a group photo that satisfies all the persons of the shooting object.

[0083] (Terminal device)

[0084] The terminal device 12 includes an imaging unit 50, an input unit 52, a display unit 54, a storage unit 56, a communication unit 58, and a control unit 60. In addition, the terminal device 12 does not necessarily need to include the imaging unit 50.

[0085] The imaging unit 50 is a camera that captures an image of an object within the imaging range. For example, the imaging unit 50 captures a group photo of a plurality of objects within the imaging range.

[0086] The input unit 52 receives various input operations to the terminal device 12. For example, the input unit 52 receives an operation of voting on a partial image to be synthesized with a reference image. The input unit 52 outputs an input signal corresponding to the received input operation to the control unit 60. The input unit 52 includes, for example, a touch panel, a button, a switch, etc. When a touch panel is used as the input unit 52, the input unit 52 is arranged on the display unit 54.

[0087] The display unit 54 displays various images. For example, the display unit 54 displays a group photo of a plurality of subject persons photographed by the photographing unit 50. For example, the display unit 54 displays a group photo of a plurality of subject persons photographed by other terminal devices 12 and stored in the server device 14. In addition, when the terminal device 12 does not include the photographing unit 50, the display unit 54 receives and displays the group photo from an external photographing device not shown in the figure via the communication unit 58. For example, the display unit 54 displays a partial image extracted by the server device 14. The display unit 54 is, for example, a display including a liquid crystal display, an organic EL, etc.

[0088] The storage unit 56 stores various information. The storage unit 56 stores information such as the calculation contents and programs of the control unit 60. The storage unit 56 includes at least one of a main storage device such as a RAM or a ROM, and an external storage device such as a HDD.

[0089] The communication unit 58 transmits and receives various information to and from an external device wirelessly. The communication unit 58 transmits and receives various information to and from the server device 14, for example, via the network N. The communication unit 58 transmits, for example, image data related to a group photo taken by the camera unit 50 to the server device 14 via the network N. The communication unit 58 transmits, for example, voting information related to voting for a partial image input to the input unit 52 to the server device 14 via the network N. The communication unit 58 is implemented by, for example, a communication module that communicates by means of a wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or the like.

[0090] The control unit 60 controls each unit of the terminal device 12. The control unit 60 includes, for example, an information processing device such as a CPU or an MPU and a storage device such as a RAM or a ROM. The control unit 60 may also be implemented by, for example, an integrated circuit such as an ASIC or an FPGA. The control unit 60 may also be implemented by a combination of hardware and software.

[0091] The control unit 60 includes a photographing control unit 70 , an image transmitting and receiving unit 72 , and a voting receiving unit 74 .

[0092] The photographing control unit 70 controls the photographing unit 50 to photograph an image. For example, the photographing control unit 70 controls the photographing unit 50 to photograph a group photo including a plurality of photographing subject persons. The photographing control unit 70 acquires the image photographed by the photographing unit 50. In addition, when the terminal device 12 does not include the photographing unit 50, the control unit 60 does not include the photographing control unit 70.

[0093] The image transceiver unit 72 controls the communication unit 58 to transmit the image related to the group photo acquired by the shooting control unit 70 from the shooting unit 50 to the server device 14 via the network N. The image transceiver unit 72 controls the communication unit 58 to receive the image related to the group photo captured by other terminal devices 12 or an external shooting device (not shown) from the server device 14 via the network N. The image transceiver unit 72 controls the communication unit 58 to receive the partial image for synthesis with the reference image.

[0094] The vote receiving unit 74 receives votes for determining a partial image to be combined with the reference image. The vote receiving unit 74 controls the communication unit 58 to transmit voting information related to the voting result to the server device 14 via the network N.

[0095] (Server device)

[0096] The server device 14 includes a storage unit 80 , a communication unit 82 , and a control unit 84 .

[0097] The storage unit 80 stores various information. The storage unit 80 stores information such as the calculation contents and programs of the control unit 84. The storage unit 80 includes at least one of a main storage device such as a RAM and a ROM, and an external storage device such as a HDD.

[0098] The storage unit 80 stores images captured by each terminal device 12 connected to the server device 14 via the network N or an external imaging device (not shown). The storage unit 80 stores the image in association with the terminal device 12 that captured the image, for example.

[0099] The communication unit 82 transmits and receives various information with external devices wirelessly. The communication unit 82 transmits and receives various information with the terminal device 12 or an external camera device (not shown) via the network N, for example. The communication unit 82 receives image data related to a group photo from the terminal device 12 or an external camera device (not shown) via the network N, for example. The communication unit 82 receives voting information about voting for a partial image from the terminal device 12 via the network N, for example. The communication unit 82 is implemented by a communication module that communicates by means of a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, or the like, for example.

[0100] The control unit 84 controls each unit of the server device 14. The control unit 84 includes, for example, an information processing device such as a CPU or an MPU and a storage device such as a RAM or a ROM. The control unit 84 may also be implemented by, for example, an integrated circuit such as an ASIC or an FPGA. The control unit 84 may also be implemented by a combination of hardware and software.

[0101] The control unit 84 includes an image acquisition unit 90 , a partial image extraction unit 92 , a partial image determination unit 94 , and an image processing unit 96 .

[0102] The image acquisition unit 90 acquires images from each terminal device 12 or an external camera (not shown) via the network N. The image acquisition unit 90 acquires images related to a group photo including a plurality of subject persons from each terminal device 12 via the network N. The image acquisition unit 90 stores the images acquired from each terminal device 12 via the network N in association with the terminal device 12 in the storage unit 80.

[0103] The partial image extraction unit 92 extracts a predetermined area as a partial image from the image stored in the storage unit 80. The partial image extraction unit 92 extracts an image of a face area of ​​each subject person as a partial image from an image related to a group photo including a plurality of subject persons stored in the storage unit 80. The partial image extraction unit 92 stores the extracted partial image in the storage unit 80.

[0104] The partial image determination unit 94 determines a partial image to be combined with the reference image based on the voting results on the partial images received from the plurality of terminal devices 12 .

[0105] The image processing unit 96 performs image processing on the corresponding region of the reference image based on the partial image determined by the partial image determining unit 94. The image processing unit 96 performs, for example, a synthesis process of synthesizing the partial image with the reference image or a deformation process of deforming the reference image.

[0106] As described above, in the third embodiment, the server device determines a partial image to be combined with the group photo based on the voting results received from the plurality of terminal devices 12. Thus, in the third embodiment, a group photo that all the subject persons are satisfied with can be generated.

[0107] In addition, the third embodiment enables each user to vote for a partial image to be synthesized with a reference image using the terminal device 12 owned by the user. Thus, the third embodiment can improve convenience.

[0108] The components of each device shown in the figure are conceptual functions and do not necessarily need to be physically configured as shown in the figure. That is, the specific method of distributing and integrating each device is not limited to the method shown in the figure, and all or part of them can be functionally or physically distributed or integrated in arbitrary units according to various loads, usage conditions, etc. In addition, the structure based on this distribution / integration can also be dynamically performed.

[0109] The above embodiments of the present invention are described, but the present disclosure is not limited to the contents of these embodiments. In addition, the above-mentioned constituent elements include elements that can be easily thought of by those skilled in the art, substantially the same elements, and elements of so-called equivalent scopes. Moreover, the above-mentioned constituent elements can be appropriately combined. Moreover, various omissions, substitutions or changes of constituent elements can be performed within the scope of the gist of the above-mentioned embodiments.

[0110] Industrial availability

[0111] The image processing device, server device, and image processing method disclosed herein can be used in information processing devices such as computers, for example.

[0112] Explanation of symbols

[0113] 10 Image processing device

[0114] 20, 50 shooting department

[0115] 22, 52 Input section

[0116] 24, 54 display unit

[0117] 26, 56, 80 storage unit

[0118] 28, 60, 84 control unit

[0119] 30, 70 Shooting control unit

[0120] 32, 92 Partial image extraction unit

[0121] 34 Personal Identification Department

[0122] 36 Voting Department

[0123] 38, 94 Partial Image Determination Unit

[0124] 40, 96 Image Processing Department

[0125] 58, 82 Ministry of Communications

[0126] 72 Image transceiver

[0127] 74 Voting Acceptance Department

[0128] 90 Image acquisition unit

Claims

1. An image processing device, comprising: A partial image extraction unit extracts partial images of a plurality of photographed subject persons contained in each of a plurality of photographed images containing the plurality of photographed subject persons and shot at different times; a voting unit that receives a plurality of votes from a plurality of voters from a partial image of a subject person who is one of the photographed subject persons among the plurality of partial images extracted by the partial image extraction unit; a partial image determining unit that determines a partial image based on the voting result on the partial image received by the voting unit; as well as The image processing unit generates a partial image of the target person based on the partial image determined by the partial image determining unit.

2. The image processing device according to claim 1, wherein: The image processing unit synthesizes the partial image as a partial image of the target person.

3. The image processing device according to claim 1, wherein: The image processing unit deforms the partial image of the target person so as to approximate the partial image.

4. The image processing device according to claim 1, wherein: The partial image determination unit determines the partial image by assigning weights corresponding to the plurality of voters.

5. The image processing device according to claim 4, wherein: The partial image determination unit determines the partial image by assigning a greater weight to the vote of the voter for the partial image than to the vote of the voters other than the voter for the partial image.

6. The image processing device according to claim 1, wherein: The system further comprises an expression analysis unit, which outputs a determination result of analyzing the expression of the photographed subject. The partial image determination unit determines the partial image by assigning a weight corresponding to a determination result of the expression analysis unit.

7. The image processing device according to claim 4, wherein: The partial image determination unit sets a person in a neutral position as a manager, and determines the partial image by giving a greater weight to the vote of the manager than to votes other than the manager.

8. A server device, comprising: A partial image extraction unit extracts partial images of a plurality of photographed subject persons contained in each of a plurality of photographed images containing the plurality of photographed subject persons and shot at different times; a partial image determining unit that determines a partial image from the partial image of the subject person who is one of the photographed subject persons among the plurality of partial images extracted by the partial image extracting unit based on a plurality of votes performed by a plurality of terminal devices; as well as The image processing unit generates a partial image of the target person based on the partial image determined by the partial image determining unit.

9. An image processing method comprising the following steps: Extracting partial images of a plurality of photographed persons contained in each of a plurality of photographed images containing a plurality of photographed persons and shot at different times; determining one partial image from among the extracted partial images of a subject person who is one of the photographed subject persons based on a plurality of votes of a plurality of voters; and Based on the determined partial image, a partial image of the target person is generated.

Images