Image processing device

Abstract

It generates images of the appropriate state depending on the situation. [Solution] The image processing device includes an image acquisition unit that acquires a captured image, a person identification unit that detects and identifies a person included in the acquired image, and a processed image generation unit that, when a person is identified, generates a processed image by processing the state of the person in the captured image according to the shooting conditions of the person in the captured image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus.

Background Art

[0002] An apparatus for generating an image in which the expression of a person is changed based on an image of the person taken is disclosed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique described in Patent Document 1, without considering the human relationship between the people photographed together and the situation at the time of shooting such as private or business, it simply changes the expression of the person to a good expression. Therefore, a technique for generating an image with an appropriate expression according to the situation at the time of shooting is required. In addition, not limited to expressions, a technique for generating an image in an appropriate state according to the situation at the time of shooting, including the standing position, pose, etc. is required.

[0005] The present invention has been made in view of the above, and an object thereof is to provide an image processing apparatus capable of generating an image in an appropriate state according to a situation.

Means for Solving the Problems

[0006] The image processing apparatus according to the present invention includes an image acquisition unit that acquires a captured image that has been captured, a person identification unit that detects and identifies a person included in the acquired captured image, and when the person is identified, a processed image generation unit that generates a processed image obtained by processing the shooting state of the person in the captured image according to the shooting situation regarding the person in the captured image. [Effects of the Invention]

[0007] According to the present invention, it is possible to generate images of an appropriate state depending on the situation. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a functional block diagram showing an example of an image processing apparatus according to an embodiment. [Figure 2] Figure 2 schematically shows the relationship between shooting conditions data and shooting status data. [Figure 3] Figure 3 is a schematic diagram illustrating an example of the process for generating a processed image. [Figure 4] Figure 4 is a flowchart showing an example of processing using the image processing apparatus according to the present embodiment. [Figure 5] Figure 5 is a flowchart showing another example of processing using the image processing apparatus according to this embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of the image processing apparatus according to the present invention will be described with reference to the drawings. However, the present invention is not limited by these embodiments. Furthermore, the components in the following embodiments include those that are easily substituted or substantially identical to those that are easily substituted by those skilled in the art.

[0010] Figure 1 is a functional block diagram showing an example of an image processing device 100 according to this embodiment. The image processing device 100 shown in Figure 1 generates a processed image of a person in a captured image based on an image captured by an external camera or the like. As shown in Figure 1, the image processing device 100 includes a communication unit 10, an input / output unit 20, a control unit 30, and a storage unit 40. The image processing device 100 may be provided as a terminal device such as a smartphone or tablet. In this configuration, if the terminal device is equipped with a camera, the processed image may be generated based on an image captured by the camera mounted on the terminal device.

[0011] The communication unit 10 performs wired or wireless communication with external devices. The communication unit 10 includes an interface such as a network interface card.

[0012] The input / output unit 20 handles the input and output of captured images. The input / output unit 20 receives captured images, for example, taken by a camera or other imaging device. The input / output unit 20 outputs processed images generated by the processed image generation unit 33, which will be described later.

[0013] The control unit 30 performs various information processing. The control unit 30 includes a processor such as a CPU (Central Processing Unit) and memory such as ROM (Read Only Memory) and RAM (Random Access Memory). The control unit 30 has an image acquisition unit 31, a person identification unit 32, and a processed image generation unit 33.

[0014] The image acquisition unit 31 can acquire various types of captured images, such as captured images input from the input / output unit 20 and captured images stored in the storage unit 40. There may be one or more captured images.

[0015] The person identification unit 32 detects and identifies people included in the captured images acquired by the image acquisition unit 31. Hereinafter, the person identified by the person identification unit 32 may be referred to as the target person.

[0016] When the target person is identified by the person identification unit 32, the processed image generation unit 33 generates a processed image obtained by processing the shooting state of the target person in the captured image according to the shooting situation of the target person in the captured image. In the present embodiment, the shooting situation includes at least one of the relationship between target persons when a plurality of target persons are included in the captured image and the background of the target person in the captured image. Further, in the present embodiment, the shooting state includes at least one of the expression of the target person, the position of the target person in the captured image, and the posture of the target person in the captured image. Therefore, the processed image generation unit 33 generates a processed image obtained by processing at least one of the expression of the target person, the position of the target person in the captured image, and the posture of the target person in the captured image according to at least one of the relationship between a plurality of target persons and the background of the target person. In addition, the processed image generation unit 33 can obtain the relationship between target persons based on the action history of the target person and process the shooting state of the target person according to the obtained relationship. Specific operations of the processed image generation unit 33 will be described later.

[0017] The storage unit 40 stores various programs, information such as data, applications, etc. for performing the processing in the control unit 30. The storage unit 40 includes a storage such as a HDD (Hard Disk Drive) or a SSD (Solid State Drive).

[0018] In addition, the storage unit 40 stores person identification data D1, shooting situation data D2, and shooting state data D3 for performing processing in the person identification unit 32 and the processed image generation unit 33. The storage unit 40 also stores image data D4.

[0019] The person identification data D1 is data for identifying the person shown in the captured image from the captured image. The person identification data D1 is data in which an image of a person is associated with the identification information of the person. The person identification data D1 includes information about one or more persons.

[0020] The shooting situation data D2 is data indicating the shooting situation regarding the people shown in the captured image. The shooting situation data D2 includes relationship data D2a and background data D2b.

[0021] The relationship data D2a is data including multiple types of items such as, for example, the intimacy indicating the depth of the private relationship between people, and the business degree indicating the depth of the business relationship between people. The relationship data D2a can be represented as numerical values indicating the respective degrees for each item such as intimacy and business degree. Note that the relationship data D2a may be composed of one item (for example, the degree of the relationship between people). Further, the items constituting the relationship data D2a are not limited to the above, and may be, for example, human emotions such as joy and sadness, or items that can be obtained by a machine learning model. The relationship data D2a can be stored in the storage unit 40 in advance, for example. In this case, the processed image generation unit 33 can obtain the relationship between people within the range included in the relationship data D2a stored in the storage unit 40.

[0022] The background data D2b is data indicating the background of the captured image as a score. The background data D2b is data obtained by scoring items such as the clothes worn by the people and the objects shown in the background of the people as shooting situations such as the degree of privacy. Note that the background in the background data D2b may include, in addition to the above clothes and objects, data on the brightness of the entire captured image, data on contrast, and data on combinations thereof, or may include various items and items that can be obtained from the captured image using a machine learning model.

[0023] The shooting state data D3 is data indicating the shooting state of a person. The shooting state data D3 includes expression data D3a, position data D3b, and posture data D3c.

[0024] Facial expression data D3a is data that represents a person's facial expression. Facial expression data D3a is set as multiple types of items for a person, such as laughter, anger, sadness, etc. Alternatively, facial expression data D3a may be set as an image of a person making a specific facial expression. Furthermore, for each item, facial expression data D3a may have a degree of expression set, such as a smile, a normal laugh, a big laugh, a roar of laughter, etc. Also, for the laughing expression, different types of items may be set, such as smiling with the corners of the mouth turned up while keeping the mouth closed, or smiling with the mouth open. Facial expression data D3a can be generated in advance for each person based on, for example, the image data D4 described later stored in the memory unit 40, or image data stored in an external memory device.

[0025] Location data D3b is data indicating the position of a person within the captured image. When changing the distance between multiple people within the captured image, for example, it can be set so that the closer the person's intimacy level, the closer they are, and the further away they are, the lower their intimacy level. Furthermore, the order in which multiple people are arranged can be set so that, for example, the sum of the intimacy levels of adjacent people is maximized.

[0026] Pose data D3c is data that indicates the posture of a person in a captured image. For postures that change continuously from a standard, such as the degree of curvature of the person's back or the degree of leg spread, the change can be made larger as the level of familiarity increases and smaller as the level of familiarity decreases. Furthermore, the magnitude of such changes from the standard posture can be made larger as the level of familiarity or business level decreases, and smaller as the level of familiarity or business level of the captured image decreases.

[0027] Furthermore, for poses such as a peace sign or clasped hands in front, where there are two possible outcomes relative to the standard (for example, standing upright with arms extended straight down), a threshold can be set. If the score of the shooting situation data D2 is higher (or lower) than the threshold, the pose is set to "present," and if the score of the shooting state data D3 is lower (or higher) than the threshold, the pose is set to "absent." For example, a peace sign is more likely to be made in private, intimate situations and less likely in business situations. Therefore, for example, if the level of intimacy is higher than the threshold, the peace sign pose can be set to "present," and if the level of intimacy is lower than the threshold, the peace sign pose can be set to "absent." Similarly, if the level of business is higher than the threshold, the clasped hands pose can be set to "present," and if the level of business is lower than the threshold, the clasped hands pose can be set to "absent."

[0028] Image data D4 includes captured image data acquired by the image acquisition unit 31 and processed image data generated by the processed image generation unit 33. Furthermore, if the image processing device 100 is an information terminal device such as a smartphone or tablet equipped with a camera, it may also include image data captured by the camera of that information terminal device.

[0029] Figure 2 schematically illustrates the relationship between shooting situation data D2 and shooting state data D3. In Figure 2, relationship data D2a is used as an example of shooting situation data D2, and facial expression data D3a is used as an example of shooting state data D3.

[0030] As shown in Figure 2, the facial expressions of individuals in the facial expression data D3a (facial expression 1, facial expression 2, facial expression 3, ...) are associated with the relationships between multiple individuals in the relationship data D2a of the shooting situation data D2 described above (relationship 1, relationship 2, relationship 3, ...). In other words, the facial expression data D3a represents the facial expressions of individuals according to the scores of predetermined items in the relationship data D2a.

[0031] Although not shown in the diagram, each item of the position data D3b and posture data D3c may be associated with the relationships between multiple people in the relationship data D2a. In addition, each item of the facial expression data D3a, position data D3b, and posture data D3c in the shooting state data D3 may be associated with a predetermined item in the background data D2b of the shooting situation data D2.

[0032] The memory unit 40 stores an image processing program that causes the computer to perform the following steps: acquiring a captured image; detecting a person if the acquired image contains one; detecting the shooting conditions of the image if a person is detected; and generating a processed image by processing the shooting conditions of the person in the image according to the shooting conditions.

[0033] In the image processing device 100, the processor in the control unit 30 reads various programs and loads them into memory, thereby executing information processing corresponding to the functions of each of the above-mentioned parts. Examples of various programs include programs received by the communication unit 10, programs stored in the storage unit 40, and programs recorded on an external recording medium. The image processing device 100 functions as an information processing device (computer) that executes various information processing. Note that other information processing devices different from the image processing device 100 may execute various programs, or the image processing device 100 and other information processing devices may cooperate to execute various programs.

[0034] Next, an example of the operation of the image processing device 100 configured as described above will be explained. When a captured image to be processed is input, the image acquisition unit 31 acquires the input captured image.

[0035] The person identification unit 32 detects a person if the captured image acquired by the image acquisition unit 31 contains a person. The person identification unit 32 can detect a person's image, for example, by performing image processing on the captured image. The person identification unit 32 can identify the person based on the detected person's image and the person identification data described above.

[0036] The processed image generation unit 33 determines whether or not there are multiple target individuals identified by the person identification unit 32. If the processed image generation unit 33 determines that there are multiple target individuals detected by the person identification unit 32, it can detect the relationships between the multiple detected target individuals as part of the shooting situation. For example, the processed image generation unit 33 can detect the relationships between the multiple target individuals based on relationship data that indicates the relationships between the target individuals.

[0037] The processed image generation unit 33 may determine the relationships between target individuals or between a target individual and other individuals based on the target individual's behavioral history. In other words, the processed image generation unit 33 may calculate the numerical values ​​for each item of the relationship data D2a based on the target individual's behavioral history. In this case, the processed image generation unit 33 can detect a person's behavioral history based on various data, such as data stored in the image processing device 100 or data from an external device acquired via the communication unit 10. Examples of such data include email data, social networking service (SNS) data, text data entered by the target individual, telephone call history data, phone book data, vital data, and image data taken separately from the captured image. For example, a person who is close to the target individual in the photograph, a person who frequently exchanges emails or other communications with the target individual, or a person who spends time with or communicates with the target individual outside of working hours can be judged to have a high degree of intimacy with the target individual. Furthermore, when detecting the behavioral history of a target person based on photographic data, the processed image generation unit 33 can detect the behavioral history based on various information such as the facial expression of the target person in the photographic data, the facial expressions of other people pictured with the target person, the target person's gaze in the photograph, the distance between the target person and other people in the photograph, and the number of photographs in which the target person and those other people are pictured together. If the image processing device 100 has the functionality of an information terminal device such as a smartphone, the various data generated by using the information terminal device can be stored in the storage unit 40. In this case, the processed image generation unit 33 can use the various data stored in the storage unit 40 to determine the behavioral history of the target person.

[0038] The processed image generation unit 33 sets the shooting state of the target person based on the determined relationship. In this case, the processed image generation unit 33 can set the shooting state of the target person based, for example, on the correspondence between the relationship data D2a of the shooting situation data D2 and the shooting state data D3. Specifically, when the processed image generation unit 33 detects multiple target people in the captured image, it calculates a relationship score for the multiple target people based on the relationship data D2a, and sets the shooting state by selecting the shooting state data D3 (at least one of the facial expression data D3a, position data D3b, and posture data D3c) corresponding to that score. The processed image generation unit 33 may also calculate a score that takes into account the brightness, contrast, etc., of the entire captured image. Through this process, the shooting state can be set according to the shooting situation.

[0039] The processed image generation unit 33 generates a processed image by processing the captured image based on the shooting state settings.

[0040] The following describes an example of the process by which the image processing device 100 generates a processed image. Figure 3 is a schematic diagram illustrating an example of the process for generating a processed image. Figure 3 shows an example where two subjects M1 and M2 are photographed. As shown in Figure 3, for example, when two cameras C1 and C2 are positioned separately to the left and right, and the two subjects are photographed with the two cameras C1 and C2, the left camera C1 generates a photographed image P1 in which the positions of the two subjects M1 and M2 are shifted to the right relative to the background trees. The right camera C2 generates a photographed image P2 in which the positions of the two people are shifted to the left relative to the background trees. The generated photographed images P1 and P2 are stored in, for example, the storage unit 40.

[0041] If, for example, one of the captured images P1 and P2, P1, is input, the image acquisition unit 31 acquires P1. The person identification unit 32 detects two target people M1 and M2 based on P1 and identifies them. The processed image generation unit 33 calculates the shooting conditions for the two target people M1 and M2. Here, we will explain using the case where the two target people M1 and M2 have a high level of intimacy as an example. The processed image generation unit 33 detects that the two target people M1 and M2 have a high level of intimacy and calculates their relationship based on that intimacy. Depending on the calculated relationship between the two target people M1 and M2, the processed image generation unit 33 can generate a processed image in which the two target people M1 and M2 are closer together, for example. In this case, the processed image generation unit 33 can generate a processed image by combining multiple images. Specifically, the processed image generation unit 33 retrieves from the storage unit 40 a different image P2 from the input image P1, out of the two captured images P1 and P2 mentioned above. The processed image generation unit 33 can create a processed image P3 in which the distance between the two subject people M1 and M2 is brought closer by combining the portion of the captured image P1 that includes the subject person M1 on the left and the portion of the captured image P2 that includes the subject person M2 on the right. The processed image generation unit 33 outputs the generated processed image P3.

[0042] Furthermore, the processed image generation unit 33 can set the shooting state of the target person based on the detected background. In this case, the processed image generation unit 33 can set the shooting state of the target person based on the correspondence between the background data D2b and the shooting state data D3 of the shooting situation data D2. Specifically, when the processed image generation unit 33 detects an object in the background of the captured image, it calculates a score for the object based on the background data D2b and sets the shooting state by selecting the shooting state data D3 (at least one of the facial expression data D3a, position data D3b, and posture data D3c) corresponding to that score. The processed image generation unit 33 may also calculate a score that takes into account the brightness, contrast, etc., of the entire captured image.

[0043] For example, if the image includes Christmas trees or cakes, the people in the photo tend to have upturned corners of their mouths because it represents a fun situation between people who are close. Also, if the overall image is bright, people tend to have open-mouthed smiles because it represents a relaxed situation. By setting the subject's shooting state based on the background, it is possible to set the shooting state to reflect these tendencies.

[0044] The processed image generation unit 33 may, for example, generate a processed image using a machine learning model that, when inputting a captured image, identification information of a subject, and the result of setting the subject's shooting state, outputs an image (processed image) in which the subject's shooting state has been changed to a shooting state corresponding to the setting result.

[0045] Figure 4 is a flowchart showing an example of processing using the image processing apparatus 100 according to this embodiment. As shown in Figure 4, the image acquisition unit 31 acquires the input captured image (step S101).

[0046] The person identification unit 32 detects whether or not a person is included in the acquired captured image (step S102). If the person identification unit 32 does not detect a person (No in step S102), it terminates the process. If the person identification unit 32 detects a person (Yes in step S102), it identifies each person based on the person identification data D1 (step S103). In step S103, if the person identification unit 32 detects multiple people, it identifies each of the multiple people.

[0047] If a person is identified, the processed image generation unit 33 calculates the shooting conditions for the identified person in the captured image (step S104) and sets the shooting state according to the calculated shooting conditions (step S105). The processed image generation unit 33 generates a processed image by processing the captured image based on the setting result (step S106).

[0048] Figure 5 is a flowchart showing another example of processing using the image processing apparatus 100 according to this embodiment. Figure 5 shows an example of the processing content in step S104 shown in Figure 4. As shown in Figure 5, the processed image generation unit 33 determines whether or not there are multiple people identified in step S103 in Figure 4 (step S201).

[0049] If there are multiple identified individuals (Yes in step S201), the processed image generation unit 33 determines whether or not it is possible to calculate the relationships between the identified individuals (step S202). In step S202, the processed image generation unit 33 can make a determination based on, for example, whether or not data for the multiple individuals exists in the relationship data D2a, and whether or not it is possible to calculate the relationships based on the behavioral history of the multiple individuals.

[0050] If the processed image generation unit 33 determines that it is possible to calculate the relationships between the identified multiple subject individuals (Yes in step S202), it calculates the relationships between these subject individuals as part of the shooting conditions (step S203). In step S203, the processed image generation unit 33 may also calculate the background of the subject individuals in the captured image as part of the shooting conditions, in addition to the relationships between the multiple subject individuals.

[0051] If the processed image generation unit 33 determines that there are not multiple target persons identified in step S201 (No. of step S201), or if it determines that it is not possible to calculate the relationships between multiple target persons identified in step S202 (No. of step S202), it calculates the background of the target persons in the captured image as the shooting situation (step S204).

[0052] The shooting conditions are calculated by performing the processing in step S203 or step S204. After the processing in step S203 or step S204, the processed image generation unit 33 performs the processing from step S105 onwards shown in Figure 4 based on the calculated shooting conditions.

[0053] As described above, the image processing apparatus 100 according to this embodiment includes an image acquisition unit 31 that acquires a captured image, a person identification unit 32 that detects a person if the acquired image contains a person, and a processed image generation unit 33 that, when a person is detected, sets the shooting state of the target person in the captured image according to the shooting conditions of the target person in the captured image, and generates a processed image by processing the captured image based on the setting result.

[0054] With this configuration, when a subject person is detected in the captured image, the shooting state of the subject person is set according to the shooting conditions of that subject person in the captured image, and a processed image is generated by processing the captured image based on the setting result, thereby generating an image in an appropriate state according to the shooting conditions.

[0055] In the image processing apparatus 100 according to this embodiment, the shooting conditions include at least one of the relationships between individuals when the captured image includes multiple individuals, and the background of the individuals in the captured image.

[0056] This configuration makes it possible to generate images in an appropriate state according to the relationship between multiple subjects and the shooting conditions of at least one of the backgrounds of the people in the captured images.

[0057] In the image processing device 100 according to this embodiment, the shooting state includes the facial expression of the subject person.

[0058] This configuration allows for the generation of images with appropriate facial expressions for the subject in the captured image, depending on the shooting conditions.

[0059] In the image processing apparatus 100 according to this embodiment, the shooting state includes at least one of the position of the subject person in the captured image and the posture of the person in the captured image.

[0060] This configuration allows for the generation of images of the subject in the captured image, with appropriate positioning and posture according to the shooting conditions.

[0061] In the image processing apparatus 100 according to this embodiment, the shooting conditions include the relationships between subjects when the captured image includes multiple subjects, and the processed image generation unit 33 determines the relationships between subjects based on the behavior history of the subjects and processes the shooting conditions of the subjects according to the determined relationships.

[0062] This configuration makes it possible to generate images of multiple subjects included in a captured image, corresponding to the relationships between those subjects based on their behavioral history.

[0063] The technical scope of the present invention is not limited to the embodiments described above, and modifications can be made as appropriate without departing from the spirit of the invention. For example, in the embodiments described above, a configuration that considers relationship data D2a and background data D2b as shooting situation data D2 was given as an example, but the invention is not limited to this case. For example, a configuration that does not consider either relationship data D2a or background data D2b as shooting situation data D2 is also possible. [Explanation of Symbols]

[0064] C1, C2... Camera, D1... Person identification data, D2... Shooting situation data, D2a... Relationship data, D2b... Background data, D3... Shooting state data, D3a... Facial expression data, D3b... Position data, D3c... Posture data, D4... Image data, P1, P2... Captured image, P3... Processed image, 10... Communication unit, 20... Input / Output unit, 33... Processed image generation unit, 30... Control unit, 31... Image acquisition unit, 32... Person identification unit, 40... Storage unit, 100... Image processing device

Claims

1. An image acquisition unit that acquires the captured image, A person identification unit that detects and identifies people included in the acquired captured image, When the person is identified, the processed image generation unit sets the shooting state of the person in the captured image according to the shooting conditions of the person in the captured image, and generates a processed image by processing the captured image based on the setting result. An image processing device equipped with the following features.

2. The aforementioned shooting conditions include at least one of the relationships between the individuals in the captured image if multiple individuals are included in the image, and the background of the individuals in the captured image. The image processing apparatus according to claim 1.

3. The aforementioned shooting conditions include the facial expression of the person. The image processing apparatus according to claim 1.

4. The aforementioned shooting state includes at least one of the position of the person in the captured image and the posture of the person in the captured image. The image processing apparatus according to claim 1.

5. The aforementioned shooting conditions include the relationships between individuals when the captured image includes multiple individuals. The processed image generation unit determines the relationship between the individuals based on their activity history and processes the image of the individuals according to the determined relationship. The image processing apparatus according to claim 1.

Images