Image Processing Apparatus and Program

Abstract

To automate a process of setting the position of an image with respect to an image frame of a template of a photobook or the like, or to save labor in the process.SOLUTION: An image processing apparatus 1 includes a storage device 16 that stores data about a template in which at least one image frame is laid out and data on an image including at least one person, and a processor 11 that executes a process of detecting a face region in the image and setting a face frame for surrounding the face region on the image, a process of initially setting the position of the image with respect to the image frame, and a process of correcting the position of the image with respect to the image frame if the face frame on the initially set image lies off the image frame.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and a program.

Background Art

[0002] In life, there are times when creating an album or photo book at life milestones such as death, marriage, retirement, graduation, etc. In recent years, with the spread of digital cameras, it has become possible to easily obtain a printed and bound photo book by pasting desired image data onto a photo book template on a computer and submitting it to a printing shop as digital data. Various editors for creating photo books are provided, and the labor-saving of the creation work has advanced.

[0003] Here, the aspect ratio of the image frame of the template does not necessarily match the aspect ratio of the image to be pasted there. Therefore, the editor needs to appropriately enlarge or reduce the image according to the image frame and further appropriately set the position of the image with respect to the image frame.

[0004] This image enlargement / reduction process and the process of setting the position of the image with respect to the image frame require individual judgment of the composition of the image and the objects shown therein, typically the arrangement and size of a portrait, etc., which is very time-consuming.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Therefore, it is desired to automate the process of setting the position of an image with respect to an image frame of a template such as a photo book or to save labor in that process.

Means for Solving the Problems

[0006] The image processing apparatus according to this embodiment includes a storage unit that stores data related to a template in which at least one image frame is laid out and data of an image in which at least one person is depicted, a face frame setting unit that detects a face region in the image and sets a face frame surrounding the face region on the image, a position setting processing unit that initially sets the position of the image with respect to the image frame, and a position correction processing unit that corrects the position of the image with respect to the image frame when the face frame on the initially set image protrudes from the image frame.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0008] Hereinafter, specific embodiments of the present invention will be described with reference to the accompanying drawings. In this embodiment, a photo book is typically taken as an example of an object to which an image is to be pasted and described below. However, the object is not limited to a photo book, and it may be an album, a photo card composed of a single sheet or a folded sheet such as a double-fold, a postcard such as a New Year's card, a calendar, a poster, or the like. Note that, in a narrow sense, a photo book is a book bound by binding sheets mainly printed with images by ink, while an album is defined as a book bound by binding photographic papers mainly printed with images by a silver salt laboratory machine. Photo books and albums are defined in a broad sense as a collection of a plurality of pages including images bound into a book. Here, the broad definition is adopted. A photo book will be applied hereinafter. An album is included in the photo book. For each page of the photo book, a template (framework) is set in which one or a plurality of image frames for pasting images are pre-laid out.

[0009] In the image processing apparatus 1 according to the present embodiment shown in FIG. 1, a user terminal 2 that is operated mainly by an orderer who orders a photo book, and a printing terminal 3 that receives manuscript data of the photo book from the image processing apparatus 1 and transfers it to a printing machine are connected via an Internet line (electronic public communication network) 4. Data of a plurality of images selected by the user is transmitted from the user terminal 2 to the image processing apparatus 1 together with a code for identifying the photo book selected by the user. In the image processing apparatus 1, a plurality of images are enlarged and reduced in accordance with the image frames on the templates of each page of the photo book, the positions of the enlarged and reduced images with respect to the image frames are set, the portions of the images that protrude from the image frames are trimmed, and manuscript data constructed in an image format such as pdf for each page with the trimmed images pasted into the image frames of the templates is generated and transmitted to the printing terminal 3.

[0010] As shown in FIG. 2, the image processing apparatus 1 includes a processor 11. Connected to the processor 11 via a control / data bus 18 are a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an input device 14 such as a keyboard, a pointing device such as a mouse or a touch panel, a display 15 such as an LCD (Liquid Crystal Display), a storage device 16, and a communication device 17 that realizes communication with the user terminal 2 and the printing terminal 3 via the Internet line 4.

[0011] The storage device 16 is implemented, for example, as a hard disk drive (HDD), a solid state drive (SSD), and a non-volatile storage device. In addition to a BIOS (Basic Input Output System) and an operating system program (OS), various applications such as an image processing program are stored in advance. Further, data regarding a plurality of types of photo book templates in which at least one image frame is laid out on each page and data of a plurality of images in which at least one person is shown, received from the user terminal 2 are stored.

[0012] The RAM 12 functions as the main memory, work area, etc. of the processor 11. The processor 11 executes various programs loaded from the storage device 16 to the RAM 12, such as the BIOS, OS, and image processing programs.

[0013] The image processing program In order to align one of the number of pixels in the vertical and horizontal directions of the image with one of the number of pixels in the vertical and horizontal directions of the image frame so that no margin occurs within the image frame, there is a scaling processing means (scaling processing section) 111 that enlarges and reduces the image while maintaining the aspect ratio, a face frame and person frame setting means (face frame and person frame setting section) 112 that detects the face area and body area of a person in the image and sets a face frame surrounding the face area and a person frame surrounding the face area and body area on the image, an image position setting means (image position setting section) 113 that initially sets the position of the image with respect to the other of the vertical and horizontal directions of the image frame, a proper position determination means (proper position determination section) 114 that determines whether the initially set position of the image is proper with respect to the image frame, for example, whether the face frame on the initially set image protrudes from the image frame, an image position correction means (image position correction section) 115 that corrects the position of the image with respect to the other of the vertical and horizontal directions of the image frame when it is determined that the initially set position of the image is improper with respect to the image frame, a trimming processing means (trimming processing section) 116 that trims the portion of the image that protrudes from the image frame in the image whose position has been initially set or corrected, an input data generation means (input data generation section) 117 that pastes the trimmed image onto the image frame of the template and generates input data constructed in a predetermined image format such as pdf. for each page, It is a program for realizing on a computer (processor 11).

[0014] The operation of the image processing apparatus 1 configured as described above will be described below. Figure 3 is a flowchart showing the process of creating photo book data executed by the image processing apparatus 1. The user selects a desired photo book template from a plurality of photo book templates and designates a plurality of images to be published in the photo book via the user terminal 2. In step S11, the processor 11 of the image processing apparatus 1 receives, via the communication device 17, an identification number for identifying the selected photo book template and data of a plurality of images to be published in the photo book from the user terminal 2. The received data of the plurality of images is written into the storage device 16 under the control of the processor 11. The photo book template data corresponding to the identification number of the selected photo book template is loaded from the storage device 16 into the RAM 12 under the control of the processor 11 (step S12).

[0015] In step S13, the processor 11 executes an image processing program and initially sets the position of the image individually designated for each image frame of the photo book template.

[0016] In step S14, the image frame and the image arranged for the image frame are displayed on the display 15.

[0017] In step S15, the editor visually checks the image frame and the image arranged for the image frame, determines the necessity of correcting the position of the image with respect to the image frame, and operates the input device 14 to input an instruction of necessary correction or unnecessary correction.

[0018] In step S17, when an instruction of unnecessary correction is input in step S15 (step S15; NO), the portion of the image protruding from the image frame is trimmed.

[0019] In step S18, the trimmed images are respectively fitted into the image frames of the photo book template and converted into an image format such as pdf. per page. Thereby, submission data (photo book data) is generated.

[0020] In step S19, the generated submission data is transmitted to the printing terminal 3 via the communication device 17. The submission data is printed on the designated paper by a printing machine and bound by a binding machine. Thereby, a photo book is completed and delivered to the designated destination by mail or the like.

[0021] Returning to the conversation, when an instruction for correction is input in step S15 (step S15; YES), in step S16, the editor operates the input device 14 to move an image on the screen with respect to the image frame, for example, by a drag-and-drop process. Returning to step S14, the image frame and the image after position correction are displayed on the display 15.

[0022] FIG. 4 shows the processing procedure of the image position setting step S13 in FIG. 3. Here, as illustrated in FIG. 5(a), one or more image frames 32 are laid out per sheet in the template 31. As the images to be handled in this process, various images such as an image in which one person is shown (FIG. 5(b)), an image in which two people are shown (FIG. 5(c)), and an image in which three or more people are shown (FIG. 5(d)) are targeted. Also, as the image frames, a square having an aspect ratio (horizontal pixel number: vertical pixel number) of 1:1 (FIG. 5(e)), a vertically long rectangle having an aspect ratio of M:N (M < N) (FIG. 5(f)), and a horizontally long rectangle having an aspect ratio of M:N (M > N) (FIG. 5(g)) are appropriately combined. The horizontal pixel number and the vertical pixel number of each image frame are also set in advance.

[0023] In step S21, as shown in FIG. 6, a rectangular face frame that surrounds the face region of a person in the image and a rectangular person frame that surrounds the entire region of the person's face and body are identified. This process typically utilizes machine learning and deep learning that enhance the learning ability by stacking multiple layers of neural networks that mimic the structure and mechanism of the brain's neural circuits, especially artificial intelligence (AI). As shown in FIG. 7(a), when a plurality of, here two, persons are depicted in the image and two face regions are detected, a single face frame is set to surround the two face regions. Also, as shown in FIG. 7(b), when two persons are depicted in the image and two person regions are detected, a single person frame is set to surround the two person regions.

[0024] Furthermore, when a large number of three or more persons are depicted in the image and three or more face regions and three or more person regions are detected, a process of extracting the target (person) as the main subject is executed. As shown in FIG. 8(a), all person frames 1-7 are identified from the image. Then, as shown in FIG. 8(b), from these person frames 1-7, the person frame with the smallest vertical length (number of vertical pixels) or the smallest area (number of pixels) of the person frame, for example, person frame 5, is selected, and here person frames 1 and 2 are extracted as the targets as person frames showing a vertical length or area exceeding a predetermined percentage, for example, 150% of the smallest person frame 5.

[0025] When the target is single, a face frame is set to surround the face region of that target, and a person frame is set to surround the face region and body region of that target. When there are multiple targets, a single face frame is set to surround the face regions of those multiple targets, and a single person frame is set to surround the face regions and body regions of those multiple targets. Also, when there is no person frame that satisfies the above conditions, it is considered that there is no target, and a single face frame is set to surround all face regions, and similarly, a single person frame is set to surround all person regions.

[0026] Note that there may be a case where an editor needs to assign an image to an image frame of a template. In this case, when all the images are received, before or in parallel with the assignment operation, a process of identifying face frames and person frames is executed, and the coordinate information of the face frames and person frames is associated with the images, so that when the editor assigns an image to the image frame of the template, there is no need to wait for the process of identifying the face frames and person frames or at least the waiting time can be shortened, and thus the work efficiency can be improved.

[0027] Next, in step S22, as shown in FIG. 9, a process of enlarging or reducing the image is executed so that the number of pixels in one of the vertical and horizontal directions of the image size (m×n) is aligned with the number of pixels in one of the vertical and horizontal directions of the image frame (M×N) while maintaining the aspect ratio so that no blank space occurs within the image frame. For example, as shown in FIG. 9(a), when the image is enlarged or reduced so that the number of pixels in the vertical direction of the vertically long image is aligned with the number of pixels in the vertical direction of the square image frame or the horizontally long image frame, blank space will occur in the horizontal direction within the image frame, so it is considered not suitable (NG). Therefore, in this example, as shown in FIG. 9(b), by enlarging or reducing the image so that the number of pixels in the horizontal direction of the image matches the number of pixels in the horizontal direction of the image frame, no blank space occurs within the image frame and the entire area of the image frame is filled with the image, so it is considered suitable (GOOD).

[0028] Also, as shown in FIG. 9(c), when the image is enlarged or reduced so that the number of pixels in the vertical direction of the horizontally long image is aligned with the number of pixels in the vertical direction of the square image frame or the horizontally long image frame, no blank space occurs within the image frame and the entire area of the image frame is filled with the image, so it is considered suitable (GOOD). On the other hand, as shown in FIG. 9(d), when the image is enlarged or reduced so that the number of pixels in the horizontal direction of the image matches the number of pixels in the horizontal direction of the image frame, blank space will occur in the vertical direction within the image frame, so it is considered not suitable (NG).

[0029] By enlarging or reducing the image so that no blank space occurs within the image frame, the image becomes larger than the image frame, and as it is, the image will protrude from the image frame. Therefore, it is necessary to appropriately set the position of the image with respect to the image frame and trim the image portion that protrudes at that position.

[0030] In step S23, as shown in FIGS. 10(a) to 10(c), the position of the image with respect to the image frame is initially set so that the center line (image center line) parallel to one of the vertical and horizontal directions passing through the center of the image frame, that is, the direction in which the number of pixels of the image is aligned with that of the image frame during the image enlargement / reduction process, and the center line (image center line) parallel to one of the vertical and horizontal directions passing through the center of the image coincide.

[0031] In step S24, it is determined whether the position of the image initially set with respect to the image frame is appropriate. Typically, as shown in FIGS. 10(a) and 10(b), when the face frame on the initially positioned image protrudes from the image frame, it is determined that the position of the image with respect to the image frame is inappropriate.

[0032] Here, as another example of inappropriateness, when a target (person) is extracted from the image, as shown in FIG. 10(c), even if the face frame of the target is within the image frame, a state where the face frame of a person who is not the target (non-target person) is cut off by the image frame, that is, the frame line of the image frame crosses or cuts through the face frame of the non-target person, is also determined to be inappropriate (not appropriate) for the position of the image with respect to the image frame.

[0033] When it is determined in step S24 that the position of the image with respect to the image frame is appropriate (YES), the position setting process ends. When it is determined in step S24 that the position of the image with respect to the image frame is inappropriate (not appropriate) (NO), and it is negated (NO) in the determination of "whether all position correction methods have been applied" in step S25, the "correction process for the position of the image with respect to the image frame" in step S26 is executed. When it is affirmed (YES) in the determination of "whether all position correction methods have been applied" in step S25, the position setting process of the image ends in an inappropriate state such as the face frame protruding from the image frame.

[0034] As multiple correction process methods, here six types of position correction processes (1) to (6) are prepared in advance as shown in FIGS. 11, 12, 13, and 14. Position correction processes (1) to (5) are applied when it is determined in step S24 that the face frame on the image with the set position is in an inappropriate position as protruding from the image frame. Position correction process (6) is applied when it is determined in step S24 that the face frame of the non-target person is inappropriate as being cut off by the image frame.

[0035] Priority orders are set in advance for position correction processes (1) to (5). The position correction process with the highest priority is applied. When it is determined in step S24 that the position of the image with respect to the image frame corrected by that position correction process is inappropriate, it is switched to and applied to the position correction process with the next highest priority.

[0036] Note that the selection and switching of position correction processes (1) to (5) may be performed according to the operation of the editor's input device 14.

[0037] In the position correction process (1), as shown in Fig. 11(a), the position of the image is corrected so that the center line of the face frame coincides with the center line of the image frame. The center line of the image frame passes through the center of the image frame and is a line parallel to one of the vertical and horizontal directions, that is, the direction in which the number of pixels of the image is aligned with that of the image frame during the image enlargement / reduction process. Similarly, the center line of the image passes through the center of the image and is a line parallel to one of the vertical and horizontal directions, that is, the direction in which the number of pixels of the image is aligned with that of the image frame during the image enlargement / reduction process.

[0038] In the position correction process (2), as shown in Fig. 11(b), the image is shifted in the direction opposite to the protruding direction by the total length of the length E by which the face frame on the image protrudes from the image frame and the preset adjustment length α. The adjustment length α is set, for example, to 10% of the number of pixels of the face frame.

[0039] In the position correction process (3), as shown in Fig. 12(a), the position of the image is corrected so that the ratio R (= Fh / G) of the gap G between the upper edge of the image frame and the upper edge of the face frame to the height of the face frame, that is, the number of pixels Fh in the vertical direction, becomes a predetermined value, for example, 20%.

[0040] In the position correction process (4), as shown in Fig. 12(b), the position of the image is corrected so that the center line of the image frame coincides with a position that is lowered by a predetermined ratio, for example, 30% of the height Ph of the person frame, from the upper edge of the person frame.

[0041] In the position correction process (5), as shown in Fig. 13, the position of the image is corrected so that the center line of the image frame coincides with the center line of the person frame.

[0042] By preparing various position correction processes in this way, it is possible to improve the possibility of arranging the image at an appropriate position with respect to the image frame for various images with different numbers of people, compositions, etc.

[0043] The position correction process (6) is applied when it is determined in step S24 that it is inappropriate because the face frame of the non-target person is cut off by the image frame. In the position correction process (6), as shown in FIG. 14, the image is shifted in the direction opposite to the protruding direction by the length F by which the face frame of the non-target person protrudes from the image frame.

[0044] After any one of the position correction processes (1) to (6) is applied, the process returns to step S24, and it is determined again whether the position of the image with respect to the image frame is appropriate.

[0045] As described above, according to the present embodiment, in order not to generate a blank space within the image frame, while maintaining the aspect ratio, the number of pixels in one of the vertical and horizontal directions of the image size (m×n) is adjusted to the number of pixels in one of the vertical and horizontal directions of the image frame (M×N). By enlarging or reducing the image so that they match, it is only necessary to adjust the position of the image with respect to the image frame in the other direction of the vertical and horizontal directions that becomes larger than the image frame (M×N). While filling the entire area of the image frame with the image, it is possible to automate the image position setting or at least increase the possibility of optimizing the image position. In addition, by using the face frame detected from the image, it is possible to optimize the position of the image with respect to the image frame so that the face frame fits within the image frame. Also, by sequentially applying a plurality of position correction processes with different position correction methods, it is possible to improve the possibility of arranging the image at an appropriate position with respect to the image frame. Since these position correction processes are constructed based on different correction principles according to differences in the number of people, composition, etc., the possibility of automation can be further increased. As described above, according to the present embodiment, it is possible to automate the process of enlarging / reducing the image with respect to the image frame of the template such as a photo book, and the process of setting the position of the image with respect to the image frame, or at least to streamline the process.

[0046] Note that any one of the above-described position corrections (1) to (5) may be applied to the initial position setting process in step S23.

[0047] Further, the distinction between the initial position setting and the position correction may be eliminated, and the initial position setting process and the position correction processes (1) to (5) of step S23 described above may be applied while sequentially switching between them.

[0048] Although several embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, as well as in the invention described in the claims and its equivalent scope.

Description of Reference Numerals

[0049] 1... Image processing apparatus, 2... User terminal, 3... Printing terminal, 11... Processor, 12... RAM, 13... ROM, 14... Input device, 15... Display, 16... Storage device, 17... Communication device, 111... Enlargement / reduction processing means, 112... Face frame / person frame specifying means, 113... Image position setting means, 114... Appropriate position determination means, 115... Image position correction means, 116... Trimming processing means, 117... Input data generation means.

Claims

1. a storage unit that stores data relating to a template in which at least one image frame is laid out and data of an image in which at least one person is photographed; a face frame setting unit that detects a face area in the image and sets a face frame surrounding the face area on the image; a position setting processing unit that initially sets a position of the image relative to the image frame; a position correction processing unit that corrects a position of the image with respect to the image frame when the face frame on the initially set image protrudes from the image frame, The position correction processing unit corrects the position of the initially positioned image relative to the image frame so that the ratio of the gap between the upper edge of the image frame and the upper edge of the face frame to the height of the face frame becomes a predetermined value.

2. a storage unit that stores data relating to a template in which at least one image frame is laid out and data of an image in which at least one person is photographed; a face frame setting unit that detects a face area in the image and sets a face frame surrounding the face area on the image; a position setting processing unit that initially sets a position of the image relative to the image frame; a position correction processing unit that corrects a position of the image with respect to the image frame when the face frame on the initially set image protrudes from the image frame; a person frame setting unit that detects a person area in the image and sets a person frame surrounding the person area on the image, The position correction processing unit corrects the initially set position of the image so that the center line of the image frame coincides with a position that is lower than the upper edge of the person frame by a predetermined percentage of the height of the person frame.

3. means for detecting a face area in an image to be placed in at least one image frame laid out in a template, and setting a face frame surrounding the face area on the image; means for initially setting a position of the image relative to the image frame; and a means for correcting a position of the image with respect to the image frame when the face frame on the initially set image protrudes from the image frame. A program in which the means for correcting the position of the image relative to the image frame corrects the position of the initially positioned image relative to the image frame so that the ratio of the gap between the upper edge of the image frame and the upper edge of the face frame to the height of the face frame becomes a predetermined value.

4. means for detecting a face area in an image to be placed in at least one image frame laid out in a template, and setting a face frame surrounding the face area on the image; means for initially setting a position of the image relative to the image frame; a means for correcting a position of the image relative to the image frame when the face frame on the initially set image protrudes from the image frame; means for detecting a person area in the image and setting a person frame surrounding the person area on the image; The means for correcting the position of the image relative to the image frame corrects the initially set position of the image so that the center line of the image frame coincides with a position that is lowered from the upper edge of the person frame by a predetermined percentage of the height of the person frame, the program.

Images