Image processing device and program

Abstract

To achieve automation or labor-saving of an image positioning process with respect to image frames in templates such as photo books.SOLUTION: An image processing apparatus 1 includes a storage device 16 configured to store data related to a template in which at least one image frame is laid out and data of an image in which at least one person appears and a processor 11 configured to perform a process for detecting a facial region in the image and setting a face frame enclosing the facial region on the image, a process for initially setting the position of the image with respect to the image frame, and a process for correcting the position of the image with respect to the image frame when the face frame on the initially positioned image protrudes from the image frame. In the process of correcting the position of the image with respect to the image frame, the image position initially set with respect to the image frame is corrected such that a ratio of a gap between either the left or right edge of the image frame and either the left or right edge of the face frame to the width of the face frame becomes a predetermined value.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and a program.

Background Art

[0002] In life, when there are milestones in life such as death, marriage, retirement, graduation, etc., one may create an album or a photo book. In recent years, with the popularization of digital cameras, by pasting desired image data on a computer onto a template of a photo book and submitting it to a printing shop as digital data, it has become possible to easily obtain a printed and bound photo book. Various editors for creating photo books are provided, and the labor-saving of the creation work has progressed.

[0003] Here, the aspect ratio of the image frame of the template and the aspect ratio of the image to be pasted there do not necessarily match. Therefore, the editor needs to appropriately enlarge or reduce the image according to the image frame and further appropriately set the position of the image with respect to the image frame.

[0004] This image enlargement / reduction process and the process of setting the position of the image with respect to the image frame require individual judgment of the composition of the image and the objects shown therein, typically the arrangement and size of a portrait, etc., which is very time-consuming.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Therefore, it is desired to automate the process of setting the position of an image with respect to an image frame of a template such as a photo book or to save labor in that process.

Means for Solving the Problems

[0006] An image processing apparatus according to an embodiment of the present invention includes a storage unit that stores data related to a template in which at least one image frame is laid out and data of an image in which at least one person is captured, a face frame setting unit that detects a face region in the image and sets a face frame surrounding the face region on the image, a position setting processing unit that initially sets the position of the image with respect to the image frame, and a position correction processing unit that corrects the position of the image with respect to the image frame when the face frame on the initially set image protrudes from the image frame. The position correction processing unit corrects the position of the image initially set with respect to the image frame so that the ratio of the gap between the left or right edge of the image frame and the left or right edge of the face frame to the width of the face frame becomes a predetermined value.

Brief Description of Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0008] Hereinafter, specific embodiments of the present invention will be described with reference to the accompanying drawings. In this embodiment, a photo book will be typically described as an example of an object to which an image is to be pasted. However, the object is not limited to a photo book and may be an album, a photo card composed of sheets such as a single sheet or a folded sheet, a postcard such as a New Year's card, a calendar, a poster, or the like. Here, in a narrow sense, a photo book is mainly a bound volume of sheets printed with images by ink, while an album is mainly defined as a bound volume of photographic papers printed with images by a silver salt laboratory machine. In a broad sense, a photo book and an album are defined as a bound volume of a plurality of pages including images. Here, the broad definition is adopted. A photo book will be applied hereinafter. An album is included in the photo book. For each page of the photo book, a template (framework) is set in which one or a plurality of image frames for pasting images are pre-laid out.

[0009] In the image processing apparatus 1 according to the present embodiment shown in FIG. 1, a user terminal 2 operated by an orderer who orders a photo book and a printing terminal 3 that receives the submission data of the photo book from the image processing apparatus 1 and transfers it to a printing machine are connected via an Internet line (electronic public communication network) 4. Data of a plurality of images selected by the user is transmitted from the user terminal 2 to the image processing apparatus 1 together with a code for identifying the photo book selected by the user. In the image processing apparatus 1, a plurality of images are enlarged and reduced according to the image frames on the template of each page of the photo book, the position of the enlarged and reduced images with respect to the image frames is set, the portions of the images that protrude from the image frames are trimmed, and the trimmed images are pasted on the image frames of the template. Submission data constructed in an image format such as pdf for each page is generated and transmitted to the printing terminal 3.

[0010] As shown in FIG. 2, the image processing apparatus 1 has a processor 11. An input device 14 such as a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, a keyboard, a pointing device such as a mouse or a touch panel, a display 15 such as an LCD (Liquid Crystal Display), a storage device 16, and a communication device 17 that realizes communication with the user terminal 2 and the printing terminal 3 via the Internet line 4 are connected to the processor 11 via a control / data bus 18.

[0011] The storage device 16 is implemented as, for example, a hard disk drive (HDD) or a solid state drive (SSD) as a non-volatile storage device. In addition to the BIOS (Basic Input Output System) and the operating system program (OS), various applications such as an image processing program are pre-stored. Further, data related to a plurality of types of photo book templates in which at least one image frame is laid out on each page and data of a plurality of images in which at least one person is shown received from the user terminal 2 are stored.

[0012] The RAM 12 functions as the main memory, work area, etc. of the processor 11. The processor 11 executes various programs such as the BIOS, OS, and image processing program loaded from the storage device 16 to the RAM 12.

[0013] The image processing program In order to align one of the number of pixels in the vertical and horizontal directions of the image with one of the number of pixels in the vertical and horizontal directions of the image frame so that no margin occurs within the image frame, there is a scaling processing means (scaling processing unit) 111 that enlarges and reduces the image while maintaining the aspect ratio, a face frame / figure frame setting means (face frame / figure frame setting unit) 112 that detects the face area and body area of a person in the image and sets a face frame surrounding the face area and a figure frame surrounding the face area and body area on the image, an image position setting means (image position setting unit) 113 that initially sets the position of the image with respect to the other of the vertical and horizontal directions of the image frame, a proper position determination means (proper position determination unit) 114 that determines whether the initially set position of the image is proper with respect to the image frame, for example, whether the face frame on the initially set image protrudes from the image frame, an image position correction means (image position correction unit) 115 that corrects the position of the image with respect to the other of the vertical and horizontal directions of the image frame when it is determined that the initially set position of the image is improper with respect to the image frame, a trimming processing means (trimming processing unit) 116 that trims the portion of the image that protrudes from the image frame in the image whose position has been initially set or corrected, an input data generation means (input data generation unit) 117 that pastes the trimmed image onto the image frame of the template and generates input data constructed in a predetermined image format such as pdf. for each page, It is a program for realizing on a computer (processor 11).

[0014] The operation of the image processing apparatus 1 configured as described above will be described below. Figure 3 is a flowchart showing the process of creating photobook data executed by the image processing apparatus 1. The user selects a desired photobook template from a plurality of photobook templates and designates a plurality of images to be published in the photobook via the user terminal 2. In step S11, the processor 11 of the image processing apparatus 1 receives, via the communication device 17, an identification number for identifying the selected photobook template and data of a plurality of images to be published in the photobook from the user terminal 2. The received data of the plurality of images is written to the storage device 16 under the control of the processor 11. The photobook template data corresponding to the identification number of the selected photobook template is loaded from the storage device 16 to the RAM 12 under the control of the processor 11 (step S12).

[0015] In step S13, the processor 11 executes an image processing program and initially sets the position of the image individually designated for each image frame of the photobook template.

[0016] In step S14, the image frame and the image arranged for the image frame are displayed on the display 15.

[0017] In step S15, the editor visually recognizes the image frame and the image arranged for the image frame, determines the necessity of correcting the position of the image with respect to the image frame, and operates the input device 14 to input an instruction of necessary correction or unnecessary correction.

[0018] In step S17, when an instruction of unnecessary correction is input in step S15 (step S15; NO), the portion of the image protruding from the image frame is trimmed.

[0019] In step S18, the trimmed images are respectively fitted into the image frames of the photobook template and converted into an image format such as pdf. per page. Thereby, submission data (photobook data) is generated.

[0020] In step S19, the generated submission data is transmitted to the printing terminal 3 via the communication device 17. The submission data is printed on designated paper by a printing machine and bound by a binding machine. Thereby, a photo book is completed and delivered to the designated destination by mail or the like.

[0021] Returning to the conversation, when an instruction for correction is input in step S15 (step S15; YES), in step S16, the editor operates the input device 14 to move an image on the screen with respect to the image frame, for example, by a drag-and-drop process. Returning to step S14, the image frame and the image after position correction are displayed on the display 15.

[0022] FIG. 4 shows the processing procedure of the image position setting step S13 in FIG. 3. Here, as illustrated in FIG. 5(a), one or more image frames 32 are laid out per sheet in the template 31. As images to be handled in this process, various images such as an image in which one person is shown (FIG. 5(b)), an image in which two people are shown (FIG. 5(c)), and an image in which three or more people are shown (FIG. 5(d)) are targeted. Also, as image frames, a square with an aspect ratio (horizontal pixel number: vertical pixel number) of 1:1 (FIG. 5(e)), a vertically long rectangle with an aspect ratio of M:N (M < N) (FIG. 5(f)), and a horizontally long rectangle with an aspect ratio of M:N (M > N) (FIG. 5(g)) are appropriately combined. The horizontal pixel number and vertical pixel number of each image frame are also preset.

[0023] In step S21, as shown in FIG. 6, a rectangular face frame that surrounds the face region of a person in the image and a rectangular person frame that surrounds the entire region of the person's face and body are specified. This process typically utilizes machine learning and deep learning that enhance the learning ability by stacking multiple layers of neural networks that mimic the structure and mechanism of the brain's neural circuits, particularly artificial intelligence (AI). As shown in FIG. 7(a), when a plurality of, here two, persons are shown in the image and two face regions are detected, a single face frame is set to surround the two face regions. Also, as shown in FIG. 7(b), when two persons are shown in the image and two person regions are detected, a single person frame is set to surround the two person regions.

[0024] Furthermore, when a large number of three or more persons are shown in the image and three or more face regions and three or more person regions are detected, a process of extracting the target (person) as the main subject is executed. As shown in FIG. 8(a), all person frames 1-7 are specified from the image. Then, as shown in FIG. 8(b), from these person frames 1-7, the person frame with the smallest vertical length (number of vertical pixels) or the smallest area (number of pixels) of the person frame, for example, person frame 5, is selected, and here person frames 1 and 2 are extracted as targets as person frames showing a vertical length or area exceeding a predetermined percentage, for example, 150% of the smallest person frame 5.

[0025] In addition, when the target is single, a face frame is set to surround the face region of that target, and a person frame is set to surround the face region and body region of that target. When there are multiple targets, a single face frame is set to surround the face regions of those multiple targets, and a single person frame is set to surround the face regions and body regions of those multiple targets. Also, when there is no person frame that satisfies the above conditions, it is assumed that there is no target, and a single face frame is set to surround all face regions, and similarly, a single person frame is set to surround all person regions.

[0026] Note that there may be a case where an editor needs to assign an image to an image frame of a template. In this case, before or in parallel with the assignment operation when all the images have been received, a process of specifying face frames and person frames is executed, and the coordinate information of the face frames and person frames is associated with the images, so that when the editor assigns an image to the image frame of the template, it is not necessary to wait for the process of specifying the face frames and person frames or at least the waiting time can be shortened, and thus the work efficiency can be improved.

[0027] Next, in step S22, as shown in FIG. 9, a process of enlarging or reducing the image is executed so that the number of pixels in one of the vertical and horizontal directions of the image size (m×n) is aligned with the number of pixels in one of the vertical and horizontal directions of the image frame (M×N) while maintaining the aspect ratio so that no blank space occurs within the image frame. For example, as shown in FIG. 9(a), when the image is enlarged or reduced so that the number of pixels in the vertical direction of the vertically long image is aligned with the number of pixels in the vertical direction of the square image frame or the horizontally long image frame, blank space will occur in the horizontal direction within the image frame, so it is considered not suitable (NG). Therefore, in this example, as shown in FIG. 9(b), by enlarging or reducing the image so that the number of pixels in the horizontal direction of the image is aligned with the number of pixels in the horizontal direction of the image frame, no blank space occurs within the image frame and the entire area of the image frame is filled with the image, so it is considered suitable (GOOD).

[0028] Also, as shown in FIG. 9(c), when the image is enlarged or reduced so that the number of pixels in the vertical direction of the horizontally long image is aligned with the number of pixels in the vertical direction of the square image frame or the horizontally long image frame, no blank space occurs within the image frame and the entire area of the image frame is filled with the image, so it is considered suitable (GOOD). On the other hand, as shown in FIG. 9(d), when the image is enlarged or reduced so that the number of pixels in the horizontal direction of the image is aligned with the number of pixels in the horizontal direction of the image frame, blank space will occur in the vertical direction within the image frame, so it is considered not suitable (NG).

[0029] By enlarging or reducing the image so that no blank space occurs within the image frame, the image becomes larger than the image frame, and as it is, the image will protrude from the image frame. Therefore, it is necessary to appropriately set the position of the image with respect to the image frame and trim the image portion that protrudes at that position.

[0030] In step S23, as shown in FIGS. 10(a) to 10(c), the position of the image with respect to the image frame is initially set so that the center line (image center line) parallel to one of the vertical and horizontal directions passing through the center of the image frame, that is, the direction in which the number of pixels of the image is aligned with that of the image frame during the image enlargement / reduction process, and the center line (image center line) parallel to one of the vertical and horizontal directions passing through the center of the image coincide.

[0031] In step S24, it is determined whether the position of the image initially set with respect to the image frame is appropriate. Typically, as shown in FIGS. 10(a) and 10(b), when the face frame on the initially positioned image protrudes from the image frame, it is determined that the position of the image with respect to the image frame is inappropriate.

[0032] Here, as another example of inappropriateness, when a target (person) is extracted from the image, as shown in FIG. 10(c), even if the face frame of the target is within the image frame, a state where the face frame of a person who is not the target (non-target person) is cut off by the image frame, that is, the frame line of the image frame crosses or longitudinally cuts the face frame of the non-target person, is also determined to be inappropriate (not appropriate) for the position of the image with respect to the image frame.

[0033] When it is determined in step S24 that the position of the image with respect to the image frame is appropriate (YES), the position setting process ends. When it is determined in step S24 that the position of the image with respect to the image frame is inappropriate (not appropriate) (NO), and it is negated (NO) in the determination of "whether all position correction methods have been applied" in step S25, the "correction process for the position of the image with respect to the image frame" in step S26 is executed. When it is affirmed (YES) in the determination of "whether all position correction methods have been applied" in step S25, the position setting process of the image ends in an inappropriate state such as the face frame protruding from the image frame.

[0034] As a plurality of correction process methods, here, six types of position correction processes (1) to (6) are prepared in advance as shown in FIGS. 11, 12, 13, and 14. Position correction processes (1) to (5) are applied when it is determined in step S24 that the face frame on the image whose position is set is in an inappropriate position as protruding from the image frame. Position correction process (6) is applied when it is determined in step S24 that the face frame of a non-target person is inappropriate as being cut off by the image frame.

[0035] Priority orders are set in advance for position correction processes (1) to (5). The position correction process with the highest priority is applied. When it is determined in step S24 that the position of the image with respect to the image frame corrected by that position correction process is inappropriate, it is switched to and applied to the position correction process with the next highest priority.

[0036] Note that the selection and switching of position correction processes (1) to (5) may be performed according to the operation of the editor's input device 14.

[0037] In the position correction process (1), as shown in Fig. 11(a), the position of the image is corrected so that the center line of the face frame coincides with the center line of the image frame. The center line of the image frame passes through the center of the image frame and is parallel to one of the vertical and horizontal directions, that is, the direction in which the number of pixels of the image is aligned with that of the image frame during the image enlargement / reduction process. Similarly, the center line of the image passes through the center of the image and is parallel to one of the vertical and horizontal directions, that is, the direction in which the number of pixels of the image is aligned with that of the image frame during the image enlargement / reduction process.

[0038] In the position correction process (2), as shown in Fig. 11(b), the image is shifted in the direction opposite to the protruding direction by the total length of the length E by which the face frame on the image protrudes from the image frame and the preset adjustment length α. The adjustment length α is set to, for example, 10% of the number of pixels of the face frame.

[0039] In the position correction process (3), as shown in Fig. 12(a), the position of the image is corrected so that the ratio R (= Fh / G) of the gap G between the upper edge of the image frame and the upper edge of the face frame to the height of the face frame, that is, the number of pixels Fh in the vertical direction, becomes a predetermined value, for example, 20%.

[0040] In the position correction process (4), as shown in Fig. 12(b), the position of the image is corrected so that the center line of the image frame coincides with a position that is lowered by a predetermined ratio, for example, 30% of the height Ph of the person frame, from the upper edge of the person frame.

[0041] In the position correction process (5), as shown in Fig. 13, the position of the image is corrected so that the center line of the image frame coincides with the center line of the person frame.

[0042] By preparing various position correction processes in this way, it is possible to improve the possibility of arranging the image at an appropriate position with respect to the image frame for various images with different numbers of people, compositions, etc.

[0043] The position correction process (6) is applied when it is determined in step S24 that it is inappropriate because the face frame of the non-target person is cut off by the image frame. In the position correction process (6), as shown in FIG. 14, the image is shifted in the direction opposite to the protruding direction by the length F by which the face frame of the non-target person protrudes from the image frame.

[0044] After any one of the position correction processes (1) to (6) is applied, the process returns to step S24, and it is determined again whether the position of the image with respect to the image frame is appropriate.

[0045] As described above, according to the present embodiment, in order not to generate a blank space within the image frame, while maintaining the aspect ratio, the number of pixels in one of the vertical and horizontal directions of the image size (m×n) is adjusted to the number of pixels in one of the vertical and horizontal directions of the image frame (M×N), and the image is enlarged or reduced. Thus, it is only necessary to adjust the position of the image with respect to the image frame in the other direction of the vertical and horizontal directions that becomes larger than the image frame (M×N). While filling the entire area of the image frame with the image, it is possible to automate the image position setting or at least increase the possibility of optimizing the image position. Further, by using the face frame detected from the image, it is possible to optimize the position of the image with respect to the image frame so that the face frame fits within the image frame. Also, by sequentially applying a plurality of position correction processes with different position correction methods, it is possible to improve the possibility of arranging the image at an appropriate position with respect to the image frame. Since these position correction processes are constructed based on different correction principles according to differences in the number of people, composition, etc., the possibility of automation can be further increased. As described above, according to the present embodiment, it becomes possible to automate the process of enlarging / reducing the image with respect to the image frame of the template such as a photo book, and the process of setting the position of the image with respect to the image frame, or at least to streamline the process.

[0046] Note that any one of the above-described position corrections (1) to (5) may be applied to the initial position setting process in step S23.

[0047] Further, it is also possible to eliminate the distinction between the initial position setting and the position correction, and to apply while sequentially switching between the initial position setting process and the position correction processes (1) to (5) of the above-described step S23.

[0048] Although some embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, as well as in the invention described in the claims and its equivalent scope.

Explanation of Signs

[0049] 1…Image processing apparatus, 2…User terminal, 3…Printing terminal, 11…Processor, 12…RAM, 13…ROM, 14…Input device, 15…Display, 16…Storage device, 17…Communication device, 111…Enlarging / reducing processing means, 112…Face frame / person frame specifying means, 113…Image position setting means, 114…Appropriate position determination means, 115…Image position correction means, 116…Trimming processing means, 117…Draft data generation means.

Claims

1. A storage unit that stores data related to a template in which at least one image frame is laid out and data of an image in which at least one person is depicted; A face frame setting unit that detects a face region in the image and sets a face frame surrounding the face region on the image; A position setting processing unit that initially sets the position of the image with respect to the image frame; A position correction processing unit that corrects the position of the image with respect to the image frame when the face frame on the initially set image protrudes from the image frame, and comprising: The position correction processing unit corrects the position of the image initially set with respect to the image frame so that the ratio of the gap between the left or right edge of the image frame and the left or right edge of the face frame to the width of the face frame becomes a predetermined value. An image processing apparatus.

2. A storage unit that stores data related to a template in which at least one image frame is laid out and data of an image in which at least one person is depicted; A face frame setting unit that detects a face region in the image and sets a face frame surrounding the face region on the image; A position setting processing unit that initially sets the position of the image with respect to the image frame; A position correction processing unit that corrects the position of the image with respect to the image frame when the face frame on the initially set image protrudes from the image frame; A person frame setting unit that detects a person region in the image and sets a person frame surrounding the person region on the image, and comprising: The position correction processing unit corrects the position of the initially set image so that the center line of the image frame coincides with a position shifted to the right or left from the left or right edge of the person frame by a predetermined ratio of the width of the person frame. An image processing apparatus.

3. Means for detecting a face region in an image arranged in at least one image frame laid out in a template and setting a face frame surrounding the face region on the image; Means for initially setting the position of the image with respect to the image frame; Means for causing a computer to correct the position of the image with respect to the image frame when the face frame on the initially set image protrudes from the image frame, The means for correcting the position of the image with respect to the image frame corrects the position of the image initially set with respect to the image frame so that the ratio of the gap between the left or right edge of the image frame and the left or right edge of the face frame to the width of the face frame becomes a predetermined value. A program.

4. Means for detecting a face region in an image disposed in at least one image frame laid out in a template, and setting a face frame surrounding the face region on the image; Means for initially setting the position of the image with respect to the image frame; Means for correcting the position of the image with respect to the image frame when the face frame on the initially set image protrudes from the image frame; Means for detecting a person region in the image, and setting a person frame surrounding the person region on the image, realized by a computer; A program, wherein the means for correcting the position of the image with respect to the image frame corrects the position of the initially set image so that the center line of the image frame coincides with a position shifted to the right or left from the left or right edge of the person frame by a predetermined ratio of the width of the person frame.

Images