Image processing device, image processing method, program, and recording medium

Abstract

To provide a device, a method, a program, and a recording medium that output an image in which a text related to an input image is properly arranged by taking into consideration a subject in the input image.SOLUTION: An image processing device includes a processor. The processor executes: processing for acquiring a text related to an input image; processing for specifying a first subject to be associated with the text, from subjects included in the input image; and processing for determining the arrangement of a text in an output image based on the input image, according to the input image and the first subject.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] One embodiment of the present invention relates to an image processing device, an image processing method, a program, and a recording medium capable of outputting an output image in which text is added to an input image. [Background technology]

[0002] As a technology for utilizing images, a technology for adding text corresponding to the subject in the image is already known, and one example is the technology described in Patent Document 1. In the technology described in Patent Document 1, a captured image is analyzed to generate text information, the captured image is processed based on the text information, and the text information is added to the processed image. This makes it possible to simultaneously display a comment and an image based on the captured image, improving the sense of matching between the two. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-229102 A Summary of the Invention [Problem to be solved by the invention]

[0004] When adding text to an image using the technology described in Patent Document 1, the text needs to be placed at an appropriate position in the image. In particular, when adding text according to an object in an image, if the text placement is determined without considering the relationship between the object and the text, there is a risk that the effect of adding the text will not be properly obtained. On the other hand, if a user tries to determine the text placement while taking the object into consideration, the task is time-consuming.

[0005] One embodiment of the present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an apparatus, method, program, and recording medium that are capable of outputting an image in which text related to an input image is placed in an appropriate position taking into account the subject in the input image. [Means for solving the problem]

[0006] The above object can be achieved by an image processing device according to any one of the following items [1] to

[16] . [1] An image processing device having a processor that processes an input image, the processor performing the following processes: obtaining text related to the input image; identifying a first object to be associated with the text from objects contained in the input image; and determining the placement of the text in an output image based on the input image in accordance with the input image and the first object. [2] The image processing device described in [1], wherein the processor acquires text based on a user's input operation. [3] An image processing device according to [1] or [2], wherein the processor generates text based on an area selected by a user in the input image. [4] An image processing device according to any one of [1] to [3], wherein the processor generates text based on the analysis results of the input image. [5] An image processing device according to any one of [1] to [4], wherein a processor generates text based on supplementary information of an input image. [6] The image processing device according to any one of [1] to [5], wherein the processor determines an arrangement in which the text does not overlap the first object. [7] The image processing device according to any one of [1] to [6], wherein the processor determines an arrangement in which the text does not overlap a specific part of the first subject. [8] An image processing device described in any of [1] to [7], wherein when outputting an output image including an input image and text, a processor determines an arrangement in which the text around a first object does not overlap with the first object or a second object different from the first object in the input image. [9] An image processing device according to any one of [1] to [8], wherein when multiple pieces of text are acquired as the text, the processor determines an arrangement of the multiple pieces of text such that the text does not overlap with each other around the first subject.

[10] An image processing device according to any of [1] to [9], wherein when outputting an output image including an input image having an image area and a margin area and text, a processor determines a placement of at least one of the image area and margin area of ​​the input image included in the output image.

[11] An image processing device according to any of [1] to

[10] , wherein when outputting an output image including an input image and text, the processor generates a margin area around at least a portion of the input image in the output image, and determines the arrangement of the input image and text in the output image.

[12] An image processing device described in any of [1] to

[11] , wherein a processor obtains an evaluation value regarding the arrangement of multiple areas in the input image based on the input image, and determines the arrangement based on the evaluation values ​​of the multiple areas.

[13] An image processing device according to any one of [1] to

[12] , wherein the processor acquires multiple pieces of text as text and determines an arrangement for the multiple pieces of text.

[14] The image processing device according to any one of [1] to

[13] , wherein the processor divides the text into multiple pieces of text and determines an arrangement for the multiple pieces of text.

[15] An image processing device described in any of [1] to

[14] , wherein a processor acquires multiple input images as input images, and generates text related to at least one of the multiple input images based on analysis results of the multiple input images.

[16] An image processing device described in any of [1] to

[15] , in which, when outputting an output image including an input image having an image area and a margin area, the processor determines based on the text whether the layout can be determined within the margin area in the input image, and if it is determined that the layout cannot be determined within the margin area in the input image, changes the layout of the image area in the input image, and determines the layout in the margin area of ​​an output image including the input image whose layout has been changed.

[0007] The above object can also be achieved by an image processing method described in any one of

[17] to

[19] below.

[17] An image processing method for processing an input image by a processor, the image processing method including the steps of: obtaining text related to the input image by the processor; identifying a first object to be associated with the text from objects contained in the input image by the processor; and determining, by the processor, a placement of the text in an output image based on the input image in accordance with the input image and the first object.

[18] The image processing method described in

[17] , wherein in the text obtaining step, a processor generates text based on the analysis results of the input image.

[19] The image processing method described in

[18] further includes a step in which the processor assigns evaluation values ​​regarding the arrangement to multiple regions in the input image, and in the step of determining the arrangement, the processor determines the arrangement based on the evaluation values ​​of the multiple regions.

[0008] Moreover, a program according to an embodiment of the present invention is a program for causing a computer to execute each step included in the image processing method according to any one of the above items

[17] to

[19] . Furthermore, a recording medium according to one embodiment of the present invention is a computer-readable recording medium having recorded thereon a program for causing a computer to execute each step included in the image processing method described in any one of

[17] to

[19] above. Effect of the Invention

[0009] According to one embodiment of the present invention, there is provided an apparatus, method, program, and recording medium capable of outputting an image in which text related to an input image is added at an appropriate position taking into account a subject in the input image. [Brief description of the drawings]

[0010] [Figure 1]FIG. 1 is a diagram illustrating an example of use of an image processing device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a procedure from acquiring an input image to outputting an output image. [Diagram 3] 1 is a diagram illustrating a hardware configuration of an image processing device according to a first embodiment of the present invention. [Figure 4] FIG. 2 is an explanatory diagram of functions of the image processing device according to the first embodiment of the present invention. [Figure 5A] FIG. 2 is a diagram showing a first example of an output image according to the first embodiment of the present invention. [Figure 5B] FIG. 4 is a diagram showing a second example of an output image according to the first embodiment of the present invention. [Figure 5C] FIG. 11 is a diagram showing a third example of an output image according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a diagram showing an example of an image processing flow according to the first embodiment of the present invention. [Figure 7] FIG. 4 is an explanatory diagram of a modified example of the first embodiment of the present invention. [Figure 8] FIG. 11 is an explanatory diagram of another modified example of the first embodiment of the present invention. [Figure 9] FIG. 11 is a diagram showing an example of use of an image processing device according to a second embodiment of the present invention. [Figure 10] FIG. 11 is a diagram showing a usage example of an image processing device according to a third embodiment of the present invention. [Figure 11] 13A and 13B are diagrams showing an input image and an output image in a third embodiment of the present invention. [Figure 12] FIG. 11 is an explanatory diagram of functions of an image processing device according to a third embodiment of the present invention. [Figure 13] FIG. 11 is an explanatory diagram of a change in image layout. [Figure 14] FIG. 13 is a diagram showing a modified example of an output image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Specific embodiments of the present invention will now be described. For convenience of explanation, the following description may be given from the viewpoint of a GUI (Graphic User Interface).Furthermore, since the basic data processing technologies (communication / transmission technology, data acquisition technology, data recording technology, data processing / analysis technology, machine learning technology, image processing technology, visualization technology, etc.) for realizing the present invention are well-known technologies, the description thereof will be omitted.

[0012] In addition, in this specification, the concept of "device" includes a single device that performs a specific function, and also includes a combination of multiple devices that exist in a distributed and independent manner but cooperate (link) to perform a specific function.

[0013] In addition, in this specification, a "user" refers to a user of the image processing device of the present invention, and specifically, for example, a person who uses an output image (to be described later) obtained by the functions of the image processing device of the present invention.

[0014] In addition, in this specification, "person" means a subject that performs a specific action, and includes individuals, groups, corporations such as companies, and organizations, and may also include computers and devices that constitute artificial intelligence (AI). Artificial intelligence (AI) is a technology that realizes intelligent functions such as inference, prediction, and judgment using hardware and software resources. The algorithm of artificial intelligence is arbitrary, and may be, for example, an expert system, case-based reasoning (CBR), Bayesian network, or subsumption architecture.

[0015] <<First embodiment of the present invention>> In a first embodiment of the present invention (hereinafter, referred to as the first embodiment), an image processing device of the present invention adds text to an image input to the image processing device (hereinafter, referred to as an input image Pi) as shown in Fig. 1, and outputs an image including the input image and the text (hereinafter, referred to as an output image Po). A user can use the output image Po. Here, adding (placing) text in an image means converting the text into an image (text image) and including it in the output image as a part of the output image.

[0016] In the present invention, an "image" is composed of a plurality of pixels, is represented by the gradation values ​​of each of the plurality of pixels, and includes at least one or more subjects. Digital image data (hereinafter, image data) that defines an image at a set resolution is generated by compressing data in which the gradation values ​​of each pixel are recorded using a predetermined compression method. Types of image data include, for example, lossy compressed image data such as JPEG (Joint Photographic Experts Group) format, and lossless compressed image data such as GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format.

[0017] In the first embodiment, the method of inputting an image is not particularly limited, but includes, for example, inputting image data of an image captured by a photographing device such as a camera, and inputting read data obtained by scanning an existing photo with a scanner or the like. In addition, if the image processing device is equipped with a photographing device, an image may be input by capturing an image with the photographing device and acquiring image data. In addition, an image may be input by downloading image data from an external device or a web server via a communication network.

[0018] Furthermore, the input image Pi may be a developed image obtained by developing a RAW image, a corrected image obtained by performing a predetermined correction process on a developed image, or an edited image obtained by performing editing and processing on a developed image or a corrected image. The edited image as the input image may be an image constructed by arranging one or more images in a predetermined layout (see Figs. 10 and 11). That is, the input image Pi may be an image having one or more image areas and other blank areas. An image area is, for example, an area that includes a subject, and a blank area is an area that does not include a subject. Note that an area that includes a design, pattern, illustration, and other decorative objects may be considered as a blank area if, for example, text can be written in that area.

[0019] In the first embodiment, the method of outputting an image is not particularly limited, but includes, for example, displaying an image on a display or monitor, printing an image, transmitting an image to another user, and providing an image as a commercial material. Commercial images include a collage image generated by compositing and decorating other images, an edited image in which one or more images (image regions) are arranged in a predetermined layout, and a booklet consisting of multiple pages on which images are published, such as an album or photo book. In addition, the mode of providing an image as a commercial material may include a mode of providing an image by printing it on a medium such as paper, as well as a mode of providing image data (digital data) in the form of an electronic commercial material without printing on a medium. In addition, the mode of providing a printed image may include sending a medium such as paper on which an image is printed to a recipient as a message card or postcard. In addition, the method of outputting an image may include posting an image to a social networking service (SNS) for public viewing.

[0020] In the first embodiment, a case will be described as an example in which an output image including one input image Pi and text Tx, but not including a blank area (to be described later), is output. In the first embodiment, an image in which text Tx is arranged within an input image Pi is output as an output image Po, as shown in FIG.

[0021] The text Tx is text related to the input image Pi, specifically, character information indicating contents according to the subject in the input image, such as comments and messages to the subject, explanations about the subject, and lines spoken by the subject. The text is composed of phrases or sentences consisting of one or more words, and when a sentence is divided, each of the divided phrases may correspond to the text. Furthermore, the text may include onomatopoeia, mimetic words, and exclamations.

[0022] In the first embodiment, the text Tx may be input by a user, or may be automatically generated by an image processing device. When automatically generating text, the image processing device analyzes an input image Pi and generates text based on the analysis result. Analyzing an image means, for example, identifying the characteristics of the image. The image characteristics are information on the image quality of each region of the image, the gradation values ​​of the pixels contained in each region, and information on the subject estimated from these pieces of information. The information on the subject may include the type of subject, the state of the subject, the position of the subject in the image, and, if the subject is a person, the facial expression, etc. Moreover, it is preferable that the image features can be quantified, vectorized, or tensorized. In this case, the image analysis result is the quantified, vectorized, or tensorized image features, that is, feature quantities.

[0023] Here, the subject refers to people, animals, objects, backgrounds, etc., contained in an image. The concept of the subject may also include the location, scene (e.g., dawn or dusk, clear skies, etc.) and theme (e.g., travel, meals, or events such as sports days) that appear in the image. Note that the image may be a landscape image, that is, the subject contained in the image may be only a landscape, and the entire image may represent the landscape as the subject.

[0024] There is no particular limitation on the technology for automatically generating text from the analysis results of an image. As an example, as in the technology described in Patent Document 1, the correspondence between the image analysis results and the contents of the text may be stored as data, and the text corresponding to the image analysis results may be selected (generated) by referring to the correspondence. Alternatively, machine learning may be performed to build a learning model for text generation, and an image may be input into the learning model to generate text related to the image. The above learning model, for example, identifies the characteristics of an input image and generates text according to the characteristics, and is built by performing machine learning using previously acquired images and text assigned to the images as learning data.

[0025] Moreover, in the first embodiment, the image processing device places (adds) the text Tx generated in the above manner in the output image Po at a position determined according to the subject included in the input image Pi. To explain in detail, the image processing device analyzes the input image Pi to generate the text Tx as shown in FIG. 2, and then identifies a subject (hereinafter referred to as a first subject) to be associated with the generated text Tx from among the subjects included in the input image Pi. The first subject is a subject that is highly related to the text Tx among one or more subjects included in the input image Pi. Specifically, for example, the first subject corresponds to a subject that utters a sound with the contents indicated by the text, a subject that is in a state indicated by the text, a subject that performs an action indicated by the text, or a subject that participates in an incident or event indicated by the text.

[0026] Then, as shown in FIG. 2, the image processing device determines the arrangement (more precisely, the arrangement position) of the text Tx in the output image Po based on the input image Pi and the identified first subject, and outputs the output image Po in which the text Tx is arranged at that position.

[0027] As described above, the image processing device according to the first embodiment specifies the first object to be associated with the generated text Tx in the input image Pi, and outputs the output image Po in which the text Tx is arranged at a position according to the specified first object. This makes it possible to arrange the text Tx at an appropriate position in the output image Po, taking into account the object (first object) in the input image Pi.

[0028] Specifically, the text can be placed in a position where the first subject and the text can be easily associated with each other, such as around the first subject. This effect is particularly effective when there are multiple subjects in the input image, and specifically, it is possible to prevent the text from being placed near a subject that has little relationship with the content of the text. In addition, in the first embodiment, as described above, the text placement is automatically determined by the function of the image processing device, so that the user does not need to determine the text placement, and the time and effort required for this can be saved.

[0029] The technique for identifying a subject (first subject) to be associated with text and determining the layout of the text according to the identified first subject and the input image is not particularly limited. As an example, the correspondence between the content of the text, the layout position of the text, and the first subject may be stored in advance as data, the first subject may be identified by referring to the correspondence, and the layout of the text may be determined so as to correspond to the first subject.

[0030] Alternatively, machine learning may be performed to construct a learning model for determining the position, and the image and text may be input to the learning model to determine the placement of the text in the image. According to the learning model, the relevance of the object in the input image to the input text is evaluated, and the first object (more specifically, the position and range of the first object in the image, etc.) is identified based on the evaluation result. The learning model then determines the placement of the text in the output image according to the first object in accordance with a predetermined rule. Such a learning model is constructed by performing machine learning using previously acquired images without text and images with text in which text is placed in association with the object in the image as learning data.

[0031] The above machine learning may be performed in stages. Specifically, a first learning for constructing a learning model for identifying a first object and a second learning for constructing a learning model for determining a text arrangement according to the first object may be performed separately.

[0032] Furthermore, in the first learning, an attention mechanism of deep learning may be applied, and a learning model for identifying the first object may be constructed based on the process of text generation by the learning model for text generation described above. In this case, when generating text to be added to an image using the learning model for text generation, the analysis result of which region in the image was given importance when generating the text can be reflected when identifying the first object.

[0033] [Example of image processing device configuration] Next, an example of the configuration of an image processing device according to the first embodiment (hereinafter, referred to as image processing device 10) will be described with reference to FIG. The image processing device 10 is a computer used by a user, specifically a client terminal, specifically a smartphone, a tablet terminal, a notebook PC (Personal Computer), etc. The image processing device 10 is not limited to a computer owned by a user, and may be configured as a terminal not owned by a user but available by entering a PIN number, a password, or the like or making a deposit when visiting a store, such as a store-installed terminal. In the following, an example will be described in which the image processing device 10 is configured by a user-owned computer, specifically, a smartphone.

[0034] As shown in FIG. 3, the computer constituting the image processing device 10 includes a processor 10a, a memory 10b, a communication interface 10c, a storage 10d, an input device 10e, and an output device 10f.

[0035] The processor 10a may be composed of, for example, a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a TPU (Tensor Processing Unit), or an ASIC (Application Specific Integrated Circuit). The memory 10b is configured by semiconductor memories such as a Read Only Memory (ROM) and a Random Access Memory (RAM).

[0036] The communication interface 10c may be, for example, a network interface card, a communication interface board, etc. A computer constituting the image processing device 10 can communicate with other devices connected to a communication network such as the Internet and a mobile communication line via the communication interface 10c.

[0037] The storage 10d may be composed of, for example, a flash memory, a hard disc drive (HDD), a solid state drive (SSD), a flexible disc (FD), a magneto-optical disc (MO disc), a compact disc (CD), a digital versatile disc (DVD), a secure digital card (SD card), or a universal serial bus memory (USB memory). The storage 10d may be built in the computer body constituting the image processing device 10, or may be attached to the computer body in an external form. Alternatively, the storage 10d may be configured by a NAS (Network Attached Storage) or the like. The storage 10d may also be an external device, such as an online storage or a database server, capable of communicating with one of the computers constituting the image processing device 10 via a communication network.

[0038] The input device 10e is a device that accepts an input operation by a user, and is configured, for example, with a touch panel, etc. The input device 10e also includes a photographing device such as a built-in camera of a smartphone, a microphone for collecting sound, etc. The output device 10f is, for example, a display.

[0039] Furthermore, a program for an operating system (OS) and an application program for executing image processing are installed as software in the computer constituting the image processing device 10. These programs are read and executed by the processor 10a, whereby the computer constituting the image processing device 10 performs its functions, specifically, executes a series of processes related to outputting the image including the above-mentioned text.

[0040] The configuration of the image processing device 10 will be described again from the functional aspect with reference to Fig. 4. As shown in Fig. 4, the image processing device 10 has a reception unit 21, an acquisition unit 22, an identification unit 23, a determination unit 24, a generation unit 25, and an output unit 26. These functional units are realized by the processor 10a of the image processing device 10 executing the above-mentioned application program for image processing and cooperating with other hardware devices of the image processing device 10. In addition, some functions may be realized by utilizing artificial intelligence (AI). Each functional unit will be described below.

[0041] (Reception Department) The reception unit 21 receives input of an image to which text is to be added. As described above, the method of inputting an image is not particularly limited, but for example, a user may photograph a subject within the angle of view using a camera of a smartphone constituting the image processing device 10. In this case, the reception unit 21 receives input of image data of the photographed image. The reception unit 21 may also receive input of an image by downloading image data from an external device (for example, a server) via a network.

[0042] (Acquisition Department) The acquisition unit 22 acquires text Tx relating to an image whose input has been accepted by the acceptance unit 21, that is, an input image Pi. In the first embodiment, the acquisition unit 22 can acquire the text Tx by two methods. The first acquisition method is a method of analyzing an input image Pi and generating text Tx based on the analysis result. Specifically, as described above, data showing the correspondence between the image analysis result and the text content may be referenced to select (generate) text Tx corresponding to the analysis result of the input image Pi. Alternatively, a learning model for text generation constructed by machine learning may be applied to the input image Pi to generate text Tx related to the input image Pi.

[0043] Furthermore, when generating text using the first acquisition method, the acquisition unit 22 may generate text Tx based on an area selected by the user in the input image Pi. To explain in more detail, when generating text based on the analysis result of the input image Pi, the acquisition unit 22 can generate text Tx by treating the area selected by the user in the input image Pi as an attention area and placing more importance on the analysis result of the attention area than other areas. This allows the user's intention to be reflected in the generation of text Tx, and for example, text Tx can be generated by using a subject selected by the user as a target for adding text.

[0044] The user may select an area (area of ​​interest) through the input device 10e, for example, by displaying the input image Pi on a touch display of a smartphone constituting the image processing device 10 and allowing the user to touch the area of ​​interest in the displayed input image Pi. The range selected as the area of ​​interest can be determined arbitrarily, and one subject included in the input image Pi may be set as the area of ​​interest, or a location touched by the user in the input image may be set as the area of ​​interest. The region of interest may be selected by the acquisition unit 22 (more specifically, the processor 10a of the image processing device 10 constituting the acquisition unit 22). That is, the acquisition unit 22 may analyze the input image Pi and select the region of interest based on the analysis result. In this case, the acquisition unit 22 may generate text Tx based on the region of interest selected by the acquisition unit 22.

[0045] Furthermore, when the input image Pi is a photographed image and the image data of the photographed image includes additional information, the acquisition unit 22 may generate the text Tx based on the analysis result of the input image and the above-mentioned additional information. The additional information is various information related to the photographed image, including the photographing location, the photographing date and time, the photographer, the photographing conditions, and the information of the camera used for photographing, and is, for example, tag information recorded in the image data file in the Exif (Exchangeable image file format) format. In a method for generating text Tx based on additional information, for example, machine learning may be performed using a previously acquired image, its additional information, and text attached to the image as learning data, and the resulting learning model for generating text may be used. As described above, in the first embodiment, the text Tx can be generated by referring to the analysis result of the input image Pi as well as the supplementary information of the input image. This makes it possible to generate more appropriate text Tx in consideration of the shooting location and shooting date and time of the input image.

[0046] The second acquisition method is a method of acquiring text based on an input operation performed by a user through the input device 10e. Specifically, the acquisition unit 22 acquires text Tx indicating the input contents based on character input performed by the user through a touch panel or voice input performed through a sound collection microphone. This makes it possible to acquire text Tx according to the user's intention and arrange (add) the text Tx in the output image Po.

[0047] In addition, the acquisition unit 22 may acquire text Tx by combining two or more of text generation based on the analysis results of the input image Pi, text generation based on a region of interest in the input image Pi, text generation based on supplementary information, and acquisition of text entered by a user.

[0048] (Specific part) The identification unit 23 identifies a first object to be associated with the text Tx acquired by the acquisition unit 22 from among the objects included in the input image Pi, and specifically identifies the position (coordinate position) and range of the first object in the input image Pi. In the first embodiment, the input image Pi includes at least one object from among a person, an object, an animal, and a background. The identification unit 23 identifies the first object from the at least one object.

[0049] To explain this by taking the case shown in Figure 2 as an example, suppose that an input image Pi containing as subjects multiple company employees attending a farewell party is acquired, and the text "Thank you for your support" is generated from the features obtained by analyzing this image. In this case, the company employee being seen off (the person in the center of the input image Pi) is identified as the first subject. Note that the number of first subjects identified in one input image may be one or more. In the following, unless otherwise specified, it is assumed that the number of first subjects identified in one input image Pi is one.

[0050] As described above, the method of identifying the first object is not particularly limited, but for example, the first object may be identified by referring to data showing the correspondence between the text and the first object. Alternatively, a learning model for identifying an object constructed by implementing machine learning may be applied to the input image Pi and the acquired text Tx to identify the first object corresponding to the text Tx. In addition, an attention mechanism of deep learning may be applied to the above learning model to identify an area (attention area) in the input image Pi that is emphasized when generating text from the input image Pi, and the first object may be identified based on the identified attention area. This makes it possible to more appropriately identify the first object corresponding to the text Tx.

[0051] (Decision Section) The determination unit 24 determines the layout (strictly speaking, the layout position) in the output image for the text Tx acquired by the acquisition unit 22. In the first embodiment, the determination unit 24 determines the layout of the text based on the input image and the first subject identified by the identification unit 23. More specifically, in the first embodiment, the layout of the text in the input image is determined in the following manner.

[0052] First, the determination unit 24 determines whether the text Tx can be arranged around the first subject in the input image Pi without overlapping with the first subject. Here, if the text Tx cannot be arranged around the first subject without overlapping with the first subject, the determination unit 24 determines the arrangement of the text so that it overlaps with the first subject. In this case, as shown in FIG. 5A, it is preferable to determine an arrangement that does not overlap with a specific part of the first subject. Here, the specific part is a part that is predetermined in the first subject, specifically, a main part in the first subject, and is a part that is required to appear clearly in the output image Po. For example, when the first subject is a person, the specific part is the upper body of the person, especially the face. Note that the specific part when the first subject is a person may be the whole body of the person. Also, when the first subject is an object, the specific part is a part that makes it difficult to identify the object if the text overlaps.

[0053] In addition, when identifying the specific part, a known technology can be used, specifically, a technology for detecting a subject in an image (image analysis technology) can be applied to identify each part of the subject in the image and identify which part each part is. Then, importance (priority) can be assigned to each identified part according to a predetermined standard or rule, and the specific part can be determined based on the importance (priority) assigned to each part.

[0054] As described above, when it is not possible to arrange the text around the first subject so as not to overlap the first subject, a position where the text will not overlap the specific part of the first subject is determined as the arrangement position of the text. This makes it possible to prevent the specific part of the first subject from being hidden by the text Tx. In addition, when it is difficult to arrange the text Tx so as not to overlap the specific part of the first subject (for example, when the input image is a zoomed image of the face of a person who is the first subject), the determination unit 24 may determine the arrangement of the text Tx so as to overlap the specific part. In this case, as shown in Fig. 5B, it is preferable to determine the arrangement of the text Tx so as to minimize the influence of the text Tx on the specific part. The arrangement position of such text Tx may be determined, for example, by using artificial intelligence (AI) or according to a predetermined rule.

[0055] On the other hand, when the text can be arranged around the first subject without overlapping with the first subject, the determination unit 24 determines whether the text can be arranged around the first subject in the input image without overlapping with both the first subject and the second subject. The second subject is a subject different from the first subject among the subjects included in the input image, and has a lower relevance to the text than the first subject. The second subject may be, for example, a subject of the same type (same category) as the first subject. Specifically, when one of a plurality of people included as subjects in the input image is the first subject, the second subject is a person of the plurality of people different from the first subject.

[0056] Then, when the text can be arranged around the first object so as not to overlap with both the first object and the second object, the determination unit 24 determines the arrangement of the text around the first object in the input image Pi so as not to overlap with both the first object and the second object, as shown in Fig. 5C. This allows the text Tx to be arranged at a position where the correspondence with the first object can be easily grasped, and prevents the first object and the second object from being hidden by the text Tx.

[0057] When it is not possible to arrange the text Tx around the first subject so as not to overlap with the first subject but not to overlap with the second subject, the determination unit 24 determines the arrangement of the text around the first subject in the input image Pi so as to overlap with the second subject but not with the first subject. This allows the text Tx to be arranged at a position where the correspondence between the text Tx and the first subject can be easily understood.

[0058] In addition, when the user selects an attention area in the input image Pi when acquiring text, the determination unit 24 may determine the arrangement of the text Tx according to the area selected as the attention area by the user when determining the arrangement of the text Tx in the above-mentioned manner. Specifically, the arrangement of the text Tx may be determined near the attention area selected by the user, or conversely, the arrangement of the text Tx may be determined by avoiding the attention area.

[0059] (Generation part) The generation unit 25 generates an output image Po corresponding to the input image Pi, based on the text Tx acquired by the acquisition unit 22 and the arrangement of the text determined by the determination unit 24. The output image Po in the first embodiment is an image including the text Tx and the input image Pi, and specifically, as shown in Fig. 1, is an image in which the text Tx is arranged (added) in the input image Pi in the arrangement determined by the determination unit 24. To explain in more detail, the generation unit 25 converts the text Tx into a text image, and generates the output image Po by superimposing the text image on the input image Pi.

[0060] (Output section) The output unit 26 outputs the output image Po generated by the generation unit 25. As described above, the image output method is not particularly limited, but for example, the output image Po may be displayed on a display (output device 10f) of a computer constituting the image processing device 10. Also, control data for printing the output image Po may be generated, and a printer (not shown) may be controlled based on the control data to print the output image Po by the printer. Alternatively, for the purpose of sharing the output image Po with other users, image data of the output image Po may be transmitted from the computer constituting the image processing device 10 to a terminal used by the other users. Alternatively, for the purpose of publishing the output image Po on an SNS, the image data of the output image Po may be uploaded to an SNS server.

[0061] [Example of image processing method according to the first embodiment] Next, as an operation example of the image processing device 10 in the first embodiment, an image processing flow using the same device will be described. In the image processing flow described below, the image processing method of the present invention is used. In other words, each step in the image processing flow described below corresponds to a component of the image processing method of the present invention. Note that the flow below is merely an example, and some steps in the flow may be deleted, new steps may be added to the flow, or the order of execution of two steps in the flow may be reversed, as long as it does not deviate from the spirit of this embodiment.

[0062] The steps in the image processing flow according to the first embodiment are performed in the order shown in Fig. 6 by the processor 10a included in the image processing device 10. That is, in each process in the image processing flow, the processor 10a executes processing corresponding to each step in Fig. 6 among data processing defined in an application program for image processing.

[0063] Specifically, in the image processing flow according to the first embodiment, the processor 10a first executes an input reception process (S001). In the input reception process, the processor 10a receives an input of an image to which text is to be added by this flow, in other words, obtains an input image Pi to be processed.

[0064] Next, the processor 10a executes an acquisition process to acquire text Tx related to the input image Pi (S002). In the acquisition process, the processor 10a may acquire text based on an input operation of text content by a user. In addition, in the acquisition process, the processor 10a may analyze the input image Pi and generate text Tx based on the analysis result. In addition, in the acquisition process, the user may be allowed to select an attention area in the input image Pi. In this case, in the acquisition process, the processor 10a may generate text Tx based on the attention area selected by the user, more specifically, by prioritizing the analysis result of the attention area over the analysis result of areas other than the attention area.

[0065] In addition, when the input image Pi is a photographed image and the image data of the photographed image includes additional information such as an Exif tag, in the acquisition process, the processor 10a may generate text Tx based on the analysis results of the input image Pi and the additional information.

[0066] Next, the processor 10a executes an identification process to identify a first object to be associated with the text Tx acquired in the acquisition process from among the objects included in the input image Pi (S003). Here, the input image Pi includes at least one object from among a person, an object, an animal, and a background. In the identification process, the processor 10a identifies the first object from the at least one object, and specifically identifies the position (coordinate position) and range of the first object in the input image Pi.

[0067] Next, the processor 10a executes a determination process for determining the arrangement of the text Tx in the output image Po (S004). In the determination process, the processor 10a determines the arrangement of the text Tx based on the first subject identified in the identification process.

[0068] Specifically, the processor 10a may determine the arrangement of the text Tx such that the text Tx does not overlap with the first object in the input image Pi around the first object. At this time, it is preferable to determine the arrangement of the text Tx such that the text Tx does not overlap with a specific part (e.g., a person's face) of the first object in the input image Pi around the first object. The processor 10a may also determine the arrangement of the text Tx such that the text Tx does not overlap with both the first object and the second object in the input image Pi around the first object.

[0069] Next, the processor 10a executes a generation process to generate an output image Po corresponding to the input image Pi (S005). In the generation process, the processor 10a generates the output image Po by adding the text Tx acquired in the acquisition process to the input image Pi in the arrangement determined in the determination process.

[0070] Thereafter, the processor 10a executes an output process to output the generated output image Po (S006). By executing the output process, an output image Po configured from the input image Pi in which the text Tx is arranged is output. Specifically, for example, the output image Po is displayed on a display or the like, printed by a printer, or image data of the output image is transmitted for the purpose of disclosure, sharing, or the like.

[0071] When the series of processes described above is completed, the image processing flow according to the first embodiment is completed. The image processing flow is performed each time an input image is received, that is, the series of processes described above is repeatedly executed by the processor 10a each time a new input image Pi is received.

[0072] [Modification of the first embodiment] The first embodiment of the present invention is not limited to the above embodiment, and may be modified as follows. The following describes the modified example. The following describes the differences between the modified example and the first embodiment.

[0073] (Regarding the first modified example) In the above-mentioned first embodiment, the image processing device 10 acquires one text Tx for one input image Pi, in other words, one output image Po includes one text Tx. However, without being limited to this, as a modified example of the first embodiment, multiple texts Tx may be acquired for one input image Pi. That is, as one embodiment of the present invention, an embodiment (hereinafter, 1A embodiment) in which multiple texts Tx are added to one input image Pi to generate an output image Po, as shown in FIG. 7, can be considered.

[0074] In the first A embodiment, a method of acquiring a plurality of pieces of text Tx relating to one input image Pi may be, for example, to construct a learning model (text generation model) that analyzes the input image Pi and generates text Tx with specifications capable of outputting a plurality of pieces of text Tx. In other words, a text generation model that generates a plurality of pieces of text from one input image Pi may be constructed by machine learning, and a plurality of pieces of text Tx may be acquired using the model. The number of pieces of text Tx generated using this text generation model may be specified by a user, or may be automatically set on the model side according to the content (subject) of the input image Pi.

[0075] The text generation model may present multiple candidates for text Tx based on the analysis result of the input image Pi (more specifically, the feature amount of the input image). Then, two or more candidates selected by the user from the multiple presented text candidates may be adopted (obtained) as multiple texts Tx. In this case, the text generation model may evaluate the validity (likelihood) of each of the multiple presented text candidates as text and present the evaluation value together with the candidate. This evaluation value corresponds to, for example, an output probability (likelihood as a candidate) calculated by applying a softmax function to the candidate output by the learning model. Then, the user may select two or more candidates based on the evaluation value of validity calculated for each candidate. Also, among the multiple candidates, the candidates with the top n evaluation values ​​(n is a natural number equal to or greater than 2) may be automatically selected.

[0076] Another method for acquiring multiple pieces of text related to one input image is to divide a single sentence of text into multiple words (phrases). In other words, multiple words separated by segmenting one piece of text may be acquired as multiple pieces of text Tx. For example, assume that the text acquired (generated) in the above acquisition process is "July 21st, Happy Birthday!!" In this case, as shown in FIG. 7, the above text is divided into two pieces of text, "July 21st" and "Happy Birthday!!". In this way, a relatively long piece of text can be divided into multiple shorter pieces of text, and each piece of divided text can be assigned to the input image Pi.

[0077] As described above, in the first embodiment, a plurality of texts Tx are obtained for one input image Pi. In this case, the processor 10a of the image processing device 10 identifies the first subject in the input image Pi for each of the plurality of texts Tx, and also determines the arrangement of the text in the output image Po. At this time, the processor 10a (more specifically, the determination unit 24) may determine an arrangement for each of the plurality of texts Tx such that the texts do not overlap each other around the first subject (a girl in FIG. 7) as shown in FIG. 7. This makes it possible to increase the utility value of the output image Po by adding a plurality of texts Tx, while avoiding the texts Tx from becoming difficult to see due to the texts overlapping each other. The first object identified for each of the multiple texts Tx may be the same between the texts, or may be different between the texts.

[0078] (Regarding the second modified example) In the above-described first embodiment, when determining the arrangement of the text Tx in the output image Po, the arrangement is determined based on the area in which the subject exists in the input image Pi. Specifically, the arrangement of the text Tx is determined in an area around the first subject in the input image Pi that does not overlap with the first subject, an area that does not overlap with a specific part of the first subject, or an area that does not overlap with both the first subject and the second subject. At this time, the suitability or insuitability of a plurality of areas in the input image Pi as an area in which the text Tx is to be arranged may be evaluated, and the arrangement of the text Tx may be determined based on the evaluation result. That is, as one embodiment of the present invention, an embodiment (hereinafter, 1B embodiment) in which evaluation values ​​based on the analysis results of the input image Pi are obtained for a plurality of areas in the input image, and the arrangement of the text is determined based on the evaluation value of each area may be considered.

[0079] There is no particular limitation on the method of acquiring evaluation values ​​for multiple regions in an input image. To explain a specific example of the method of acquiring evaluation values, a subject in the input image is detected using a known subject detection technology and a segmentation technology. Here, when multiple subjects are detected, the input image Pi is partitioned into multiple regions according to the multiple subjects, as shown in FIG. 8. Furthermore, when a person is included in the multiple subjects, a known face extraction technology is applied to the region in which the person exists to extract a face (particularly a face facing forward), and the region in which the face is included is identified.

[0080] Then, as shown in FIG. 8, for each of the multiple regions, an evaluation value (a numerical value added to each region in the image in FIG. 8) is calculated and acquired based on the relevance between the subject included in each region and the acquired text Tx. The evaluation value is an evaluation value regarding the arrangement of the text Tx, and specifically, a score regarding the suitability of the region in which the text Tx is to be arranged. This score may be smaller the higher the relevance with the text Tx, and conversely, may be larger the higher the relevance with the text Tx. Furthermore, for a region in which a person who is highly relevant to the text Tx exists, as shown in FIG. 8, the evaluation value assigned to the region in which the person's face exists (face region) may be set to a value in which the relevance is higher than the evaluation value assigned to a region other than the face region.

[0081] When evaluating the relevance between each region in the input image Pi and the text Tx, it is advisable to apply the learning model used when generating the text. Specifically, for each region of the input image, the degree to which the region was given importance when generating the text is estimated, and the relevance with the text Tx is evaluated based on the estimated degree of importance, and the evaluation value is assigned to the region.

[0082] As another example of a method for assigning an evaluation value, an evaluation value may be assigned according to the importance of a subject included in each region, regardless of the relevance to the text Tx. To be more specific, the input image Pi is analyzed to detect a subject in the input image Pi by a procedure similar to that described above, and when multiple subjects are detected, an importance is set for each subject. The importance is set according to a predetermined rule, for example, a correspondence between the type of subject and the importance is determined in advance for each type of subject, and the importance is set based on the correspondence. Then, for multiple regions in the input image Pi, an evaluation value according to the importance of the region is acquired (calculated). At this time, the evaluation value may be larger as the importance is higher, or conversely, may be smaller as the importance is higher. In addition, for a region where a subject with high importance (for example, a person) exists, the evaluation value assigned to the region where the person's face exists (face region) may be set to a value when the importance is higher than the evaluation value assigned to a region other than the face region.

[0083] Then, the processor 10a (more specifically, the determination unit 24) of the image processing device 10 determines the placement of the text Tx based on the evaluation value acquired for each region in the input image Pi. To be more specific, for example, if the evaluation value decreases as the importance of the subject increases, the placement of the text Tx is determined in the region with the high evaluation value.

[0084] As described above, in the first B embodiment, the processor 10a of the image processing device 10 determines the arrangement of the text based on the evaluation values ​​acquired for multiple regions in the input image Pi. This makes it possible to appropriately determine the arrangement of the text Tx based on the relevance between each subject and the text Tx or the importance of each subject.

[0085] More specifically, in the first B embodiment, an evaluation value for each region in the input image Pi is acquired so that the possibility that the arrangement of the text Tx is determined in the target region in the input image Pi is lower than the possibility that the arrangement of the text Tx is determined in other regions. This allows the text Tx to be arranged in the output image Po at a position that does not overlap with the target region in the input image Pi included in the output image Po. Here, the target region is a region in the input image Pi that is required to avoid overlapping with the text Tx. Specifically, for example, a region in which a subject with high importance exists, particularly when such a subject is a person, a region in which the face is located corresponds to the target region.

[0086] <<Second embodiment of the present invention>> In the above-described first embodiment, one input image Pi is acquired, but a case where multiple input images Pi are acquired may be considered. This case will be described below as a second embodiment of the present invention (hereinafter, the second embodiment). In the second embodiment, differences from the first embodiment will be mainly described, and descriptions of points in common with the first embodiment will be omitted.

[0087] In the second embodiment, the processor 10a of the image processing device 10 acquires a plurality of images as input images Pi. The image acquisition procedure is the same as that of the first embodiment. The processor 10a then analyzes the plurality of input images Pi, and generates text Tx related to at least one of the plurality of input images Pi based on the analysis results of the plurality of input images Pi.

[0088] Specifically, as shown in Fig. 9, when two or more input images Pi are input, one input image Pi is a main image to which text is added, and the remaining input images Pi are auxiliary images for generating text. The processor 10a analyzes each of the main image and the auxiliary image, and generates text Tx for the main image based on the analysis results of each image. The text generation procedure is the same as in the first embodiment, except that the analysis results of multiple images are used.

[0089] After generating the text Tx, the processor 10a identifies a first subject to be associated with the generated text Tx from among the subjects included in the main image. The procedure for identifying the first subject is the same as that of the first embodiment.

[0090] Then, the processor 10a determines the arrangement of the text Tx in the output image Po including the main image based on the specified first object. At this time, it is preferable that the arrangement of the text Tx is determined in an area around the first object in the main image that does not overlap with the first object, particularly in an area that does not overlap with a specific portion of the first object, as shown in Fig. 9. The procedure for determining the arrangement of the text Tx is the same as that of the first embodiment.

[0091] Furthermore, the second embodiment is not limited to the case where text to be added to a main image is generated based on a plurality of input images Pi including the main image. For example, the processor 10a of the image processing device 10 may simultaneously receive m (m is a natural number) input images Pi related to each other, and generate text Tx for each input image Pi based on the analysis results of each of the m input images Pi, the number of which is equal to the number of input images Pi. That is, one piece of text Tx may be added to each of the m input images Pi. In addition, the m input images Pi that are related to each other may be, for example, images taken at the same shooting location and the same shooting time, or may be images taken with the same shooting theme (specifically, images obtained by photographing the same subject multiple times at different shooting times).

[0092] The second embodiment may also include a case in which a process of generating text Tx and a process of determining the layout of the text Tx are repeatedly executed each time an input image Pi is acquired (received). In this case, the text Tx is added to each acquired input image Pi.

[0093] As described above, in the second embodiment, it is possible to generate text Tx based on the analysis results of multiple input images Pi. This effect is particularly effective when creating an album or photo book and adding text to at least one or more images among multiple images included therein. In the second embodiment, when analyzing multiple input images Pi to obtain text, it is not necessary to use all of the multiple input images Pi, and the multiple input images Pi may include input images Pi that are not analyzed (input images Pi that are not subject to analysis).

[0094] <<Third embodiment of the present invention>> In the above-described first embodiment, an image including a subject, or more precisely, an image consisting only of an image region including a subject, is used as the input image Pi. On the other hand, there may be a case where an edited image generated by arranging one or more images (material images) on a layout image, that is, an image having an image area and a margin area, is used as the input image Pi, as shown in Fig. 10. This case will be described below as the third embodiment of the present invention (hereinafter, the third embodiment). In the third embodiment, differences from the first embodiment will be mainly described, and descriptions of commonalities with the first embodiment will be omitted.

[0095] As described above, the input image Pi in the third embodiment is an image having an image area Pa1 and a blank area Pa2, and is generated, for example, by using image editing software. The image area Pa1 is an image including a subject, and one or more blank areas Pa1 are arranged in the input image Pi. The blank area Pa2 is an area located outside the image area Pa1 in the input image Pi, and may be a plain area or an area to which a color, design, pattern, or the like is added. In the following, as shown in FIG. 11, an example will be described in which a plurality of image regions Pa1 are included in an input image Pi, and a blank region Pa2 is a white, plain region.

[0096] In the third embodiment, an output image including the above-mentioned input image Pi and text Tx is output. To be more specific, the processor 10a of the image processing device 10 (hereinafter, image processing device 10X) according to the third embodiment determines the arrangement of the text Tx in the margin area Pa2 in the input image Pi included in the output image Po, as can be seen from Fig. 11. To be more specific, the processor 10a analyzes the input image Pi in the above-mentioned manner, more precisely, analyzes a plurality of image areas Pa1 included in the input image Pi, and generates one or more texts Tx based on the respective analysis results.

[0097] After generating the text Tx, the processor 10a identifies a first object to be associated with the generated text Tx from each object in a plurality of image regions Pa1 included in the input image Pi. The processor 10a then determines the placement of the text Tx based on the identified first object and the input image Pi. At this time, the processor 10a, for example, identifies an image region Pa1 that includes the first object from among the plurality of image regions Pa1 included in the input image Pi, and determines the placement of the text Tx in a blank region Pa2 adjacent to the image region Pa1.

[0098] Finally, the processor 10a generates an output image Po in which text is arranged in the margin area Pa2, as shown in Fig. 11, and outputs the generated output image Po. In this way, in the third embodiment, when the input image Pi has a margin area Pa2, the margin area Pa2 is effectively used to arrange the text Tx, thereby generating an edited image (output image Po) with text. This effect is particularly effective when adding text to images constituting postcards or each page of a photo book. The effect of the third embodiment is also useful in cases where it is difficult to place the text Tx within the image area Pa1, such as when the face of a person who is the subject is zoomed in and takes up the entire image area Pa1.

[0099] Note that, in the third embodiment, the text Tx may be arranged in the image area Pa1 of the input image Pi. That is, in the third embodiment, the arrangement of the text Tx may be determined in at least one of the image area Pa1 and the margin area Pa2 of the input image Pi.

[0100] In the third embodiment, the image processing device 10X may further include a change unit 27 in addition to the above-mentioned functions, as shown in Fig. 12. The change unit 27 is a function for changing the layout of the image regions Pa1 in the input image Pi. The layout of the image regions Pa1 may include the number of image regions Pa1 arranged in the input image Pi, as well as the size, arrangement position, outer edge shape (aspect ratio) of each image region Pa1, and the inclination of the image region Pa1 with respect to the input image Pi (i.e., the amount of rotation of the image region Pa1).

[0101] When using the function of the modification unit 27, in a previous step, the processor 10a acquires text Tx related to the input image Pi and generates the text Tx based on, for example, the analysis result of each image area Pa1 in the input image Pi. In a subsequent step, the modification unit 27 determines whether it is possible to determine the layout of the text Tx within the blank area Pa2 in the input image Pi based on the generated text Tx.

[0102] Specifically, the modification unit 27 specifies the number of characters and size of the generated text Tx, and specifies the position and size of a blank space area Pa2 (hereinafter, adjacent blank space area) adjacent to an image area Pa1 including a first object corresponding to the text Tx in the input image Pi. At this time, the modification unit 27 may specify whether or not an adjacent blank space area exists in the current input image Pi. Based on the result of the specification, the modification unit 27 determines whether or not the layout of the text Tx can be determined in the blank space area Pa2 in the current input image Pi, more specifically, in the adjacent blank space area.

[0103] When the layout of the text can be determined in the adjacent blank space area in the current input image Pi, the processor 10a (more specifically, the determination unit 24) determines the layout of the text Tx in the adjacent blank space area.

[0104] On the other hand, if the layout of the text Tx cannot be determined in the adjacent blank area in the current input image Pi, or if there is no adjacent blank area, the modification unit 27 modifies the layout of the image area Pa1 in the input image Pi as shown in Fig. 13. The modification of the layout may include not only changing the number and layout of the image areas Pa1 in the input image Pi, but also performing movement, enlargement / reduction, change in aspect ratio, trimming, and the like on one or more image areas Pa1.

[0105] The layout change may be performed automatically according to a preset rule, or may be performed based on a user's change operation. Alternatively, by utilizing artificial intelligence (AI), a layout that ensures a sufficient adjacent blank area for arranging the text Tx may be calculated, and the input image Pi may be re-edited with that layout, and the position, size, etc. of the image area Pa1 in the input image Pi may be changed.

[0106] When the change unit 27 changes the layout of the image area Pa1 in the input image Pi, the processor 10a (more specifically, the determination unit 24) determines the arrangement of the text Tx in the output image Po including the input image Pi whose layout has been changed. At this time, the processor 10a determines the arrangement of the text Tx in the margin area Pa2 in the input image Pi whose layout has been changed, or more precisely, in the adjacent margin area, of the output image Po, as shown in FIG. According to the above configuration, when the layout of the image area Pa1 in the input image Pi is such that the text Tx cannot be placed in the adjacent margin area, the layout can be appropriately revised so that the text Tx can be appropriately placed in the adjacent margin area.

[0107] <<Other embodiments>> Although specific embodiments of the present invention have been described above, the above-mentioned embodiments are merely examples given to facilitate understanding of the present invention and do not limit the present invention. That is, the present invention may be modified or improved from the embodiments described below without departing from the spirit of the present invention. The present invention also includes equivalents thereof. Furthermore, the embodiments of the present invention may include a combination of the above-mentioned embodiments and one or more of the following modifications.

[0108] (Modification of output image) In the above embodiment, the output image Po is generated by adding text Tx to the input image Pi. That is, the output image Po according to the above embodiment is the same as the input image Pi, except that the text Tx is added. However, this is not limited to this, and as shown in FIG. 14, the output image Po may include the input image Pi and the text Tx, and may have a blank area Pa3 around at least a part of the input image Pi in the output image Po.

[0109] When obtaining the output image Po, the processor 10a of the image processing device 10 accepts an input of an image, acquires text Tx related to the input image Pi, and identifies a first object from the objects in the input image Pi, in the same procedure as in the above-described embodiment. Furthermore, as shown in Fig. 14, an image to be used as a mount on which the input image Pi is attached (hereinafter, mount image Pd) is selected. The mount image Pd is an image having a larger display size (angle of view) than the input image Pi, and constitutes the output image Po together with the input image Pi and text Tx (strictly speaking, an image of the text Tx). To be more specific, the output image Po is generated by combining the input image Pi and text Tx so that these images are arranged (superimposed) on the mount image Pd. Also, as described above, since the mount image Pd is an image having a larger display size than the input image Pi, the area of ​​the mount image Pd included in the output image Po where the input image Pi is not arranged becomes a blank area Pa3, as shown in FIG.

[0110] Then, the processor 10a of the image processing device 10 determines the arrangement of the input image Pi and the text Tx in the output image Po when generating the above-mentioned output image Po. The arrangement of the input image Pi may be determined according to a user's input operation, or may be automatically determined by the processor 10a according to the contents of the input image Pi (specifically, the subject in the input image Pi). The arrangement of the text Tx is determined based on a first subject specified in the input image Pi, for example, in the periphery of the first subject so as not to overlap the first subject or its specific portion. At this time, the arrangement of the text Tx may be determined in at least one of the area in which the input image Pi is arranged in the output image Po and the margin area Pa3.

[0111] (About the computer that constitutes the image processing device) In the above embodiment, the image processing device of the present invention is configured by a computer that is directly used by a user, such as a terminal (client terminal) owned by the user. However, the present invention is not limited to this, and the image processing device of the present invention may be configured by a computer that is indirectly available to the user, for example, a server computer. Here, the above server computer may be, for example, a server computer for cloud services, specifically, a server computer for ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). In this case, when necessary information is input at the client terminal, the above server computer performs various processes (calculations) based on the input information, and the calculation results are output at the client terminal. In other words, the functions of the server computer that constitutes the image processing device of the present invention can be used at the client terminal.

[0112] (Processor configuration) The processor included in the image processing device of the present invention includes various processors, including, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. The various processors also include programmable logic devices (PLDs) such as field programmable gate arrays (FPGAs) whose circuit configurations can be changed after manufacture. Furthermore, various processors include dedicated electric circuits such as ASICs (Application Specific Integrated Circuits), which are processors having a circuit configuration designed exclusively for performing specific processing.

[0113] Furthermore, one functional unit of the image processing device of the present invention may be configured by one of the various processors described above, or may be configured by a combination of two or more processors of the same or different types, for example, a combination of multiple FPGAs, or a combination of an FPGA and a CPU, etc. Furthermore, the multiple functional units of the image processing device of the present invention may be configured by one of various processors, or two or more of the multiple functional units may be integrated into one processor. Also, as in the above-described embodiment, one processor may be configured by combining one or more CPUs and software, and this processor may function as multiple functional units.

[0114] Also, for example, a processor may be used that realizes the functions of the entire system including the multiple functional units in the image processing device of the present invention on a single IC (Integrated Circuit) chip, as typified by SoC (System on Chip), etc. Also, the hardware configuration of the various processors described above may be an electric circuit (Circuitry) that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0115] 10,10X Image Processing Device 10a processor 10b Memory 10c Communication Interface 10d Storage 10e Input Device 10f output device 21 Reception 22 Acquisition Department 23 Specific section 24 Decision Section 25 Generation part 26 Output section 27 Changes Pa1 image area Pa2,Pa3 margin area Pd mount image Pi Input image Po output image Tx Text

Claims

1. An image processing device comprising a processor for processing an input image, The processor, obtaining text relating to the input image; A process of identifying a first object to be associated with the text from objects included in the input image; and determining a layout of the text in an output image based on the input image in accordance with the input image and the first object.

2. The image processing device according to claim 1 , wherein the processor acquires the text based on an input operation of a user.

3. The image processing apparatus of claim 1 , wherein the processor generates the text based on a region selected by a user in the input image.

4. The image processing device according to claim 1 , wherein the processor generates the text based on a result of analysis of the input image.

5. The image processing device according to claim 1 , wherein the processor generates the text based on supplementary information of the input image.

6. The image processing device of claim 1 , wherein the processor is configured to determine the placement such that the text does not overlap the first object.

7. The image processing device according to claim 1 , wherein the processor determines the arrangement such that the text does not overlap a specific portion of the first subject.

8. 6. The image processing device of claim 1, wherein when outputting the output image including the input image and the text, the processor determines the arrangement in which the text does not overlap the first object and a second object different from the first object in the input image around the first object.

9. The image processing device according to claim 1 , wherein, when a plurality of pieces of text are acquired as the text, the processor determines the arrangement of the plurality of pieces of text such that the pieces of text do not overlap each other around the first subject.

10. 6. The image processing device according to claim 1, wherein when outputting an output image including the input image having an image region and a margin region and the text, the processor determines the arrangement in at least one of the image region and the margin region of the input image included in the output image.

11. 6. The image processing device according to claim 1, wherein, when outputting an output image including the input image and the text, the processor generates a margin area around at least a portion of the input image in the output image and determines an arrangement of the input image and the text in the output image.

12. The processor, obtaining an evaluation value for the arrangement for a plurality of regions in the input image based on the input image; The image processing device according to claim 1 , wherein the layout is determined based on the evaluation values ​​of the plurality of regions.

13. The image processing device according to claim 1 , wherein the processor acquires a plurality of pieces of text as the text, and determines the arrangement of the plurality of pieces of text.

14. The image processing device according to claim 1 , wherein the processor divides the text into a plurality of pieces of text, and determines the arrangement for the plurality of pieces of text.

15. The processor acquires a plurality of input images as the input image; The image processing device according to claim 1 , further comprising: a processor configured to generate the text relating to at least one of the plurality of input images based on a result of analysis of the plurality of input images.

16. When outputting the output image including the input image having an image region and a white space region and the text, the processor: determining whether the placement can be determined within the blank area in the input image based on the text; changing a layout of the image area in the input image when it is determined that the arrangement cannot be determined in the blank area in the input image; The image processing device according to claim 1 , wherein the arrangement is determined in the margin area of ​​the output image including the input image whose layout has been changed.

17. 1. An image processing method for processing an input image by a processor, comprising: obtaining, by the processor, text associated with the input image; identifying, by the processor, a first object to be associated with the text from among objects included in the input image; determining, by the processor, a placement of the text in an output image based on the input image in response to the input image and the first object.

18. The image processing method according to claim 17 , wherein the step of obtaining text comprises generating the text by the processor based on a result of analysis of the input image.

19. The method further includes a step of assigning, by the processor, an evaluation value relating to the arrangement to a plurality of regions in the input image; The image processing method according to claim 18 , wherein in the step of determining the layout, the layout is determined by the processor based on the evaluation values ​​of the plurality of regions.

20. A program for causing a computer to execute each step included in the image processing method according to any one of claims 17 to 19.

21. A computer-readable recording medium, 20. A recording medium having recorded thereon a program for causing a computer to execute each step included in the image processing method according to any one of claims 17 to 19.

Images