Image processing apparatus, image processing method, program product, and recording medium

Abstract

The invention provides an image processing device, an image processing method, a program, and a recording medium capable of outputting an image in which text related to an input image is appropriately arranged in consideration of a subject in the input image. An image processing device according to one embodiment of the present invention is provided with a processor that executes the following processes: acquiring text relating to an input image; determining a first subject corresponding to the text from among subjects included in the input image; and determining a configuration of text in an output image based on the input image according to the input image and the first subject.

Description

Technical Field

[0001] One embodiment of the present invention relates to an image processing device, an image processing method, a program product, and a recording medium capable of outputting an output image in which text is added to an input image. Background Art

[0002] As a technology for utilizing images, a technology for adding text corresponding to a subject in an image is known, and as an example, the technology described in Patent Document 1 can be cited. In the technology described in Patent Document 1, a photographic image is analyzed to generate text information, the photographic image is processed based on the text information, and the text information is added to the processed image. In this way, comments based on the photographic image and the image can be displayed at the same time, and the matching sense between the two can be improved.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2017-229102

[0004] When adding text to an image using the technology described in Patent Document 1, the text needs to be placed at an appropriate position in the image. In particular, when adding text corresponding to a subject in an image, if the placement of the text is determined without considering the relationship between the subject and the text, the effect of adding the text may not be properly obtained. On the other hand, if the user determines the placement of the text in consideration of the subject, the operation takes time. Summary of the invention

[0005] An embodiment of the present invention is completed in view of the above situation, and its purpose is to provide an image processing device, image processing method, program and recording medium that can output an image in which text related to the input image is arranged at an appropriate position taking into account the subject in the input image.

[0006] The above object can be achieved by the image processing device described in any one of the following [1] to

[16] .

[0007] [1] An image processing device includes a processor and processes an input image, wherein the processor performs the following processing:

[0008] Get the text associated with the input image;

[0009] determining a first object corresponding to the text from among the objects included in the input image; and

[0010] The arrangement of text in an output image based on the input image is determined according to the input image and the first object.

[0011] [2] The image processing device according to [1], wherein:

[0012] The processor obtains the text according to the user's input operation.

[0013] [3] The image processing device according to [1] or [2], wherein:

[0014] The processor generates text based on the area selected by the user in the input image.

[0015] [4] The image processing device according to any one of [1] to [3], wherein:

[0016] The processor generates text based on the analysis results of the input image.

[0017] [5] The image processing device according to any one of [1] to [4], wherein:

[0018] The processor generates text based on the incidental information of the input image.

[0019] [6] The image processing device according to any one of [1] to [5], wherein:

[0020] The processor determines a placement in which the text does not overlap with the first subject.

[0021] [7] The image processing device according to any one of [1] to [6], wherein:

[0022] The processor determines the arrangement of the text so that it does not overlap with the specific portion of the first object.

[0023] [8] The image processing device according to any one of [1] to [7], wherein:

[0024] When outputting an output image including an input image and text, the processor determines an arrangement around the first subject so that the text does not overlap with the first subject and a second subject different from the first subject in the input image.

[0025] [9] The image processing device according to any one of [1] to [8], wherein:

[0026] When a plurality of texts are acquired as the texts, the processor determines an arrangement of the plurality of texts so that the texts do not overlap each other around the first subject.

[0027]

[10] The image processing device according to any one of [1] to [9], wherein:

[0028] When outputting an output image including an input image having an image area and a blank area and text, the processor determines the arrangement on at least one of the image area and the blank area of ​​the input image included in the output image.

[0029]

[11] An image processing device according to any one of [1] to

[10] , wherein:

[0030] When outputting an output image including an input image and text, the processor generates a blank area on at least a portion of a periphery of the input image in the output image, and determines the arrangement of the input image and the text in the output image.

[0031]

[12] An image processing device according to any one of [1] to

[11] , wherein:

[0032] The processor performs the following processing:

[0033] According to an input image, obtaining evaluation values ​​related to configuration for a plurality of regions in the input image; and

[0034] The configuration is determined based on the evaluation values ​​of the multiple areas.

[0035]

[13] An image processing device according to any one of [1] to

[12] , wherein:

[0036] The processor obtains a plurality of texts as texts and determines a configuration for the plurality of texts.

[0037]

[14] An image processing device according to any one of [1] to

[13] , wherein:

[0038] The processor splits the text into multiple texts and determines the configuration for the multiple texts.

[0039]

[15] An image processing device according to any one of [1] to

[14] , wherein:

[0040] The processor performs the following processing:

[0041] acquiring a plurality of input images as input images; and

[0042] A text related to at least one of the plurality of input images is generated according to the analysis results of the plurality of input images.

[0043]

[16] An image processing device according to any one of [1] to

[15] , wherein:

[0044] When outputting an output image including an input image having an image area and a blank area and text, the processor performs the following processing:

[0045] Determine whether the configuration can be determined in a blank area of ​​the input image based on the text;

[0046] If it is determined that the arrangement cannot be determined in the blank area in the input image, changing the layout of the image area in the input image; and

[0047] In the output image including the input image whose layout is changed, the arrangement is determined in the blank area.

[0048] Furthermore, the above-mentioned purpose can also be achieved by the image processing method described in any one of the following

[17] to

[19] .

[0049]

[17] An image processing method, which is an image processing method in which a processor processes an input image, and the image processing method comprises the following steps:

[0050] The processor obtains text associated with the input image;

[0051] The processor determines a first object corresponding to the text from among the objects included in the input image; and

[0052] The processor determines, based on the input image and the first object, the arrangement of text in the output image based on the input image.

[0053]

[18] The image processing method according to

[17] , wherein:

[0054] In the step of acquiring text, the processor generates text according to the analysis result of the input image.

[0055]

[19] The image processing method according to

[18] further comprises the following steps:

[0056] The processor assigns configuration-related evaluation values ​​to a plurality of regions in the input image.

[0057] In the step of determining the arrangement, the processor determines the arrangement based on the evaluation values ​​of the plurality of regions.

[0058] Furthermore, a program product according to one embodiment of the present invention includes a program for causing a computer to implement each step included in the image processing method described in any one of

[17] to

[19] above.

[0059] Furthermore, a recording medium according to one embodiment of the present invention is a computer-readable recording medium, which records a program for causing a computer to implement each step included in the image processing method described in any one of

[17] to

[19] above.

[0060] Effects of the Invention

[0061] According to one embodiment of the present invention, there is provided an image processing device, an image processing method, a program product, and a recording medium capable of outputting an image in which text related to an input image is assigned to an appropriate position in consideration of a subject in the input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a diagram showing a usage example of the image processing device according to the first embodiment of the present invention.

[0063] Figure 2 This is a diagram showing a flow from acquiring an input image to outputting an output image.

[0064] Figure 3 It is a diagram showing the hardware configuration of the image processing device according to the first embodiment of the present invention.

[0065] Figure 4 It is an explanatory diagram regarding the function of the image processing device according to the first embodiment of the present invention.

[0066] Figure 5A It is a diagram showing a first example of an output image according to the first embodiment of the present invention.

[0067] Figure 5B It is a diagram showing a second example of the output image according to the first embodiment of the present invention.

[0068] Figure 5C It is a diagram showing a third example of the output image according to the first embodiment of the present invention.

[0069] Figure 6 This is a diagram showing an example of the image processing flow according to the first embodiment of the present invention.

[0070] Figure 7 It is an explanatory diagram of a modified example of the first embodiment of the present invention.

[0071] Figure 8 It is an explanatory diagram of another modified example of the first embodiment of the present invention.

[0072] Fig. 9 It is a diagram showing a usage example of the image processing device according to the second embodiment of the present invention.

[0073] Fig.10 It is a diagram showing a usage example of the image processing device involved in the third embodiment of the present invention.

[0074] Fig.11 It is a diagram showing an input image and an output image in the third embodiment of the present invention.

[0075] Fig.12 It is an explanatory diagram regarding the functions of the image processing device according to the third embodiment of the present invention.

[0076] Fig.13 This is a diagram for explaining changes to the image layout.

[0077] Fig.14 It is a diagram showing a modified example of the output image. DETAILED DESCRIPTION

[0078] Hereinafter, specific embodiments of the present invention will be described.

[0079] In addition, for the sake of convenience, the following description is sometimes made from the perspective of a GUI (Graphic User Interface). Furthermore, the basic data processing technologies (communication / transmission technology, data acquisition technology, data recording technology, data processing / analysis technology, machine learning technology, image processing technology, and visualization technology, etc.) used to implement the present invention are well-known technologies, and therefore the description thereof is omitted.

[0080] Furthermore, in this specification, the concept of "device" includes a single device that performs a specific function, and also includes a combination of multiple devices that exist independently of each other and cooperate (coordinate) to perform a specific function.

[0081] Furthermore, in this specification, a “user” is a user of the image processing apparatus of the present invention, and specifically, for example, a person who uses an output image (described later) obtained by the function of the image processing apparatus of the present invention.

[0082] Furthermore, in this specification, "person" refers to the subject that performs a specific behavior, including individuals, groups, legal persons such as enterprises, and groups, and further, it may also include computers and devices that constitute artificial intelligence (AI). Artificial intelligence is the use of hardware resources and software resources to achieve intelligent functions such as reasoning, prediction and judgment. The algorithm of artificial intelligence is arbitrary, such as expert system, case-based reasoning (CBR), Bayesian network or subsumption architecture.

[0083] <<First embodiment of the present invention>>

[0084] In the first embodiment of the present invention (hereinafter referred to as the first embodiment), Figure 1 As shown, the image processing device of the present invention adds text to an image input to the image processing device (hereinafter, input image Pi), and outputs an image including the input image and the text (hereinafter, output image Po). The user can use the output image Po.

[0085] Here, assigning (arranging) text to an image means converting the text into an image (text image) and including the text in the output image as a part of the output image.

[0086] The "image" in the present invention is composed of a plurality of pixels, represented by the grayscale values ​​of the plurality of pixels, and includes at least one or more subjects. Furthermore, digital image data (hereinafter referred to as image data) defining the image at a set resolution is generated by compressing the data recording the grayscale value of each pixel in a predetermined compression method. Examples of the types of image data include irreversibly compressed image data in the form of JPEG (Joint Photographic Experts Group), and reversibly compressed image data in the form of GIF (Graphics Interchange Format) or PNG (Portable Network Graphics).

[0087] In the first embodiment, the image input method is not particularly limited, and includes, for example, inputting image data of an image captured by a camera or other photographic device, and inputting read data obtained by reading an existing photo with a scanner or the like. Furthermore, when a photographic device is mounted on the image processing device, the image can be input by capturing an image with the photographic device and acquiring image data. Furthermore, the image can also be input by downloading image data from an external device or a Web server or the like via a communication network.

[0088] Furthermore, the input image Pi may be a developed image obtained by developing a RAW image, a corrected image obtained by performing a predetermined correction process on a developed image, or an edited image obtained by editing and processing a developed image or a corrected image.

[0089] The edited image as the input image may be an image formed by arranging one or more images in a predetermined layout (see Fig.10 and 11 ). That is, the input image Pi may be an image having one or more image regions and blank regions other than the image regions. The image region is, for example, a region containing a subject, and the blank region is a region not containing a subject. In addition, a region containing patterns, patterns, designs, illustrations, and other decorative objects, for example, can be regarded as a blank region as long as text can be written in the region.

[0090] In the first embodiment, the output method of the image is not particularly limited, and includes, for example, displaying the image on a display or monitor, printing the image, sending the image to other users, and providing the image as a commodity. The image as a commodity includes: a collage image generated by synthesizing and decorating with other images, an edited image in which one or more images (image areas) are arranged in a predetermined layout, and a booklet consisting of multiple pages on which images are recorded, such as a photo album and a photo book. In addition, in the method of providing the image as a commodity, in addition to the method of providing the image by printing it on a medium such as paper, it may also include a method of providing image data (digital data) in the form of an electronic commodity without printing it on a medium. In addition, in the method of providing the printed image, it may include sending a medium such as paper printed with the image to the requester as a message card or postcard. Furthermore, in the method of outputting the image, it may include publishing the image on SNS (Social Networking Service) for public disclosure.

[0091] In the first embodiment, the output image including an input image Pi and text Tx and not including a blank area described later is described as an example. Figure 1 As shown in FIG. 1 , an image in which text Tx is arranged in the input image Pi is output as an output image Po.

[0092] The text Tx is text related to the input image Pi, specifically, character information representing the content corresponding to the subject in the input image, such as comments and messages about the subject, descriptions related to the subject, and lines spoken by the subject. The text is composed of a sentence or article including one or more words, and when the article is segmented, each of the multiple sentences segmented is equivalent to the text. Furthermore, the text may include mimetic words, onomatopoeia, and interjections.

[0093] In the first embodiment, the text Tx can be input by the user, or it can be automatically generated by the image processing device. In the case of automatically generating text, the image processing device analyzes the input image Pi and generates text based on the analysis result. Analyzing an image refers to, for example, determining the features of the image. The features of the image are information related to the image quality of each area of ​​the image, the grayscale values ​​of the pixels contained in each area, and the information of the subject inferred from this information. The information of the subject may include the type of the subject, the state of the subject, the position of the subject in the image, the facial expression when the subject is a person, etc.

[0094] Furthermore, the image feature is preferably a feature that can be quantified, vectorized, or tensorized. In this case, the image analysis result becomes a feature of the image quantified, vectorized, or tensorized, that is, a feature amount.

[0095] Here, the subject refers to the people, animals, objects, and background contained in the image. In addition, the concept of subject may also include places, scenes (e.g., dawn or dusk, sunny days, etc.) and themes (e.g., travel, dining, sports events, etc.) that appear in the image. In addition, the image may be a landscape image, that is, the subject contained in the image is only the landscape, and the image as a whole may represent the landscape as the subject.

[0096] The technology for automatically generating text based on the analysis results of an image is not particularly limited. As an example, the correspondence between the image analysis results and the text content can be stored as data, as in the technology described in Patent Document 1, and the text corresponding to the image analysis results can be selected (generated) with reference to the correspondence. Alternatively, machine learning can be implemented to construct a learning model for text generation, and text related to the image can be generated by inputting an image into the learning model. The above-mentioned learning model, for example, determines the features of the input image and generates text corresponding to the features, and can be constructed by implementing machine learning using images acquired in the past and text assigned to the image as learning data.

[0097] Furthermore, in the first embodiment, the image processing device arranges (assigns) the text Tx generated by the above-mentioned method to a position in the output image Po determined according to the subject included in the input image Pi. Figure 2 As shown in the figure, after the image processing device analyzes the input image Pi and generates the text Tx, it determines the subject (hereinafter referred to as the first subject) corresponding to the generated text Tx from the subjects included in the input image Pi. The first subject is a subject that has a high correlation with the text Tx among one or more subjects included in the input image Pi. Specifically, for example, a subject that makes a sound represented by the text, a subject that is in a state represented by the text, a subject that performs an action represented by the text, or a subject that participates in an event or activity represented by the text is equivalent to the first subject.

[0098] Moreover, if Figure 2 As shown, the image processing device determines the arrangement (strictly speaking, the arrangement position) of the text Tx in the output image Po based on the input image Pi and the determined first subject, and outputs the output image Po in which the text Tx is arranged at the position.

[0099] As described above, the image processing device according to the first embodiment determines the first subject corresponding to the generated text Tx in the input image Pi, and outputs the output image Po in which the text Tx is arranged at a position corresponding to the determined first subject. Thus, the text Tx can be arranged at an appropriate position in the output image Po in consideration of the subject (first subject) in the input image Pi.

[0100] Specifically, for example, the text can be arranged at a position such as the periphery of the first subject where the first subject and the text can be easily associated. This effect is particularly effective when there are multiple subjects in the input image. Specifically, it is possible to suppress the arrangement of the text near a subject that has a low association with the text content. In addition, in the first embodiment, as described above, since the arrangement of the text is automatically determined by the function of the image processing device, it is not necessary for the user to determine the arrangement of the text, and the time and effort can be omitted.

[0101] In addition, the technology for determining the subject (first subject) corresponding to the text and determining the configuration of the text based on the determined first subject and the input image is not particularly limited. As an example, the correspondence between the text content and the configuration position of the text and the first subject can be stored as data in advance, and the first subject can be determined with reference to the correspondence, and the configuration of the text can be determined in a manner corresponding to the first subject.

[0102] Alternatively, machine learning can be implemented to construct a learning model for position determination, and an image and text are input to the learning model to determine the configuration of the text in the image. According to the above learning model, the relevance of the subject in the input image to the input text is evaluated, and the first subject (specifically, the position and range of the first subject in the image, etc.) is determined based on the evaluation result. Then, through the above learning model, the configuration of the text in the output image is determined based on the first subject and in accordance with established rules. This learning model can be constructed by implementing machine learning using a text-free image acquired in the past and a text-containing image in which text is configured in a corresponding association with the subject in the image as learning data.

[0103] Furthermore, the above-mentioned machine learning may be implemented in stages. Specifically, a first learning for constructing a learning model for determining the first subject and a second learning for constructing a learning model for determining the arrangement of texts based on the first subject may be implemented separately.

[0104] Furthermore, in the first learning, the Attention mechanism of deep learning can be applied to construct a learning model for determining the first subject taking into account the process of generating text based on the aforementioned text generation learning model. In this case, when the text to be assigned to the image is generated using the text generation learning model, it is possible to reflect which area of ​​the image is analyzed to generate the text when determining the first subject.

[0105] [Configuration Example of Image Processing Device]

[0106] Next, refer to Figure 3 , a configuration example of an image processing device (hereinafter, image processing device 10) according to the first embodiment is described.

[0107] The image processing device 10 is composed of a computer used by a user, specifically a client terminal, specifically, a smartphone, a tablet terminal, or a notebook PC (Personal Computer), etc. In addition, the image processing device 10 is not limited to a computer owned by the user, and may be composed of a terminal such as a store-installed terminal that is not owned by the user but can be used by inputting a code or password or paying an amount when visiting a store, etc.

[0108] In addition, the following description will take as an example a case where the image processing device 10 is constituted by a computer owned by a user, specifically, a smartphone.

[0109] like Figure 3 As shown, the computer constituting the image processing apparatus 10 includes a processor 10a, a memory 10b, a communication interface 10c, a storage device 10d, an input device 10e, and an output device 10f.

[0110] The processor 10a is composed of, for example, a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a TPU (Tensor Processing Unit), or an ASIC (Application Specific Integrated Circuit).

[0111] The memory 10 b is composed of, for example, a semiconductor memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory).

[0112] The communication interface 10c may be constituted by, for example, a network interface card or a communication interface board, etc. The computer constituting the image processing apparatus 10 can communicate with other devices connected to a communication network such as the Internet and a mobile communication line via the communication interface 10c.

[0113] The storage device 10d is composed of, for example, a flash memory, an HDD (Hard Disc Drive), an SSD (Solid State Drive), an FD (Flexible Disc), an MO disk (Magneto-Optical disc), a CD (Compact Disc), a DVD (Digital Versatile Disc), an SD card (Secure Digital card) or a USB memory (Universal Serial Bus memory).

[0114] The storage device 10d may be built into the computer main body constituting the image processing device 10, or may be installed in the computer main body in an external form. Alternatively, the storage device 10d may be constituted by NAS (Network Attached Storage) or the like. Furthermore, the storage device 10d may be an external device capable of communicating with a computer constituting the image processing device 10 via a communication network, such as an online storage device or a database server.

[0115] The input device 10e is a device that receives input operations from a user, and is composed of, for example, a touch panel, etc. The input device 10e also includes a camera such as a built-in camera of a smartphone, a microphone for collecting sound, and the like.

[0116] The output device 10f is constituted by, for example, a display or the like.

[0117] In addition, a program for an operating system (OS) and an application program for executing image processing are installed as software in the computer constituting the image processing device 10. By reading and executing these programs by the processor 10a, the computer constituting the image processing device 10 performs its functions, specifically, performs a series of processes related to the output of the image including the above-mentioned text.

[0118] refer to Figure 4 , the structure of the image processing device 10 is re-explained from the perspective of its functions. Figure 4 As shown in FIG. 1 , the image processing device 10 includes a receiving unit 21, an acquiring unit 22, a determining unit 23, a determining unit 24, a generating unit 25, and an output unit 26. These functional units are implemented by the processor 10a of the image processing device 10 executing the above-mentioned image processing application and cooperating with other hardware devices of the image processing device 10. In addition, some functions can be implemented using artificial intelligence (AI). The following describes each functional unit.

[0119] (Receiving Department)

[0120] The receiving unit 21 receives input of an image to which text is to be assigned. As described above, the image input method is not particularly limited, and for example, a user may use a camera of a smartphone constituting the image processing device 10 to capture a subject within the field of view. In this case, the receiving unit 21 receives input of image data of the photographed image. In addition, the receiving unit 21 may also receive input of an image by downloading image data from an external device (e.g., a server) via a network.

[0121] (Acquisition Department)

[0122] The acquisition unit 22 acquires text Tx related to the input image Pi, which is the image received and input by the receiving unit 21. In the first embodiment, the acquisition unit 22 can acquire the text Tx by two methods.

[0123] The first acquisition method is a method for analyzing an input image Pi and generating text Tx based on the analysis result. Specifically, as described above, the text Tx corresponding to the analysis result of the input image Pi can be selected (generated) with reference to data indicating the correspondence between the image analysis result and the text content. Alternatively, a learning model for text generation constructed by machine learning can be applied to the input image Pi to generate text Tx related to the input image Pi.

[0124] Furthermore, when generating text by the first acquisition method, the acquisition unit 22 can generate text Tx based on the area selected by the user in the input image Pi. To explain in more detail, when generating text based on the analysis result of the input image Pi, the acquisition unit 22 can set the area selected by the user in the input image Pi as the focus area, and generate text Tx by giving more importance to the analysis result of the focus area than other areas. Thus, when generating text Tx, the user's intention can be reflected, for example, the subject selected by the user can be used as a text assignment object to generate text Tx.

[0125] In addition, the selection of the region (region of interest) by the user is performed through the input device 10e. For example, the input image Pi can be displayed on the touch display of the smartphone constituting the image processing device 10, and the user can touch the region of interest in the displayed input image Pi. The range selected as the region of interest can be determined arbitrarily, and a subject included in the input image Pi can be set as the region of interest, or a part of the input image touched by the user can be set as the region of interest.

[0126] Furthermore, the region of interest may be selected by the acquisition unit 22 (specifically, the processor 10a of the image processing device 10 constituting the acquisition unit 22). That is, the acquisition unit 22 may analyze the input image Pi and select the region of interest based on the analysis result. In this case, the acquisition unit 22 may generate the text Tx based on the region of interest selected by the acquisition unit 22.

[0127] Furthermore, when the input image Pi is a photographic image and the image data of the photographic image includes supplementary information, the acquisition unit 22 can generate text Tx based on the analysis result of the input image and the supplementary information. The supplementary information is various information related to the photographic image, represented by the photographing location, photographing date and time, photographer, photographing conditions, and information of the camera used in the photographing, and is, for example, tag information recorded in the image data file in the form of Exif (Exchangeable image file format).

[0128] In the method of generating text Tx based on incidental information, for example, it is conceivable to use a previously acquired image, incidental information of the image, and text assigned to the image as learning data to perform machine learning and use a learning model for text generation obtained as a result.

[0129] As described above, in the first embodiment, the text Tx can be generated by referring to the incidental information of the input image together with the analysis result of the input image Pi. Thus, a more appropriate text Tx can be generated taking into account the shooting location and shooting date and time of the input image.

[0130] The second acquisition method is a method for acquiring text based on the input operation performed by the user through the input device 10e. Specifically, the acquisition unit 22 acquires the text Tx representing the input content based on the character input performed by the user through the touch panel or the voice input performed through the sound pickup microphone. In this way, the text Tx corresponding to the user's intention can be acquired and configured (assigned) to the output image Po.

[0131] In addition, the acquisition unit 22 may acquire the text Tx by combining two or more of generating text based on the analysis result of the input image Pi, generating text based on the focus area in the input image Pi, generating text based on the supplementary information, and acquiring text input by the user.

[0132] (Confirmation Department)

[0133] The determination unit 23 determines the first subject corresponding to the text Tx acquired by the acquisition unit 22 from the subjects included in the input image Pi, and specifically determines the position (coordinate position) and range of the first subject in the input image Pi. In the first embodiment, the input image Pi includes at least one subject among a person, an object, an animal, and a background. The determination unit 23 determines the first subject from the at least one subject.

[0134] If listed Figure 2 Taking the case shown as an example, an input image Pi including a plurality of company employees attending a farewell party is obtained as the subject, and a text "Thank you for your care" is generated based on the features obtained by analyzing the image. In this case, the resigned company employee (the person reflected in the center of the input image Pi) is determined as the first subject.

[0135] In addition, the first subject identified in one input image may be one or more. In the following, unless otherwise specified, the first subject identified in one input image Pi is one.

[0136] As mentioned above, the method for determining the first subject is not particularly limited. For example, the first subject can be determined by referring to data representing the correspondence between the text and the first subject. Alternatively, the first subject corresponding to the text Tx can be determined by applying a learning model for subject determination constructed by implementing machine learning to the input image Pi and the acquired text Tx. In addition, the Attention mechanism of deep learning can be applied to the above-mentioned learning model to determine the area (focus area) in the input image Pi that is important when generating text based on the input image Pi, and the first subject can be determined based on the determined focus area. In this way, the first subject corresponding to the text Tx can be determined more appropriately.

[0137] (Decision Department)

[0138] The determination unit 24 determines the arrangement (strictly speaking, the arrangement position) of the text Tx acquired by the acquisition unit 22 in the output image. In the first embodiment, the determination unit 24 determines the arrangement of the text based on the input image and the first subject determined by the determination unit 23. To be more specific, in the first embodiment, the arrangement of the text is determined in the input image as follows.

[0139] First, the decision unit 24 determines whether the text Tx can be arranged around the first subject in the input image Pi in a manner that does not overlap with the first subject. Here, when the text Tx cannot be arranged around the first subject in a manner that does not overlap with the first subject, the decision unit 24 determines to arrange the text so as to overlap with the first subject. In this case, Figure 5A As shown, it is sufficient to determine a configuration that does not overlap with the specific part of the first subject. Here, the specific part refers to a predetermined part of the first subject, specifically, a main part of the first subject, and is a part that is required to be clearly shown in the output image Po. For example, in the case where the first subject is a person, the specific part is the upper body of the person, especially the face. In addition, when the first subject is a person, the specific part can be the whole body of the person. And, in the case where the first subject is an object, the specific part is a part of the object that is difficult to identify when it is assumed to overlap with the text.

[0140] In addition, when determining a specific part, a known technology can be used. Specifically, a technology for detecting a subject in an image (image analysis technology) can be applied to identify each part of the subject in the image and determine which part each part is. On this basis, importance (priority) is assigned to each determined part according to a predetermined benchmark or rule, and the specific part is determined based on the importance (priority) assigned to each part.

[0141] As described above, when the text cannot be arranged around the first subject so as not to overlap the first subject, the position not overlapping the specific part of the first subject is determined as the arrangement position of the text. This can prevent the specific part of the first subject from being blocked by the text Tx and being invisible.

[0142] In addition, when it is difficult to arrange the text Tx so as not to overlap with the specific part of the first subject (for example, when the input image is a zoomed image of the face of a person as the first subject), the determination unit 24 may determine the arrangement of the text Tx so as to overlap with the specific part. Figure 5B As shown, the arrangement of the text Tx is determined so as to minimize the influence of the text Tx on the specific part. The arrangement position of the text Tx can be determined by artificial intelligence (AI) or according to a predetermined rule.

[0143] On the other hand, when it is possible to arrange text around the first subject in a manner that does not overlap with the first subject, the decision unit 24 determines whether it is possible to arrange text around the first subject in the input image in a manner that does not overlap with both the first subject and the second subject. The second subject refers to a subject that is different from the first subject among the subjects included in the input image and has a lower correlation with the text than the first subject. Furthermore, the second subject may be, for example, a subject of the same type (same category) as the first subject. Specifically, when one of the multiple persons included as subjects in the input image is the first subject, the second subject is a person among the multiple persons who is different from the first subject.

[0144] Furthermore, when the text can be arranged around the first subject so as not to overlap with both the first subject and the second subject, as in Figure 5C As shown, the determination unit 24 determines the arrangement of text that does not overlap with the first subject and the second subject in the vicinity of the first subject in the input image Pi. Thus, the text Tx can be arranged at a position where the correspondence with the first subject is easy to grasp, and the first subject and the second subject can be prevented from being blocked by the text Tx and invisible.

[0145] When it is impossible to arrange the text Tx around the first subject so as not to overlap with both the first subject and the second subject, the decision unit 24 decides to arrange the text so as to overlap with the second subject but not overlap with the first subject around the first subject in the input image Pi. Thus, the text Tx can be arranged at a position where the correspondence between the text Tx and the first subject can be easily understood.

[0146] Furthermore, when the user selects a focus area in the input image Pi when acquiring text, when the determination unit 24 determines the arrangement of the text Tx according to the above-mentioned method, the arrangement of the text Tx can be determined according to the area selected by the user as the focus area. Specifically, the arrangement of the text Tx can be determined near the focus area selected by the user, or conversely, the arrangement of the text Tx can be determined avoiding the focus area.

[0147] (Generation Department)

[0148] The generating unit 25 generates an output image Po corresponding to the input image Pi based on the text Tx acquired by the acquiring unit 22 and the arrangement of the text determined by the determining unit 24. The output image Po of the first embodiment is an image including the text Tx and the input image Pi. Specifically, Figure 1As shown, the image is an image in which the text Tx is arranged (given) in the arrangement determined by the determination unit 24 in the input image Pi. To explain in more detail, the generation unit 25 converts the text Tx into a text image and superimposes the text image on the input image Pi to generate the output image Po.

[0149] (Output section)

[0150] The output unit 26 outputs the output image Po generated by the generation unit 25. As mentioned above, the method of outputting the image is not particularly limited. For example, the output image Po can be displayed on the display (output device 10f) of the computer constituting the image processing device 10. In addition, control data for printing the output image Po can also be generated, and a printer not shown in the figure can be controlled according to the control data to print out the output image Po. Alternatively, for the purpose of sharing the output image Po with other users, the image data of the output image Po can be sent from the computer constituting the image processing device 10 to the terminal used by the other user. Alternatively, for the purpose of publishing the output image Po through SNS, the image data of the output image Po can be uploaded to an SNS server.

[0151] [Example of the Image Processing Method According to the First Embodiment]

[0152] Next, as an example of the operation of the image processing device 10 in the first embodiment, an image processing flow using the same device is described. The image processing method of the present invention is used in the image processing flow described below. That is, each step in the image processing flow described below is equivalent to a constituent element of the image processing method of the present invention.

[0153] The following process is merely an example, and part of the steps in the process may be deleted, a new step may be added, or the execution order of two steps in the process may be replaced without departing from the spirit of the present embodiment.

[0154] Each step in the image processing flow according to the first embodiment is performed by the processor 10a included in the image processing device 10 according to Figure 6 That is, in each step of the image processing flow, the processor 10a executes the data processing specified in the image processing application. Figure 6 The processing corresponding to each step in .

[0155] Specifically, in the image processing flow of the first embodiment, the processor 10a first executes an input receiving process (S001). In the input receiving process, the processor 10a receives an input of an image to which text is added in this flow, in other words, acquires an input image Pi to be processed.

[0156] Next, the processor 10a performs an acquisition process (S002) of acquiring text Tx related to the input image Pi. In the acquisition process, the processor 10a can acquire text according to the input operation of the text content performed by the user. In addition, in the acquisition process, the processor 10a can analyze the input image Pi and generate text Tx according to the analysis result.

[0157] Furthermore, during the acquisition process, the user may be allowed to select a focus area in the input image Pi. In this case, during the acquisition process, the processor 10a generates text Tx based on the focus area selected by the user, specifically, giving more importance to the analysis results of the focus area than to the analysis results of areas other than the focus area.

[0158] Furthermore, when the input image Pi is a photographic image and the image data of the photographic image includes supplementary information such as an Exif tag, in the acquisition process, the processor 10a may generate the text Tx based on the analysis result of the input image Pi and the supplementary information.

[0159] Next, the processor 10a performs a determination process (S003) of determining the first subject corresponding to the text Tx acquired in the acquisition process from the subjects included in the input image Pi. Here, the input image Pi includes at least one subject among a person, an object, an animal, and a background. In the determination process, the processor 10a determines the first subject from the at least one subject, specifically, determines the position (coordinate position) and range of the first subject in the input image Pi.

[0160] Next, the processor 10a executes a determination process (S004) for determining the arrangement of the text Tx in the output image Po. In the determination process, the processor 10a determines the arrangement of the text Tx based on the first subject determined in the determination process.

[0161] Specifically, as the configuration of the text Tx, the processor 10a may determine a configuration that does not overlap with the first subject in the periphery of the first subject in the input image Pi. In this case, the configuration of the text Tx that does not overlap with a specific part (e.g., a face) of the first subject may be determined in the periphery of the first subject. Furthermore, as the configuration of the text Tx, the processor 10a may also determine a configuration that does not overlap with both the first subject and the second subject in the periphery of the first subject in the input image Pi.

[0162] Next, the processor 10a executes a generation process of generating an output image Po corresponding to the input image Pi (S005). In the generation process, the processor 10a generates the output image Po by adding the text Tx acquired in the acquisition process to the input image Pi in the arrangement determined in the determination process.

[0163] Then, the processor 10a performs output processing (S006) to output the generated output image Po. By performing the output processing, the output image Po composed of the input image Pi with the text Tx arranged in the image is output. Specifically, for example, the output image Po is displayed on a display, etc., printed with a printer, or transmitted for the purpose of publicizing or sharing the image data of the output image.

[0164] When the series of processes described above are completed, the image processing flow according to the first embodiment is completed. The image processing flow is performed each time an input image is received, that is, each time a new input image Pi is received, the processor 10a repeatedly executes the series of processes described above.

[0165] [Modification of the first embodiment]

[0166] The first embodiment of the present invention is not limited to the above-described embodiment, and for example, the following modified examples can be considered. The modified examples are described below. In addition, the modified examples are described below with respect to the differences from the above-described first embodiment.

[0167] (Regarding the first variant)

[0168] In the first embodiment described above, the image processing device 10 obtains one text Tx for one input image Pi, in other words, one text Tx is included in one output image Po. However, this is not limited to this. As a variation of the first embodiment, multiple texts Tx may be obtained for one input image Pi. That is, as an embodiment of the present invention, Figure 7 As shown in FIG. 1 , an embodiment (hereinafter referred to as Embodiment 1A) can be considered in which a plurality of texts Tx are added to one input image Pi to generate an output image Po.

[0169] In the first embodiment, as a method for obtaining a plurality of texts Tx related to an input image Pi, for example, a learning model (text generation model) that generates texts Tx by analyzing the input image Pi with specifications capable of outputting a plurality of texts Tx can be cited. That is, a text generation model that generates a plurality of texts from an input image Pi can be constructed by machine learning, and the plurality of texts Tx can be obtained using the model. The number of texts Tx generated using the text generation model can be specified by the user, or can be automatically set on the model side according to the content (subject) of the input image Pi.

[0170] Furthermore, the text generation model may also provide multiple candidates for the prompted text Tx based on the analysis results of the input image Pi (specifically, the feature values ​​of the input image). Furthermore, two or more candidates selected by the user from the multiple suggested text candidates may be adopted (acquired) as multiple texts Tx. In this case, the text generation model may evaluate the appropriateness (likelihood) of each of the multiple suggested text candidates as a text, and prompt the evaluation value together with the candidate. The evaluation value is, for example, equivalent to the output probability (reliability of the candidate) calculated by applying a softmax function to the candidate output by the learning model. Furthermore, the user may select more than two candidates based on the evaluation values ​​of the appropriateness calculated for each candidate. Furthermore, candidates whose evaluation values ​​are in the top n (n is a natural number greater than 2) may be automatically selected from multiple candidates.

[0171] As another method of acquiring multiple texts related to one input image, a method of dividing a text composed of one article into multiple words (phrases) can be cited. That is, multiple words (phrases) divided into sections of one text can be acquired as multiple texts Tx. For example, suppose that the text acquired (generated) in the above acquisition process is "Happy Birthday on July 21st!!". In this case, Figure 7 As shown, the above text is divided into two texts, "July 21" and "Happy Birthday!!" By setting in this way, a long text can be divided into a plurality of shorter texts and each divided text can be added to the input image Pi.

[0172] As described above, in the first embodiment, a plurality of texts Tx are obtained for one input image Pi. In this case, the processor 10a of the image processing device 10 determines the first subject in the input image Pi for each of the plurality of texts Tx, and determines the arrangement of the texts in the output image Po. At this time, the processor 10a (specifically, the determination unit 24) may determine the first subject in the input image Pi for each of the plurality of texts Tx, such as Figure 7 As shown, in the first subject ( Figure 7 Thus, multiple texts Tx can be assigned to improve the utilization value of the output image Po, and it is possible to avoid the difficulty in seeing each text Tx due to overlapping of texts.

[0173] In addition, the first subject determined for each of the plurality of texts Tx may be the same or different between the texts.

[0174] (Regarding the second modification)

[0175] In the first embodiment described above, when determining the configuration of the text Tx in the output image Po, the configuration is determined based on the area where the subject in the input image Pi exists. Specifically, the configuration of the text Tx is determined in an area around the first subject in the input image Pi that does not overlap with the first subject, an area that does not overlap with a specific part of the first subject, or an area that does not overlap with both the first subject and the second subject. At this time, the appropriateness or inappropriateness of multiple areas in the input image Pi as areas for configuring the text Tx can be evaluated, and the configuration of the text Tx can be determined based on the evaluation results. That is, as an embodiment of the present invention, the following embodiment can be considered: evaluation values ​​based on the analysis results of the input image Pi are obtained for multiple areas in the input image, and the configuration of the text is determined based on the evaluation values ​​of each area (hereinafter, the 1B embodiment).

[0176] The method for obtaining evaluation values ​​for multiple regions in the input image is not particularly limited. A specific example of the method for obtaining evaluation values ​​is described. The subject in the input image is detected using a known subject detection technology and segmentation technology. Here, when multiple subjects are detected, Figure 8 As shown, the input image Pi is divided into multiple regions according to multiple subjects. Furthermore, when a person is included in the multiple subjects, a known face extraction technique is applied to the region where the person exists to extract the face (especially the face facing forward), and the region including the face is determined.

[0177] Moreover, if Figure 8 As shown, for each of the multiple regions, an evaluation value ( Figure 8 The evaluation value is an evaluation value related to the configuration of the text Tx, specifically, a score related to the appropriateness of the region where the text Tx is configured. The score can be set to be smaller as the relevance to the text Tx is higher, and conversely, it can be set to be larger as the relevance to the text Tx is higher. In addition, for regions where people with high relevance to the text Tx exist, Figure 8 As shown, the evaluation value assigned to the region where the face of the person exists (face region) can be set to a value with a higher correlation than the evaluation value assigned to the region other than the face region.

[0178] In addition, when evaluating the relevance between each region in the input image Pi and the text Tx, the learning model used when generating the text can be applied. Specifically, for each region of the input image, the degree of importance attached to the region when generating the text is inferred, and the relevance with the text Tx is evaluated based on the inferred degree of importance, and the evaluation value is assigned to the region.

[0179] Furthermore, as another example of a method of assigning an evaluation value, an evaluation value may be assigned according to the importance of the subject contained in each region, regardless of the relevance of the text Tx. Specifically, the input image Pi is analyzed in the same order as the above order to detect the subject in the input image Pi, and when multiple subjects are detected, the importance is set for each subject. The importance is set according to a predetermined rule, for example, the correspondence between the type of subject and the importance is determined in advance according to the type of each subject, and is set according to the correspondence. Furthermore, for multiple regions in the input image Pi, an evaluation value corresponding to the importance of the region is obtained (calculated). At this time, the evaluation value may be larger as the importance is higher, or conversely, it may be smaller as the importance is higher. Furthermore, with respect to a region where a subject with high importance (for example, a person) exists, the evaluation value assigned to the region (facial region) where the face of the person exists may be set to a value when the importance is higher than the evaluation value assigned to the region other than the facial region.

[0180] The processor 10a (determining unit 24 in detail) of the image processing device 10 determines the arrangement of the text Tx based on the evaluation values ​​obtained for each region in the input image Pi. Specifically, for example, when the importance of the subject is higher and the evaluation value is smaller, the arrangement of the text Tx is determined in the region with the higher evaluation value.

[0181] As described above, in the first embodiment, the processor 10a of the image processing device 10 determines the arrangement of the text based on the evaluation values ​​obtained for the plurality of regions in the input image Pi. Thus, the arrangement of the text Tx can be appropriately determined taking into account the relevance between each subject and the text Tx or the importance of each subject.

[0182] To explain in more detail, in the 1B embodiment, the evaluation value of each area in the input image Pi is obtained in such a way that the possibility of determining the configuration of the text Tx in the area other than the object area in the input image Pi is lower than the possibility of determining the configuration of the text Tx in the area other than the object area. As a result, in the output image Po, the text Tx can be configured at a position that does not overlap with the object area in the input image Pi contained in the output image Po. Here, the object area refers to an area in the input image Pi that is required to avoid overlapping with the text Tx. Specifically, for example, an area where a highly important subject exists, especially when such a subject is a person, the area where the face of the subject is located corresponds to the object area.

[0183] <<Second embodiment of the present invention>>

[0184] In the first embodiment described above, one input image Pi is acquired, but it is also possible to acquire a plurality of input images Pi. This case will be described below as a second embodiment of the present invention (hereinafter, second embodiment).

[0185] In addition, in the second embodiment, the points different from the first embodiment will be mainly described, and the description of the points common to the first embodiment will be omitted.

[0186] In the second embodiment, the processor 10a of the image processing device 10 acquires a plurality of images as input images Pi. The image acquisition step is the same as that of the first embodiment. Furthermore, the processor 10a analyzes the plurality of input images Pi and generates a text Tx related to at least one of the plurality of input images Pi based on the analysis results of the plurality of input images Pi.

[0187] If we explain it in detail, Fig. 9 As shown in the figure, when two or more input images Pi are input, one input image Pi is a main image to which text is to be assigned, and the remaining input images Pi are auxiliary images for text generation. The processor 10a analyzes the main image and the auxiliary images respectively, and generates text Tx for the main image based on the analysis results of each image. The text generation order is the same as the first embodiment except that the analysis results of multiple images are used.

[0188] After generating the text Tx, the processor 10a specifies the first subject corresponding to the generated text Tx from the subjects included in the main image. The procedure for specifying the first subject is the same as that of the first embodiment.

[0189] Then, the processor 10a determines the arrangement of the text Tx in the output image Po including the main image according to the determined first subject. Fig. 9 As shown, the arrangement of the text Tx is preferably determined in an area around the first subject in the main image that does not overlap with the first subject, especially in an area that does not overlap with a specific portion of the first subject. In addition, the order of determining the arrangement of the text Tx is the same as that of the first embodiment.

[0190] Furthermore, the second embodiment is not limited to the case where the text assigned to the main image is generated based on a plurality of input images Pi including the main image. For example, the processor 10a of the image processing device 10 can simultaneously receive m (m is a natural number) mutually related input images Pi and generate text Tx related to each input image Pi in the same number of portions as the input images Pi based on the analysis results of each of the m input images Pi. That is, a text Tx can be assigned to each of the m input images Pi.

[0191] In addition, the m mutually related input images Pi may be images taken at the same photography location and at the same photography time, or images taken with the same photography theme (specifically, images obtained by photographing the same subject multiple times with changing photography times).

[0192] Furthermore, the second embodiment may include a case where the process of generating text Tx and the process of determining the arrangement of the text Tx are repeatedly performed each time the input image Pi is acquired (received). In this case, the text Tx is assigned to each of the acquired input images Pi.

[0193] As described above, in the second embodiment, text Tx can be generated based on the analysis results of the plurality of input images Pi. This effect is particularly effective when text is added to at least one of the plurality of images included in the creation of a photo album or a photo book.

[0194] In the second embodiment, when analyzing a plurality of input images Pi to obtain text, it is not necessary to use all of the plurality of input images Pi, and the plurality of input images Pi may include input images Pi that are not analyzed (input images Pi excluded from analysis targets).

[0195] <<Third embodiment of the present invention>>

[0196] In the above-described first embodiment, an image including a subject, or strictly speaking, an image consisting only of an image region including a subject, is used as the input image Pi.

[0197] On the other hand, Fig.10 As shown, it is possible to consider the case where an edited image generated by arranging one or more images (raw material images) on a layout image, that is, an image having an image area and a blank area, is used as the input image Pi. This case is described below as a third embodiment of the present invention (hereinafter, the third embodiment).

[0198] In addition, in the third embodiment, points different from the first embodiment will be mainly described, and description of points common to the first embodiment will be omitted.

[0199] As described above, the input image Pi in the third embodiment is an image having an image area Pa1 and a blank area Pa2, and can be generated, for example, by using image editing software. The image area Pa1 is an image containing a subject, and one or more of them are arranged in the input image Pi. The blank area Pa2 is an area outside the image area Pa1 in the input image Pi, and can be an area without texture, or an area with color or pattern added.

[0200] The following, such as Fig.11 As shown, the case where the input image Pi includes a plurality of image areas Pa1 and the blank area Pa2 is a white area without texture is taken as an example for description.

[0201] In the third embodiment, an output image including the input image Pi and the text Tx is output. Fig.11 As can be seen, the processor 10a of the image processing device 10 (hereinafter, image processing device 10X) according to the third embodiment determines the arrangement of the text Tx in the blank area Pa2 in the input image Pi included in the output image Po. To explain in more detail, the processor 10a analyzes the input image Pi, more specifically, analyzes the plurality of image areas Pa1 included in the input image Pi, in the aforementioned manner, and generates one or more texts Tx based on the respective analysis results.

[0202] After generating the text Tx, the processor 10a determines the first subject corresponding to the generated text Tx from the subjects of each of the plurality of image regions Pa1 included in the input image Pi. Furthermore, the processor 10a determines the configuration of the text Tx based on the determined first subject and the input image Pi. At this time, the processor 10a determines the image region Pa1 including the first subject from among the plurality of image regions Pa1 included in the input image Pi, for example, and determines the configuration of the text Tx in the blank region Pa2 adjacent to the image region Pa1.

[0203] Finally, if Fig.11 As shown, the processor 10a generates an output image Po in which text is arranged in the blank area Pa2, and outputs the generated output image Po. In this way, in the third embodiment, when the input image Pi has a blank area Pa2, the blank area Pa2 is effectively used to arrange the text Tx, thereby generating an edited image (output image Po) with text. This effect is particularly effective when text is added to images of each page constituting a postcard or a photo book.

[0204] Furthermore, the effect of the third embodiment is also useful in a case where it is difficult to arrange the text Tx in the image area Pa1, such as when zooming is performed and the face of a person as the subject is reflected throughout the entire image area Pa1.

[0205] In the third embodiment, the text Tx may be arranged in the image region Pa1 of the input image Pi. That is, in the third embodiment, the arrangement of the text Tx may be determined in at least one of the image region Pa1 and the blank region Pa2 of the input image Pi.

[0206] Furthermore, in the third embodiment, as Fig.12As shown in FIG. 1 , the image processing device 10X may further include a change unit 27 in addition to the above functions. The change unit 27 is a function for changing the layout of the image area Pa1 in the input image Pi. The layout of the image area Pa1 may include the number of image areas Pa1 arranged in the input image Pi, the size, arrangement position, outer edge shape (aspect ratio) of each image area Pa1, and the inclination of the image area Pa1 relative to the input image Pi (i.e., the rotation amount of the image area Pa1).

[0207] When the function of the change unit 27 is used, in the previous stage, the processor 10a obtains the text Tx related to the input image Pi, for example, generates the text Tx based on the analysis results of each image area Pa1 in the input image Pi. In the subsequent process, the change unit 27 determines whether the configuration of the text Tx can be determined in the blank area Pa2 in the input image Pi based on the generated text Tx.

[0208] Specifically, the change unit 27 determines the number of characters and size of the generated text Tx, and determines the position and size of the blank area Pa2 (hereinafter, the adjacent blank area) adjacent to the image area Pa1 containing the first subject corresponding to the text Tx in the input image Pi. At this time, the change unit 27 can determine whether the adjacent blank area exists in the input image Pi at the current time point. Based on the determination result, the change unit 27 determines whether the configuration of the text Tx can be determined in the blank area Pa2, more specifically, the adjacent blank area, in the input image Pi at the current time point.

[0209] When the arrangement of the text can be determined in the adjacent blank area in the input image Pi at the current time point, the processor 10 a (specifically, the determination unit 24 ) determines the arrangement of the text Tx in the adjacent blank area.

[0210] On the other hand, when the configuration of the text Tx cannot be determined in the adjacent blank area of ​​the input image Pi at the current time point, or when the adjacent blank area does not exist, such as Fig.13 As shown, the changing unit 27 changes the layout of the image area Pa1 in the input image Pi. The layout change may include, in addition to changing the number and arrangement of the image areas Pa1 in the input image Pi, moving, enlarging or shrinking, changing the aspect ratio, and trimming one or more image areas Pa1.

[0211] The layout change can be performed automatically according to a predetermined rule or according to a user's change operation. Alternatively, artificial intelligence (AI) can be applied to calculate a layout that ensures that there are enough adjacent blank areas to place the text Tx, and the input image Pi can be re-edited with the layout to change the position and size of the image area Pa1 in the input image Pi.

[0212] When the changing unit 27 changes the layout of the image area Pa1 in the input image Pi, the processor 10a (specifically, the determining unit 24) determines the arrangement of the text Tx in the output image Po including the input image Pi whose layout is changed. Fig.13 As shown, the processor 10a determines the arrangement of the text Tx in the blank area Pa2 in the input image Pi whose layout is changed in the output image Po, or more precisely, in the adjacent blank area.

[0213] According to the above configuration, when the layout of the image area Pa1 in the input image Pi cannot arrange the text Tx in the adjacent blank area, the layout can be appropriately reviewed so that the text Tx can be appropriately arranged in the adjacent blank area.

[0214] <<Other implementation methods>>

[0215] The specific embodiments of the present invention are described above, but the above embodiments are only an example listed for easy understanding of the present invention, and do not limit the present invention. That is, the present invention can be changed or improved from the embodiments described below as long as it does not deviate from its purpose. And, the present invention includes its equivalent. Furthermore, in the embodiments of the present invention, the above embodiments can be included in combination with one or more of the following variations.

[0216] (Variation example of output image)

[0217] In the above embodiment, the output image Po is generated by adding text Tx to the input image Pi. That is, except for the fact that the text Tx is added, the output image Po involved in the above embodiment is the same image as the input image Pi. However, this is not limited to this. Fig.14 As shown, the output image Po may include the input image Pi and the text Tx, and may have a blank area Pa3 in at least a portion of the periphery of the input image Pi in the output image Po.

[0218] When acquiring the output image Po, the processor 10a of the image processing device 10 receives the image input, acquires the text Tx related to the input image Pi, and determines the first subject from the subjects in the input image Pi in the same order as the above-mentioned embodiment. Fig.14 As shown in FIG. 1 , an image used as a backing paper to which the input image Pi is pasted (hereinafter referred to as a backing paper image Pd) is selected.

[0219] The backing image Pd is an image whose display size (viewing angle) is larger than that of the input image Pi, and together with the input image Pi and the text Tx (strictly speaking, the image of the text Tx), the output image Po is formed. Specifically, the output image Po is generated by compositing the input image Pi and the text Tx in such a manner that these images are arranged (overlapped) on the backing image Pd. As described above, the backing image Pd is an image whose display size is larger than that of the input image Pi, and therefore, Fig.14 As shown in FIG. 1 , a region in which the input image Pi is not arranged in the inserting paper image Pd included in the output image Po becomes a blank region Pa3.

[0220] Moreover, when generating the above-mentioned output image Po, the processor 10a of the image processing device 10 determines the configuration of the input image Pi and the text Tx in the output image Po. The configuration of the input image Pi can be determined according to the user's input operation, or it can be automatically determined by the processor 10a according to the content of the input image Pi (specifically, the subject in the input image Pi). The configuration of the text Tx is determined based on the first subject determined in the input image Pi, for example, in a manner that does not overlap with the first subject or a specific part thereof around the first subject. At this time, the configuration of the text Tx can be determined in at least one of the area in the output image Po where the input image Pi is configured and the blank area Pa3.

[0221] (Regarding Computer Constituting Image Processing Device)

[0222] In the above-mentioned embodiment, the image processing device of the present invention is constituted by a computer directly used by the user, such as a terminal (client terminal) owned by the user. However, it is not limited to this, and the image processing device of the present invention may also be constituted by a computer that can be indirectly used by the user, such as a server computer. Here, the above-mentioned server computer may be, for example, a server computer for cloud services, specifically, a server computer for ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service) or IaaS (Infrastructure as a Service). In this case, if the necessary information is input into the client terminal, the above-mentioned server computer performs various processing (calculations) according to the input information, and the calculation results are output on the client terminal side. That is, the functions of the server computer constituting the image processing device of the present invention can be utilized on the client terminal side.

[0223] (About the structure of the processor)

[0224] The processor included in the image processing device of the present invention includes various processors, and includes, for example, a general-purpose processor, namely, a CPU, which executes software (programs) to function as various processing units.

[0225] In addition, various processors include PLDs (Programmable Logic Devices) which are processors whose circuit configuration can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays).

[0226] Furthermore, various processors include processors such as ASIC (Application Specific Integrated Circuit) which have a circuit configuration specifically designed to execute specific processing, that is, dedicated circuits.

[0227] Furthermore, a functional unit of the image processing device of the present invention may be composed of one of the various processors described above. Alternatively, a functional unit of the image processing device of the present invention may be composed of a combination of two or more processors of the same or different types, such as a combination of multiple FPGAs or a combination of an FPGA and a CPU.

[0228] Furthermore, the plurality of functional units included in the image processing apparatus of the present invention may be constituted by one of various processors, or two or more of the plurality of functional units may be constituted by one processor.

[0229] Furthermore, as in the above-described embodiment, a mode may be adopted in which a processor is constituted by a combination of one or more CPUs and software, and the processor functions as a plurality of functional units.

[0230] Furthermore, for example, the following method may be adopted: a processor that realizes the functions of the entire system including multiple functional units in the image processing device of the present invention by using one IC (Integrated Circuit) chip, as represented by SoC (System On Chip). Furthermore, the hardware structure of the above-mentioned various processors may be a circuit (Circuitry) composed of circuit elements such as semiconductor elements.

[0231] Explanation of symbols

[0232] 10, 10X-image processing device, 10a-processor, 10b-memory, 10c-communication interface, 10d-memory, 10e-input device, 10f-output device, 21-receiving unit, 22-acquisition unit, 23-determination unit, 24-determination unit, 25-generation unit, 26-output unit, 27-changing unit, Pa1-image area, Pa2, Pa3-blank area, Pd-backing paper image, Pi-input image, Po-output image, Tx-text.

Claims

1. An image processing device comprising a processor and processing an input image, The processor performs the following processing: Obtaining text related to the input image; determining a first object corresponding to the text from among the objects included in the input image; and The arrangement of the text in an output image based on the input image is determined according to the input image and the first object.

2. The image processing device according to claim 1, wherein: The processor acquires the text according to an input operation of a user.

3. The image processing device according to claim 1, wherein: The processor generates the text based on an area selected by a user in the input image.

4. The image processing device according to claim 1, wherein: The processor generates the text according to a result of analyzing the input image.

5. The image processing device according to claim 1, wherein: The processor generates the text according to incidental information of the input image.

6. The image processing device according to any one of claims 1 to 5, wherein: The processor determines the arrangement in which the text does not overlap with the first object.

7. The image processing device according to any one of claims 1 to 5, wherein: The processor determines the arrangement in which the text does not overlap with a specific portion of the first object.

8. The image processing device according to any one of claims 1 to 5, wherein: When outputting the output image including the input image and the text, the processor determines the arrangement around the first subject so that the text does not overlap with the first subject and a second subject different from the first subject in the input image.

9. The image processing device according to any one of claims 1 to 5, wherein: The processor determines, when a plurality of texts are acquired as the texts, the arrangement of the plurality of texts so that the texts do not overlap each other around the first subject.

10. The image processing device according to any one of claims 1 to 5, wherein: When outputting the output image including the input image having an image area and a blank area and the text, the processor determines the arrangement within at least one of the image area and the blank area of ​​the input image included in the output image.

11. The image processing device according to any one of claims 1 to 5, wherein: When outputting an output image including the input image and the text, the processor generates a blank area around at least a portion of the input image in the output image, and determines the arrangement of the input image and the text in the output image.

12. The image processing device according to any one of claims 1 to 5, wherein: The processor performs the following processing: According to the input image, obtaining evaluation values ​​related to the configuration for a plurality of regions in the input image; and The arrangement is determined based on the evaluation values ​​of the plurality of regions.

13. The image processing device according to any one of claims 1 to 5, wherein: The processor acquires a plurality of texts as the texts, and determines the configuration for the plurality of texts.

14. The image processing device according to any one of claims 1 to 5, wherein: The processor divides the text into a plurality of texts and determines the configuration for the plurality of texts.

15. The image processing device according to any one of claims 1 to 5, wherein: The processor performs the following processing: acquiring a plurality of input images as the input images; and The text related to at least one of the plurality of input images is generated according to analysis results of the plurality of input images.

16. The image processing device according to any one of claims 1 to 5, wherein: When outputting the output image including the input image having an image area and a blank area and the text, the processor performs the following processing: determining whether the configuration can be determined within the blank area in the input image based on the text; If it is determined that the arrangement cannot be determined within the blank area in the input image, changing the layout of the image area in the input image; and In the output image including the input image whose layout is changed, the arrangement is determined within the blank area.

17. An image processing method, wherein a processor processes an input image, the image processing method comprising the following steps: The processor obtains text related to the input image; The processor determines a first object corresponding to the text from among the objects included in the input image; and The processor determines, based on the input image and the first object, an arrangement of the text in an output image based on the input image.

18. The image processing method according to claim 17, wherein: In the step of acquiring the text, the processor generates the text according to the analysis result of the input image.

19. The image processing method according to claim 18, further comprising the following steps: The processor assigns evaluation values ​​related to the configuration to a plurality of regions in the input image, In the step of determining the arrangement, the processor determines the arrangement based on the evaluation values ​​of the plurality of areas. 20 . A program product comprising a program for causing a computer to execute each step included in the image processing method according to claim 17 . 21 . A recording medium readable by a computer, wherein a program for causing a computer to execute each step included in the image processing method according to claim 17 is recorded on the recording medium.

Images