Text information generating device, text information generating method, program, and recording medium

Abstract

To provide a device, a method, a program, and a recording medium that generate a proper text by taking into account a background image or an image layout as well as an image to be arranged in a medium.SOLUTION: A text information generating device 10 comprises a processor 10a. The processor 10a executes: first acquisition processing for acquiring first information related to at least either an image layout in a background image, or the background image; second acquisition processing for acquiring second information related to an image; and generation processing for generating text information, based on the first information and the second information.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] One embodiment of the present invention relates to a text information generating device, a text information generating method, a program, and a recording medium capable of generating appropriate text information in consideration of a background image and an image to be placed on the background image. [Background technology]

[0002] A technique for extracting keywords corresponding to the image to be used and arranging them in or around the image when creating a so-called photo book or the like is already known. One example of such a technique is the technique described in Patent Document 1. In the technique described in Patent Document 1, keywords are extracted and a title is determined by analyzing image data used to create a photo book, and the image and title are assigned to a design template corresponding to the title. This makes it possible to create a stylish photo book without spending a lot of time and effort. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-149020 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a photo book, not only the images but also the atmosphere of the mount on which the images are placed (hereinafter referred to as background image) or the layout of the images on the background image have a significant impact on the impression of the target page or the entire photo book. If the image size is smaller than the general image size or if there are only a few images placed, the impact of the background image and layout is likely to be greater. In addition, there are cases where the atmosphere and impression created by the background image or layout and the images can affect each other. Therefore, simply generating text based only on the images may not be appropriate as text to be added to the photo book.

[0005] On the other hand, when adding text to images using the technology described in Patent Document 1, keywords extracted without considering the background image or layout as described above are used. Such keywords may not match the impression given by the background image or layout, and may leave an awkward impression on the entire photo book.

[0006] One embodiment of the present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an apparatus, method, program, and recording medium that are capable of generating appropriate text information taking into account a background image and an image to be placed in the background image. [Means for solving the problem]

[0007] The above object can be achieved by a text information generating device according to any one of [1] to [9] below. [1] A text information generation device comprising a processor that generates text information based on a background image and one or more images, wherein the processor executes a first acquisition process to acquire a layout of images in the background image and first information relating to at least one of the background images, a second acquisition process to acquire second information relating to the images, and a generation process to generate text information based on the first information and the second information. [2] The text information generation device described in [1], wherein the processor, in the generation process, sets a style based on at least one of the first information and the second information, and performs an output process to output text according to the style. [3] The text information generating device according to [2], wherein the information about the background image is information about the color and pattern of the background image. [4] The text information generating device according to any one of [1] to [3], wherein the information about the layout is information about a blank area of ​​a background image in which no image is placed. [5] The text information generation device described in [4], wherein the processor, in the generation process, determines at least one of the number of characters of text contained in the text information or the display size of the text based on the size of the blank area. [6] A text information generation device according to any one of [1] to [5], wherein a processor generates text information in a generation process based on first information regarding a layout, rules for generating text information according to the layout, and second information. [7] A text information generation device according to any of [1] to [6], wherein when the layout is a layout in which two or more images of different sizes are arranged on a background image, in the generation process, the processor generates text information based on at least second information relating to the largest image. [8] A text information generation device described in any of [1] to [7], wherein when the layout is a layout in which two or more images of equal size are arranged on a background image, the processor generates text information based on second information regarding each of the multiple images in the generation process. [9] A text information generation device described in any of [1] to [8], wherein a processor executes a process of generating an image with text information in which the text information is placed in at least one of a background image or an image, and the processor further executes a process of determining a method of placing text indicated by the text information in the image with text information based on at least one of the first information and the second information.

[0008] The above object can also be achieved by a text information generation method described in

[10] below.

[10] A method for generating text information based on a background image and one or more images by a processor, the method including the steps of: obtaining, by the processor, first information relating to a layout of images in the background image and at least one of the background image; obtaining, by the processor, second information relating to the images; and generating, by the processor, the text information based on the first information and the second information.

[0009] Moreover, a program according to one embodiment of the present invention is a program for causing a computer to execute each step included in the text information generation method described in

[10] above. Furthermore, a recording medium according to one embodiment of the present invention is a computer-readable recording medium having recorded thereon a program for causing a computer to execute each step included in the text information generation method described in

[10] above. Effect of the Invention

[0010] According to one embodiment of the present invention, there are provided an apparatus, a method, a program, and a recording medium capable of generating appropriate text information in consideration of a background image and an image to be placed on the background image. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of an operation concept of a text information generating device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing the correspondence relationship between the image analysis result and the contents of text information according to the first embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram showing a correspondence relationship between a layout of an image arrangement and an image arrangement to be used for generating text information according to the first embodiment of the present invention. [Figure 4]FIG. 2 is a diagram showing a correspondence relationship between an image analysis result and a text style according to the first embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing the corresponding relationship between blank areas and the number of characters of text, etc. according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a diagram showing a determination table that associates the number of images used to generate text information with the arrangement position of the text information according to the first embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing an example of an output image in which text is arranged according to the first embodiment of the present invention. [Figure 8] FIG. 2 is a diagram showing a conceptual example of subject selection from an image layout according to the first embodiment of the present invention. [Figure 9] FIG. 2 is a diagram showing a concept of controlling the placement of text in accordance with the inclination of a subject in an arrangement image according to the first embodiment of the present invention. [Figure 10] 1 is a diagram illustrating an example of a hardware configuration of a text information generating device according to a first embodiment of the present invention. [Figure 11] 1 is a diagram illustrating functional units of a text information generating device according to a first embodiment of the present invention. [Figure 12] FIG. 1 is a flow diagram of a text information generating method according to a first embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Specific embodiments of the present invention will now be described. For convenience of explanation, the following description may be given from the viewpoint of a GUI (Graphic User Interface).Furthermore, since the basic data processing technologies (communication / transmission technology, data acquisition technology, data recording technology, data processing / analysis technology, machine learning technology, image processing technology, visualization technology, etc.) for realizing the present invention are well-known technologies, the description thereof will be omitted.

[0013] In addition, in this specification, the concept of "device" includes a single device that performs a specific function, and also includes a combination of multiple devices that exist in a distributed and independent manner but cooperate (link) to perform a specific function.

[0014] Furthermore, in this specification, a "user" refers to a user of the text information generation device of the present invention, and more specifically, to a person who uses, for example, an image (more specifically, an output image described below) to which the text information described below is added, which is obtained by the functions of the text information generation device of the present invention.

[0015] In addition, in this specification, "person" means a subject that performs a specific action, and includes individuals, groups, corporations such as companies, and organizations, and may also include computers and devices that constitute artificial intelligence (AI). Artificial intelligence (AI) is a technology that realizes intelligent functions such as inference, prediction, and judgment using hardware and software resources. The algorithm of artificial intelligence is arbitrary, and may be, for example, an expert system, case-based reasoning (CBR), Bayesian network, or subsumption architecture.

[0016] <<First embodiment of the present invention>> In a first embodiment of the present invention (hereinafter, referred to as the first embodiment), the text information generating device of the present invention generates text information (hereinafter, referred to as text information Tx) based on a background image Pb such as a mount image and an image to be placed on this background image Pb (hereinafter, referred to as a placement image Pi), as shown in Fig. 1, and outputs an image (hereinafter, referred to as an output image Po) including this text information Tx. The user can use the output image Po. In order to implement these functions, the text information generating device has functional units such as a receiving unit 21, a generating unit 22, a determining unit 23, and an output unit 24 (see FIG. 11). The specific functions of these functional units will be described in turn in the following explanation. Here, an example of an output image Po can be an image in which one or more placed images Pi are placed on a background image Pb, with text information Tx added to the background image Pb or to the placed images Pi, or with text information Tx added across the background image Pb and the placed images Pi. Adding (placing) text information Tx on an image means converting the text information Tx into an image (text image) and including it in the output image Po as a part of the output image Po.

[0017] In the present invention, an "image" is composed of a plurality of pixels, is represented by the gradation values ​​of each of the plurality of pixels, and includes at least one or more subjects. Digital image data (hereinafter, image data) that defines an image at a set resolution is generated by compressing data in which the gradation values ​​of each pixel are recorded using a predetermined compression method. Types of image data include, for example, lossy compressed image data such as JPEG (Joint Photographic Experts Group) format, and lossless compressed image data such as GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format. In the following description, when the term "image" is used simply, the concept includes any of the background image Pb, the layout image Pi, and the output image Po.

[0018] (Reception Department) The reception unit 21 receives input of an image used for generating text information. In the first embodiment, the method of acquiring an image in the reception unit 21 is not particularly limited, but includes, for example, the reception unit 21 receiving input of image data of an image captured by a photographing device such as a camera through a predetermined interface and a communication line, and receiving data read by a scanner or the like for an existing photograph from a device such as a scanner. Furthermore, in the case where the text information generating device is equipped with a photographing device, the reception unit 21 may acquire image data of an image photographed by the photographing device. Furthermore, the reception unit 21 may acquire an image by downloading the image data from an external device or a Web server via a communication network. The image data acquired by the reception unit 21 is assumed to be, for example, a material for producing a photo book or the like, but is not limited to this. Any image data for producing or editing some electronic medium by placing it in a background image can be applicable.

[0019] The arranged image Pi may be a developed image obtained by developing a RAW image, or a corrected image obtained by performing a predetermined correction process on the developed image. The developed image or corrected image as the arranged image Pi is one or more images arranged in a predetermined layout with respect to the background image Pb (see FIG. 1). The background image Pb is an image on which the arrangement image Pi is arranged (superimposed), and functions as a mount for the arrangement image Pi. The background image Pb is an image larger in size than the arrangement image Pi, has color, and may be a black and white image or a color image. The background image Pb may have a pattern, or an illustration of a character or the like may be drawn as one aspect of the pattern. The output image Po is an image in which one or more placement images Pi are arranged in a predetermined layout on a background image Pb, has one or more image areas and blank areas Pv other than the image areas, and further includes text information Tx. An image area is an area in the background image Pb in which the placement image Pi is arranged, and a blank area Pv is an area in the background image Pb that does not include the placement image Pi. Taking the images of each page constituting a photobook as an example, an area on that page with no image, i.e., a margin, corresponds to the blank area Pv. The layout is the arrangement of the arranged images Pi in the background image Pb. The layout may include the number of arranged images Pi, as well as the size, arrangement position, outer edge shape (aspect ratio) of each arranged image Pi, and the inclination of the arranged image Pi with respect to the background image Pb (i.e., the amount of rotation of the arranged image Pi).

[0020] (Output section) The output unit 24 outputs the output image Po including the text information Tx generated by the generation unit 22. In the first embodiment, the method of outputting the output image Po is not particularly limited, but includes, for example, the output unit 24 in the text information generation device displaying the output image Po on a display or a monitor, printing the output image Po, transmitting the output image Po to another user, and providing the output image Po as a commercial product. The output image Po as a commercial product is assumed to be a booklet (medium) consisting of a plurality of pages on which images are published, such as an album, a photo book, a postcard, a message card, an electronic album, or the like. Moreover, the manner in which the output image Po is provided as a commercial product may include providing the output image Po by printing it on a medium such as paper, as well as providing image data (digital data) in the form of an electronic commercial product without printing on a medium. The manner in which the printed output image Po is provided may include sending a medium such as paper on which the output image Po is printed to a recipient as a message card or postcard. Furthermore, the method of outputting the output image Po may include posting the output image Po on a social networking service (SNS) for public viewing.

[0021] The output image Po is output by the output unit 24 as an image including text information Tx. The text information Tx is information (character information) displaying text related to the arranged image Pi, and specifically, is information indicating a sentence (text) corresponding to a subject in one or more arranged images Pi. Such text information Tx displays text such as comments and messages to the subject, explanations related to the subject, and lines uttered by the subject. The text indicated by the text information Tx is composed of a phrase or sentence consisting of one or more words, and when a sentence is divided, each of the divided phrases may correspond to text. Furthermore, the text indicated by the text information Tx may include onomatopoeia, mimetic words, and exclamations.

[0022] (Generation part) In the first embodiment, the text information Tx is automatically generated by a generation unit 22 of a text information generation device. When automatically generating the text information Tx, the generation unit 22 of the text information generation device generates the text information Tx based on the layout of the arrangement images Pi in the background image Pb, and first information regarding at least one of the background image Pb and second information regarding the arrangement images Pi. The generation unit 22 is assumed to acquire the above-mentioned first information by analyzing at least one of the above-mentioned layout and background image Pb. Similarly, the generation unit 22 is assumed to acquire the second information by analyzing the layout image Pi.

[0023] Such an analysis is, for example, to identify the characteristics of various phenomena related to the image. Such characteristics are, for example, information on the hue and tone of each region of the image, information on the image quality, the gradation value of the pixel, the size and position of the blank region, the layout of the arranged images Pi (e.g., the number of images arranged, the arrangement interval, the arrangement shape, the size, the size variation, etc.), and information on the subject estimated from the above information. The information on the subject may include the type of the subject, the condition of the subject, the position of the subject in the image, and, if the subject is a person, the facial expression, pose, gesture, age group, gender, etc. Moreover, it is preferable that the image features can be quantified, vectorized, or tensorized. In this case, the image analysis result is the quantified, vectorized, or tensorized image features, that is, feature quantities.

[0024] Here, the subject refers to people, animals, objects, backgrounds, etc. included in the arranged image Pi. The concept of the subject may also include the location, scene (e.g., dawn or dusk, clear skies, etc.) and theme (e.g., travel, meals, or events such as sports days) that appear in the arranged image Pi. The arranged image Pi may be a landscape image, that is, the subject included in the arranged image Pi may be only a landscape, and the entire image may represent the landscape as the subject.

[0025] There is no particular limitation on the technology for automatically generating text from the image analysis results. As an example, the correspondence between the image analysis results and the contents of the text information Tx (see FIG. 2) may be stored as data, and the generating unit 22 may select (generate) the text information Tx corresponding to the image analysis results by referring to the correspondence. As shown in FIG. 2, the correspondence between the image analysis result and the contents of the text information Tx is, for example, the image analysis result of the background image Pb and the arranged image Pi and the contents of the text information Tx (more specifically, the text indicated by the text information Tx) are associated with each other. The image analysis result includes, for example, the color and pattern of the background image Pb, and the type, facial expression, pose, attributes (age group and gender), shooting location, scene, theme, and the like of the subject included in the arranged image Pi. In the example shown in FIG. 2, text information Tx such as "Happy birthday! You've learned to crawl well now," "We had a lot of fun at the pool in the park during the summer vacation," "The sunset over the Sea of ​​Japan is special," "You got a hole-in-one in the rain!?", "Mom, look over here!", "Thank you for your many years of hard work," "Congratulations on your new home!", and "Wow, that's amazing" are associated with the above image analysis result.

[0026] Alternatively, a learning model for generating text may be constructed by performing machine learning, and the generating unit 22 may generate text information Tx related to an image by inputting an image into the learning model. The above learning model, for example, identifies the characteristics of an input image and generates text information Tx according to the characteristics, and is constructed by performing machine learning using previously acquired images and text information assigned to the images as learning data.

[0027] In addition, when generating the text information Tx in the above-mentioned generating unit, a form may be adopted in which the results of the image analysis to be used for generating the text information Tx are selectively used based on the layout of the arranged image Pi. In this case, the text information generating device may store, as data, a correspondence relationship (see FIG. 3) between the layout of the arranged image Pi and the arranged image Pi to be used for generating the text information Tx as a rule for generating the text information Tx according to the layout, and the generating unit may refer to the correspondence relationship to select the arranged image Pi to be the target of the image analysis result.

[0028] As shown in Fig. 3, the above-mentioned correspondence relationship associates the layout of the arrangement image Pi with the selection method of the arrangement image Pi. In the correspondence relationship shown in Fig. 3, the layout of the arrangement image Pi includes a layout in which a plurality of arrangement images Pi of different sizes are arranged on a background image Pb, a layout in which a plurality of arrangement images Pi of the same size are arranged on a background image Pb, and the like. In addition, in the correspondence relationship shown in Fig. 3, the selection method of the arrangement image Pi includes a case in which the largest arrangement image Pi of the plurality of arrangement images Pi is selected, and a case in which each of the plurality of arrangement images Pi is selected. When such a correspondence relationship is used, the text information Tx is generated in a manner that takes into account the similarity in size among the plurality of arrangement images Pi.

[0029] The size of the above-mentioned layout image Pi may be expressed in terms of the total number of pixels, the number of pixels in either the vertical or horizontal direction, or the number of pixels on the long side. In addition, the concept of the size of the placed image Pi being "equivalent" refers to a difference that is not visually perceptible, for example, a condition that the size difference between the target image and the comparison image is smaller than a predetermined value (e.g., ratio, number of pixels, area).

[0030] (Decision Section) In the first embodiment, the determination unit 23 of the text information generating device can set the style of the text information Tx generated in the above manner. The style corresponds to the type of font (character style) of the text indicated by the text information Tx, the display color, the display size, the presence or absence of decoration, and the presence and the type of an object for displaying text such as a speech bubble. The determination unit 23 of the text information generation device sets the above-mentioned style based on, for example, at least one of first information (information relating to at least one of the layout of the arranged images Pi in the background image Pb and the background image Pb) and second information (information relating to the arranged images Pi).

[0031] Specifically, the determination unit 23 of the text information generation device determines the corresponding style by inputting, for example, at least one of information on the layout of the arranged image Pi in the background image Pb and the background image Pb (first information), and information on the arranged image Pi (second information) into a pre-trained learning model. This learning model is a style setting model constructed by implementing machine learning using as training data a set of at least one of a background image Pb and a background image Pb in which an arranged image Pi is arranged, and at least one of the arranged images Pi, which have been used in the past or have been prepared for learning, and an appropriate style for text information Tx.

[0032] Alternatively, at least one of the above-mentioned first information and second information, i.e., only the background image Pb, the background image Pb with the layout image Pi arranged thereon, and the image analysis results of the layout image Pi, and the correspondence relationship (see FIG. 4) between the style of the text information Tx may be stored as data, and the determination unit 23 may refer to the correspondence relationship to select a style corresponding to at least one of the first information and second information (the image analysis results). As shown in FIG. 4, the correspondence between the image analysis result and the style is such that the image analysis result regarding the background image Pb, the arrangement image Pi, and the layout thereof is associated with the style. In the correspondence shown in FIG. 4, the image analysis result of the background image Pb includes, for example, the color pattern of the background image Pb, the size or shape of the blank area, etc. The analysis result of the layout of the arrangement image Pi includes, for example, the arrangement number, arrangement interval, arrangement shape, etc. of the arrangement image Pi. The analysis result of the arrangement image Pi includes, for example, the overall hue, color tone, and subject attributes (e.g., type, age group, sex, size, pose, gesture, etc.) in the arrangement image Pi. In addition, in the correspondence shown in FIG. 4, the style of the text information Tx includes the font type of the text indicated by the text information Tx, the decoration at the time of display, the display color, the display size, and the type of object such as a speech bubble, etc.

[0033] Furthermore, the determination unit 23 of the text information generating device can control the generation of text information Tx based on attribute information of a blank area Pv occurring in the background image Pb as information on the background image Pb. The attribute information of the blank area Pv is information on the blank area Pv, such as the size of the blank area Pv and the position of the blank area Pv in the background image Pb (hereinafter referred to as the position). Therefore, the text information generating device may store as data the correspondence (see FIG. 5) between the size and position of the blank area Pv and the number of characters and display size of the text indicated by the text information Tx, and the determination unit may refer to the correspondence to determine the number of characters of text to be placed in the blank area Pv, etc.

[0034] 5, the above-mentioned correspondence relationship is one in which information about blank area Pv in background image Pb corresponds to attribute information of text indicated by text information Tx. Information about blank area Pv includes the area of ​​blank area Pv, the occupancy rate (area basis) and position (e.g., center, top, bottom of background image Pb) of blank area Pv in background image Pb, etc. Text attribute information includes the number of characters of text indicated by text information Tx (e.g., 10 characters or less, 20 characters or less, etc.) and character size (e.g., 3 point, 5 point, 10 point, etc.), etc.

[0035] Furthermore, the determination unit 23 of the text information generating device determines a method of arranging the text information Tx when generating an output image Po as an image with text information in which the text information Tx is arranged in at least one of the background image Pb and the arrangement image Pi. The concept of the method of arranging the text information Tx may include a method of arranging the text information Tx in the background image Pb or the arrangement image Pi (including a concept spanning both images). The above-mentioned determination of the arrangement method by the determination unit 23 is executed based on at least one of the first information (the layout of the arranged images Pi in the background image Pb, and at least one of the image analysis results of the background image Pb) and the second information (the image analysis results of the arranged images Pi). In other words, the arrangement of the text information Tx in the output image Po is determined based on at least one of the first information and the second information.

[0036] In this case, the determination unit 23 of the text information generating device determines the arrangement method of the text information Tx based on the images (at least one of the arranged images Pi and the background image Pb) used to generate the text information Tx and the layout of the arranged images Pi. The arrangement method specified by this determination may include at least one of the concepts of the arrangement position and the arrangement direction of the text information Tx. The arrangement position of the text information Tx may include concepts such as a predetermined specified position (e.g., the center, top end, bottom end, left end, right end, or corner of the image) in at least one of the background image Pb and the arrangement image Pi, or a position according to a specific subject included in the arrangement image Pi, etc. The specific subject may be, for example, a main subject among the subjects included in the arrangement image Pi, specifically, a subject arranged at the center of the angle of view, a subject captured at the largest size, or a subject in focus. Moreover, the arrangement direction of the text information Tx means the inclination (rotation) of the text information Tx on the image where it is arranged.

[0037] The determination unit 23 of the text information generating device, which determines the above-mentioned arrangement method, determines the arrangement position of the text information Tx using a determination table or a determination algorithm shown in Fig. 6. In the table of Fig. 6, the number of arrangement images Pi used in the text information Tx (in other words, the number of arrangement images Pi arranged in the background image Pb) is associated with the arrangement position of the text information Tx. Note that in the table shown in Fig. 6, the arrangement positions of the text information Tx include the top of the background image Pb, the periphery of the arrangement image Pi, and the periphery of the subject in the arrangement image Pi.

[0038] In this case, the determination unit 23 of the text information generation device will, for example, refer to the above-mentioned judgment table and determine the placement position of the text information Tx depending on information on whether multiple images (background image Pb or placement image Pi) or a single image were used to generate the text information Tx. When the determination table specifies the placement position with respect to the periphery of the subject, the determination unit 23 of the text information generating device identifies the subject in the layout image Pi, and the output unit 24 outputs the output image Po in which the text information Tx is placed at a position (e.g., above, below, left, right, etc.) according to the subject identified by the determination unit (see FIG. 7). This allows the text information Tx to be placed at an appropriate position in the output image Po, taking into account the subject in the layout image Pi.

[0039] Specifically, the text information Tx can be arranged at a position where the subject and the text information Tx can be easily associated with each other, such as around the subject. When multiple subjects exist in the layout image Pi, the determination unit of the text information generating device selects a specific subject (e.g., the subject with the largest size in the image, the subject present in the center of the image, the subject located in the middle of the multiple subjects, etc.) based on a predetermined rule, and determines the layout position of the text information Tx around the subject (see FIG. 8).

[0040] When controlling the arrangement method based on an arrangement image Pi including a plurality of subjects as described above, the effect of arranging the text information Tx at an appropriate position in the output image Po in consideration of the subjects in the arrangement image Pi becomes even more effective. Specifically, it is possible to prevent the text information Tx from being arranged near a subject that has little relationship with the contents of the text information Tx. In addition, since the arrangement of the text information Tx is automatically determined by the function of the text information generating device, the user does not need to determine the arrangement of the text information Tx by himself, and the time and effort required for this can be saved.

[0041] In determining the arrangement direction of the arrangement method, the determination unit 23 of the text information generating device determines the inclination (rotation) of the arrangement image Pi used to generate the text information Tx on the background image Pb, and controls the text information Tx to be inclined according to this inclination. Also, instead of the inclination of the arrangement image Pi, the inclination of the subject in the arrangement image Pi may be determined, and the text information Tx may be inclined according to this inclination (see FIG. 9).

[0042] The technique for identifying an image (placement image Pi or background image Pb) to which text information Tx is to be added or the layout of the placement image Pi and determining the placement position of the text information Tx accordingly is not limited to the judgment table and algorithm already described. For example, in addition to the above, a learning model for determining the placement position of the text information Tx may be constructed by performing machine learning, and the determination unit 23 may input an image (at least one of the background image Pb and the placement image Pi) to this learning model to determine the placement position of the text information Tx in the image. The subject in the placement image Pi may also be included in the subject for which the placement position is to be determined. Such a learning model is constructed by performing machine learning using, for example, a previously acquired image without text and an image with text in which the text information Tx is placed in association with the image or the subject in the image as learning data.

[0043] The above machine learning may be performed in stages. Specifically, a first learning for constructing a learning model for identifying an object in an image and a second learning for constructing a learning model for determining the position of text information according to the object may be performed separately.

[0044] Furthermore, in the first learning, a deep learning attention mechanism may be applied, and a learning model for identifying a subject may be constructed based on the process of generating text information by the learning model for text generation described above. In this case, when generating text information to be added to an image using the learning model for text generation, the analysis result of which region in the image was given importance in generating the text information can be reflected when identifying the subject.

[0045] [Example of the configuration of a text information generation device] Next, a configuration example of a text information generating device according to the first embodiment (hereinafter, text information generating device 10) will be described with reference to FIG. The text information generation device 10 is composed of a computer used by a user, specifically a client terminal, and specifically composed of a smartphone, a tablet terminal, a notebook PC (Personal Computer), etc. Note that the text information generation device 10 is not limited to a computer owned by a user, and may be composed of a terminal that is not owned by a user but can be used by entering a PIN or password or making a deposit when visiting a store, such as a store-installed terminal. In the following, an example will be described in which the text information generation device 10 is configured by a user's computer, specifically, a smartphone.

[0046] As shown in FIG. 10, the computer constituting the text information generation device 10 includes a processor 10a, a memory 10b, a communication interface 10c, a storage 10d, an input device 10e, and an output device 10f.

[0047] The processor 10a may be composed of, for example, a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a TPU (Tensor Processing Unit), or an ASIC (Application Specific Integrated Circuit). The memory 10b is configured by semiconductor memories such as a Read Only Memory (ROM) and a Random Access Memory (RAM).

[0048] The communication interface 10c may be, for example, a network interface card or a communication interface board. The computer constituting the text information generation device 10 can communicate with other devices connected to a communication network such as the Internet and a mobile communication line via the communication interface 10c.

[0049] The storage 10d may be composed of, for example, a flash memory, a hard disc drive (HDD), a solid state drive (SSD), a flexible disc (FD), a magneto-optical disc (MO disc), a compact disc (CD), a digital versatile disc (DVD), a secure digital card (SD card), or a universal serial bus memory (USB memory). The storage 10d may be built in the computer main body constituting the text information generation device 10, or may be attached to the computer main body in an external form. Alternatively, the storage 10d may be configured by a NAS (Network Attached Storage) or the like. The storage 10d may also be an external device, such as an online storage or a database server, capable of communicating with one of the computers constituting the text information generation device 10 via a communication network.

[0050] The input device 10e is a device that accepts an input operation by a user, and is configured, for example, with a touch panel, etc. The input device 10e also includes a photographing device such as a built-in camera of a smartphone, a microphone for collecting sound, etc. The output device 10f is, for example, a display.

[0051] Furthermore, a program for an operating system (OS) and an application program for executing image processing including text information generation processing are installed as software in the computer constituting text information generation device 10. By reading and executing these programs by processor 10a, the computer constituting text information generation device 10 performs the functions of reception unit 21, generation unit 22, determination unit 23, and output unit 24 (see FIG. 11) already described, and specifically, executes a series of processes related to outputting an image including the above-mentioned text.

[0052] 11, the text information generation device 10 has a reception unit 21, a generation unit 22, a determination unit 23, and an output unit 24. These functional units are realized by a processor 10a of the text information generation device 10 executing the above-mentioned application program for image processing and cooperating with other hardware devices of the text information generation device 10. In addition, some functions may be realized by utilizing artificial intelligence (AI).

[0053] [Example of text information generation method according to the first embodiment] Next, as an operation example of the text information generating device 10 in the first embodiment, an image processing flow using the device will be described. In the image processing flow described below, the text information generating method of the present invention is used. In other words, each step in the image processing flow described below corresponds to a component of the text information generating method of the present invention. Note that the flow below is merely an example, and some steps in the flow may be deleted, new steps may be added to the flow, or the order of execution of two steps in the flow may be reversed, as long as it does not deviate from the spirit of this embodiment.

[0054] The steps in the image processing flow according to the first embodiment are performed in the order shown in Fig. 12 by the processor 10a included in the text information generation device 10. That is, in each process in the image processing flow, the processor 10a executes processing corresponding to each step in Fig. 12 among data processing defined in the application program for image processing.

[0055] Specifically, in the image processing flow according to the first embodiment, the processor 10a first executes an image reception process (S001). In this reception process, the processor 10a receives an input of an image to which text is to be added by this flow, in other words, obtains a background image Pb and a layout image Pi to be processed. At this time, these images may be input in a state in which the layout image Pi is arranged on the background image Pb (i.e., as a single input image), or the layout image Pi and the background image Pb may be input in a state in which they are separated from each other.

[0056] Next, the processor 10a executes an acquisition process to acquire first information regarding at least one of the layout of the arrangement image Pi in the background image Pb and the background image Pb, and second information regarding the arrangement image Pi (S002). In the acquisition process, the process of acquiring the first information corresponds to the first acquisition process, and the process of acquiring the second information corresponds to the second acquisition process. In the above-mentioned acquisition process, the processor 10a inputs the background image Pb in which the above-mentioned arranged image Pi is arranged, and the arranged image Pi, into an image analysis program that is held or can be called up so as to be usable. As a result, each image is analyzed to identify the characteristics of each image, and the processor 10a acquires the identified characteristics as the first information and the second information.

[0057] In the above analysis, the processor 10a acquires, for example, information on hue and tone, information on image quality, pixel gradation values, and information on the size and position of blank areas (which may constitute either or both of the first information and the second information) in each area of ​​the image (specifically, the background image Pb or the arranged image Pi, and the same applies below). In the above analysis, the processor 10a further acquires information on the layout of the arranged image Pi (information that may constitute the first information, such as the number of images arranged, the arrangement interval, the arrangement shape, the size, and the size variation) and information on the subject estimated from this information (information that may constitute the second information). The above-mentioned subject information may include the type of subject, the state of the subject, the position of the subject in the image, and, if the subject is a person, facial expression, pose, gesture, age group, gender, and the like.

[0058] Next, the processor 10a automatically generates text information Tx based on the first information and the second information obtained in the above-mentioned step S002 (S003). In this automatic generation, the processor 10a acquires the above-mentioned first information by analyzing at least one of the above-mentioned layout and background image Pb. Similarly, the processor 10a acquires the above-mentioned second information by analyzing the arrangement image Pi. The image analysis method itself for analyzing the characteristics of the image can be appropriately adopted from existing technology.

[0059] When automatically generating the text information Tx, the processor 10a generates text information Tx corresponding to the image analysis results obtained as the first information and the second information, for example, by referring to the correspondence between the above-mentioned image analysis results and the contents of the text information Tx (see Figure 2).

[0060] The text information Tx generated by the above procedure reflects the background image Pb and the layout of the layout images Pi in the background image Pb. To be more specific, the color and pattern of the background image Pb is an element that greatly affects the impression of the arranged images Pi arranged thereon, or the medium generated by arranging the arranged images Pi on the background image Pb, depending on the tendency, etc. For example, if the color of the background image Pb is dark blue, dark purple, or gray, and the pattern is a Japanese pattern (e.g., hemp leaf, blue sea wave, or wave chidori, etc.), the impression that a person who sees this tends to have is one of calmness, tradition, dignity, discipline, etc. On the other hand, if the background image Pb is pink, sky blue, or gold and has a pop pattern (e.g., stars, hearts, musical notes, or puppies), the impression that people who see it will have is likely to be one of vitality, youth, dynamism, freedom, cuteness, etc.

[0061] Furthermore, with regard to the arranged images Pi arranged in such a background image Pb, the number and arrangement pattern of the arranged images Pi, as well as the type, facial expression, pose, age group, sex, location, scene, and theme of the subject included in the arranged images Pi, are factors that greatly affect the impression of the arranged images Pi or the medium generated by arranging the arranged images Pi in the background image Pb. From the viewpoint of the influence on such impressions, it is possible that the overall impression of the edited image in which the arranged images Pi are arranged in the background image Pb changes depending on the combination of the impression regarding the arranged images Pi and the impression regarding the background image Pb. In the correspondence relationship between the image analysis results and the contents of the text information Tx described above, it is assumed that the text information Tx is set including the viewpoint of the combination of the first information and the second information.

[0062] For example, suppose that the color pattern of the background image Pb is subdued, and the layout of the arranged images Pi in the background image Pb is conservative, for example, the arranged images Pi are tiled vertically and horizontally of the background image Pb. Also, suppose that the analysis results obtained for the arranged image Pi indicate that the subject in the image is a middle-aged man, the location is an office, and the scene is a bouquet presentation scene. In this case, the generated text information Tx is a sentence with polite and subdued words and tone that evokes the image of a farewell party for a retiring person, for example, text information such as "Thank you very much for your hard work after your long service."

[0063] On the other hand, suppose that the color pattern of the background image Pb is flashy and pop, and the arrangement pattern of the arranged images Pi in the background image Pb is edgy, for example, a large number of arranged images Pi are randomly arranged on the background image Pb. Also, suppose that the analysis results obtained for the arranged images Pi indicate that the subject in the image is a female student, the location is a concert hall, and the scene is a scene in which the audience is dancing to the music. In this case, the generated text information Tx is a sentence with lively and casual words and tone that evokes the image of attending a music festival, for example, text information such as "The music is amazing and it's so much fun!" is generated.

[0064] As already mentioned, such a text generation method can be replaced with a learning model for text generation constructed by machine learning. That is, the processor 10a may input a background image Pb (at least one of the background image Pb alone and the background image Pb with the arranged image Pi already arranged) and the arranged image Pi to the learning model to generate text information Tx related to those images. The above learning model, for example, identifies the characteristics of the input image and generates text information Tx according to the characteristics, and is constructed by performing machine learning using previously acquired images and text information assigned to the images as learning data.

[0065] Next, the processor 10a converts the text information Tx generated in the above-mentioned step S003 into an image (text image), arranges it in the output image Po, and outputs it (S004). This step S004 corresponds to the output process. The output image Po in which text information Tx is placed can be, for example, an image in which a placement image Pi is placed on a background image Pb, with the text information Tx added either on the background image Pb or on the placement image Pi, or the text information Tx added across both the background image Pb and the placement image Pi.

[0066] When the series of processes described above is completed, the image processing flow according to the first embodiment is completed. The image processing flow is executed each time an image input is received, that is, the series of processes described above is repeatedly executed by the processor 10a each time a new image input is received.

[0067] [Modification of the first embodiment] The first embodiment of the present invention is not limited to the above embodiment, and may be modified as follows. The following describes the modified example. The following describes the differences between the modified example and the first embodiment.

[0068] (Regarding the first modified example) In the above-described first embodiment, the text information generating device 10 generates (obtains) text information Tx and outputs an output image Po on the premise that only one arrangement image Pi is arranged on the background image Pb, or one arrangement image Pi is determined as a processing target from the beginning. However, without being limited to this, as a modified example of the first embodiment, a mode can be assumed in which a plurality of arrangement images Pi are arranged on the background image Pb, and the text information generating device proactively selects one of the plurality of arrangement images Pi as an image to be used for generating the text information Tx.

[0069] In this case, when generating the text information Tx, the processor 10a selectively uses the results of the image analysis to be used for generating the text information Tx based on the layout of the arranged image Pi. Specifically, the processor 10a stores a correspondence relationship (see FIG. 3) between the layout and the arranged image Pi to be used for generating the text information Tx as a rule for generating the text information Tx according to the layout of the arranged image Pi, and selects the arranged image Pi to be analyzed for generating the text information by referring to the correspondence relationship.

[0070] As a selection method in this case, the processor 10a may selectively execute two types of logic: a logic for selecting the largest of the multiple placement images Pi when the layout has multiple placement images Pi of different sizes arranged on the background image Pb; and a logic for selecting each of the multiple placement images Pi when the layout has multiple placement images Pi of the same size arranged on the background image Pb.

[0071] (Regarding the second modified example) The above-mentioned style may be set for the text information Tx generated in the above-mentioned first embodiment. Therefore, as a second modified example of the first embodiment, a form in which the style is set for the text information Tx will be described. In the second modified example, the processor 10a sets the above-mentioned style based on, for example, at least one of first information (information on at least one of the layout of the arranged images Pi in the background image Pb and the background image Pb) and second information (information on the arranged images Pi).

[0072] Specifically, the processor 10a determines a corresponding style by inputting at least one of first information regarding at least one of the layout of the arranged image Pi in the background image Pb and the background image Pb, and second information regarding the arranged image Pi, into a pre-trained learning model. For example, the processor 10a may input the analysis results of the background image Pb and the arranged images Pi to the above-mentioned learning model, so that style-related information such as the font type, decoration, color, size, and object type such as speech bubbles of the text indicated by the text information Tx is output. Note that the image analysis results input to the learning model include the color pattern of the background image Pb, the size and shape of the blank area, the layout of the arranged images Pi in the background image Pb (e.g., the number of arranged images, the arrangement interval, the arrangement shape, etc.), the overall hue and tone of the arranged images Pi, and the attributes of the subject (e.g., the type, age group, sex, size, pose, gesture, etc.), etc. The above learning model is a style setting model constructed by implementing machine learning using as training data a set of a background image Pb (which may be a background image Pb in which an arranged image Pi is arranged) and at least one of the arranged images Pi, which have been used in the past or prepared for learning, and a suitable style for the text information Tx.

[0073] Alternatively, at least one of the above-mentioned first information and second information, i.e., only the background image Pb, the background image Pb with the arranged image Pi arranged thereon, and the image analysis results for the arranged image Pi, and the correspondence relationship (see FIG. 4) between the style of the text information Tx may be stored as data. Then, by referring to the correspondence relationship, a style corresponding to the image analysis results indicated by at least one of the first information and second information may be selected.

[0074] (Regarding the third modified example) As a modified example other than the first and second modified examples described above, it is also possible to employ an aspect in which the number of characters of text indicated by the text information Tx to be placed in a blank area Pv is controlled based on the blank area Pv in the background image Pb. In this case, the processor 10a controls the generation of the text information Tx based on attribute information of the blank area Pv occurring in the background image Pb as information related to the background image Pb.

[0075] Therefore, the processor 10a refers to the correspondence between the attribute information of the blank area Pv and the number of characters and display size of the text (FIG. 5), and determines the number of characters of the text indicated by the text information Tx to be placed in the blank area Pv, etc. In addition, the above-mentioned correspondence relationship does not simply stipulate that the larger the blank area Pv, the more text characters can be placed. For example, it may also include a correspondence relationship in which the number of text characters is extremely small and the display size is large when the position of the blank area Pv is near the center of the background image Pb. If the number of text characters is controlled from the viewpoint of the position of such blank area Pv, it is possible to generate text information Tx that succinctly mentions the subject and scene of the placed image Pi with text (sentence) of a smaller number of characters in a prominent place on the background image Pb.

[0076] More specifically, the processor 10a acquires attribute information of the blank area Pv, such as the area, occupancy rate (area basis), position (e.g., center, top, bottom of the background image Pb) of the blank area Pv in the background image Pb, etc. Then, by referring to the above-mentioned correspondence, the processor 10a determines the style, including the number of characters and character size of the text indicated by the text information Tx to be placed in the blank area Pv. When there are a plurality of blank areas Pv in which the text information Tx can be placed, the user may be allowed to select the blank area Pv in which the text information Tx is to be placed from among the plurality of blank areas Pv. Alternatively, the processor 10a may automatically select the blank area Pv in which the text information Tx is to be placed according to attribute information (specifically, area or position) of each blank area Pv.

[0077] (Regarding the fourth modified example) In the above-described first embodiment, the method of arranging the generated text information Tx may be controlled. Therefore, as a fourth modification of the first embodiment, a mode in which the method of arranging the text information Tx is controlled will be described. The processor 10a executes the determination of the arrangement method based on at least one of the first information (the layout of the arrangement image Pi in the background image Pb and the image analysis result of at least one of the background image Pb) and the second information (the image analysis result of the arrangement image Pi). That is, the arrangement of the text information Tx in the output image Po is determined based on at least one of the first information and the second information.

[0078] In this case, the processor 10a determines the arrangement of the text information Tx based on the image (at least one of the layout image Pi and the background image Pb) used to generate the generated text information Tx and its layout. The determined arrangement is at least one of the layout position and the layout direction of the text information Tx.

[0079] The processor 10a that determines the above-mentioned arrangement specifies whether the arrangement image Pi used in generating the text information Tx is a plurality of arrangement images Pi arranged on the background image Pb, or a single arrangement image Pi arranged on the background image Pb, in order to determine the arrangement position of the text information Tx. Information on the arrangement image Pi used in generating the text information Tx may be stored in a storage means when the text information Tx is generated. Then, the processor 10a compares the information on the arrangement image Pi used in generating the text information Tx with a determination table. As shown in FIG. 6, the determination table defines the layout position of the text information Tx depending on whether a plurality of layout images Pi are used to generate the text information Tx or whether one image is used.

[0080] In this case, the processor 10a determines, by the above-mentioned comparison, for example, that the position of the text information Tx is to be on the top of the background image Pb when a plurality of layout images Pi are used, or, for example, that the processor 10a determines, by the above-mentioned comparison, that the position of the text information Tx is to be around the layout image Pi or around the subject in the layout image Pi when a single layout image Pi is used. When the determination table specifies the layout position of the text information Tx at a position corresponding to the subject, the processor 10a identifies the subject in the layout image Pi, and identifies the positions around the subject.

[0081] This allows the text information Tx to be arranged at a position where the subject and the text information Tx can be easily associated with each other, such as around the subject. When multiple subjects exist in the layout image Pi, the processor 10a selects a specific subject based on a predetermined rule (e.g., the largest subject in the image, the subject in the center of the image, the subject in the center of multiple subjects in the image, etc.), and determines the layout position of the text information Tx around the selected subject.

[0082] In determining the arrangement direction of the text information Tx, the processor 10a determines the inclination (rotation angle) of the arrangement image Pi used to generate the text information Tx on the background image Pb, and controls the text information Tx to rotate according to this rotation angle. Alternatively, the processor 10a may determine the inclination of the subject in the arrangement image Pi, instead of the inclination of the arrangement image Pi, and controls the text information Tx to tilt according to this inclination.

[0083] Although specific embodiments of the present invention have been described above, the above-mentioned embodiments are merely examples given to facilitate understanding of the present invention and do not limit the present invention. That is, the present invention may be modified or improved from the embodiments described below without departing from the spirit of the present invention. The present invention also includes equivalents thereof. Furthermore, the embodiments of the present invention may include a combination of the above-mentioned embodiments and one or more of the following modifications.

[0084] (About the computer that constitutes the text information generating device) In the above embodiment, the text information generating device of the present invention is configured by a computer that is directly used by a user, such as a terminal (client terminal) owned by the user. However, the present invention is not limited to this, and the text information generating device of the present invention may be configured by a computer that is indirectly available to the user, for example, a server computer. Here, the above server computer may be, for example, a server computer for cloud services, specifically, a server computer for ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). In this case, when necessary information is input at the client terminal, the above server computer performs various processes (calculations) based on the input information, and the calculation results are output on the client terminal side. In other words, the functions of the server computer that constitutes the text information generating device of the present invention can be used on the client terminal side.

[0085] (Processor configuration) The processor included in the text information generation device of the present invention includes various processors, including, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. The various processors also include programmable logic devices (PLDs) such as field programmable gate arrays (FPGAs) whose circuit configurations can be changed after manufacture. Furthermore, various processors include dedicated electric circuits such as ASICs (Application Specific Integrated Circuits), which are processors having a circuit configuration designed exclusively for performing specific processing.

[0086] Furthermore, one functional unit of the text information generating device of the present invention may be configured by one of the various processors described above, or by a combination of two or more processors of the same or different types, for example, a combination of multiple FPGAs, or a combination of an FPGA and a CPU, etc. Furthermore, the multiple functional units of the text information generation device of the present invention may be configured by one of various processors, or two or more of the multiple functional units may be integrated into one processor. Also, as in the above-described embodiment, one processor may be configured by combining one or more CPUs and software, and this processor may function as multiple functional units.

[0087] Also, for example, a processor may be used that realizes the functions of the entire system including the multiple functional units in the text information generation device of the present invention on a single IC (Integrated Circuit) chip, as typified by SoC (System on Chip), etc. Also, the hardware configuration of the various processors described above may be an electric circuit (Circuitry) that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0088] 10 Text information generator 10a processor 10b Memory 10c Communication Interface 10d Storage 10e Input Device 10f output device 21 Reception 22 Generation part 23 Decision Section 24 Output section Pv blank area Pb background image Pi placement image Po output image Tx Text

Claims

1. 1. A text information generating device comprising a processor, the device generating text information based on a background image and one or more images, the device comprising: The processor, a first acquisition process for acquiring first information related to at least one of a layout of the image in the background image and the background image; a second acquisition process for acquiring second information related to the image; a generation process of generating the text information based on the first information and the second information; A text information generating device that executes the above.

2. The text information generating device according to claim 1 , wherein the processor, in the generating process, sets a style based on at least one of the first information and the second information, and executes an output process of outputting the text information in accordance with the style.

3. The text information generating device according to claim 2 , wherein the information about the background image is information about a color and pattern of the background image.

4. 4. The text information generating device according to claim 1, wherein the information about the layout is information about a blank area of ​​the background image where the image is not placed.

5. The text information generating device according to claim 4 , wherein the processor, in the generation process, determines at least one of a number of characters of text included in the text information or a display size of the text based on a size of the blank area.

6. 4. The text information generating device according to claim 1, wherein the processor generates the text information based on the first information regarding the layout, a rule for generating the text information according to the layout, and the second information in the generation process.

7. 4. The text information generation device according to claim 1, wherein when the layout is a layout in which two or more images of different sizes are arranged on the background image, the processor generates text information in the generation process based on at least the second information relating to the largest image.

8. 4. The text information generation device according to claim 1, wherein when the layout is a layout in which two or more images of equal size are arranged on the background image, the processor generates text information in the generation process based on the second information regarding each of the two or more images.

9. the processor executes a process of generating an image with text information by arranging the text information in at least one of the background image or the image; The processor, The text information generating device according to claim 1 , further comprising a process of determining a method of arranging the text indicated by the text information in the text-information-added image based on at least one of the first information and the second information.

10. 1. A method for generating text information based on a background image and one or more images, comprising: obtaining, by the processor, first information relating to at least one of a layout of the images in the background image and the background image; obtaining, by the processor, second information related to the image; generating, by the processor, the text information based on the first information and the second information; A method for generating text information comprising:

11. A program for causing a computer to execute each step included in the text information generating method according to claim 10.

12. A computer-readable recording medium, A recording medium having recorded thereon a program for causing a computer to execute each step included in the text information generating method according to claim 10.

Images