Image processing device, image processing method, program, and recording medium

Abstract

To provide an image processing device, an image processing method, a program, and a recording medium that allow proper text information to be imparted to an image.SOLUTION: In an image processing device comprising a processor, the processor determines a text style based on an image, and generates text information corresponding to the image, based on the image and the text style.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] One embodiment of the present invention relates to an image processing device, an image processing method, a program, and a recording medium. [Background technology]

[0002] As a technique for utilizing images, a technique for adding text to an image is already known. One example of such a technique is the technique described in Patent Document 1. In the technology described in Patent Document 1, keywords are extracted by analyzing image data, and decorative text that decorates the image data is determined based on the keywords. More specifically, the decorative text is determined by referring to a decorative text database in which keywords and decorative texts are associated with each other, and searching for decorative text that corresponds to the extracted keywords. This allows for tasteful decoration without spending time and effort. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-149709 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1, once a keyword is determined, the text information (decorative text) is limited to text information that is predetermined to correspond to the keyword. However, even if an appropriate keyword corresponding to an image is extracted, depending on the image, the text information that is predetermined to correspond to the keyword may not be appropriate text information corresponding to the image. In this case, appropriate text information is not added to the image.

[0005] One embodiment of the present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an image processing device, an image processing method, a program, and a recording medium capable of adding appropriate text information to an image. [Means for solving the problem]

[0006] The above object can be achieved by an image processing device according to any one of the following items [1] to

[19] . [1] An image processing device having a processor, the processor determining a text style based on an image, and generating text information corresponding to the image based on the image and the text style. [2] The image processing device described in [1], wherein the processor sets a first decision mode that determines a text style based on an image, or a second decision mode that determines a text style based on first input information, and determines the text style based on the set first decision mode or second decision mode. [3] An image processing device according to [1] or [2], wherein when the second determination mode is set, the processor determines the text style based on first input information from the user regarding the text style. [4] An image processing device described in any of [1] to [3], wherein a processor outputs the determined text style, redetermines the text style based on second input information regarding the text style, and generates text information based on the image and the redetermined text style. [5] The image processing device according to any one of [1] to [4], wherein the processor determines a text style based on a specific subject area extracted by analyzing the image. [6] An image processing device according to any one of [1] to [5], wherein a processor determines two or more text styles based on an image, and generates text information based on the two or more text styles. [7] An image processing device described in any of [1] to [6], wherein the processor determines two or more text styles having different themes based on the image, and generates text information for each combination of the two or more text styles having different themes. [8] An image processing device described in any of [1] to [7], wherein a processor generates text information using a first trained model trained to output text information by inputting an image and a text style. [9] An image processing device described in any of [1] to [8], wherein a processor identifies text information corresponding to an input image and text style by referring to a first table that pre-sets a correspondence between images and text styles and text information.

[10] An image processing device described in any of [1] to [9], wherein a processor determines the text style using a second trained model trained to output a text style by inputting an image.

[11] An image processing device according to any one of [1] to

[10] , wherein the processor identifies the text style corresponding to the input image by referring to a second table in which a correspondence between images and text styles is preset.

[12] The processor: An image processing device according to any one of [1] to

[11] , which generates two or more text information candidates based on an image and a text style, outputs the generated two or more text information candidates, and determines text information from among the two or more text information candidates based on user selection information.

[13] An image processing device described in any of [1] to

[12] , wherein a processor determines two or more text styles based on an image, assigns an evaluation score to each of the two or more text styles based on a correlation with the content of the image, generates text information for each text style to which an evaluation score has been assigned based on the image and the text style, and outputs the generated two or more pieces of text information based on the evaluation scores of the text styles.

[14] An image processing device described in any of [1] to

[13] , wherein a processor determines two or more text styles based on an image, assigns an evaluation score to each of the two or more text styles based on its correlation with the content of the image, generates text information based on the image and the text style among the two or more text styles having the highest evaluation score, outputs the generated text information, and, if a request to regenerate the text information is received, outputs a text style different from the text style with the highest evaluation score, and regenerates the text information based on the image and the text style selected by a user.

[15] An image processing device described in any of [1] to

[14] , wherein the processor outputs the generated text information, and when third input information regarding correction of the text information is received, corrects the text information based on the third input information.

[16] An image processing device described in any of [1] to

[15] , wherein a processor reconstructs a first trained model, which is constructed by machine learning using training data including images, text styles, and text information, using training data including corrected text information.

[17] An image processing device according to any one of [1] to

[16] , wherein the processor modifies a first table, which pre-sets correspondences between images and text styles and text information, using the modified text information.

[18] An image processing device described in any of [1] to

[17] , wherein a processor acquires multiple images, determines a text style common to the acquired multiple images, and generates text information common to each of the multiple images based on the multiple images and the common text style.

[19] An image processing device according to any one of [1] to

[18] , wherein the processor stores at least one of the determined text style and the generated text information as auxiliary information of the corresponding image.

[0007] The above object can also be achieved by an image processing method described in

[20] below.

[20] An image processing method comprising: determining a text style based on an image; and generating text information corresponding to the image based on the image and the text style, the method comprising: executing, by a processor, a process for determining a text style based on the image and the text style.

[0008] Moreover, a program according to one embodiment of the present invention is a program for causing a computer to execute each step included in the image processing method described in

[20] above. Furthermore, a recording medium according to one embodiment of the present invention is a computer-readable recording medium having recorded thereon a program for causing a computer to execute each step included in the image processing method described in

[20] above. Effect of the Invention

[0009] According to one embodiment of the present invention, there are provided an image processing device, an image processing method, a program, and a recording medium capable of adding appropriate text information to an image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of use of an image processing device according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram showing a procedure from obtaining an input image to generating text information. [Diagram 3] 1 is a diagram illustrating a hardware configuration of an image processing apparatus according to an embodiment of the present invention. [Figure 4] FIG. 2 is an explanatory diagram of functions of an image processing device according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing an operation screen for generating text information according to an embodiment of the present invention. [Figure 6] FIG. 4 is a diagram showing an example of an image processing flow according to an embodiment of the present invention. [Figure 7] FIG. 11 is an explanatory diagram of functions of an image processing device according to a first modified example of the present invention. [Figure 8]FIG. 11 is an explanatory diagram of functions of an image processing device according to a second modified example of the present invention. [Figure 9] FIG. 13 is an explanatory diagram of functions of an image processing device according to a third modified example of the present invention. [Figure 10] FIG. 13 is an explanatory diagram of functions of an image processing device according to a fourth modified example of the present invention. [Figure 11] FIG. 13 is an explanatory diagram regarding evaluation scores in the fourth modified example of the present invention. [Figure 12] FIG. 13 is an explanatory diagram of functions of an image processing device according to a fifth modified example of the present invention. [Figure 13] FIG. 13 is an explanatory diagram of functions of an image processing device according to a sixth modified example of the present invention. [Figure 14] FIG. 13 is a diagram showing a procedure from obtaining an input image to generating text information in the seventh modified example of the present invention. [Figure 15] FIG. 13 is a diagram showing a procedure from obtaining an input image to generating text information in an eighth modified example of the present invention. [Figure 16] 11 is a diagram showing a procedure from acquisition of an input image to generation of text information in an image processing device according to a comparative example. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Specific embodiments of the present invention will now be described. For convenience of explanation, the following description may be given from the viewpoint of a GUI (Graphic User Interface).Furthermore, since the basic data processing technologies (communication / transmission technology, data acquisition technology, data recording technology, data processing / analysis technology, machine learning technology, image processing technology, visualization technology, etc.) for realizing the present invention are well-known technologies, the description thereof will be omitted.

[0012] In addition, in this specification, the concept of "device" includes a single device that performs a specific function, and also includes a combination of multiple devices that exist in a distributed and independent manner but cooperate (link) to perform a specific function.

[0013] In the present specification, a "user" refers to a user of the image processing device of the present invention, and more specifically, for example, a person who uses text information, which will be described later, obtained by the functions of the image processing device of the present invention.

[0014] In addition, in this specification, "person" means a subject that performs a specific action, and includes individuals, groups, corporations such as companies, and organizations, and may also include computers and devices that constitute artificial intelligence (AI). Artificial intelligence (AI) is a technology that realizes intelligent functions such as inference, prediction, and judgment using hardware and software resources. The algorithm of artificial intelligence is arbitrary, and may be, for example, an expert system, case-based reasoning (CBR), Bayesian network, or subsumption architecture.

[0015] <<One embodiment of the present invention>> In one embodiment of the present invention (hereinafter, this embodiment), an image processing device (hereinafter, image processing device 10) generates text information (hereinafter, text information Tx) corresponding to an image (hereinafter, input image Pi) input to the image processing device 10, as shown in FIG. 1. The user can add the generated text information Tx to a desired position within or around the input image Pi. Adding the text information Tx means generating an image (text object) indicating the text information Tx and synthesizing it with the input image Pi.

[0016] More specifically, an example of an operation screen for generating text information shown in FIG. 5 will be described. In the example shown in FIG. 5, first, an input image Pi showing a "baby eating scene" is selected by the user. Then, the user clicks on an icon for generating text information displayed on the screen (the "generate" icon in FIG. 5). As a result, "delicious" is generated as text information Tx in the text information field shown in the lower right of FIG. 5 as text information Tx explaining the "baby eating scene". After the text information Tx is generated, the generated text information Tx is placed in or around the input image Pi by a predetermined operation by the user. In this way, in the image processing device 10, the generated text information Tx can be used by being added to or around an output image output on the screen.

[0017] In the present invention, an "image" is composed of a plurality of pixels, is represented by the gradation values ​​of each of the plurality of pixels, and includes at least one or more subjects. Digital image data (hereinafter, image data) that defines an image at a set resolution is generated by compressing data in which the gradation values ​​of each pixel are recorded using a predetermined compression method. Types of image data include, for example, lossy compressed image data such as JPEG (Joint Photographic Experts Group) format, and lossless compressed image data such as GIF (Graphics Interchange Format) or PNG (Portable Network Graphics) format.

[0018] In this embodiment, the method of inputting an image is not particularly limited, but includes, for example, inputting image data of an image captured by a photographing device such as a camera, and inputting read data obtained by scanning an existing photo with a scanner or the like. In addition, if a photographing device is installed in the image processing device, an image may be input by capturing an image with the photographing device and acquiring image data. In addition, an image may be input by downloading image data from an external device or a web server via a communication network.

[0019] The "input image Pi" may be a developed image obtained by developing a RAW image, a corrected image obtained by performing a specified correction process on the developed image, or an edited image obtained by performing editing and processing on the developed image or the corrected image. The edited image as the input image Pi may be an image constructed by arranging one or more images in a predetermined layout. In other words, the input image Pi may be an image having one or more image areas and blank areas other than the image area. The image area is an area that includes a subject, and the blank area is an area that does not include a subject.

[0020] The "text information Tx" is text information corresponding to the input image Pi, that is, character information indicating contents according to the input image Pi. More specifically, the input image Pi is character information indicating contents according to the subject in the input image Pi, such as comments and messages to the subject, explanations about the subject, and lines spoken by the subject. The text information is composed of a phrase or sentence consisting of one or more words. The text information Tx may be character information expressed in at least one of a font and decoration determined by a text style St described later, or may be character information fixed in a predetermined font and not including decoration.

[0021] "Subject" refers to people, animals, objects, backgrounds, etc., contained in an image. The concept of subject may also include the location, scene (e.g., dawn or dusk, clear skies, etc.) and theme (e.g., travel, meals, or events such as sports days) that appear in the image. Note that the image may be a landscape image, i.e., the only subject contained in the image may be a landscape, and the entire image may represent the landscape as the subject.

[0022] Here, as shown in FIG. 2, the text information Tx is generated based on an input image Pi and a text style St, and the text style St is determined based on the input image Pi.

[0023] The "text style St" refers to a style related to the expression of the text information Tx. The "text style St" may include, for example, information such as atmosphere, writing style, language system, expression technique, font, and decoration. The "atmosphere" refers to the impression that an expression creates or the feelings that can be felt from an expression when the scene of an image or the state or situation of a subject in an image is expressed (converted into text), and may include, for example, "serious," "fun," and "lively." The "writing style" may include, for example, colloquial and literary styles, and the "colloquial style" may include, for example, baby talk and dialects. The "language system" may include, for example, Japanese and English. The "expression technique" may include, for example, inversion, onomatopoeia such as onomatopoeia, and exclamations. The "font" may include, for example, Western fonts such as serifs, and handwritten fonts. The "decoration" may include, for example, bold, underline, italics, and color schemes.

[0024] Also, the text style St may be, for example, a further limited or detailed version of the text style St exemplified above. For example, the text style St may be "colloquial style popular with teenagers" by adding "popular with teenagers" indicating the gaze, aim, and intention to the "colloquial style" exemplified above.

[0025] In this way, the text style St is information for generating text information Tx, i.e., information for converting the content corresponding to the input image Pi into text, and may include information estimated from the content corresponding to the input image Pi. In addition, the text style St may include information indicating the characteristics of the input image Pi, specifically, information indicating the characteristics of the subject in the input image Pi. For example, if the subject in the input image Pi includes a "dog," then "dog" may be the text style St.

[0026] Furthermore, a combination of the text styles St exemplified above may be used as the text style St. For example, an onomatopoeia expressing a fun atmosphere may be created by combining "fun" as an example of an atmosphere with an onomatopoeia as an example of an expression technique.

[0027] As described above, the image processing apparatus according to this embodiment determines the text style St based on the input image Pi, and generates text information Tx corresponding to the input image Pi based on the input image Pi and the text style St. This makes it possible to add appropriate text information Tx to the input image Pi.

[0028] This point will be described more specifically using a comparative example (see FIG. 16) showing the conventional technology of Patent Document 1 and the like. In the image processing device according to the comparative example shown in FIG. 16, a keyword Kw is extracted by analyzing an input image Pi, and text information Tx0 corresponding to the keyword Kw is selected by referring to a database (not shown). Therefore, once the keyword Kw is determined, the content of the text information Tx0 is limited to the text information predetermined corresponding to the keyword Kw. However, even if an appropriate keyword Kw corresponding to an input image Pi is extracted, depending on the input image Pi, the text information predetermined corresponding to the keyword Kw may not correspond to the appropriate text information Tx0 corresponding to the input image Pi. In this case, the appropriate text information Tx0 is not assigned to the input image Pi.

[0029] In contrast, in the image processing device according to this embodiment, as shown in Fig. 2, text information Tx is generated based on both the input image Pi and the text style St, not just the text style St, so that appropriate text information Tx can be imparted to the input image Pi. In particular, in this embodiment, the text information Tx is generated and is not predetermined text information, so that more appropriate text information Tx can be imparted to the input image Pi. For the above reasons, the image processing device according to this embodiment can add appropriate text information Tx to the input image Pi.

[0030] [Example of image processing device configuration] Next, an example of the configuration of an image processing device according to this embodiment (hereinafter, referred to as image processing device 10) will be described with reference to FIG. The image processing device 10 is a computer used by a user, specifically a client terminal, specifically a smartphone, a tablet terminal, a notebook PC (Personal Computer), etc. The image processing device 10 is not limited to a computer owned by a user, and may be configured as a terminal not owned by a user but available by entering a PIN number, a password, or the like or making a deposit when visiting a store, such as a store-installed terminal. In the following, an example will be described in which the image processing device 10 is configured by a user-owned computer, specifically, a smartphone.

[0031] As shown in FIG. 3, the computer constituting the image processing device 10 includes a processor 10a, a memory 10b, a communication interface 10c, a storage 10d, an input device 10e, and an output device 10f.

[0032] The processor 10a may be composed of, for example, a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a TPU (Tensor Processing Unit), or an ASIC (Application Specific Integrated Circuit). The memory 10b is configured by semiconductor memories such as a Read Only Memory (ROM) and a Random Access Memory (RAM).

[0033] The communication interface 10c may be, for example, a network interface card, a communication interface board, etc. A computer constituting the image processing device 10 can communicate with other devices connected to a communication network such as the Internet and a mobile communication line via the communication interface 10c.

[0034] The storage 10d may be composed of, for example, a flash memory, a hard disc drive (HDD), a solid state drive (SSD), a flexible disc (FD), a magneto-optical disc (MO disc), a compact disc (CD), a digital versatile disc (DVD), a secure digital card (SD card), or a universal serial bus memory (USB memory). The storage 10d may be built in the computer body constituting the image processing device 10, or may be attached to the computer body in an external form. Alternatively, the storage 10d may be configured by a NAS (Network Attached Storage) or the like. The storage 10d may also be an external device, such as an online storage or a database server, capable of communicating with one of the computers constituting the image processing device 10 via a communication network.

[0035] The input device 10e is a device that accepts an input operation by a user, and is configured, for example, with a touch panel, etc. The input device 10e also includes a photographing device such as a built-in camera of a smartphone, a microphone for collecting sound, etc. The output device 10f is, for example, a display.

[0036] In addition, a program for an operating system (OS) and an application program for executing image processing are installed as software in the computer constituting the image processing device 10. These programs are read and executed by the processor 10a, whereby the computer constituting the image processing device 10 performs its functions, specifically, executes a series of processes related to the generation of the above-mentioned text information Tx. The application program for executing image processing may be obtained by reading it from a computer-readable recording medium, or may be obtained by downloading it via a communication network such as the Internet or an intranet.

[0037] The configuration of the image processing device 10 will be described again from the functional aspect with reference to Fig. 4. As shown in Fig. 4, the image processing device 10 has an image receiving unit 21, a text style determination unit 22, a text information generating unit 23, a text information output unit 24, a text information correcting unit 25, and a storage unit 26. These functional units are realized by the processor 10a of the image processing device 10 executing the above-mentioned application program for image processing and cooperating with other hardware devices of the image processing device 10. In addition, some functions may be realized by utilizing artificial intelligence (AI). Each functional unit will be described below.

[0038] (Image Reception Department) The image receiving unit 21 receives input of an image to which text information is to be added. As described above, the method of inputting an image is not particularly limited, but for example, a user may take a picture of a subject within the angle of view using a camera of a smartphone constituting the image processing device 10. In this case, the image receiving unit 21 receives input of image data of the taken image.

[0039] (Text style determination part) The text style determination unit 22 determines a text style St based on an image whose input is accepted by the image accepting unit 21, i.e., an input image Pi. Specifically, the text style determination unit 22 analyzes the input image Pi, and determines a text style St based on the analysis result.

[0040] "Analyzing the input image Pi" means, for example, identifying the characteristics of the image. "Image characteristics" are information about the image quality of each region of the image, the gradation values ​​of the pixels contained in each region, and information about the subject estimated from this information. The information about the subject may include the type of subject, the condition of the subject, the position of the subject in the image, and, if the subject is a person, the facial expression, etc. Moreover, it is preferable that the image features can be quantified, vectorized, or tensorized. In this case, the image analysis result is the quantified, vectorized, or tensorized image features, that is, feature quantities.

[0041] As an example of a method for determining the text style St based on the analysis result of the input image Pi, the text style determination unit 22 may determine the text style St using a determination model (corresponding to a second trained model) trained to output a text style by inputting an input image. The above determination model may be constructed by performing machine learning using, for example, previously acquired images and text styles associated with the images as training data. This allows the image processing device 10 to appropriately determine the text style St.

[0042] Alternatively, as another example of a determination method for determining a text style St based on the analysis results of an input image Pi, the text style determination unit 22 may identify the text style St corresponding to the input image Pi by referring to a determination table (corresponding to a second table) in which a correspondence relationship between an input image and a text style is previously set. This allows the image processing device 10 to appropriately determine the text style St.

[0043] Furthermore, the text style determination unit 22 may determine the text style St based on a specific subject area extracted by analyzing the input image Pi. More specifically, when determining the text style St based on the analysis result of the input image Pi, the text style determination unit 22 may select a specific subject region in the input image Pi as a region of interest and determine the text style St by prioritizing the analysis result of the selected region of interest over other regions. Note that in this embodiment, the selection of the region of interest is automatically performed by the image processing device 10.

[0044] An example of a "specific subject area" is an area of ​​a main subject in the input image Pi. Examples of the main subject include a subject located at the center of the input image Pi, a subject in focus, or a subject occupying the largest area among the subjects in the input image Pi. Thereby, in the image processing device 10, the text style St that focuses on the specific subject region is applied, and as a result, it is possible to generate text information Tx that emphasizes the specific subject region.

[0045] In the above description, the selection of the area of ​​interest is automatically performed by the image processing device 10, but the present invention is not limited to this and the area of ​​interest may be selected by the user. In this case, the area of ​​interest is selected by the user through the input device 10e. For example, the input image Pi may be displayed on a touch display of a smartphone constituting the image processing device 10, and the user may touch the area of ​​interest in the displayed input image Pi. This allows the image processing device 10 to reflect the user's intention in determining the text style St.

[0046] (Text information generation section) The text information generating unit 23 generates text information Tx corresponding to the input image Pi, based on the input image Pi and the text style St. Specifically, the text information generating unit 23 generates the text information Tx, based on the analysis result of the input image Pi and the determined text style St.

[0047] To explain an example of a method for generating the text information Tx, the text information generating unit 23 may generate the text information Tx using a generation model (corresponding to a first trained model) trained to output text information by inputting an input image and a text style. The above generation model may be constructed by performing machine learning using, for example, a combination of an image and a text style acquired in the past and text information associated with the combination as training data. This allows the image processing device 10 to generate the text information Tx more appropriately.

[0048] Alternatively, the text information generation unit 23 may refer to a generation table (corresponding to a first table) in which a correspondence relationship between the input image and text style and the text information is preset, and identify the text information Tx corresponding to the input image Pi and text style St. This allows the image processing device 10 to generate the text information Tx more appropriately.

[0049] (Text information output section) The text information output unit 24 outputs the text information Tx generated by the text information generating unit 23. The method of outputting the text information Tx is not particularly limited, but for example, as shown in FIG. 5, the text information Tx may be displayed on the screen of a display (output device 10f) of a computer constituting the image processing device 10.

[0050] (Text information correction section) When the text information correction unit 25 receives input information (corresponding to the third input information) related to the correction of the text information Tx, the text information correction unit 25 corrects the text information Tx based on the input information. For example, as shown in Fig. 5, when the user clicks on the "correct" icon displayed on the screen, the text information correction unit 25 enables editing of the text information Tx displayed in the text information field on the screen. After that, when the user clicks on the "confirm" icon displayed on the screen, the text information correction unit 25 corrects the text information Tx based on the input information related to the correction by the user. In this way, the image processing device 10 can reflect the user's intention in the generation of the text information Tx.

[0051] (Preservation Department) The storage unit 26 stores at least one of the determined text style St and the generated text information Tx as incidental information of the corresponding input image Pi. The input image Pi and the incidental information may be stored, for example, in the storage 10d of the image processing device 10, or may be stored in the storage of a computer indirectly available to the user, for example, a server computer capable of communicating with the image processing device 10. Furthermore, the supplementary information may include text information Tx corrected by the text information correcting unit 25.

[0052] The input image Pi and the above-mentioned incidental information may be stored together as the same data file, or may be stored as separate data files. When stored as separate data files, one data file may store path information to the other data file. Also, a management database may be provided for managing the correspondence between the two data files. The management database may indicate, for example, the correspondence between the identification ID of the data file of the input image Pi and the identification ID of the data file of each of the incidental information. This allows the image processing device 10 to organize the input image Pi, the text style St, and the text information Tx in association with each other. Moreover, in the image processing device 10, by storing each piece of information on the text style St and the text information Tx, it is possible to use the information in future image processing by the image processing device 10. For example, as in a sixth modified example of the present invention (see FIG. 13) described later, it is possible to use the above-mentioned pieces of information when updating the above-mentioned generative model, generative table, and the like.

[0053] [Example of image processing method according to this embodiment] Next, as an operation example of the image processing device 10 in this embodiment, an image processing flow using the same device will be described. In the image processing flow described below, the image processing method of the present invention is used. In other words, each step in the image processing flow described below corresponds to a component of the image processing method of the present invention. Note that the flow below is merely an example, and some steps in the flow may be deleted, new steps may be added to the flow, or the order of execution of two steps in the flow may be reversed, as long as it does not deviate from the spirit of this embodiment.

[0054] Each step in the image processing flow according to this embodiment is performed in the order shown in Fig. 6 by the processor 10a included in the image processing device 10. That is, in each process in the image processing flow, the processor 10a executes processing corresponding to each step in Fig. 6 among data processing defined in an application program for image processing.

[0055] Specifically, in the image processing flow according to the present embodiment, the processor 10a first executes a reception process (S001). In the reception process, the processor 10a receives an input of an image to which text information Tx is to be added by this flow, in other words, obtains an input image Pi to be processed. Next, the processor 10a executes a determination process to determine a text style St based on the input image Pi (S002).

[0056] Next, the processor 10a executes a generation process to generate text information Tx corresponding to the input image Pi, based on the input image Pi and the text style St (S003).

[0057] Next, the processor 10a executes an output process to output the generated text information Tx to the user (S004). The processor 10a presents the text information Tx to the user, for example, through a screen shown in FIG. 5. Next, the processor 10a executes a determination process to determine whether or not input information related to the correction of the text information Tx has been received (S005). When input information related to the correction of the text information Tx has been received, the processor 10a executes a correction process to correct the text information Tx based on the input information (S006).

[0058] If the processor 10a does not receive input information regarding modification of the text information Tx, the processor 10a executes a saving process to save at least one of the determined text style St and the generated text information Tx as auxiliary information of the corresponding input image Pi (S007).

[0059] When the series of processes described above is completed, the image processing flow according to this embodiment is completed. The image processing flow is executed each time an input image Pi is received. In other words, each time a new input image Pi is received, the series of processes described above is repeatedly executed by the processor 10a.

[0060] <<Other embodiments>> Although specific embodiments of the present invention have been described above, the above-described embodiments are merely examples given to facilitate understanding of the present invention and are not intended to limit the present invention. In other words, the present invention may be modified or improved from the embodiments described below without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof. The modified example will be described below, focusing on the differences between the modified example and the above embodiment.

[0061] (Regarding the first modified example) In the above embodiment, the image processing device 10 determines the text style St based on the input image Pi. However, this is not limited to this, and the text style St may be determined based on input information (corresponding to first input information) related to the text style St, as in the image processing device 10A shown in FIG. More specifically, the image processing device 10A may further include a determination mode setting unit 27 as a functional unit in addition to the image processing device 10. The determination mode setting unit 27 sets a first determination mode in which a text style St is determined based on an input image Pi, or a second determination mode in which a text style St is determined based on input information related to the text style St. More specifically, the determination mode setting unit 27 sets one of the first determination mode or the second determination mode according to an instruction of the determination mode by the user.

[0062] Next, when the first determination mode is set by the determination mode setting unit 27, the text style determination unit 22 determines the text style St based on the input image Pi, as in the above embodiment. On the other hand, when the second determination mode is set by the determination mode setting unit 27, the text style determination unit 22 determines the text style St based on input information related to the text style St. The input information related to the text style St includes, for example, the above-mentioned atmosphere, writing style, language system, expression technique, font, decoration, and other information. For example, the user inputs "fun" or the like, which indicates the atmosphere of the text information Tx, as input information related to the text style St, to the image processing device 10 through the input device 10e. As a result, the text style determination unit 22 determines the text style St based on the input information related to the input text style St. In this way, the image processing device 10A can reflect the user's intention in determining the text style St.

[0063] Note that the input information regarding the text style St is not limited to the input information from the user. For example, the determination mode setting unit 27 may determine the text style St based on the input information regarding the text style St from an external information processing device different from the image processing device 10A. An example will be used to explain in more detail. Assume that a photo book consisting of a plurality of images including an input image Pi has a character image (e.g., an image showing a message, etc.) showing character information different from the text information Tx arranged on the cover and in the margins of each page. In this case, an external information processing device may obtain input information related to the text style St (e.g., the above-mentioned "fun" etc.) by analyzing the character image. In this manner, the text style determination unit 22 may determine the text style St based on input information relating to the text style St acquired from an external information processing device or the like.

[0064] (Regarding the second modified example) In the above embodiment, the text information Tx is generated based on the text style St that is initially determined based on the input image Pi. However, the present invention is not limited to this, and the text information Tx may be generated based on a redetermined text style St as in an image processing device 10B shown in FIG.

[0065] More specifically, compared to the image processing device 10, the image processing device 10B may further include a text style output unit 28, a second input information receiving unit 29, and a text style redetermining unit 30 as functional units. The text style output unit 28 outputs the text style St determined by the text style determination unit 22, specifically, displays it on the screen of the output device 10f. When the user changes the text style St displayed on the screen, the user inputs input information (corresponding to second input information) related to the text style St. This causes the second input information receiving unit 29 to receive the input information related to the text style St. Then, the text style redetermining unit 30 redetermines the text style based on the received input information related to the text style St. Thereafter, the text information generating unit 23 generates text information Tx based on the input image Pi and the redetermined text style St.

[0066] As described above, in the image processing device 10B, if the user checks the text style St initially determined based on the input image Pi and determines that the text style St needs to be changed, the user can redetermine the text style St. This allows the user's intention to be reflected in the determination of the text style St.

[0067] (Regarding the third modified example) In the above embodiment, one piece of text information Tx is generated. However, the present invention is not limited to this, and two or more candidates of text information Tx may be generated as in the image processing device 10C shown in Fig. 9, and one piece of text information Tx to be added to the input image Pi may be determined from the two or more pieces of text information Tx generated.

[0068] More specifically, compared to the image processing device 10, the image processing device 10C may further include a candidate generating unit 31, a candidate output unit 32, a selected information receiving unit 33, and a text information determining unit 34 as functional units. The candidate generation unit 31 generates two or more candidates for text information Tx based on the input image Pi and the text style St. More specifically, first, the text style determination unit 22 determines two or more text styles St based on the input image Pi. Then, the candidate generation unit 31 generates two or more candidates for text information Tx based on a combination of the input image Pi and each of the two or more text styles St. The candidate output unit 32 outputs the two or more candidates of text information Tx that have been generated, and displays the two or more candidates of text information Tx on the screen of the output device 10f, for example. The user selects the desired text information Tx from the two or more candidates of text information Tx displayed on the screen. As a result, the selection information receiving unit 33 receives the text information Tx selected by the user as selection information. The text information determination unit 34 determines the text information Tx from the two or more candidates of text information Tx based on the received selection information.

[0069] As described above, in the image processing device 10C, the user can select the text information Tx from among candidates for the text information Tx, and therefore the user's intention can be reflected in the generation of the text information Tx.

[0070] (Regarding the fourth modified example) Moreover, a functional unit may be further added to the image processing device 10C described above to form an image processing device 10D shown in Fig. 10. The image processing device 10D may further include an evaluation score assigning unit 35 in addition to the functional units of the image processing device 10C.

[0071] Specifically, first, the text style determination unit 22 determines two or more text styles St based on the input image Pi, similarly to the third modified example described above. Next, the evaluation score assignment unit 35 assigns an evaluation score to each of the determined two or more text styles St according to the correlation with the contents of the input image Pi. Note that, for example, a higher evaluation score may be assigned to a more specific or detailed text style St according to the relevance. For example, a higher evaluation score may be assigned to a "colloquial style popular among teenagers," which is more specific information, compared to the "colloquial style" exemplified in the above description, as the evaluation score of the text style St. After the evaluation score assigning unit 35 assigns an evaluation score to each text style St, the candidate generating unit 31 generates text information Tx for each text style St to which the evaluation score has been assigned, based on the input image Pi and the text style St.

[0072] Thereafter, the candidate output unit 32 outputs the two or more pieces of generated text information Tx based on the evaluation score of the text style St. More specifically, as in the example shown in FIG. 11, the text information Tx "Delicious" is displayed on the screen in association with an evaluation score of "80" of the corresponding text style St "Baby talk." Also, the text information Tx "Ah~n" is displayed on the screen in association with an evaluation score of "60" of the corresponding text style St "Onomatopoeia." The two or more candidates for text information Tx may be displayed in ascending or descending order of the evaluation score of the text style St. The user selects desired text information Tx from two or more candidates of text information Tx displayed on the screen while referring to the evaluation score of the text style St. The selection information receiving unit 33 receives the text information Tx selected by the user as selection information. The text information determination unit 34 determines the text information Tx from two or more candidates of text information Tx based on the received selection information by the user.

[0073] As described above, in the image processing device 10D, the user can reflect his / her own intention in the generation of text information Tx while referring to the evaluation score of the text style St by the image processing device 10D.

[0074] (Regarding the fifth modified example) In addition, as shown in FIG. 12, the image processing device 10E may further have, as functional units compared to the image processing device 10, a regeneration receiving unit 36, a text style output unit 37, and a selection information receiving unit 38 in addition to the aforementioned evaluation score assignment unit 35.

[0075] More specifically, first, the text style determination unit 22 determines two or more text styles St based on the input image Pi. Next, the evaluation score assignment unit 35 assigns an evaluation score to each of the determined two or more text styles St according to the correlation with the content of the input image Pi. Thereafter, the text information generating unit 23 generates text information Tx based on the input image Pi and the text style St with the highest evaluation score among the two or more text styles St. More specifically, as shown in Fig. 11, when the text style St with the highest evaluation score among the two or more text styles St is "baby talk", "delicious" is generated as text information Tx based on the input image Pi and the text style St as "baby talk". The text information output unit 24 outputs this text information Tx to, for example, the screen of the output device 10f.

[0076] Here, when the user makes a request for regeneration of the text information Tx through the input device 10e, the regeneration receiving unit 36 ​​receives the request for regeneration of the text information Tx. As a result, the text style output unit 37 outputs a text style St different from the text style St with the highest evaluation score. More specifically, the text style output unit 37 outputs "onomatopoeia" as a text style St different from the text style St with the highest evaluation score to the screen of the output device 10f, as shown in FIG. 11. Then, when the user selects a desired text style St from the one or more output text styles St, the selection information receiving unit 38 receives it as selection information related to the text style St. After that, the text information generating unit 23 regenerates text information Tx based on the input image Pi and the text style St selected by the user. In the example shown in Fig. 11, when the text style St "onomatopoeia" is selected by the user, the text information Tx "aa~n" is regenerated.

[0077] As described above, the image processing device 10F can generate text information Tx based on the text style St with the highest evaluation score with priority, while being able to generate text information Tx based on a text style St different from that in response to the user's intention.

[0078] (Regarding the sixth modified example) 13, the image processing device 10F may further include a reconstruction unit 39 as a functional unit in addition to the image processing device 10. The following two examples of the function of the reconstruction unit 39 are given. As a first example, the reconstruction unit 39 may reconstruct the generative model used in the text information generation unit 23. As described above, the generative model is constructed by machine learning using training data including an input image, a text style, and text information. As shown in FIG. 13, the reconstruction unit 39 acquires corrected text information Tx from at least one of the text information correction unit 25 and the storage unit 26. Then, the reconstruction unit 39 reconstructs the generative model using the training data including the corrected text information Tx. Note that "reconstructing" means performing re-learning to update an already constructed trained model, more specifically, updating parameters, coefficients, etc. of the trained model.

[0079] As a second example, the reconstruction unit 39 may modify the generation table used in the text information generation unit 23. As described above, the generation table is a table in which a correspondence relationship between the input image and the text style and the text information is set in advance. As shown in Fig. 13, the reconstruction unit 39 acquires the modified text information Tx from at least one of the text information modification unit 25 and the storage unit 26. Then, the reconstruction unit 39 modifies the generation table using the modified text information Tx. As described above, the image processing device 10F can reconstruct at least one of the generative model and the generative table, and as a result, can provide more appropriate text information Tx.

[0080] (Regarding the seventh modified example) In addition, in the above embodiment, one piece of text information Tx is generated based on one text style St, but this is not limited to this, and one or more pieces of text information Tx may be generated based on two or more text styles St, as shown in FIG. 14. In this case, the text style determination unit 22 may determine two or more text styles St based on the input image Pi, and the text information generation unit 23 may generate one or more pieces of text information Tx based on the two or more text styles St.

[0081] Here, the two or more text styles St may be two or more text styles St with different themes. That is, the text style determination unit 22 may determine two or more text styles St with different themes based on the input image Pi, and the text information generation unit 23 may generate text information Tx for each combination of two or more text styles St with different themes.

[0082] The themes of the text style St may be classified based on information such as the above-mentioned atmosphere, writing style, language system, expression technique, font, decoration, etc. Furthermore, each classified theme may be further classified into more detailed categories.

[0083] Although the method of using one determination model has been given as an example of the method of determining the text style St, for example, multiple determination models may be used. That is, the text style determination unit 22 may determine the text style St output from each of the multiple determination models as two or more text styles St. Alternatively, the text style determination unit 22 may use a plurality of determination models constructed for each theme of the text style St. In this case, the text style determination unit 22 may determine the text styles St having different themes output from each of the plurality of determination models as two or more text styles St.

[0084] As described above, in the image processing device according to the seventh modification, more appropriate text information Tx can be generated by using a plurality of text styles St. In particular, by using text styles St with different themes, text information Tx is generated based on a plurality of viewpoints. This makes it possible to provide more appropriate text information Tx to the input image Pi.

[0085] (Regarding the eighth modified example) Furthermore, in the above embodiment, one input image Pi is input (acquired), but this is not limiting. For example, as shown in FIG. 15, a plurality of input images Pi may be input. In this case, first, the image receiving unit 21 acquires a plurality of input images Pi. Next, the text style determination unit 22 determines a text style St common to the acquired multiple input images Pi. More specifically, the image receiving unit 21 analyzes each of the multiple input images Pi, and determines one text style St for the multiple input images Pi based on the analysis result of each input image Pi. In other words, determining a common text style St means determining one text style St for the multiple input images Pi. Next, the text information generating unit 23 generates text information Tx common to each of the multiple input images Pi based on the multiple input images Pi and the common text style St. Note that generating common text information Tx means generating one piece of text information Tx for the multiple input images Pi. As described above, the image processing device according to the eighth modification can generate one common text information Tx for a plurality of input images Pi. This makes it possible to generate text information Tx common to a plurality of input images Pi arranged on a page, for example, as a title in the page of a photo book. Note that the image processing device according to the eighth modification may generate common text information Tx as a title for each page of a photo book, while also generating text information Tx for each of the plurality of input images Pi arranged on the page.

[0086] (About the computer that constitutes the image processing device) In the above embodiment, the image processing device of the present invention is configured by a computer that is directly used by a user, such as a terminal (client terminal) owned by the user. However, the present invention is not limited to this, and the image processing device of the present invention may be configured by a computer that is indirectly available to the user, for example, a server computer. Here, the above server computer may be, for example, a server computer for cloud services, specifically, a server computer for ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). In this case, when necessary information is input at the client terminal, the above server computer performs various processes (calculations) based on the input information, and the calculation results are output at the client terminal. In other words, the functions of the server computer that constitutes the image processing device of the present invention can be used at the client terminal.

[0087] (Processor configuration) The processor included in the image processing device of the present invention includes various processors, including, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. The various processors also include programmable logic devices (PLDs) such as field programmable gate arrays (FPGAs) whose circuit configurations can be changed after manufacture. Furthermore, various processors include dedicated electric circuits such as ASICs (Application Specific Integrated Circuits), which are processors having a circuit configuration designed exclusively for performing specific processing.

[0088] Furthermore, one functional unit of the image processing device of the present invention may be configured by one of the various processors described above, or may be configured by a combination of two or more processors of the same or different types, for example, a combination of multiple FPGAs, or a combination of an FPGA and a CPU, etc. Furthermore, the multiple functional units of the image processing device of the present invention may be configured by one of various processors, or two or more of the multiple functional units may be integrated into one processor. Also, as in the above-described embodiment, one processor may be configured by combining one or more CPUs and software, and this processor may function as multiple functional units.

[0089] Also, for example, a processor may be used that realizes the functions of the entire system including the multiple functional units in the image processing device of the present invention on a single IC (Integrated Circuit) chip, as typified by SoC (System on Chip), etc. Also, the hardware configuration of the various processors described above may be an electric circuit (Circuitry) that combines circuit elements such as semiconductor elements. [Explanation of symbols]

[0090] 10, 10A, 10B, 10C, 10D, 10E, 10F Image processing device 10a processor 10b Memory 10c Communication Interface 10d Storage 10e Input Device 10f output device 21 Image Reception Section 22 Text style determination section 23 Text information generation unit 24 Text information output section 25 Text information correction section 26 Preservation Department 27 First input information receiving unit 28,37 Text style output section 29 Second input information receiving unit 30 Text style redetermination unit 31 Candidate generation section 32 Candidate output section 33,38 Selection information reception section 34 Text information determination unit 35 Evaluation Score Assignment Department 36 Regeneration reception section 39 Reconstruction Department Kw Keywords Pi Input image St Text Style Tx,Tx0 Text information

Claims

1. An image processing device including a processor, The processor, Determine text style based on image, An image processing device that generates text information corresponding to the image based on the image and the text style.

2. The processor, setting a first determination mode for determining the text style based on the image or a second determination mode for determining the text style based on first input information; The image processing device according to claim 1 , wherein the text style is determined based on the first determination mode or the second determination mode that has been set.

3. The image processing device according to claim 2 , wherein, when the second determination mode is set, the processor determines the text style based on the first input information regarding the text style from a user.

4. The processor, outputting the determined text style; Re-determining the text style based on second input information regarding the text style; The image processing apparatus of claim 1 , further comprising: generating the text information based on the image and the re-determined text style.

5. The processor, The image processing apparatus according to claim 1 , further comprising: determining the text style based on a specific subject area extracted by analyzing the image.

6. The processor, determining two or more of the text styles based on the image; The image processing device according to claim 1 , wherein the text information is generated based on the two or more text styles.

7. The processor, determining the two or more thematically distinct text styles based on the image; The image processing apparatus according to claim 6 , wherein the text information is generated for each combination of the two or more text styles having different themes.

8. The image processing device according to claim 1 , wherein the processor generates the text information using a first trained model trained to output text information by inputting an image and a text style.

9. The image processing device according to claim 1 , wherein the processor refers to a first table in which a correspondence relationship between an image and a text style and text information is preset, and specifies the text information corresponding to the input image and the text style.

10. The image processing device according to claim 1 , wherein the processor determines the text style using a second trained model trained to output a text style by inputting an image.

11. The image processing device according to claim 1 , wherein the processor refers to a second table in which a correspondence relationship between an image and a text style is preset, and specifies the text style corresponding to the input image.

12. The processor, generating two or more candidates for the text information based on the image and the text style; Outputting the two or more generated candidates of text information; The image processing apparatus according to claim 1 , wherein the text information is determined from among the two or more text information candidates based on information selected by a user.

13. The processor, determining two or more of the text styles based on the image; assigning an evaluation score to each of the two or more text styles according to a correlation with the content of the image; generating the text information for each of the text styles to which the evaluation score has been assigned based on the image and the text style; The image processing apparatus according to claim 1 , wherein the generated two or more pieces of text information are output based on the evaluation score of the text style.

14. The processor, determining two or more of the text styles based on the image; assigning an evaluation score to each of the two or more text styles according to a correlation with the content of the image; generating the text information based on the image and the text style having the highest evaluation score among the two or more text styles; Outputting the generated text information; When a request to regenerate the text information is received, a text style different from the text style with the highest evaluation score is output; The image processing apparatus according to claim 1 , further comprising: a processor configured to regenerate the text information based on the image and the text style selected by a user.

15. The processor, Outputting the generated text information; The image processing device according to claim 1 , wherein, when third input information relating to the correction of the text information is received, the text information is corrected based on the third input information.

16. The image processing device according to claim 15 , wherein the processor reconstructs a first trained model, which is constructed by machine learning using training data including an image, a text style, and text information, using the training data including the modified text information.

17. The image processing apparatus according to claim 15 , wherein the processor modifies a first table, in which a correspondence relationship between an image and text style and text information is previously set, by using the modified text information.

18. The processor, acquiring a plurality of said images; determining the text style common to the captured images; The image processing apparatus according to claim 1 , further comprising: generating the text information common to each of the plurality of images based on the plurality of images and the common text style.

19. The image processing device according to claim 1 , wherein the processor stores at least one of the determined text style and the generated text information as supplementary information of the corresponding image.

20. determining a text style based on the image; generating text information corresponding to the image based on the image and the text style, the image processing method being executed by a processor.

21. A program for causing a computer to execute each step included in the image processing method according to claim 20.

22. A computer-readable recording medium, 21. A recording medium having recorded thereon a program for causing a computer to execute each process included in the image processing method according to claim 20.

Images