Image processing apparatus, image processing method, program, and recording medium

Abstract

An apparatus, method, program, and recording medium are provided in which a user can efficiently perform accurate correction of an image generated by an AI. An image processing apparatus (10) according to an embodiment of the present application includes a processor (11) that receives an image and information associated with the image, and determines a processing item related to the image based on at least one of the image and the information associated with the image.

Description

Technical Field

[0001] One embodiment of the present invention relates to an image processing apparatus, image processing method, program, and recording medium that enables a user to efficiently and accurately correct images generated by an AI. Background Technology

[0002] The need to depict and edit images on a computer for various purposes exists not only in fields such as photography and graphic design, but also widely in fields such as medicine and manufacturing. Therefore, technologies for freely editing image quality, tone, exposure, etc., are known. As an example of such technology, the technology described in Patent Document 1 can be cited. Patent Document 1 discloses a technology for the interface between an image processing device and a camera device, as well as an image processing program, which allows for easy operation even when using grayscale conversion processing, which offers high freedom but whose parameters are difficult to intuitively grasp.

[0003] This technology relates to an image processing apparatus, characterized by comprising: a display mechanism for displaying image data; a perceptual word designation mechanism for designating perceptual words that perceptually express the image processing effect; a target value setting mechanism for setting a target value corresponding to the perceptual word; a region designation mechanism for designating a specific region on the image displayed on the display mechanism; an image quality evaluation quantity calculation mechanism for calculating an image quality evaluation quantity based on the specific region; an image processing mechanism for performing image processing on the image data according to predetermined image processing parameters corresponding to the perceptual word designated by the perceptual word designation mechanism, and generating a processed result image; and a control mechanism for comparing the image quality evaluation quantity calculated by the image quality evaluation quantity calculation mechanism from the region on the processed result image designated by the region designation mechanism with the target value set by the target value setting mechanism, correcting the image processing parameters based on the comparison result, and controlling the image processing to be performed again.

[0004] Furthermore, Patent Document 2 discloses a technique for generating another quantitative image or emphasis image based on a quantitative image or emphasis image even when the theoretical formula or formula representing the relationship between a quantitative image or emphasis image and another quantitative image or emphasis image is unknown.

[0005] This technology relates to a medical image diagnostic support device, comprising: a learning image receiving unit that receives two or more learning images and one or more forward resolution images; a synthesis parameter determination unit that, when synthesizing pixel values ​​of corresponding pixels in the two or more learning images using synthesis parameter values, determines synthesis parameter values ​​whose synthesized pixel values ​​are close to the pixel values ​​of corresponding pixels in the forward resolution image; an examination target image receiving unit that receives images of the same type as the two or more learning images as examination target images, wherein the images are generated for a first subject; and an image synthesis execution unit that, by using... The pixel values ​​of corresponding pixels in two or more inspection object images are synthesized using the synthesis parameter values ​​obtained by the synthesis parameter determination unit to generate a synthesized image desired by the user. The two or more learning images are two or more types of images generated for the first subject or a second subject different from the first subject. The one or more correct solution images are images generated for the second subject or a third subject different from the first and second subjects. The image is an image of a different type from the learning images and an image of the same type as the synthesized image desired by the user.

[0006] Previous technical documents Patent documents Patent Document 1: Japanese Patent Application Publication No. 2006-093757 Patent Document 2: Japanese Patent Application Publication No. 2019-180637 Summary of the Invention

[0007] The technical problem to be solved by the invention In recent years, so-called generative AI (Artificial Intelligence) has been able to be utilized in internet services or prescribed applications, and can be easily and conveniently used in various situations. In this type of generative AI, simply inputting text information corresponding to the desired image into its user interface will output the corresponding image.

[0008] On the other hand, it's not uncommon for AI-generated images to differ from the user's intended image. While users can easily obtain an image simply by inputting text information, efficiently obtaining the desired image is also difficult. In such situations, users repeatedly modify, add to, and delete the input text information in an effort to make the AI-generated image approximate their desired one.

[0009] On the other hand, it is difficult for humans to understand which algorithms AI uses to generate images. Therefore, as mentioned above, it is uncertain whether changes made by the user to textual information will result in the desired image being generated. Users find it difficult to input / express through text anything other than simple modifications such as changing the subject from male to female, or even subtle changes that resonate with human emotions, such as altering the atmosphere or impression of objects in the background.

[0010] One embodiment of the present invention was made in view of the above circumstances, and its object is to provide an apparatus, method, program and recording medium that enables a user to efficiently perform accurate corrections on images generated by AI.

[0011] means for solving technical problems The above objective is achieved by any of the image processing apparatus described in any one of [1] to [9].

[0012] [1] An image processing apparatus comprising a processor, wherein the processor receives an image and information associated with the image, and determines processing items associated with the image based on at least one of the image and the information associated with the image.

[0013] [2] According to the image processing apparatus of [1], wherein the image is an image generated based on information associated with the image.

[0014] [3] According to the image processing apparatus of [1], the information related to the image is text generated from the image.

[0015] [4] According to the image processing apparatus of [1], the processor receives information related to an object that the user wishes to depict in the image as information related to the image, and determines the processing item based on at least one of the image and the received information related to the object.

[0016] [5] According to the image processing apparatus of [4], the processor receives information related to a depicted region of the object in the image, and also uses the information related to the depicted region to determine the processing item.

[0017] [6] According to the image processing apparatus of [1], the processor determines the amount of correction for the processing item based on at least one of the image and information related to the image.

[0018] [7] According to the image processing apparatus of [6], the processor determines the correction amount of the processing item as information recorded in text.

[0019] [8] According to the image processing apparatus of [6], wherein the correction amount of the processing item represents at least one of the forms or states of a concept.

[0020] [9] According to the image processing apparatus of [6], the processor prompts the user to receive a specified amount of correction for the processing item from the user, and performs the image correction processing according to the amount of correction received from the user.

[0021] Furthermore, the above objectives can also be achieved by the image processing method described below

[10] .

[0022]

[10] An image processing method includes the following steps: receiving an image and information associated with the image by a processor; and determining, by the processor, processing items associated with the image based on at least one of the image and the information associated with the image.

[0023] Furthermore, the above objective can also be achieved through the procedure described below

[11] .

[0024]

[11] A program for causing a computer to perform the steps included in the image processing method described in

[10] .

[0025] Furthermore, the above objectives can also be achieved using the recording medium described below

[12] .

[0026]

[12] A recording medium, which is a computer-readable recording medium, wherein the recording medium records a program for causing a computer to perform the steps included in the image processing method described in

[10] .

[0027] Invention Effects According to one embodiment of the present invention, an apparatus, method, program, and recording medium are provided that enable a user to efficiently and accurately correct images generated by AI. Attached Figure Description

[0028] Figure 1 This is a diagram illustrating a structural example of the image processing system according to this embodiment.

[0029] Figure 2 This is a diagram illustrating an example of the hardware structure of the image processing apparatus according to this embodiment.

[0030] Figure 3 This diagram illustrates the functional units of the image processing apparatus according to this embodiment.

[0031] Figure 4 This is a diagram illustrating an example of a text information table as described in this embodiment.

[0032] Figure 5 This is a diagram illustrating an example of an input image table according to this embodiment.

[0033] Figure 6 This is a diagram showing an example of a table of modifications involved in this embodiment.

[0034] Figure 7 This is a diagram illustrating an example of a generated image table according to this embodiment.

[0035] Figure 8 This is a flowchart illustrating (1) of the image processing method according to this embodiment.

[0036] Figure 9 This is a diagram showing an example of a processing rule table involved in this embodiment.

[0037] Figure 10 This is a flowchart illustrating (3) of the image processing method according to this embodiment.

[0038] Figure 11 This is a diagram showing an example (1) of the output screen involved in this embodiment.

[0039] Figure 12 This is a diagram showing an example (2) of the output screen involved in this embodiment.

[0040] Figure 13 This is a diagram showing an example (3) of the output screen in this embodiment.

[0041] Figure 14 This is a diagram showing an example (4) of the output screen involved in this embodiment.

[0042] Figure 15 This is a flowchart (2) illustrating the image processing method according to this embodiment.

[0043] Figure 16 This is a diagram showing an example (5) of the output screen in this embodiment.

[0044] Figure 17 This is a diagram showing an example (6) of the output screen involved in this embodiment. Detailed Implementation

[0045] The specific embodiments of the present invention will be described below.

[0046] Furthermore, for ease of explanation, the following descriptions will sometimes be presented from a GUI (Graphical User Interface) perspective. Also, the fundamental data processing technologies used to implement this invention (communication / transmission technologies, data acquisition technologies, data recording technologies, data processing / analysis technologies, machine learning technologies, image processing technologies, and visualization technologies, etc.) are well-known technologies, and therefore related descriptions are omitted.

[0047] Furthermore, in this specification, the concept of "device" includes a single device that performs a specific function, and also includes a combination of multiple devices that exist separately and independently of each other and are linked (jointly) to perform a specific function.

[0048] Furthermore, in this specification, "user" refers to the user of the image processing apparatus of the present invention, specifically, for example, a person who performs image correction processing based on image generation AI using the functions of the image processing apparatus of the present invention.

[0049] Furthermore, in this specification, "human" refers to the subject performing a specific action, including individuals, groups, legal persons and organizations such as enterprises, and further, it may include computers and devices that constitute artificial intelligence (AI). Artificial intelligence (AI) uses hardware and software resources to achieve intelligent functions such as reasoning, prediction, and judgment. The algorithms of artificial intelligence are arbitrary, such as expert systems, case-based reasoning (CBR), convolutional neural networks (CNN), deep neural networks (DNN), Bayesian networks, or subsumption architectures, etc.

[0050] <<One embodiment of the present invention>> [Structure of an image processing system] In one embodiment of the present invention (hereinafter referred to as this embodiment), it is assumed that the connection to Figure 1 The image processing system 5 shown in the diagram comprises an image processing device 10, a user terminal 30, a photographic device 40, and a server computer 50. The user terminal 30 and the photographic device 40 are information processing devices used by the user to capture images as needed to generate the base image and upload them to the image processing device 10. In addition to uploading the base image, the user also uploads text information required for image generation based on image generation AI to the image processing device 10.

[0051] Alternatively, the user may choose not to use the user terminal 30 or the camera device 40, but instead use the UI (User Interface) of the image processing device 10. For example, the user may scan or print a handheld photograph via media, view the screen of their own user terminal 30 (the screen displaying the photograph image), or read the data as the base image, or directly input text information using a keyboard, mouse, touchscreen, or other UI-based device. In this configuration / application, the image processing system 5 may consist solely of the image processing device 10.

[0052] Image processing device 10 stores the base image uploaded from user terminal 30 or photographic device 40 in input image table 212 (in Figure 3 , Figure 5 (described later), and the text information is stored in text information table 211 (in Figure 3 , Figure 4 (described later), and used for image generation or explanatory text generation.

[0053] The image processing apparatus 10 is an information processing apparatus that uses its own image generation AI and descriptive text generation AI, or those provided by the server computer 50, to perform image generation or modification processing based on the aforementioned basic image and text information. However, this processing is performed according to the image processing method of the present invention and based on user instructions received by a specified UI.

[0054] Furthermore, as described above, the server computer 50 is a server device that provides image generation AI and explanatory text generation AI to the image processing apparatus 10 via the network 1. Of course, if the image processing apparatus 10 is configured to not require the provision of image generation AI and explanatory text generation AI, this server computer 50 may not be included in the image processing system 5.

[0055] In addition to the configuration where the image processing device 10 and the photographing device 40 are connected by network 1, the configuration can also be a configuration where the internal bus wiring of the image processing device 10 is directly connected to the interface of the photographing device 40.

[0056] Furthermore, it is conceivable that the photographer performing various operations related to photography or photo printing output in the photography device 40, the registered user inputting the image (base image) obtained through the photography or its photo printing input (which may include scanning or uploading from the user terminal 30) to the image processing device 10, and the viewer browsing the generated image or its correction result as the processing result in the image processing device 10 on the user terminal 30 are different people, or all of them are the same person.

[0057] [Example of the structure of an image processing device] Next, refer to Figure 2 and Figure 3 The structure of the image processing apparatus 10 according to this embodiment will be described below. The image processing apparatus 10 is composed of a computer used by a user. Specifically, in addition to a PC (Personal Computer), it can also be composed of a smartphone, tablet terminal, or laptop PC. Furthermore, the image processing apparatus 10 is not limited to a computer owned by the user, and can also be composed of a terminal that the user does not own but can use when visiting a store or facility, such as a store setup terminal. Hereinafter, the case where the image processing apparatus 10 is composed of a computer owned by the user, specifically a PC, will be described as an example.

[0058] like Figure 2 As shown, the computer constituting the image processing apparatus 10 includes a processor 11, an auxiliary storage device 12, a main storage device 13, an input device 14, an output device 15, and a communication device 16.

[0059] The processor 11 may be composed of, for example, a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a TPU (Tensor Processing Unit), or an ASIC (Application Specific Integrated Circuit).

[0060] Auxiliary storage device 12 may be composed of flash memory, HDD (Hard Disc Drive), SSD (Solid State Drive), FD (Flexible Disc), MO (Magneto-Optical Disc), CD (Compact Disc), DVD (Digital Versatile Disc), SD card (Secure Digital Card), or USB memory (Universal Serial Bus memory).

[0061] This auxiliary storage device 12 can be built into the computer body constituting the image processing apparatus 10, or it can be externally installed on the computer body. Alternatively, the auxiliary storage device 12 can also be a NAS (Network Attached Storage) or the like. Furthermore, the auxiliary storage device 12 can also be an external device capable of communicating with a computer constituting the image processing apparatus 10 via a communication network, such as an online storage device or a database server.

[0062] Furthermore, the auxiliary storage device 12 stores programs 121, including the operating system (OS) and image processing execution applications. By having the processor 11 read and execute the programs 121, the computer constituting the image processing device 10 functions... Figure 3 The functions of the receiving unit 21, generating unit 22, determining unit 23, and output unit 24 shown are, in particular, to perform a series of processes related to the correction of an image based on a base image or text information.

[0063] The main storage device 13 is composed of semiconductor memory such as ROM (Read Only Memory) and RAM (Random Access Memory). The processor 11 loads the program 121 described above onto the main storage device 13 and executes it thereon.

[0064] Input device 14 is a device that accepts user input operations, such as a keyboard or mouse, a touch panel, or a camera unit. Furthermore, input device 14 may include photographic equipment mounted in a digital camera and a microphone for sound collection. Output device 15 is, for example, a display and a speaker.

[0065] The communication device 16 may be configured as, for example, a network interface card or a communication interface board. The computer constituting the image processing device 10 can communicate with other devices connected to the network 1, such as the Internet and mobile communication lines, via the communication device 16.

[0066] like Figure 3 As shown, the image processing apparatus 10 includes a receiving unit 21, a generating unit 22, a determining unit 23, and an output unit 24. These functional units are implemented by the processor 11 of the image processing apparatus 10 executing the aforementioned program 121 and cooperating with other hardware devices of the image processing apparatus 10. Furthermore, the image generation AI 221 and the explanatory text generation AI 222 in the generating unit 22 can also be invoked and executed from the server computer 50.

[0067] [Various definitions related to image processing, etc.] In this embodiment, such as Figure 12 , Figure 14, Figure 16 As shown, the image processing apparatus 10 of the present invention modifies the additional configuration image Pa desired by the user based on the user's operation of sliding Us in the slider Ub in a base image such as an input image Pi provided by the user or P1 generated by the image processing apparatus 10. The base image is an image including a background image Pb such as a sky or landscape and one or more configuration images Ph disposed on the background image Pb.

[0068] The background image Pb is an image on which the configuration image Ph is disposed, and functions as a substrate for the configuration image Ph. The background image Pb is an image of the same size as or larger than the configuration image Ph, and can be a specific monochrome image such as white or black, in addition to landscapes. Furthermore, patterns can be implemented on the background image Pb, and illustrations such as characters can be depicted as patterns. Moreover, the background image Pb can be the input image Pi itself, as described later.

[0069] Furthermore, the image generated accompanying the correction based on the image processing device 10 is the first candidate image P1, and the image generated accompanying the further correction by the user operation is the second candidate image P2. For ease of explanation, the first and second candidate images have been described, but each time a correction is performed through user operation, a third candidate image, a fourth candidate image, and so on up to the nth candidate image are generated. In addition, the image that satisfies the correction completed by the user is output as the output image Po.

[0070] Furthermore, the "image" in this invention is composed of multiple pixels, expressed by the grayscale values ​​of each pixel, and includes at least one configuration image Ph. The digital image data (hereinafter, image data) defining the image at a set resolution is generated by compressing data recording the grayscale values ​​of each pixel using a predetermined compression method. Examples of image data include irreversibly compressed image data in the form of JPEG (Joint Photographic Experts Group), and reversibly compressed image data in the form of GIF (Graphics Interchange Format) or PNG (Portable Network Graphics).

[0071] (Receiving Department) The receiving unit 21 is a functional unit that receives an input image Pi and text Ti (hereinafter referred to as input text Ti) as input text information from the user terminal 30 or the camera device 40 via the input device 14 or the communication device 16, and stores them, for example, in the auxiliary storage device 12. The receiving unit 21 stores the input text Ti received as described above in the text information table 211, and stores the input image Pi in the input image table 212.

[0072] Figure 4 The diagram shows a structural example of the text information table 211 in this embodiment. The text information table 211 is a collection of records containing user ID, date and time, and text information values. The user ID is the identification information of the user who entered the input text Ti, and may also be the identification information of the user terminal 30. Furthermore, in applications where the image processing device 10 is used by a single user, this user ID is not required. The date and time are the date and time when the input text Ti was entered.

[0073] The text information is information such as user-inputted text information stored by each keyword. This text information is information that lists statements made up by the UI space when the user inputs using input device 14 or user terminal 30, and stores them in each field. Alternatively, the receiving unit 21 may assign the input sentence to the morphological analysis engine, thereby extracting one or more statements and storing them in each field. Furthermore, the morphological analysis engine only needs to be maintained by the receiving unit 21 or be accessible via network 1.

[0074] Figure 5 The diagram shows a structural example of the input image table 212 in this embodiment. The input image table 212 is a collection of records containing values ​​for user ID, date and time, image data, and descriptive text. The user ID is the identification information of the user who input the input image Pi, and may also be the identification information of the user terminal 30. Furthermore, in applications where the image processing device 10 is used by a single user, this user ID is not required. The date and time are the date and time when the input image Pi was input.

[0075] The description refers to keywords obtained by assigning a user-inputted image to the description text generation AI 222. Such description text is one or more statements obtained by the receiving unit 21 from the input image obtained from the input device 14 or the user terminal 30 and assigned to the description text generation AI 222. Furthermore, the description text generation AI 222 only needs to be configured to be held by the receiving unit 21 or be accessible via the network 1.

[0076] AI222, the descriptive text generation AI, is a pre-built learning model for generating descriptive text using machine learning. This learning model, for example, determines the features of an input image Pi and generates descriptive text corresponding to those features. It is constructed by implementing machine learning by using previously acquired input images Pi and descriptive text assigned to those images (e.g., appropriate descriptions given by knowledgeable individuals) as training data.

[0077] Regarding the technique for obtaining descriptive text from the input image Pi, as described above, it is not limited to the technique of generating AI222 using descriptive text. For example, the correspondence between the image analysis results of the input image Pi and the descriptive text can be stored as data, and the receiving unit 21 can select (generate) descriptive text by referring to this correspondence. The correspondence between the image analysis results and the descriptive text can be, for example, establishing a correspondence between image analysis results such as the color pattern of the background image Pb, the type, shape, size, and theme of the object used as the configuration image Ph, and descriptive text such as "sea," "lake," "city," "mountain," "sea at dusk," "grassland in the rain," and "lion family."

[0078] Furthermore, the method for receiving the input image Pi or input text Ti in the receiving unit 21 is not particularly limited, and also includes methods where the camera unit reads a photographic printout of an image captured by the user terminal 30 or the photographic device 40. Additionally, the receiving unit 21 can also acquire an image by downloading image data from an external device or a web server via the network 1.

[0079] (Generation Department) The generation unit 22 is a functional unit that generates candidate images P1 to PN 1 to n as needed, based on the input image Pi and input text Ti obtained by the receiving unit 21. Alternatively, the input image Pi may include an image generated by the generation unit 22 assigning the input text Ti to the image generation AI 221. Furthermore, the input text Ti may include text generated by the generation unit 22 assigning the input image Pi to the descriptive text generation AI 222. The image generation AI 221 can be utilized through the image generation service provided in the network 1, but it can also be maintained by the image processing apparatus 10 itself. The image generation AI 221 is artificial intelligence that automatically generates the image desired by the user based on the text information input by the user.

[0080] In this embodiment, when generating candidate images P1 to PN from the first to the nth, the generation unit 22 generates an additional configuration image Pa corresponding to the input text Ti using the image generation AI 221, and then synthesizes it with the input image Pi. This synthesis is performed by configuring the additional configuration image Pa in a predetermined area of ​​the background image Pb or the configuration image Ph of the input image Pi. The generation unit 22 has the following function: when performing the above configuration, for example, it analyzes a blank area in the background image Pb of the input image Pi that does not overlap with the configuration image Ph by a certain area, and determines the configuration area of ​​the additional configuration image Pa. Alternatively, it may have the following function: it analyzes the area in the input image Pi that is in a predetermined positional relationship with the configuration image Ph (e.g., forming a horizontal column, connected vertically, or located at the center of the configuration image Ph), and determines it as the configuration area of ​​the additional configuration image Pa. Furthermore, the AI ​​can determine the configuration of the configuration image Ph in the input image Pi, or the user can specify the position.

[0081] The aforementioned analysis refers, for example, to determining the characteristics of various aspects related to an image. These characteristics include, for example, information related to hue and tone in different regions of the image, information related to image quality, grayscale values ​​of pixels, size and location of blank areas, layout of the image Ph (e.g., number of images, spacing, shape, size, size deviation, etc.), and information about the main objects (e.g., buildings or people, etc.) estimated based on this information.

[0082] The information about the object can include its type, state, location in the image, and, if the object is a living organism, its posture, gestures, age, and gender. Furthermore, the image features are preferably those that can be quantized, vectorized, or tensorized. In this case, the image analysis results become quantized, vectorized, or tensorized image features, i.e., feature quantities. Additionally, the configured image Ph can be a landscape image, meaning that the objects included in the configured image Ph are only landscapes, or it can be an image representing the landscape as the entire object.

[0083] (Determination Department) Furthermore, the determination unit 23 functions as follows: Regarding the first to nth candidate images P1 to PN generated by the generation unit 22, it provides a UI indicating the correction items and amounts to the user, and determines the content (items, correction amounts) selected by the user through this UI as the correction content. For example, the correction items are equivalent to... Figure 12 , Figure 14 , Figure 16 , Figure 17The slider Ub, as shown, displays the type, attributes, state, or values ​​of objects such as grassland, pond, lake, Western style, Japanese style, stable, egg, chick, chicken, skeleton, etc. The user operates the slider Us on the slider Ub via the user terminal 30, thereby allowing them to freely adjust the type or attributes of the object corresponding to the additional configuration image Pa. The determination unit 23 accepts the user's operation on the slider Us, obtains the type or attributes of the additional configuration image Pa, etc., desired by the user, and determines them as generation parameters for candidate images after the second candidate image P2. These generation parameters are transmitted from the determination unit 23 to the generation unit 22, where candidate image generation is performed.

[0084] Furthermore, when the determining unit 23 sets the aforementioned correction item or correction amount to the slider Ub, for example, it sets the input text Ti or explanatory text (the explanatory text generated by the explanatory text generation AI222) and... Figure 6 The values ​​of each item in the records of the correction item table 231 are checked to determine the records that include the corresponding values ​​in the items, and the value shown in the record is obtained as the aforementioned item or correction amount. The correction item table 231 can be replaced by a machine learning model that has been trained with training data corresponding to the same content. That is, the correction item or correction amount can also be determined using this model. Furthermore, both the correction item-based method and the model-based method can be implemented, and the results obtained can be combined separately. Moreover, the text such as the above-mentioned explanatory text generated from various images such as candidate images or input images Pi, or the input text Ti entered by the user, or the text selected by the user in these text classes can be input into the automatic generation model of correction items, thereby automatically generating intermediate concepts (e.g., egg, chick, ...) associated with the input text (e.g., "growth" or "chicken") as one of the correction items. The correction items obtained in this way become the index values ​​of the scale in the slider Ub. In addition, the above-mentioned automatic generation model is a machine learning model that uses information representing a certain concept and the combination of intermediate concepts as training data. In addition to the methods inherent in the image processing device 10 itself, the aforementioned automatically generated model can also be invoked and utilized from an external device via the network 1.

[0085] Figure 6The modified item table 231 shown is a collection of records containing values ​​for broad concepts, intermediate concepts, items, and items 1 to N. A broad concept represents the highest-level concept among the concepts of the things corresponding to the item, and an intermediate concept represents a concept subordinate to the broad concept. For example, regarding the broad concept "nature," the intermediate concepts "water area" and "mountain" are defined. Furthermore, items are the items that can be included in the intermediate concepts; for the intermediate concept "water area," these could be "size," "water surface / waves," or "color." Items 1 to N represent the item itself or its modified value corresponding to the item. For example, regarding the intermediate concept "water area," the item "size" is defined as item 1 "sea," item 2 "lake," item 3 "pond," and item N "puddle." Additionally, the modified item table 231 can also be set / edited by the user.

[0086] In addition, in the above example, the correction amount of the project was defined by text such as "lake" or "pond", but it could also be defined by specific values ​​such as "area of ​​more than 100,000 square kilometers", "more than 50 square kilometers and less than 100,000 square kilometers", or "more than 0.5 square kilometers and less than 50 square kilometers" to indicate the sense of size.

[0087] Furthermore, as mentioned above, in addition to items and their modifications from a physical perspective such as the type or size of the matter, these items or modifications can also be specified based on the form or state of the concept. For example, such as Figure 6 As shown in Table 231 of the correction items, regarding the item "water surface / wave" of the intermediate concept "water area," the item and correction quantity can be defined by the state of items such as item 1 "stable," item 2 "small wave," item 3 "medium wave," and item N "severe weather." Alternatively, regarding the item "style" of the intermediate concept "building," the item and correction quantity can also be defined by the form of the "style" of "buildings" such as item 1 "Western style," item 2 "Greek style," item 3 "Southeast Asian style," and item N "Japanese style." Furthermore, regarding the item "growth" of the intermediate concept "organism," the item and correction quantity can be defined by the form of the "growth" of "organisms" such as item 1 "egg," item 2 "chick / infant," item 3 "adult / human," and item N "skeleton." In this case, the morphological changes accompanying a series of time passes from the growth of an "organism" to its extinction are defined as items and their correction quantities. Applying this concept of time elapsed to projects and adjustments can also be applied to other higher-level and middle-level concepts.

[0088] In addition, such as Figure 16 As shown in screen G7, the determination unit 23 can also accept user-specified positions, i.e., depiction areas, for the additional configuration image Pa in the input image Pi or each candidate image through the movement operation of the position specification object G72. Figure 16In the example of screen G7, an elliptical object with a dashed line is shown as object G72. However, this is just one example; any object can be moved and a specific area can be specified by the user. Furthermore, the user can also define the drawing area by drawing an outline (so-called freehand drawing) without using an object. The determination unit 23 also notifies the generation unit 22 of the information about the drawing area received from object G72, along with the aforementioned correction items or correction amounts, and uses this information for candidate image generation.

[0089] (Output Department) The output unit 24 is a functional unit for displaying each candidate image below the first candidate image P1 generated by the generation unit 22, or an output image Po obtained by the user agreeing on and confirming the depiction content of the candidate image, on an output device 15 or user terminal 30. Therefore, the output unit 24 stores and maintains the image generated by the generation unit 22 in the generated image table 232 (see reference). Figure 7 The function is called and output appropriately based on user requests or the arrival of specified times.

[0090] Furthermore, the output method of the output image Po is not particularly limited. For example, the output unit 24 may include displaying the output image Po on a monitor or display screen of the output device 15 or user terminal 30, printing the output image Po, sending the output image Po to other users, or providing the output image Po as a product. The output image Po as a product may include media consisting of one or more pages or cards containing images, such as photo albums, photo books, postcards, message cards, electronic photo albums, and photo cards.

[0091] [Example of an image processing method] Next, as an example of the operation of the image processing apparatus 10 in this embodiment, the image processing flow using this apparatus will be described. In the image processing flow described below, the image processing method of the present invention is used. That is, each step in the image processing flow described below corresponds to a constituent element of the image processing method of the present invention.

[0092] Furthermore, the following process is merely an example, and some steps in the process may be deleted, new steps may be added, or the execution order of two steps in the process may be replaced without departing from the spirit of this embodiment.

[0093] Each step in the image processing flow involved in this embodiment is performed by the processor 11 provided by the image processing device 10. Figure 8 The sequence shown is implemented. That is, in each step of the image processing flow, the processor 11 executes the data processing specified in the image processing application. Figure 8 The processing corresponding to each step in the process.

[0094] Specifically, in the image processing flow of this embodiment, firstly, the receiving unit 21 receives at least one of an input image Pi and text information (text information related to the object being depicted; hereinafter, input text Ti), for example, from the user terminal 30 (S1). During this processing, the receiving unit 21 will... Figure 11 The receiving screen G1 is displayed on the user terminal 30, and the input text Ti is obtained as the input value through the input field G11 of the receiving screen G1. Furthermore, the receiving unit 21 obtains the file of the input image Pi specified by the user in the file search field G12 of the receiving screen G1.

[0095] Acquiring this input text Ti or input image Pi is necessary when generating the first candidate image, which becomes the base image for user correction. If the receiving unit 21 only receives the input text Ti, the value of the input text Ti is sent to the generation unit 22, and the image generation AI 221 generates the first candidate image P1 (S2). Figure 12 In screen G2, an example of the first candidate image P1 generated based on the input text "mountain grassland" is shown. This first candidate image P1 is an image in which a grassland is arranged in front of a background image Pb depicting the sky, and a mountain is arranged in the back.

[0096] Furthermore, if only the input image Pi is received, the input image Pi is sent to the generation unit 22, becoming the first candidate image P1. On the other hand, if both the input text Ti and the input image Pi are received in the receiving screen G1, they are sent to the generation unit 22, where the image generation AI 221 of the generation unit 22 performs the process of generating an additional configuration image Pa based on the input text Ti and configuring the additional configuration image Pa on the input image Pi to generate the first candidate image P1.

[0097] When receiving input text Ti in the supplementary information bar G21 of screen G2 and clicking the generate button G22 (S3), receiving unit 21 notifies determining unit 23 as a correction request. Determining unit 23 matches the input text Ti received from receiving unit 21 with correction item table 231 to determine the correction item (S4). When determining the correction item, for example, if the input text Ti is "lake", determining unit 23 determines the record in correction item table 231 related to the item "size" of the median concept "water area". This is because "item 2" in "item 1" to "item N" shown in the record contains "lake".

[0098] In addition to prompting the first candidate image P1 and receiving correction instructions from the user, the following slider Ub can also be generated and displayed without generating and prompting the first candidate image P1.

[0099] The determining unit 23 generates a slider Ub based on the correction amount (in the case of sea, lake, pond, and puddle) of the correction item ("size" of "water area") determined in the correction item table 231, and sends it to the user terminal 30 (S5). In the case of "lake," regarding the correction item "size" of "water area," a slider Ub is generated with the correction amount, i.e., "sea," "lake," "pond," "swamp," and "puddle," set as the scale. Figure 12 In the case of screen G3A, the left end of the scale on the slider Ub is marked with "Grassland". Furthermore, the scale adjacent to the right of the "Grassland" scale does not have a correction value set. The scale without a correction value corresponds to the state where the additional configuration image Pa, before the user-specified correction item, is displayed in the second candidate image P2. On the other hand, the "Grassland" scale corresponds to the state before the additional configuration image Pa is displayed, i.e., the first candidate image P1.

[0100] Furthermore, in the above example, a slider Ub was used as an interface for users to select correction items and correction amounts. However, it is also possible to configure radio buttons corresponding to each correction amount instead of this slider Ub, for example, to receive the correction amount selection from the user through the radio buttons.

[0101] The determination unit 23 receives a user operation on the slider Us on the slider Ub (S6). At this time, the determination unit 23 acquires the value of the scale on the slider Ub where the slider Us is placed. The determination unit 23 notifies the generation unit 22 of the received scale value. Based on this notification, the generation unit 22 changes the type or attribute of the additional configuration image Pa displayed on the screen G3A to generate or modify the second candidate image P2, and sends this response to the user terminal 30 (S7). An example of this second candidate image P2 as an output screen is shown below. Figure 12 The image is G3A.

[0102] If the OK button is not clicked on screen G3A, but during the operation of sliding Us on the slider Ub or clicking the back button (S8: No), the generation unit 22 repeats the above steps S3 to S7. On the other hand, if the OK button is clicked on screen G3A (S8: Yes), the generation unit 22 will generate the candidate image displayed on screen G3A at this time (in... Figure 12 In the case of the second candidate image P2), the output image Po is sent to the user terminal 30 (S9), and the process ends.

[0103] Furthermore, the change of the type or attribute of the aforementioned additional configuration image Pa can also be performed, for example, using a rule table pre-held by the generation unit 22. In this case, the rule table becomes a table storing rules for the image processing content determined according to the correction amount of each scale, i.e., correction item (see reference). Figure 9 The processing rules table specifies that, when the correction item is "size," the processing rules in this table enlarge or reduce the size of the original image by a specified ratio based on the concept of each item, i.e., the correction amount. Furthermore, when the correction item is "water surface / waves," the rules increase or decrease the height, width, and frequency of occurrence of the wave portion of the original image by a specified ratio based on the concept of each item, i.e., the correction amount.

[0104] Alternatively, the values ​​assigned to each scale can be the applicability when the user-inputted text Ti (and its corresponding item or state, etc.) is set to a maximum of 100. In this case, the aforementioned values ​​can be determined, for example, by the size ratio between the corresponding objects, or by dividing the number of intermediate concepts of the item by 100 (for example, if there are 3 intermediate concepts between 0 and 100, and there are a total of 4 stages between the concepts, dividing 100 by 4 to obtain 25 is used as one scale, and the values ​​are determined as 0, 25, 50, 75, 100, etc.). In this case, it is best to also record what the input text is, i.e., the name of the object, etc. Furthermore, even in the correction axis determined automatically or in the table, the values ​​of the scales where 100 is set to the maximum value can be recorded in the same way as above. In this case, it is best to also record what the axis with that scale represents (e.g., the size of an object, atmosphere, etc.).

[0105] Furthermore, when the modification item is "atmosphere," rules can be used to change the object's color or shape in the original image based on the concept of each item, i.e., the modification amount, according to that atmosphere. For example, if the modification amount for "atmosphere" is "Western style," the object's surface can be constructed using the shape and color of stone; if the modification amount is "Japanese style," the object's surface can be constructed using the shape and color of wood and a tiled roof. Moreover, when the modification item is "growth," rules can be used to change the image itself or its shape in the original image based on the concept of each item, i.e., the modification amount, according to that growth. For example, if the modification amount for "growth" is "egg," the object can be changed to an image of an egg; if the modification amount is "chick," the object can be changed to an image of a chick; if the modification amount is "adult bird," the object can be changed to an image of an adult bird; and if the modification amount is "skeleton," the object can be changed to an image of a bird's skeleton.

[0106] Furthermore, when the correction item is the "painter," it's possible to employ rules based on the concept of each item, i.e., the amount of correction, to change the color or shape of the object in the original image according to the painter. For example, if the correction amount for the "painter" is "Van Gogh," the object's surface might be processed into a spiral shape to emphasize yellow and blue; if the correction amount is "Gauguin," the object's color might be set to a warm color scheme, emphasizing the roundness of the lines while making them thicker. In this case, for example, as... Figure 10 As shown in the process, the receiving unit 21 receives an image of a painting by a specific painter from the user as the base image of the input image Pi (S30). The determining unit 23 then inputs the base image obtained by the receiving unit 21 into an art style determination AI to determine the art style (S31). This AI can, for example, use a determination model that has undergone machine learning in training data that establishes a corresponding association between the painting categories representing each painter's work and their style descriptions. The receiving unit 21 also receives input text Ti related to the object being depicted (S32). The generating unit 22 provides the input text Ti and the depiction instructions from the aforementioned rules to the image generation AI 221, generating a first candidate image P1 in the corresponding painter's style.

[0107] [Modifications of this embodiment] This embodiment is not limited to the above-described embodiment; for example, the following modified examples can be considered. These modified examples will be described below. Furthermore, the differences between these modified examples and the above-described embodiment will be explained below.

[0108] (Regarding the first variation) In the above-described embodiment, the image processing apparatus 10 receives input text Ti related to the additional configuration image Pa on the screen G2, matches it with the correction item table 231, thereby determining the correction items and generating a slider Ub to identify the user's correction intention. However, it is not limited to this. As a variation of this embodiment, it may also be done as follows: when the user issues a correction request for the first candidate image P1 (for example, the correction button is clicked on the screen of the user terminal 30), for example, the determination unit 23 analyzes the input text Ti used to generate the first candidate image P1 to estimate the candidates of the correction items desired by the user.

[0109] For example, suppose in Figure 13In the input field G41 of screen G4, the words "sea" and "city" are received as input text Ti for generating the first candidate image. Assuming that generation unit 22 assigns these two words to image generation AI 221, the first candidate image P1 shown in screen G5 is obtained. In this case, determination unit 23 analyzes the phrase "city, sea" in the input text Ti for the first candidate image and determines the candidate correction items that the user wants to correct. The determination of the correction item candidates is done by matching the above phrase with the correction item table 231 to determine the corresponding item (e.g., city → Western style, Japanese style, sea → water surface / waves, city × sea → sense of distance between city and sea). As described above, the correction item candidates can also be determined using a machine learning model trained on training data that is the set of input text Ti and the correct correction items.

[0110] exist Figure 14 An example of a scene corresponding to the first variation is shown. Figure 14 In screen G5, the following structure is shown: In the first candidate image P1 generated from the input text "Sea City", the configuration image Ph, which is an image of either a city or a sea, is configured on a background image Pb depicting the sky; and a slider Ub and a slider Us are displayed, which are specified to receive the correction amount of the correction items made by the user. The slider Ub is generated and displayed for "city" and "sea" respectively.

[0111] Furthermore, in Figure 14 In screen G6A, the following structure is shown: Upon receiving a user operation on slider Us on slider Ub, which is related to "city" and "sea" respectively, for the first candidate image P1, a second candidate image P2 is displayed, showing the city in a Japanese style and the sea in a turbulent state. Furthermore, in Figure 14 In screen G6B, the following structure is shown: the user clicks the OK button to end the correction for the second candidate image P2, and the output image Po is displayed.

[0112] (Regarding the second variation) Furthermore, it is also possible to determine the correction item not by using the input text Ti obtained from the user as the key, but by extracting words by assigning the first candidate image P1 to the descriptive text generation AI 222, and determining the correction item based on this. In this case, the determination unit 23 inputs the input image Pi (including the concept of the first candidate image P1) received from the user by the receiving unit 21 to the descriptive text generation AI 222. The descriptive text generation AI 222 analyzes the input image Pi and generates a sentence as descriptive text. This descriptive text generation AI 222 is an AI that has performed machine learning on a learning model that uses the input image Pi and a set of correct sentences to be extracted from the input image Pi as training data.

[0113] The determination unit 23 obtains the aforementioned statements from the descriptive text generation AI 222 (for example, obtaining the statements "wave" and "city" from the first candidate image P1), and determines the correction items and correction amounts by comparing the statements with the correction item table 231, and generates a slider Ub corresponding to each statement. Alternatively, the following control can be implemented: in the statements extracted by the descriptive text generation AI 222, "city and sea" and similar words originally input for generating the first candidate image are not suggested as correction items.

[0114] Furthermore, as described above, a group of correction items and correction amounts is determined for each statement, but it is also possible to generate only a predetermined number of sliders Ub and respond to the user terminal 30. In this case, the determination unit 23 can evaluate the differences in priority between correction items and only respond to sliders Ub related to correction items with a priority of a threshold or higher.

[0115] On the other hand, the user searches for the correction item they want to correct in the aforementioned slider Ub, and executes the correction instruction using the slider Us in the slider Ub that is the target (for example, regarding "wave", sliding the slider Us to the "severe weather" side). The determination unit 23 notifies the generation unit 22 of the correction amount obtained from the slider Ub and the statement (wave, city) obtained from the explanatory text generation AI 222. The generation unit 22 assigns the text information "city, sea, severe weather" to the image generation AI 221, and generates a second candidate image P2 based on the text information. At this time, the first candidate image P1 can also be input as a reference image to the image generation AI 221.

[0116] Alternatively, the following structure can be used: if AI222 is generated from the descriptive text, for example, the phrase "wave" is extracted, then the phrase is input into other AIs (not the correction item table 231) and keywords such as "rough / smooth" indicating the degree of adjustment are extracted.

[0117] (Regarding the third variation) Additionally, the first candidate image P1 can itself be an image input by the user. Figure 15 The following is a flowchart illustrating this process. Assume that prior to this process, an image (real shot) of a city and sea captured by the user using a camera device 40, etc., is input to the receiving unit 21 as the first candidate image P1. The receiving unit 21 transmits this input image P1 as the first candidate image P1 to the generation unit 22. The generation unit 22 inputs the first candidate image P1 to the descriptive text generation AI 222 (S15). The generation unit 22 obtains the descriptive text "city" and "sea" generated by the descriptive text generation AI 222 regarding the first candidate image P1 (S16).

[0118] Furthermore, the receiving unit 21 receives input text Ti from the user regarding the additional configuration image Pa (S17). The determining unit 23 matches the statement obtained in S16 and the input text Ti obtained in S17 with the correction item table 231 to determine the correction item and correction amount (S18). Afterwards, with... Figure 8 In the process, S5 to S9 are executed in the same way as S19 to S22.

[0119] exist Figure 16 , 17 Examples of images corresponding to the aforementioned variations are shown below. Figure 16 In the screens G7 and G8 shown, the following structure is illustrated: the input image Pi is analyzed to obtain the phrase "grassland," and "lake" is obtained as the input text Ti related to the additional configuration image Pa. A scale for the correction amount from "grassland" to "lake" is set on the slider Ub. Furthermore, a control example is shown where the first to fourth candidate images P1 to P4, generated according to each scale, are displayed side-by-side with the scale. In this case, the generation unit 22 generates candidate images corresponding to the scale and displays them on screen G8.

[0120] Additionally, at this time, the system can display an object G72 that the user has specified regarding which region of the first candidate image P1 to place the additional configuration image Pa in, and accept the user's movement operation on that object G72. In this case, the configuration position of the object G72 becomes the configuration position of the additional configuration image Pa.

[0121] Furthermore, in Figure 17 In the screens G9 and G10 shown, the following structure is illustrated: the input image Pi is analyzed to extract the statement, and the word "chicken" is obtained as the input text Ti related to the additional configuration image Pa. A scale for correction amounts from "none" to "skeleton" is set on the slider Ub. Furthermore, a control example is shown where candidate images P1 to P5, generated according to each scale, are displayed side-by-side with the scale. In this case, the generation unit 22 generates candidate images corresponding to the scales and displays them on screen G10.

[0122] Additionally, at this time, an object G92 can be displayed, specifying which region of the first candidate image P1 the additional configuration image Pa "chicken" should be placed in, based on the user's acceptance of this specification, and the user can move the object G92. In this case, the configuration position of the object G92 becomes the configuration position of the additional configuration image Pa.

[0123] Furthermore, the values ​​of the scale in the aforementioned slider Ub correspond to the values ​​of the items related to the "growth" of "animals" in the correction item table 231, specifically the values ​​of "egg to skeleton".

[0124] The specific embodiments of the present invention have been described above, but these embodiments are merely examples for ease of understanding and do not limit the present invention. That is, the present invention can be modified or improved from the embodiments described below without departing from its spirit. Furthermore, the present invention includes equivalents. In addition, embodiments of the present invention may include combinations of the above embodiments with one or more of the following variations.

[0125] (Regarding computers that constitute image processing devices) In the above embodiments, the image processing apparatus 10 of the present invention is constituted by a computer directly used by the user, such as a PC (Personal Computer). However, it is not limited to this; the image processing apparatus 10 of the present invention can also be constituted by a computer that the user can indirectly utilize, such as a server computer 50. Here, the server computer 50 can be, for example, a cloud service server computer, specifically, an ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service) server computer. In this case, if the required information is input into the user terminal 30 owned by the user, the server computer 50 performs various processing (operations) including image generation based on the input information, and the results of the operations are output on the user terminal 30 side. That is, the functions of the server computer 50 constituting the image processing apparatus of the present invention can be utilized on the user terminal 30 side. Alternatively, in addition to the above-mentioned PC or server computer, the image processing apparatus 10 can also be constituted by a camera device 40 or a smartphone (a type of user terminal 30) used by the user.

[0126] (Regarding the processor's architecture) The image processing apparatus 10 of the present invention includes a processor 11 comprising various processors. Among these various processors, for example, there is a general-purpose processor, namely a CPU, that executes software (programs) to perform functions as various processing units.

[0127] Furthermore, various processors, including FPGAs (Field Programmable Gate Arrays), are processors whose circuit structure can be changed after manufacturing, namely PLDs (Programmable Logic Devices).

[0128] In addition, various processors include ASICs (Application Specific Integrated Circuits), which have circuit structures specifically designed for specific processing, i.e., dedicated circuits.

[0129] Furthermore, a functional unit of the image processing apparatus 10 of the present invention can be composed of one of the various processors described above. Alternatively, a functional unit of the image processing apparatus 10 of the present invention can also be composed of a combination of two or more processors of the same or different types, such as a combination of multiple FPGAs or a combination of an FPGA and a CPU.

[0130] Furthermore, the multiple functional units of the image processing apparatus 10 of the present invention can be constituted by one of various processors, or two or more of the multiple functional units can be constituted by one processor. Also, as in the above-described embodiment, a processor can be constituted by a combination of one or more CPUs and software, and this processor can function as multiple functional units.

[0131] Furthermore, it can also be implemented in the following manner: for example, using a SoC (System on Chip) or similar device, a processor that implements the overall system functions, including multiple functional units in the image processing apparatus 10 of the present invention, using a single IC (Integrated Circuit) chip. Moreover, the hardware structure of the various processors described above can be a circuit composed of circuit elements such as semiconductor elements.

[0132] Symbol Explanation 1-Network, 5-Image processing system, 10-Image processing device, 11-Processor, 12-Auxiliary storage device, 121-Program, 13-Main storage device, 14-Input device, 15-Output device, 16-Communication device, 21-Receiving unit, 211-Text information table, 212-Input image table, 22-Generation unit, 221-Image generation AI, 222-Description text generation AI, 23-Determination unit, 231-Correction item table, 232-Generated image table, 24-Output unit, 30-User terminal, 40-Photographic device, 50-Server computer, Ti-Input text, Tx-Output text, Pi-Input image, Po-Generated image, P1-First candidate image, P2-Second candidate image, Pb-Background image, Ph-Configuration image, Pa-Append configuration image, Ub-Slider, Us-Slide, Uc-Item.

Claims

1. An image processing apparatus comprising a processor, wherein the image processing apparatus, The processor receives an image and information associated with the image, and determines processing items related to the image based on at least one of the image and the information associated with the image.

2. The image processing apparatus according to claim 1, wherein, The image is generated based on information associated with the image.

3. The image processing apparatus according to claim 1, wherein, The information associated with the image is text generated from the image.

4. The image processing apparatus according to claim 1, wherein, The processor receives information related to an object that the user wishes to depict in the image as information associated with the image, and determines the processing item based on at least one of the image and the received object-related information.

5. The image processing apparatus according to claim 4, wherein, The processor receives information related to the depicted area of ​​the object in the image, and also uses the information related to the depicted area to determine the processing item.

6. The image processing apparatus according to claim 1, wherein, The processor determines the amount of correction for the processing item based on at least one of the image and information associated with the image.

7. The image processing apparatus according to claim 6, wherein, The processor determines the correction amount of the processing item as information recorded in text.

8. The image processing apparatus according to claim 6, wherein, The modification amount of the processing item represents at least one of the forms or states of the concept.

9. The image processing apparatus according to claim 6, wherein, The processor prompts the user to provide an interface for receiving the user's specified correction amount for the processing item, and performs the image correction processing according to the correction amount received from the user.

10. An image processing method, comprising the following steps: The processor receives images and information associated with those images; and The processor determines processing items related to the image based on at least one of the image and information associated with the image.

11. A program for causing a computer to perform the steps included in the image processing method of claim 10.

12. A recording medium, which is a computer-readable recording medium, wherein, The recording medium contains a program for causing a computer to perform the steps included in the image processing method of claim 10.

Images