Information processing system, information processing method, and program

Abstract

To provide an information processing system, an information processing method, and a program that can stably execute the image processing desired by a user on the system.SOLUTION: The information processing system according to the present invention is an information processing system that applies an image processing algorithm to image data. The information processing system is equipped with: an input information acquisition unit that acquires user input information regarding the image data and information concerning the image processing algorithm; an image processing candidate acquisition unit that, by inputting a prompt based on the input information and the information concerning the image processing algorithm into a generative model based on the deep learning, acquires an image processing candidate, which is at least one of the image processing algorithm and a parameter constituting the image processing algorithm; and a display control unit that displays information about the image processing candidate on a display unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosure of this specification relates to an information processing system, an information processing method, and a program that can stably cause the system to execute image processing desired by a user. [Background technology]

[0002] There is known a technology for generating an image desired by a user by performing various image processing on image data. Non-Patent Document 1 discloses a technology in which a user inputs a predetermined image processing name or the like as a prompt to a generative model, which is a large-scale language model, and the generative model outputs the result of the predetermined image processing. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Chenfei Wu, et al. “Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models”, computer Vision and Pattern Recognition 2023 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology described in Non-Patent Document 1 automatically executes image processing based on prompts entered by the user and outputs the results of the executed image processing. Therefore, depending on the accuracy of the generative model and the prompts entered by the user, there is a possibility that unintended image processing may be selected and executed.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an information processing system that can stably perform the image processing desired by the user by displaying image processing candidates to the user based on information input by the user.

[0006] In addition to the above-mentioned objectives, the achievement of effects derived from the various configurations shown in the description of the invention below, which cannot be obtained by conventional techniques, can also be positioned as another objective of the disclosure of this specification. [Means for solving the problem]

[0007] In order to solve the above problem, the information processing system of the present invention is an information processing system that applies an image processing algorithm to image data, and includes: an input information acquisition unit that acquires user input information for the image data and information related to the image processing algorithm; an image processing candidate acquisition unit that acquires an image processing algorithm corresponding to the input information and an image processing candidate that is at least one of the parameters that constitute the image processing algorithm by inputting a prompt based on the input information and the information related to the image processing algorithm into a generative model based on deep learning; and a display control unit that displays information related to the image processing candidate on a display unit. The present invention is characterized by comprising: [Effects of the Invention]

[0008] The technology disclosed in this specification allows the system to stably execute image processing desired by the user. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the functional configuration of an information processing system according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing system according to a first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of a processing procedure of the information processing system according to the first embodiment. [Figure 4] FIG. 3 is a view showing an example of information relating to image processing algorithms that can be used by the information processing system according to the first embodiment. [Figure 5] FIG. 2 is a view showing an example of a display of the information processing system according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of the functional configuration of an information processing system according to a second embodiment. [Figure 7] FIG. 10 is a diagram showing an example of a processing procedure of an information processing system according to a second embodiment. [Figure 8] FIG. 10 is a view showing an example of a display of an information processing system according to a second embodiment. [Figure 9] FIG. 10 is a diagram showing an example of the functional configuration of an information processing system according to a third embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a processing procedure of an information processing system according to a third embodiment. [Figure 11] FIG. 11 is a view showing an example of a display of an information processing system according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the information processing system disclosed in this specification will be described with reference to the drawings. The same or equivalent components, members, and processes shown in each drawing will be assigned the same reference numerals, and duplicate descriptions will be omitted as appropriate. In addition, some of the components, members, and processes will be omitted as appropriate in each drawing.

[0011] The present invention will be described below using an information processing system that displays CT image data captured by an X-ray computed tomography (X-ray CT) device and the results of image processing performed on the CT image data as an example of an information processing system. Note that the embodiments of the present invention are not limited to the following embodiments, and can be applied to any images, including medical images such as images captured by a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, and an ultrasound diagnostic device.

[0012] First Embodiment The information processing system in the first embodiment is a system that can receive input information from a user, and input the input information and information related to an image processing algorithm into a generative model based on deep learning to obtain image processing candidates and present them to the user. Furthermore, the information processing system may receive an acceptance or rejection from the user of the presented image processing candidates.

[0013] The device configuration of an information processing system 1000 according to this embodiment will be described below with reference to FIG.

[0014] The information processing system 1000 of the present invention is an information processing system that applies an image processing algorithm to image data, and is configured to be connectable via a network to a memory unit 1010 that stores various data and a generative model 2000 based on deep learning.

[0015] The information processing system 1000 includes an input information acquisition unit 1020 that acquires user input information for image data and information related to the image processing algorithm. It also includes an image processing candidate acquisition unit 1030 that inputs a prompt based on the input information and the information related to the image processing algorithm to a deep learning-based generative model 2000. The image processing candidate acquisition unit 1030 acquires, from the generative model 2000, an image processing candidate that is an image processing algorithm corresponding to the input information and at least one of parameters constituting the image processing algorithm. The information processing system 1000 further includes a display control unit 1040 that displays information related to the image processing candidate on the display unit. The information processing system 1000 may also include a selection information acquisition unit 1050 that acquires selection information for the image processing candidate displayed on the display unit.

[0016] The following describes the functional components that make up the information processing system 1000. Note that the storage unit 1010 and the generative model 2000 may be realized as components of the same system, and each functional component of the information processing system 1000 may be configured from other devices via a network or the like.

[0017] The storage unit 1010 is a part of a computer-readable storage medium, and is a large-capacity storage device such as a hard disk drive (HDD) or a solid-state drive (SSD). The storage unit 1010 stores information about available image processing algorithms.

[0018] The input information acquisition unit 1020 acquires user input information and information related to the image processing algorithm from the storage unit 1010. The user input information is text information entered by the user or text information selected by the user, and is a text prompt.

[0019] The image processing candidate acquisition unit 1030 inputs a prompt based on the user's input information and information related to the image processing algorithm to the deep learning-based generative model 2000. Then, it acquires image processing candidates, which are the image processing algorithm corresponding to the input information and at least one of the parameters constituting the image processing algorithm. The image processing candidate acquisition unit 1030 may acquire the image processing candidates by performing lexical analysis on the answer text information acquired from the generative model 2000.

[0020] The display control unit 1040 causes the display unit to display information about the candidate image processing acquired by the candidate image processing acquisition unit 1030.

[0021] The selection information acquisition unit 1050 receives the user's selection result for the information on the candidate image processing, i.e., the user's decision on whether or not to apply the image processing corresponding to the information on the candidate image processing. In other words, it acquires the user's decision on whether or not to adopt the candidate image processing displayed on the display unit.

[0022] By configuring the information processing system 1000 as described above, the system can stably execute the image processing desired by the user.

[0023] (Hardware configuration) Next, the hardware configuration of the information processing system 1000 will be described with reference to Fig. 2. The information processing system 1000 has the configuration of a known computer (information processing system). The information processing system 1000 includes, as its hardware configuration, a CPU 201, a main memory 202, a magnetic disk 203, a display memory 204, a monitor 205, a mouse 206, and a keyboard 207.

[0024] A CPU (Central Processing Unit) 201 mainly controls the operation of each component element. A main memory 202 stores control programs executed by the CPU 201 and provides a work area when the CPU 201 executes the programs. A magnetic disk 203 stores programs for implementing various application software, including an OS (Operating System), device drivers for peripheral devices, and programs for performing the processes described below. The CPU 201 executes programs stored in the main memory 202, magnetic disk 203, etc., thereby realizing the functions (software) of the information processing system 1000 shown in FIG. 1 and the processes in the flowcharts described below. The magnetic disk 203 may be the same as the storage unit 1010.

[0025] Display memory 204 temporarily stores display data. Monitor 205 is an example of a display unit, and is composed of, for example, a CRT monitor or LCD monitor, and displays images, text, etc. based on the data from display memory 204. Mouse 206 and keyboard 207 are used by the user to input pointing and characters, etc., respectively. The above components are connected to each other via a common bus 208 so that they can communicate with each other.

[0026] The CPU 201 corresponds to an example of a processor or a control unit. In addition to the CPU 201, the information processing system 1000 may have at least one of a GPU (Graphics Processing Unit) and an FPGA (Field-Programmable Gate Array). Furthermore, at least one of a GPU and an FPGA may be included instead of the CPU 201. The main memory 202 and the magnetic disk 203 correspond to an example of a memory or a storage device.

[0027] (Processing Procedure) Next, the processing procedure of the information processing system 1000 according to this embodiment will be described with reference to FIG.

[0028] (Step S3010) In step S3010, the input information acquisition unit 1020 acquires input information from a user. Here, the input information is text information representing the content of processing the user desires for the original image data. In this embodiment, a case where a text prompt such as "I would like to lower the detection threshold and perform the lesion detection process again" is acquired as input information will be described as an example. This text prompt may be acquired as information entered by the user using the mouse 206 or keyboard 207 connected to the information processing system 1000. Alternatively, it may be acquired by converting information entered by voice by the user using a microphone (not shown) or the like into a text prompt using a known voice recognition technology. Alternatively, typical user requests may be displayed on the monitor 205, allowing the user to select from among them using the mouse 206 or keyboard 207. Examples of such requests include "I would like to increase the overall brightness of the image data" and "I would like to reduce the blurring caused by the smoothing process applied to the image data."

[0029] (Step S3020) In step S3020, the input information acquisition unit 1020 further acquires information about image processing algorithms available in the information processing system 1000 from the storage unit 1010. In this embodiment, a case where smoothing processing, brightness adjustment processing, and lesion detection processing are acquired as examples of available image processing algorithms will be described. The acquired information includes, for each image processing algorithm, the algorithm name, processing content, input / output information format, configurable parameter names, and the effects of changing the parameters. In this embodiment, text information containing this information is acquired. FIG. 4(a) shows an example of information about the image processing algorithm, and FIG. 4(b) shows an example of text information actually acquired by the image processing information acquisition unit 1050. However, the information processing system 1000 may execute processing with this information stored in advance in the main memory 202 or the like. In this case, this processing step can be omitted. The input information acquisition unit 1020 transmits the user's input information and information about the image processing algorithm to the image processing candidate acquisition unit 1030, and the processing proceeds to the next step.

[0030] (Step S3030) In step 3030, the image processing candidate acquisition unit 1030 generates an augmented text prompt, which is a prompt for the generative model, based on the text prompt, which is user input information, and information about available image processing algorithms. The image processing candidate acquisition unit 1030 then inputs the augmented text prompt to the generative model 2000 and acquires answer text information including information about the image processing candidate.

[0031] The configuration of the generative model will now be described. The generative model in this embodiment is a model centered around a Transformer and is composed of processing blocks such as a Tokenization block, an Embeddings block, a Positional Encoding block, a Transformer block, and a Decoding block. In the generative model, the Tokenization block first processes an input text prompt, dividing the text prompt into tokens such as words and phrases, and obtains a group of tokens corresponding to the text prompt. Each token is associated with an ID (numerical data). Tokenization is achieved using a known technique such as the Byte-Pair Encoding algorithm. Next, the Embeddings block converts each token into a vector representation (embedding vector) to make it easier for the model to understand the meaning. Within the embedding space, for example, tokens that are semantically similar are mapped to nearby positions. The conversion to the vector representation is achieved by a trained embedding layer. Next, the Positional Encoding block adds positional relationship information between tokens to the embedding vector corresponding to each token, so that the positional relationship between tokens can be taken into account in the subsequent Transformer block. More specifically, a position vector representing the position of the token in the text prompt is added to the embedding vector to obtain a position-encoded embedding vector. The transformer block then processes the position-encoded embedding vector using an attention mechanism or other method to convert it into an abstract representation while capturing the relationships and context between tokens, and predicts the output token. The predicted token is then combined with the group of tokens corresponding to the input text prompt, generating new input data, and the above process is repeated. Once a token indicating the end of the sentence is output, the decoding block finally converts the output tokens into text (de-tokenization) or formats them into human-understandable text.The above generative model utilizes large-scale text data and is trained using a known method such as Masked Language Modeling, and is therefore capable of generating answer text to a text prompt (question text). Note that any model may be used as the generative model, regardless of its configuration, processing procedure, or training method, as long as it is capable of generating answer text to a text prompt.

[0032] Next, the augmented text prompt generated by the image processing candidate acquisition unit 1030 and the output of the generative model will be described in detail. In this embodiment, an augmented text prompt is generated by inserting a text prompt and information about available image processing algorithms into a prompt template. The prompt template may be, for example, the following sentence, and may be set in advance: "Propose a process that satisfies the input information based on information about the following image processing algorithm. Information about the image processing algorithm: {(A)}, input information: {(B)}. The output format should be the following: '{image processing algorithm name}:{parameters}'".

[0033] Here, text information regarding the image processing algorithm acquired by the input information acquisition unit 1020 is inserted into (A) in the prompt template. Also, input information acquired by the input information acquisition unit 1020 is inserted into (B).

[0034] The image processing candidate acquisition unit 1030 generates an expanded text prompt as described above, inputs it into the generative model 2000, and processes it to acquire answer text information, for example, "lesion detection processing: Th=0.3," from the generative model 2000. Here, "lesion detection processing" in the example answer text information is the name of the image processing algorithm included in the information about the image processing algorithm inserted in (A) in the expanded text prompt, and is one of the image processing algorithms that the information processing system 1000 can use.

[0035] The image processing candidate acquisition unit 1030 acquires information about the image processing candidate from the response text information. Here, the information about the image processing candidate refers to a set of the image processing algorithm name and parameters related to the image processing algorithm for realizing the desired processing content expressed by the user in the text prompt (input information) on the information processing system 1000. The image processing candidate acquisition unit 1030 lexically analyzes the response text information "Lesion detection processing: Th=0.3" acquired in step S3050 and separates it at the ":" character to extract the image processing algorithm name (lesion detection processing) and parameters (Th=0.3). Then, based on the extracted results, the image processing candidate acquisition unit 1030 acquires the image processing algorithm name and parameters related to image processing that can be implemented by the information processing system 1000 as information about the image processing candidate, and sends this information to the display control unit 1040, before proceeding to the next step.

[0036] (Step 3040) The display control unit 1040 displays information about candidate image processing on the monitor 205, which is the display unit. An example display is shown in FIG. 5. The information display window 500 displayed on the display unit by the display control unit 1040 is composed of suggested text 510, suggested content 520, and selection buttons 530 and 531. The suggested text 510 is fixed text that prompts the user to decide whether to apply the candidate image processing. The suggested content 520 is text generated based on information about the candidate image processing, and in this embodiment, text expressing the differences in information about each image processing is displayed. More specifically, text is set that suggests changing the parameter Th of the lesion detection processing, which is set by default, from 0.5 to 0.3. Note that the method of presenting the suggested text is not limited to displaying it on the monitor, and may be other means, such as text reading.

[0037] (Step S3050) The selection information acquisition unit 1050 receives information on the selection results made by the user. If the selection button 530 indicating affirmative action is pressed, it is determined that the proposed content 520 for the image processing candidate has been accepted, and if the selection button 531 indicating negative action is pressed, it is determined that the proposed content 520 has been rejected. In other words, the user's acceptance or rejection of the image processing candidate is determined based on which selection button was pressed. Note that the selection results may also be received by other means, such as voice input. The information processing system 1000 determines the processing content and parameters to be applied based on the user's selection.

[0038] If the proposal is accepted in step S3050, the parameter setting values ​​for the image processing corresponding to the information on the candidate image processing are changed to the proposed parameter values. The display control unit 1030 may then display the obtained image processing results on the monitor 205. In this embodiment, the lesion detection process with the parameter Th changed to 0.3 is applied to the original image data, and the lesion detection result data after the parameter change is obtained and displayed on the monitor 205. The current image data is, for example, CT image data. If the proposal is rejected, a message prompting the user to resubmit the parameter proposal may be displayed on the monitor 205. For example, a message such as "Do you want to set the parameter Th to a value other than 0.3? If so, please enter a specific value" may be displayed on the monitor 205. Alternatively, a message prompting the user to reenter the text prompt may be displayed on the monitor 205. If the user wishes to reenter the text prompt, the process returns to step S3010.

[0039] Through the above processing procedure, the information processing system 1000 can identify information about candidate image processing desired by the user by processing the text prompt indicating the image processing content desired by the user and information about the image processing algorithm using a generative model.

[0040] <Second embodiment> System for acquiring pre-image processing information (information about pre-image processing) The information processing system 6000 in this embodiment further uses information related to preliminary image processing for display by a display control unit 6040 and for acquisition of image processing candidate by an image processing candidate acquisition unit 6030. This configuration reduces the user's workload while enabling the user to specify the image processing candidate desired, and stably execute the image processing desired by the user. Note that the same functional configuration as in the first embodiment will be assigned the same reference numerals, and explanations will be omitted where appropriate.

[0041] Hereinafter, the functional configuration of the information processing system 6000 according to this embodiment will be described with reference to FIG.

[0042] The storage unit 6010 is a part of a computer-readable storage medium, and is a large-capacity storage device such as a hard disk drive (HDD) or a solid-state drive (SSD). The storage unit 6010 stores original image data (an example of second image data) and pre-image processing result data (an example of first image data) that is the result of applying any image processing to the original image data in accordance with a previous user operation. Furthermore, the storage unit 6010 stores information about available image processing algorithms, as well as a history of pre-image processing that has been applied to the original image data (or pre-image processing data) in the past.

[0043] The input information acquisition unit 6020 acquires information input by the user and information relating to the image processing algorithm from the storage unit 6010, as well as information relating to the preliminary image processing.

[0044] The image processing candidate acquisition unit 6030 further inputs information about pre-image processing into the generative model 2000 to acquire image processing candidates.

[0045] The display control unit 6040 displays information about the image processing candidates acquired by the image processing candidate acquisition unit 4030. Alternatively, the display unit displays information about the preliminary image processing and information about the image processing candidates.

[0046] (Processing Procedure) Next, the processing procedure of the information processing system 6000 according to this embodiment will be described with reference to Fig. 7. When the information processing system 6000 starts processing, it first proceeds to step S7010.

[0047] (Step S7010) In step S7010, the input information acquisition unit 6020 acquires the original image data to be processed and preliminary image processing result data, which is the result of any image processing applied to the original image data by a past user operation. In addition, the preliminary image processing information acquisition unit 1080 acquires information regarding the history of image processing previously applied to the image data (information regarding preliminary image processing).

[0048] In this embodiment, the original image data is CT image data (an example of first image data), and the preliminary image processing result data is lesion detection result data (an example of second image data) that represents the lesion detection result.

[0049] For example, when the preliminary image processing is a lesion detection process, the preliminary image processing result data is image data in which a lesion area is expressed in a distinguishable manner from other areas, for example, image data in which pixels belonging to a lesion are expressed as 1 and other pixels are expressed as 0. The preliminary image processing result data such as lesion detection result data is not limited to image data, but may also be data other than image data, such as coordinate information or labels. Furthermore, the display control unit 6040 may display the CT image data, which is the original image data, and the lesion detection result data, which is the preliminary image processing result data, on the monitor 205.

[0050] (Steps S3010 to S3020) Steps S3010 and S3020 executed by the input information acquisition unit 6020 are the same as those in the first embodiment, and therefore a description thereof will be omitted.

[0051] (Step S7020) The image processing candidate acquisition unit 6030 acquires information about pre-image processing, a text prompt indicating the desired processing of image data, and information about available image processing algorithms. The image processing candidate acquisition unit 6030 generates an expanded text prompt, which serves as input to the generative model 2000, by inserting the text prompt and information about the image processing algorithm. A prompt template might be, for example, the following: "Propose a process that satisfies the input information based on the information about the following image processing algorithm. Information about the image processing algorithm: {(A)}, Input information: {(B)}, Pre-image processing: {(C)}. The output format should be the following: '{Image processing algorithm name}:{Parameters}'." Here, (A) in the prompt template is where text information about the image processing algorithm is inserted. Furthermore, (B) is where user input information is inserted. Furthermore, (C) is where information about the pre-image processing acquired in step S7010 is inserted. Here, the information about pre-image processing is, for example, a set of the name of an image processing algorithm previously applied to image data such as raw image data or pre-image-processed data, and parameters related to the image processing algorithm. In this example, information about the image processing algorithm that was most recently applied to the image data is obtained from the history. As a specific example, a case will be described in which a history indicating that lesion detection processing was applied to the original image data with a parameter Th=0.5 is obtained as information about the preliminary image processing.

[0052] (Step S7030) In step S7030, the display control unit 6040 generates and displays on the monitor 205 drawing content based on the information on the image processing candidates and the information on the preliminary image processing. An example display is shown in FIG. 8. The information display window 800 is composed of suggested text 810, suggested content 820, and selection buttons 830 and 831. The suggested text 810 is fixed text that prompts the user to decide whether to apply the image processing candidates. The suggested content 820 is text generated based on the information on the preliminary image processing and the information on the image processing candidates. In this embodiment, text expressing the differences between the information on each image processing is displayed. More specifically, the image processing algorithm corresponding to the information on the preliminary image processing and the image processing algorithm corresponding to the information on the image processing candidates differ only in parameters. Therefore, text is set suggesting that the lesion detection processing parameter Th be changed from 0.5 to 0.3. Note that the method of presenting the suggested text is not limited to displaying it on a monitor; other means, such as text reading, may also be used.

[0053] (Step S3050) Step S3050 is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0054] Through the above processing procedure, the information processing system 6000 can use information related to preliminary image processing to more accurately identify information related to candidate image processing desired by the user.

[0055] (Variation) In the first and second embodiments, an example was given in which a user desires to adjust the threshold for lesion detection for image data and inputs a text prompt. However, the information processing system may be configured to automatically propose a prompt by performing image analysis on image data such as raw image data or pre-image processing result data.

[0056] For example, the information regarding image processing may be the entire text of a manual or instruction manual for an information processing system, or a prompt may be generated by extracting relevant portions of the manual or instruction manual using a known technique such as Retrieval-Augmented Generation (RAG) and inserting the extracted text into a prompt template.

[0057] In addition, examples of information on usable image processing algorithms and examples of input and output for generative models are only examples, and an appropriate format may be used depending on the system.

[0058] Furthermore, the format of the information input to the generative model is not limited to text; it can also be data converted into features used within the generative model (such as IDs or embedding vectors corresponding to tokens). Image data or audio data can also be input as prompts, and these can be used in combination with text. In such cases, the generative model must be configured to be able to handle multimodal input.

[0059] Alternatively, the generative model can be trained to provide information about image processing algorithms, in which case the text prompts can use the information entered by the user as is.

[0060] In the first and second embodiments, examples of changing the parameter setting values ​​of image processing have been shown. However, in addition to changing the parameter setting values, the proposal may also be a proposal regarding a change in a model used in an image processing algorithm, or a proposal for the image processing algorithm itself.

[0061] A specific example of a proposal to change a model used in an image processing algorithm will be given. For example, assume that an information processing system receives a text prompt from a user, such as "I would like the lesion detection sensitivity to be increased." The information processing system generates input data (augmented text prompt) for the generative model 2000 using a procedure similar to that of this embodiment. At this time, the image processing information in the input data includes information on available models (e.g., the characteristics, training data, and training methods of each of multiple deep learning models) as well as the names and parameters of image processing algorithms available to the information processing system. The information processing system inputs the input data containing this information into the generative model 2000 and executes a series of generative model processes to obtain answer text information. Then, based on the answer text information, the system performs lexical analysis and compares it with information on prior image processing, and makes a proposal to the user, such as "Image processing algorithm X: Change model A to model B." A specific example of a proposal for an image processing algorithm itself will be given. For example, assume that an information processing system receives a text prompt from a user, such as "I would like to blur the image." The information processing system generates input data for the generative model using a procedure similar to that of this embodiment, executes a series of generative model processes, and obtains answer text information. Then, from the response text information, the name of an image processing algorithm, for example, "Gaussian filter processing," is obtained as information about the image processing candidate.

[0062] In the first and second embodiments, the image processing candidate acquisition unit has been described as acquiring answer text information related to one image processing candidate. However, the present invention is not limited to this example, and answer text information including multiple image processing candidates may be acquired. Specifically, answer text information related to multiple image processing candidates can be acquired by inserting text such as "Propose three processes in order of priority for output" into the augmented text prompt input to the generative model. In this case, each of the multiple image processing candidates included in the acquired answer text information can be displayed in a manner that allows the user to compare them. Then, in step S3090, the user may be allowed to select a suitable image processing candidate from the three image processing candidates. This has the effect of increasing the probability of acquiring the user's desired image processing candidate.

[0063] In the first and second embodiments, an example has been described in which the information processing system prompts the user to re-propose parameters or re-enter a text prompt when the user rejects a candidate image processing. However, the present invention is not limited to this example. For example, when the user rejects a candidate image processing, the process can be returned to step S3020, and an augmented text prompt can be generated to cause the generative model to propose an image processing other than the candidate image processing already proposed. Specifically, a prompt such as "Output a candidate other than 'lesion detection processing: Th=0.3'" can be generated and the process can be executed. This, as with the previous example, has the effect of increasing the probability of obtaining the candidate image processing desired by the user.

[0064] <Third embodiment> In the first and second embodiments, the information processing system identifies information about candidate image processing algorithms, such as image processing algorithm names and parameters, for obtaining the results desired by the user, based on a prompt input by the user. Then, based on the information about the candidate image processing algorithms, the information processing system proposes changes to parameter setting values ​​to the user, and receives a decision on whether or not to accept the proposal (application decision) from the user, thereby determining whether or not to adopt the proposal, such as changes to parameter setting values.

[0065] In this embodiment, after identifying information about candidate image processing in the same procedure as in the first or second embodiment, the image processing unit applies image processing corresponding to the information about the candidate image processing to the original image data, and presents the resulting image to the user. The image processing unit also accepts approval or disapproval (application decision) from the user for the resulting image, and determines whether or not to adopt the image processing based on that.

[0066] (Functional configuration) Hereinafter, the device configuration of the information processing system 9000 according to this embodiment will be described with reference to FIG.

[0067] The storage unit 9010 is a part of a computer-readable storage medium, and is a large-capacity storage device such as a hard disk (HDD) or a solid-state drive (SSD). The storage unit 9010 stores original image data (an example of second image data) and preliminary image processing result data (an example of first image data) that is the result of applying arbitrary image processing to the original image data through a previous user operation. Furthermore, the storage unit 9010 stores information regarding image processing algorithms that can be used on the information processing system 9000.

[0068] The image data acquisition unit 9020 acquires the original image data and the preliminary image processing result data from the storage unit 9010 .

[0069] The display control unit 9050 displays image data such as original image data and preliminary image processing result data, instruction text for the user, and selection buttons for accepting selections from the user on the display unit.

[0070] The input information acquisition unit 9040 acquires the image processing content desired by the user as a text prompt input by the user, and also acquires information regarding the image processing algorithm.

[0071] The image processing candidate acquisition unit 9050 acquires information about image processing that can be used by the information processing system 6000 .

[0072] The image processing candidate acquisition unit 9050 inputs a text prompt including the input information acquired by the input information acquisition unit 9040 and information related to image processing to the generative model 2000. Then, after processing by the generative model 2000, answer text information including the image processing and information related to the image processing is acquired.

[0073] An image processing candidate acquisition unit 9040 performs lexical analysis on the answer text information acquired from the generative model, and acquires information about the image processing candidates.

[0074] The image processing unit 9060 applies image processing corresponding to the information about the image processing candidates acquired by the image processing candidate acquisition unit 9040 to the original image data acquired by the image data acquisition unit 9020. Then, it acquires post-processing result data (an example of third image data) that is the result of the image processing.

[0075] The display control unit 9050 generates comparison image data of the pre-image processing result data and the post-image processing result data, and displays it on the display unit.

[0076] The selection information acquisition unit 9070 accepts a user's decision (user's decision to apply) regarding the information on the candidate image processing displayed on the display unit as to whether or not to apply the image processing corresponding to the information on the image processing. The information processing system 9000 determines the image processing to be applied to the original image data and the preliminary image processing result data based on the user's selection information accepted by the selection information acquisition unit 9070.

[0077] (Hardware configuration) The configuration is the same as that of the first embodiment, so a description thereof will be omitted.

[0078] (Processing Procedure) Next, the processing procedure of the information processing system 9000 according to this embodiment will be described with reference to Fig. 10. When the information processing system 9000 starts processing, the process first proceeds to step S10010.

[0079] (Step S10010) The image data acquisition unit 9020 acquires the original image data and the preliminary image processing result data from the storage unit 9010 .

[0080] (Steps S7010 to S7020) The processing is the same as steps S7010 to S7020 in the second embodiment, so a description thereof will be omitted.

[0081] (Step S10020) In step S10020, the image processing unit 9060 acquires post-image processing result data by applying image processing corresponding to the information on the image processing candidate acquired from the image processing candidate acquisition unit 9040 to the original image data acquired in step S10010. In this embodiment, lesion detection process with the parameter Th changed to 0.3 is applied to the CT image data, which is the original image data, to acquire lesion detection result data after the parameter change (an example of third image data), which is post-image processing result data.

[0082] (Step S10030) In step S10030, the display control unit 9050 compares the pre-image processing result data (an example of first image data) with the post-image processing result data, and displays the generated comparison image data on the display unit. In this embodiment, superimposed image data, which is comparison image data, is generated by superimposing the pre-image processing result data on the post-image processing result data. The generation of the comparison image data is not limited to superimposition, and the difference between the two image processing result data may also be calculated using an exclusive OR or the like.

[0083] The display control unit 9050 displays each image data on a display unit such as the monitor 205. In addition to the comparison image data, the original image data, the pre-image processing result data, and the post-image processing result data may be displayed side by side. FIG. 11 shows an example of how these image data are displayed. The information display window 1100 displayed on the display unit by the display control unit 9050 is composed of proposed text 1110, an original image 1120 (original image data), a pre-result 1130 (pre-image processing result data), a processing application result 1140 (post-image processing result data), a comparison result 1150 (comparison image data), and selection buttons 1160 and 1161. The original image 1120 is CT image data including an organ region 1121.

[0084] (Step S10040) In step S10040, the selection information acquisition unit 9070 accepts the selection result from the user. If the user presses the selection button 1160, it is determined that the proposal based on the information about the image processing candidates has been accepted, and if the user presses the selection button 1161, it is determined that the proposal has been rejected.

[0085] The information processing system 9000 determines the processing content and parameters to be applied based on the user's selection information. If the proposal based on the information on the image processing candidates is accepted in step S10040, the setting values ​​of the image processing parameters corresponding to the information on the image processing candidates are changed to the proposed parameter values. Then, the display control unit 9050 displays the post-processing result data on the monitor 205. If the proposal is rejected, a message prompting the user to re-propose parameters or re-enter the text prompt is displayed on the monitor 205. If the user wishes to re-enter the text prompt, the process returns to step S3010.

[0086] Through the above processing procedure, the information processing system 9000 can identify information about the image processing candidates desired by the user by processing the text prompt indicating the image processing content desired by the user and information about the image processing algorithm using a generative model.

[0087] (Variation) In this example, an example was given in which a user desired segmentation (image recognition) based on deep learning for image data, but image editing or image analysis may also be desired as image processing. Image editing includes brightness conversion and smoothing processing, and image analysis includes feature calculation.

[0088] Furthermore, in this embodiment, the user is prompted to make a selection by displaying an image corresponding to each image processing, but the parameters used in the image processing may also be displayed.

[0089] In addition, in this embodiment, the image comparison unit generates a superimposed image by superimposing each image data, but it is also possible to generate a difference image, generate an image by arranging each image, or calculate features based on evaluation indices other than images.

[0090] The disclosure of this specification includes the following information processing system, information processing method, and program.

[0091] (Item 1) 1. An information processing system for applying an image processing algorithm to image data, comprising: an input information acquisition unit that acquires user input information for image data and information related to image processing algorithms; an image processing candidate acquisition unit that acquires an image processing candidate that is an image processing algorithm corresponding to the input information and at least one of parameters constituting the image processing algorithm by inputting a prompt based on the input information and information related to the image processing algorithm into a generative model based on deep learning; a display control unit that displays information about the candidate image processing methods on a display unit; An information processing system comprising:

[0092] (Item 2) 2. The information processing system according to item 1, further comprising an acquisition unit for acquiring information on image processing candidates selected by a user.

[0093] (Item 3) 3. The information processing system according to item 2, wherein the selection information acquisition unit acquires whether or not to adopt the image processing candidates displayed on the display unit.

[0094] (Item 4) the image processing candidate acquisition unit acquires information on a plurality of image processing candidate from the generative model; 4. The information processing system according to item 2 or 3, wherein the selection information acquisition unit acquires selection information by the user from information relating to the plurality of image processing candidates.

[0095] (Item 5) the image processing candidate acquisition unit acquires information on a plurality of image processing candidate from the generative model; The information processing system according to any one of items 2 to 4, characterized in that the display control unit, if the image processing candidate displayed on the display unit is not adopted by the user, causes another image processing candidate from among the plurality of image processing candidates to be displayed on the display unit.

[0096] (Item 6) The input information acquisition unit further acquires information regarding prior image processing that has been previously applied to the image data; 6. The information processing system according to any one of items 1 to 5, wherein the display control unit further displays information relating to the preliminary image processing.

[0097] (Item 7) 7. The information processing system according to any one of items 1 to 6, wherein the image processing candidate acquisition unit acquires the image processing candidate by inputting information about the pre-image processing into the generative model.

[0098] (Item 8) 8. The information processing system according to any one of items 1 to 7, wherein the information regarding the pre-image processing is information including an image processing algorithm that has been executed in the past or parameters of an image processing algorithm that has been executed in the past.

[0099] (Item 9) 9. The information processing system according to any one of items 1 to 8, wherein the user input information for the image data is text information input by the user or text information selected by the user.

[0100] (Item 10) 10. The information processing system according to any one of items 1 to 9, wherein the image processing candidate acquisition unit generates the prompt using a prompt template.

[0101] (Item 11) 5. The information processing system according to any one of items 2 to 4, further comprising an image processing unit that executes image processing corresponding to the candidate image processing selected by the user.

[0102] (Item 12) an image data acquisition unit that acquires first image data and second image data before pre-image processing is applied to the first image data; 12. The information processing system according to any one of items 1 to 11, further comprising an image processing unit that generates third image data by applying the candidate image processing to the first image data or the second image data.

[0103] (Item 13) The information processing system described in item 12, characterized in that the display control unit displays at least one of the first image data and the second image data and the third image data in a comparable manner on the display unit.

[0104] (Item 14) Item 14. The information processing system according to item 12 or 13, wherein the display control unit generates comparison image data representing a comparison result between at least one of the first image data and the second image data and the third image data, and causes the display unit to display the comparison image data.

[0105] (Item 15) 1. An information processing method for applying an image processing algorithm to image data, comprising: an input information acquisition step of acquiring user input information for image data and information regarding an image processing algorithm; an image processing candidate acquisition step of acquiring an image processing candidate that is an image processing algorithm corresponding to the input information and at least one of parameters constituting the image processing algorithm by inputting a prompt based on the input information and information related to the image processing algorithm into a generative model based on deep learning; a display control step of displaying information about the candidate image processing methods on a display unit; An information processing method comprising:

[0106] (Item 16) Item 16. A program for causing a computer to execute the information processing method according to Item 15.

[0107] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0108] 1010 Storage section 1020 Input information acquisition unit 1030 Image processing candidate acquisition unit 1040 Display control unit 1050 Selection information acquisition section 2000 Generative Model

Claims

1. 1. An information processing system for applying an image processing algorithm to image data, comprising: an input information acquisition unit that acquires user input information for image data and information related to image processing algorithms; an image processing candidate acquisition unit that acquires an image processing candidate that is an image processing algorithm corresponding to the input information and at least one of parameters constituting the image processing algorithm by inputting a prompt based on the input information and information related to the image processing algorithm into a generative model based on deep learning; a display control unit that displays information about the candidate image processing methods on a display unit; An information processing system comprising:

2. 2. The information processing system according to claim 1, further comprising an acquisition unit for acquiring information on image processing candidates selected by a user.

3. 3. The information processing system according to claim 2, wherein the selection information acquisition unit acquires whether or not the candidate image processing displayed on the display unit is adopted.

4. the image processing candidate acquisition unit acquires information on a plurality of image processing candidate from the generative model; 3. The information processing system according to claim 2, wherein the selection information acquisition unit acquires information selected by the user from the information regarding the plurality of image processing candidates.

5. the image processing candidate acquisition unit acquires information on a plurality of image processing candidate from the generative model; The information processing system according to claim 2, characterized in that the display control unit causes other image processing candidates from among the plurality of image processing candidates to be displayed on the display unit when the image processing candidate displayed on the display unit is not adopted by the user.

6. The input information acquisition unit further acquires information regarding prior image processing that has been previously applied to the image data; The information processing system according to claim 1 , wherein the display control unit further displays information relating to the preliminary image processing.

7. The information processing system according to claim 6 , wherein the image processing candidate acquisition unit acquires the image processing candidate by inputting information about the pre-image processing into the generative model.

8. 7. The information processing system according to claim 6, wherein the information relating to the preliminary image processing includes an image processing algorithm that has been executed in the past, or parameters of an image processing algorithm that has been executed in the past.

9. 9. The information processing system according to claim 1, wherein the user input information for the image data is text information input by the user or text information selected by the user.

10. 9. The information processing system according to claim 1, wherein the image processing candidate acquisition unit generates the prompt using a prompt template.

11. 5. The information processing system according to claim 2, further comprising an image processing unit that executes image processing corresponding to candidate image processing selected by a user.

12. an image data acquisition unit that acquires first image data and second image data before pre-image processing is applied to the first image data; 9. The information processing system according to claim 1, further comprising an image processing unit that generates third image data by applying the image processing candidates to the first image data or the second image data.

13. The information processing system according to claim 12, characterized in that the display control unit causes at least one of the first image data and the second image data and the third image data to be displayed on the display unit in a comparable manner.

14. The information processing system according to claim 13, characterized in that the display control unit generates comparison image data representing a comparison result between at least one of the first image data and the second image data and the third image data, and displays the comparison image data on the display unit.

15. 1. An information processing method for applying an image processing algorithm to image data, comprising: an input information acquisition step of acquiring user input information for image data and information regarding an image processing algorithm; an image processing candidate acquisition step of acquiring an image processing candidate that is an image processing algorithm corresponding to the input information and at least one of parameters constituting the image processing algorithm by inputting a prompt based on the input information and information about the image processing algorithm into a generative model based on deep learning; a display control step of displaying information about the candidate image processing methods on a display unit; An information processing method comprising:

16. A program for causing a computer to execute the information processing method according to claim 15.

Images