Information processing device, method, program, and system

Abstract

[Problem] To acquire desired information with higher accuracy while minimizing effort. [Solution] A program to be executed by a computer provided with a processor and a memory, wherein the program causes the processor to execute: a step for acquiring an image in which a prescribed region is captured, and presenting the image to a user; a step for receiving a request for an explanation about the image; a step for retrieving, in response to the request for the explanation being received, a sensing value which was measured when the image was captured and which relates to the requested explanation, from a database that stores the captured image and a sensing value measured with respect to the prescribed region; a step for inputting, to an LLM, the image for which the explanation was requested and the sensing value acquired through said retrieval, and causing an explanation text to be output; and a step for presenting the output explanation text to the user.

Description

Information Processing Apparatus, Method, Program, and System

[0001] The present disclosure relates to an information processing apparatus, method, program, and system.

[0002] Patent Document 1 discloses an information retrieval apparatus that can output information highly relevant to a person and their actions based on the results of sensing the person and their actions.

[0003] Japanese Patent Application Laid-Open No. 2022-014034

[0004] In Patent Document 1, in order to output information highly relevant to a person and their actions based on the results of sensing the person and their actions, a general-purpose knowledge database in which language information and various types of information are stored in association with each other is searched. However, in Patent Document 1, since a general-purpose knowledge database in which language information and various types of information are stored in association with each other is required, it is time-consuming.

[0005] An object of the present disclosure is to provide information desired by a user with higher accuracy while avoiding labor.

[0006] In order to solve the above problems, a program according to one aspect of the present disclosure is a program for causing a computer including a processor and a memory to execute. The program causes the processor to execute steps of: acquiring an image of a predetermined area, and presenting the image to the user; receiving a request for an explanation about the image; searching a database for a sensing value measured at the time when the image was taken and related to the requested explanation, where the taken image and the sensing value measured for the predetermined area are stored in the database; inputting the image for which an explanation was requested and the sensing value acquired by the search into a large language model (LLM) to output an explanatory text; and presenting the output explanatory text to the user.

[0007] According to the present disclosure, it is possible to provide information desired by a user with higher accuracy while avoiding labor.

[0008] This is a block diagram showing an example of the overall configuration of System 1. This is a diagram showing an example of the functional configuration of Terminal Device 10. This is a diagram showing an example of the functional configuration of Server 20. This is a diagram showing the data structure of User Information Table 2021. This is a diagram showing the data structure of Image Information Table 2022. This is a diagram showing the data structure of Sensing Value Table 2023. This is a flowchart showing an example of the operation of Server 20. This is a schematic diagram showing an example of the display screen of Display 141. This is a schematic diagram showing an example of the display screen of Display 141. This is a block diagram showing the basic hardware configuration of Computer 90.

[0009] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0010] Furthermore, in the following description, "processor" refers to one or more processors. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

[0011] Furthermore, at least one processor may be a broad-sense processor, such as a hardware circuit that performs some or all of the processing (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0012] Furthermore, in the following explanation, we may use expressions such as "xxx table" to describe information from which an output is obtained for a given input. This information can be data of any structure, or it can be a learning model such as a neural network that generates an output for a given input. Therefore, "xxx table" can be referred to as "xxx information."

[0013] Furthermore, in the following explanation, the configuration of each table is just an example; one table may be divided into two or more tables, or all or part of two or more tables may be a single table.

[0014] Furthermore, in the following explanation, the subject of the process may sometimes be "program," but since a program is executed by a processor and performs defined processes using the memory and / or interface as appropriate, the subject of the process may also be the processor (or a device such as a controller that has that processor).

[0015] The program may be installed on a device such as a computer, or it may reside on a program distribution server or a computer-readable (e.g., non-temporary) recording medium. Furthermore, in the following description, two or more programs may be implemented as a single program, or one program may be implemented as two or more programs.

[0016] Furthermore, in the following explanation, identification numbers are used as identification information for various objects, but other types of identification information (for example, identifiers including letters or symbols) may also be used.

[0017] Furthermore, in the following explanations, when describing similar elements without distinction, a reference code (or a common code among reference codes) may be used, and when describing similar elements with distinction, the element's identification number (or reference code) may be used.

[0018] Furthermore, in the following explanation, only control lines and information lines deemed necessary for the explanation are shown, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

[0019] Each information processing device consists of a computer equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by said hardware configuration will be described later. For each of the terminal device 10 and the server 20, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0020] (Outline) The system according to this embodiment has a function for outputting a descriptive text about an image. The system stores, for example, images and sensing values ​​associated with each other in a database. When the system receives, for example, a request for a description of an image from a user, it uses RAG (Retrieval-Augmented Generation) to search the database for the sensing values ​​associated with the image for which a description has been requested. The system inputs, for example, the image and the sensing values ​​obtained through the search into the LLM, and has the LLM output a descriptive text about the image. The system presents the descriptive text output from the LLM to the user. This allows the user to obtain the desired information with higher accuracy while minimizing effort.

[0021] Images include, for example, images taken at a predetermined time and place. Specifically, for example, images include images taken in an operating room at a predetermined time, i.e., images related to surgery. Images also include, for example, images taken by a surveillance camera installed in a predetermined monitoring area at a predetermined time. Sensing values ​​include, for example, values ​​measured by a predetermined device at a predetermined time and place. Specifically, for example, sensing values ​​include temperature, heart rate, blood pressure, body temperature, operating parameters of devices used in surgery (signals from the operating pedal of an endoscope), etc., measured by a measuring device installed in an operating room (or on the patient) at a predetermined time. Sensing values ​​also include, for example, temperature, humidity, sound volume, etc., measured by a measuring device installed in a predetermined monitoring area at a predetermined time.

[0022] <Configuration> The configuration of the information processing system will now be described. Figure 1 is a block diagram showing an example of the overall configuration of System 1. System 1 shown in Figure 1 includes, for example, a terminal device 10, a server 20, a camera 30, a sensor 31, a first database 32, a second database 33, and an LLM system 34. The terminal device 10, the server 20, the camera 30, the sensor 31, the first database 32, the second database 33, and the LLM system 34 are connected by communication via, for example, a network 80.

[0023] Figure 1 shows an example where System 1 includes one terminal device 10, but the number of terminal devices 10 included in System 1 is not limited to one. System 1 may include two or more terminal devices 10.

[0024] Figure 1 shows an example where system 1 includes one LLM system 34, but the number of LLM systems 34 included in system 1 is not limited to one. System 1 may include two or more LLM systems 34.

[0025] Figure 1 shows an example where server 20 is independent of the LLM system 34, but server 20 may also include the functions of the LLM system 34. In other words, server 20 may store the LLM.

[0026] In this embodiment, a collection of multiple devices may be treated as a single server. The method of allocating the multiple functions required to implement the server 20 according to this embodiment to one or more hardware can be appropriately determined in view of the processing capacity of each hardware and / or the specifications required for the server 20.

[0027] The terminal device 10 shown in Figure 1 is an information processing device operated by the user to view the image. The terminal device 10 can be implemented as, for example, a stationary PC (Personal Computer), a laptop PC, etc. The terminal device 10 may also be implemented as a mobile device such as a smartphone or tablet.

[0028] The terminal device 10 includes a communication interface 12, an input device 13, an output device 14, a memory 15, storage 16, and a processor 19. The input device 13 is a device for receiving input operations from the user (e.g., a touch panel, touchpad, mouse or other pointing device, keyboard, etc.). The output device 14 is a device for presenting information to the user (display, speaker, etc.).

[0029] Server 20 is an information processing device that, for example, uses RAG to provide a service that displays descriptive text about an image. Specifically, when Server 20 receives a request from a user for a description of an image, it searches the second database 33 for the sensing values ​​associated with the image for which a description has been requested. In addition to the image and instructions for creating a description of the image, Server 20 also inputs the retrieved sensing values ​​as prompts to the LLM system 34. Server 20 causes the LLM system 34 to output the description of the image. Server 20 then presents the description of the image output from the LLM system 34 to the user. In this way, Server 20 uses RAG by searching for sensing values ​​from the second database 33 and inputting the retrieved sensing values ​​as prompts to the LLM system 34.

[0030] The server 20 is an information processing device implemented by, for example, a computer connected to the network 80. As shown in Figure 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an interface to an input device for receiving input operations from the user and an output device for outputting information to the user.

[0031] Camera 30 is a device that receives light using a photodetector and outputs it as a shooting signal. Camera 30 is installed in a predetermined space, for example, and takes pictures in a direction corresponding to the operator's input within that space. Alternatively, Camera 30 may be installed in a predetermined space, for example, and take pictures in a predetermined direction within that space.

[0032] Sensor 31 is, for example, a device installed in a predetermined space that measures a preset value. Alternatively, sensor 31 is a device installed on an object placed in a predetermined space that measures a preset value. The preset values ​​may include, for example, temperature, humidity, volume, heart rate, blood pressure, body temperature, or operating parameters of a device used in surgery.

[0033] The first database 32 is an information processing device that stores information about images. The first database 32 includes, for example, a communication interface, an input / output interface, memory, storage, and a processor. The first database 32 stores, for example, an image information table 2022. The image information table 2022 will be described later. The images managed in the image information table 2022 may be, for example, DICOM (Digital Imaging and Communications in Medicine).

[0034] In the example shown in Figure 1, the first database 32 is built outside of the server 20. "Outside" indicates, for example, that it is managed by an administrator separate from the server 20. Note that the first database 32 may also be built within the server 20.

[0035] The second database 33 is an information processing device that stores information related to sensing values. The second database 33 includes, for example, a communication interface, an input / output interface, memory, storage, and a processor. The second database 33 stores, for example, a sensing value table 2023. The sensing value table 2023 will be described later.

[0036] In the example shown in Figure 1, the second database 33 is built outside the server 20. However, the second database 33 may also be built within the server 20. Figure 1 also shows the case where there is only one second database 33. The second database 33 may be provided for each different sensing value. If there is a different second database 33 for each sensing value, each second database 33 may be managed by a different administrator.

[0037] The LLM system 34 is a system in which a large-scale artificial intelligence model (LLM) used, for example, in the field of natural language processing (NLP) has been constructed. By learning from large amounts of text data (web pages, books, articles, etc.), the LLM can understand patterns in human language and effectively perform natural language generation (NLG) tasks.

[0038] LLM is used in many NLP tasks, such as generating responses to specific questions, automatically generating text, summarizing text, translation, and sentiment analysis. LLM can also be used in a variety of applications, including education, entertainment, customer service, and product development. LLM is, for example, multimodal LLM. Examples of LLM include: • GPT-4 (registered trademark) (OpenAI Inc.) • Gemini (registered trademark) (Google Inc.) • StableLM (StableAI Inc.) • Llama3.2 (Meta Inc.)

[0039] The LLM system 34 receives instructions from the server 20 to create an image, sensing values, and a description of the image, and instructs the LLM to output a response based on the input image, sensing values, and instructions for creating a description of the image. The LLM system 34 then sends the response output from the LLM to the server 20.

[0040] Each information processing device consists of a computer equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by said hardware configuration will be described later. For each of the terminal device 10, server 20, and LLM system 34, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0041] <1.1 Configuration of the Terminal Device> Figure 2 is a diagram showing an example of the functional configuration of the terminal device 10. As shown in Figure 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a storage unit 180, and a control unit 190. Each block included in the terminal device 10 is electrically connected, for example, by a bus or the like.

[0042] The communication unit 120 performs processes such as modulation / demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to the outside (for example, the server 20). The communication unit 120 performs reception processing on the signal received from the outside and outputs it to the control unit 190.

[0043] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 includes a keyboard 131 and a mouse 132. The input device 13 converts an instruction input from the user into an electrical signal and outputs the electrical signal to the control unit 190. Note that the input device 13 may include, for example, a reception port for receiving an electrical signal input from an external input device.

[0044] The output device 14 is a device for presenting information to the user operating the terminal device 10. The output device 14 is realized, for example, by a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, or the like.

[0045] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of an audio signal. The audio processing unit 17 converts the signal given from the microphone 171 into a digital signal and gives the converted signal to the control unit 190. Also, the audio processing unit 17 gives the audio signal to the speaker 172. The audio processing unit 17 is realized, for example, by a processor for audio processing. The microphone 171 receives an audio input and gives an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal given from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0046] The camera 160 is a device that receives light by a light receiving element and outputs it as a shooting signal.

[0047] The position information sensor 150 is a sensor that detects the position of the terminal device 10 and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals from at least three or four satellites are received, and based on the received signals, the current position of the terminal device 10 equipped with the GPS module is detected. The position information sensor 150 may detect the current position of the terminal device 10 from the position of the radio base station to which the terminal device 10 is connected.

[0048] The storage unit 180 is realized, for example, by the memory 15, the storage 16, etc., and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0049] The user information 181 includes, for example, information about the user who uses the terminal device 10. Information about the user includes, for example, the user's name, age, address, date of birth, contact information, etc.

[0050] The control unit 190 is realized when the processor 19 reads a program stored in the memory unit 180 and executes instructions contained in the program. The control unit 190 controls the operation of the terminal device 10. By operating according to the program, the control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193.

[0051] The operation reception unit 191 processes instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from the keyboard 131, mouse 132, etc.

[0052] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives voice signals input from the microphone 171 and converted into digital signals by the voice processing unit 17. The operation reception unit 191 obtains instructions from the user by, for example, analyzing the received voice signals and extracting predetermined nouns.

[0053] The transmitting / receiving unit 192 performs processing to enable the terminal device 10 to send and receive data with an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 sends information input by the user or instructions from the user to the server 20. The transmitting / receiving unit 192 also receives information provided by the server 20.

[0054] The presentation control unit 193 controls the output device 14 in order to present information provided by the server 20 to the user. Specifically, for example, the presentation control unit 193 causes the display 141 to display summary information transmitted from the server 20. The presentation control unit 193 also causes the speaker 172 to output information transmitted from the server 20.

[0055] <1.2 Functional Configuration of the Server> Figure 3 shows an example of the functional configuration of the server 20. As shown in Figure 3, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0056] The communication unit 201 performs processing to enable the server 20 to communicate with external devices.

[0057] The storage unit 202 includes, for example, a user information table 2021. However, the tables stored in the storage unit 202 are not limited to the user information table 2021.

[0058] User information table 2021 is a table that stores information about users. Further details will be provided later.

[0059] The control unit 203 is realized when the processor 29 reads a program stored in the memory unit 202 and executes instructions contained in the program. The program includes applications such as a web browser application. The program includes a programming language such as JavaScript® that is executed on the web browser application stored in the terminal device 10. By operating according to the program, the control unit 203 performs the functions indicated as the reception control module 2031, the transmission control module 2032, the presentation control module 2033, the search control module 2034, the image processing module 2035, and the LLM processing control module 2036.

[0060] The reception control module 2031 controls the process by which the server 20 receives signals from an external device according to a communication protocol.

[0061] The transmission control module 2032 controls the process by which the server 20 transmits signals to an external device according to a communication protocol.

[0062] The presentation control module 2033 controls the process of presenting information to the user. Specifically, for example, the presentation control module 2033 presents an image to the user. The presentation control module 2033 also presents the explanatory text transmitted from the LLM system 34 to the user. More specifically, for example, the presentation control module 2033 presents the explanatory text about the image, which was output as a response from the LLM system 34, to the user.

[0063] The search control module 2034 receives a request for an image description and controls the process of searching for sensing values ​​from the second database 33 based on the received request. Specifically, for example, the search control module 2034 receives an instruction from the user to create an image description as a request for an image description. In response to the receipt of the description request, the search control module 2034 identifies additional information attached to the image. Additional information includes, for example, metadata of the data to be attached. For example, additional information attached to an image includes information about the date and time the image was taken, the location where the image was taken, the equipment used to take the image, the operator who operated the equipment, etc.

[0064] The search control module 2034 searches the second database 33 for sensing values ​​with additional information that partially or completely matches the identified additional information, based on the identified additional information and the timing at which the user requested an explanation. For example, the additional information attached to the sensing value includes information about the date and time the sensing value was measured, the location where the sensing value was measured, the sensor that measured the sensing value, and the equipment to which the sensor is attached. As a result, the search control module 2034 searches the second database 33 for sensing values ​​that were measured when the image for which the user requested an explanation was taken, and that were measured at the location where the image was taken.

[0065] The image processing module 2035 controls the process of deleting supplementary information displayed in an image. Supplementary information is, for example, information attached to an image, and includes information about the object represented in the image (name, ID, etc.). Supplementary information also includes information about the state of the object represented in the image (parameters, etc.). Supplementary information is not an image captured by the camera 30. Specifically, for example, the image processing module 2035 takes an image with supplementary information attached as input data and inputs the image into a trained model that has been trained using the specifications for the supplementary information as ground truth data. The image processing module 2035 deletes the supplementary information from the image by having the trained model output an image from which the supplementary information has been removed. If the position where supplementary information is attached in the image is predetermined, the image processing module 2035 may delete the supplementary information from the image based on the predetermined position.

[0066] The LLM processing control module 2036 inputs an image, sensing values, and instructions for creating a descriptive text about the image to the LLM, and outputs a descriptive text about the image. The image may be a still image or a video. Specifically, the LLM processing control module 2036 inputs the image for which a description is requested, the sensing values ​​obtained by the search, and instructions for creating a descriptive text about the image to the LLM. In other words, the LLM processing control module 2036 inputs the instructions for creating a descriptive text about the image and the sensing values ​​obtained by the search as prompts to the LLM, and also inputs the image for which a description is requested to the LLM. As a result, the LLM processing control module 2036 causes the LLM to output a descriptive text about the image that takes the sensing values ​​into account.

[0067] The LLM processing control module 2036 may input instructions to the LLM, for example, instructions input by the user to request an image description. Alternatively, the LLM processing control module 2036 may create instructions to input to the LLM based on, for example, instructions to request an image description. The LLM processing control module 2036 may, for example, create instructions to output an image description that takes sensing values ​​into account, based on instructions to request an image description.

[0068] <2. Data Structure> Figures 4 to 6 show the data structure of the tables stored by the server 20. Note that Figures 4 to 6 are examples and do not exclude data that is not shown. Also, even if data is listed in the same table, it may be stored in separate memory areas in the storage unit 202.

[0069] Figure 4 shows the data structure of the user information table 2021. The user information table 2021 shown in Figure 4 is a table that uses User ID as the key and has columns for Name, Age, Gender, Date of Birth, and Contact Information.

[0070] User ID is an item that stores an identifier to uniquely identify the user. Name is an item that stores the user's name. Age is an item that stores the user's age. Gender is an item that stores the user's gender. Date of birth is an item that stores the user's date of birth. Contact information is an item that stores the contact information (e.g., telephone number, email address, etc.) of the terminal device 10 that the user possesses.

[0071] Figure 5 shows the data structure of the image information table 2022. The image information table 2022 shown in Figure 5 is a table that uses the image ID as the key and contains the date and time, image data, and an ID related to the acquisition location.

[0072] The Image ID is an item that stores an identifier to uniquely identify the image. The Date and Time is an item that stores the date and time the image was created. The Image Data is an item that stores the image data. The Location ID is an item that stores an identifier to uniquely identify the location where the image was taken. The Date and Time and the Location ID are additional information associated with the image.

[0073] Figure 6 shows the data structure of the sensing value table 2023. The sensing value table 2023 shown in Figure 6 is a table that uses the sensing value ID as the key and contains the year, month, and time, an ID related to the acquisition location, and the sensing value.

[0074] The Sensing Value ID is an item that stores an identifier to uniquely identify the sensing value. The Date and Time is an item that stores the date and time the sensing value was generated. The Location ID is an item that stores an identifier to uniquely identify the location where the sensing value was obtained. The Sensing Value is an item that stores the sensing value. The Date and Time and the Location ID are additional information associated with the sensing value.

[0075] <3 Operation> This section describes the operation of server 20 when using RAG to display explanatory text about an image.

[0076] Figure 7 is a flowchart illustrating an example of the operation of server 20 when presenting descriptive text about an image. In the explanation of Figure 7, server 20 provides the service of presenting descriptive text about an image, for example, as SaaS (Software as a Service). In this case, the target image is, for example, an image related to surgery (an image taken with an endoscope). The image is taken, for example, by an endoscopic device installed in an operating room and stored in the image information table 2022. The user is, for example, a medical professional such as a doctor or nurse.

[0077] First, a user who wants to obtain a description of an image accesses server 20. The user logs in to the service provided by server 20 that displays descriptions of images, for example, by entering their user ID and password.

[0078] In step S11, the server 20 presents an image to the user. Specifically, for example, the user operates the terminal device 10 and selects an image (a video related to surgery) that they wish to view. The operation reception unit 191 accepts the image selection input by the user. The transmission / reception unit 192 sends the image ID of the video selected by the user to the server 20. The reception control module 2031 receives the image ID sent from the terminal device 10. In other words, the server 20 accepts the video selection by the user. The presentation control module 2033 reads the video data stored in the image information table 2022 of the first database 32 based on the received image ID. The presentation control module 2033 presents the video data to the user. The presentation control unit 193 controls the display 141 to display the video based on the video data to the user.

[0079] The user views the video displayed on the display 141. The user operates the terminal device 10 and inputs instructions to create a descriptive text for a predetermined frame of the video displayed on the terminal device 10. The predetermined frame of the video is, for example, an image with a predetermined timestamp. An example of an instruction to create a descriptive text for an image is, "This is an endoscopic image of laparoscopic surgery. Please describe the surgical situation." The operation reception unit 191 receives the instruction to create a descriptive text for an image as a request for an explanation of the image. The transmission / reception unit 192 sends the instruction to create a descriptive text for an image, the image ID of the video relating to the image for which an explanation is requested, and the timestamp of the video at the time the instruction was received to the server 20.

[0080] In step S12, the server 20 receives a request for an explanation of the image. Specifically, the receiving control module 2031 receives an instruction from the terminal device 10 to create an explanation of the image, the image ID of the video relating to the image for which an explanation is requested, and the timestamp of the video at the time the instruction was received. In other words, the server 20 receives a request from the user for an explanation of the image.

[0081] In step S13, the server 20 searches for sensing values ​​(various sensing values ​​measured during surgery). Specifically, for example, the search control module 2034 performs a vector search based on the instruction and targets sensing values ​​for the search. The search control module 2034 identifies additional information (date and time and ID related to the acquisition location) associated with the received image ID in the image information table 2022 of the first database 32. The search control module 2034 searches the sensing value table 2023 of the second database 33 for sensing values ​​to which the identified additional information (date and time and ID related to the acquisition location) and additional information including the video timestamp are attached. In other words, in response to the receipt of a request for explanation, the search control module 2034 searches the second database 33 for sensing values ​​measured at the timing of the video timestamp and measured in the operating room where the surgery is performed. Furthermore, in response to a request for explanation, the search control module 2034 retrieves sensing values ​​measured at the timing of the video's timestamp and for patients undergoing surgery from the second database 33.

[0082] In step S14, the server 20 deletes the ancillary information. Specifically, the image processing module 2035 reads the video identified by the received image ID from the image information table 2022 of the first database 32. The image processing module 2035 inputs the read video into, for example, a trained model in the storage unit 202. Ancillary information displayed in the video, such as the patient's name, patient ID, and gender, is deleted by the trained model. Ancillary information displayed in the video, such as the operation parameters of the surgical instrument arms, is also deleted by the trained model. The image processing module 2035 retrieves the video from which the ancillary information has been deleted, which is output from the trained model.

[0083] In step S15, the server 20 obtains a description of the video. Specifically, the LLM processing control module 2036 inputs the instruction received in step S12 to create a description of the image, the sensing values ​​retrieved in step S13, and the video from which the accompanying information has been removed in step S14 into the LLM. In other words, the LLM processing control module 2036 inputs the instruction received in step S12 to create a description of the image, the sensing values ​​retrieved in step S13, and the video from which the accompanying information has been removed in step S14 into the LLM as prompts.

[0084] In this case, the LLM processing control module 2036 may create an instruction to be input to the LLM, for example, an instruction to request an image description, and an instruction to output an explanation that takes sensing values ​​into consideration.

[0085] The LLM processing control module 2036 retrieves the explanatory text output from the LLM. An example of the explanatory text is: "This image is an endoscopic image of laparoscopic surgery. A needle holder to hold the needle and Versatile Grasping Forceps to grasp the tissue are being used for suturing. At this time, the patient's temperature is 37.0°C, heart rate is 80 beats / min, systolic blood pressure is 115 mmHg, and diastolic blood pressure is 75 mmHg. The surgery is progressing smoothly."

[0086] In step S16, the server 20 presents the descriptive text to the user. Specifically, the presentation control module 2033 reads the descriptive text output from the LLM in step S15. The presentation control module 2033 presents the descriptive text about the image to the user. The presentation control unit 193 controls the display 141 to display the descriptive text about the image to the user.

[0087] <4. Screen Examples> Figures 8 and 9 are schematic diagrams showing examples of the display screen of the display 141 of the terminal device 10. Figure 8 explains the case where the image is, for example, an image related to surgery (an image taken with an endoscope).

[0088] The screen shown in Figure 8 includes a first region 1411 and a second region 1412. The first region 1411 is, for example, a region for displaying images related to surgery. The images displayed in the first region 1411 are, for example, images obtained from image data read from the image information table 2022 by the presentation control module 2033.

[0089] The image displayed in the first region 1411 includes supplementary information A1, A2, A3, and A4. Supplementary information A1 is information related to the operation of the endoscope's operating pedal. Supplementary information A2 is information related to the surgery. Supplementary information A3 is information indicating the date and time the image was taken. Supplementary information A4 is information indicating the position of the endoscope. Supplementary information A1, A2, A3, and A4 are information within the image, but they are not information generated by the camera 30 taking the image.

[0090] The second area 1412 is an area that displays instructions, for example, for creating a descriptive text about an image displayed in the first area 1411, which have been entered by the user. In the screen shown in Figure 8, this area includes a button 1413. Button 1413 is a button for entering a request for a description of the image. In other words, button 1413 is a button for sending the prompt displayed in the second area 1412 to the server 20.

[0091] In step S11, when the server 20 accepts the selection of an image, the presentation control module 2033 reads the selected image from the image information table 2022. The presentation control module 2033 presents the read image to the user. The presentation control unit 193 displays the image in the first area 1411.

[0092] The user checks the image displayed in the first area 1411 and enters instructions to create a description of the image. The instructions to create a description of the image are displayed in the second area 1412. The user presses button 1413. When button 1413 is pressed, the operation reception unit 191 receives a request for a description of the image. The transmission / reception unit 192 sends the request for a description of the image to the server 20.

[0093] Figure 9 is a schematic diagram showing another example of the display screen of the display 141 of the terminal device 10. In Figure 9, the example is given when the image is, for example, an image related to surgery (an image taken with an endoscope).

[0094] The screen shown in Figure 9 includes a third area 1414, a fourth area 1415, and a fifth area 1416. The third area 1414 is, for example, an area for displaying images related to surgery. The fourth area 1415 is, for example, an area for displaying instructions entered by the user to create a descriptive text about the image displayed in the third area 1414. The fifth area 1416 displays, for example, a descriptive text about the image displayed in the third area 1414, output by the LLM system 34.

[0095] In step S16, the presentation control module 2033 reads the explanatory text output from the LLM in step S15. The presentation control unit 193 displays the explanatory text about the image in the fifth area 1416.

[0096] Improving the accuracy of the output information from the LLM can sometimes be difficult. Therefore, a method (Retrieval-Augmented Generation: RAG) is being used to improve the accuracy of the output information from the LLM by inputting input information linked to external information into the LLM, without modifying the LLM itself.

[0097] As described above, in this embodiment, the search control module 2034 receives a request for an image description and controls the process of searching for sensing values ​​from the second database 33. The LLM processing control module 2036 inputs the image and sensing values ​​into the LLM and outputs a description of the image. Therefore, according to this embodiment, the information desired by the user can be provided with higher accuracy while minimizing effort.

[0098] In the above embodiment, the image and sensing value are associated with each other using additional information (IDs for time and acquisition location) already associated with the image and sensing value, and then input into the LLM. Therefore, it is not necessary to pre-associate information (primary key) for associating the image and sensing value with each other, and information processing can be made more efficient.

[0099] Furthermore, in the above embodiment, the image from which the supplementary information has been removed is input to the LLM. Therefore, according to this embodiment, the information desired by the user can be provided with higher accuracy.

[0100] <Modified Examples> In the above embodiment, the example described was that the image is an image related to surgery and the sensing values ​​are various sensing values ​​measured during surgery. However, the image may be an image captured by a surveillance camera, and the sensing values ​​may be various sensing values ​​measured in the space being monitored.

[0101] Furthermore, the above embodiment described an example in which one database (second database 33) stores the sensing values. However, the sensing values ​​may be recorded in multiple databases. For example, the sensing values ​​may be recorded in different databases for each type of sensing value.

[0102] Furthermore, the administrator of at least one of the sensing values ​​may be different from the administrator of the other sensing values.

[0103] Furthermore, in the above embodiment, the case in which the instruction text for creating a descriptive text about the image is input via the input device 13 was described as an example. However, the instruction text for creating a descriptive text about the image may also be input via the voice processing unit 17.

[0104] Furthermore, the above embodiment was described using the example of a server presenting a descriptive text about an image. However, the server may also present a descriptive text about video. The system stores, for example, associated video and sensing values ​​in a database. When the server receives a request from a user for a description of a video, for example, it retrieves information about the video from the database. The server retrieves, for example, an image of the video for which a description has been requested at the moment the request was entered. The server uses RAG, as described above, to search the database for sensing values ​​associated with the retrieved image for the video for which a description has been requested. The server inputs, for example, the image and the sensing values ​​obtained through the search into LLM, and has LLM output a descriptive text about the video at the time the user requested the description. The system presents the descriptive text output from LLM to the user.

[0105] Furthermore, in the above embodiment, the example described was in which the search control module 2034 performs a vector search based on an instruction and searches for the sensing value using additional information (IDs for time and acquisition location) already associated with the image and sensing value. However, the search control module 2034 may also search for the sensing value using a multimodal LLM. The multimodal LLM may be stored in the server 20 or an external service may be used. For example, when the search control module 2034 receives a request from the user for a description of an image, it inputs the image and the instruction into the multimodal LLM. The multimodal LLM outputs a response about the image. Based on the response output from the multimodal LLM, the search control module 2034 performs a vector search, keyword search, hybrid search, and / or semantic search to retrieve the sensing value from the database. The search control module 2034 inputs the image, the retrieved sensing value, and an instruction to generate a descriptive text about the image as prompts to the LLM, and the LLM outputs a descriptive text about the image. The system then presents the descriptive text output by the LLM to the user.

[0106] <Basic Hardware Configuration of Computer> Figure 10 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 includes at least a processor 901, main memory 902, auxiliary storage 903, and a communication IF 991 (interface). These are electrically connected to each other by a communication bus 921.

[0107] The processor 901 is hardware for executing the instruction set described in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0108] The main memory 902 is for temporarily storing programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0109] The auxiliary storage device 903 is a storage device for storing data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, etc.

[0110] A communication interface (IF991) is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. The network consists of various mobile communication systems, such as the Internet, LANs, and wireless base stations. For example, networks include 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks (e.g., Wi-Fi®) that can connect to the Internet via designated access points. When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via wired connections, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0111] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0112] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 (Figure 10) is described below. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0113] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0114] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0115] The memory unit is implemented by a main memory 902 and an auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform addition, update, and deletion operations on data stored in the memory unit according to the various programs.

[0116] The term "database" refers to a relational database, which is used to manage and associate data sets called tables and masters, which are structured in a tabular format defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and each master has a primary key column to uniquely identify a record, but setting a primary key for a column is not mandatory. The control unit can cause the processor 901 to add, delete, and update records in specific tables and masters stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, the information processing device and information processing system described in this disclosure can be considered manufactured.

[0117] Furthermore, the databases and masters in this disclosure may include any data structures (lists, dictionaries, associative arrays, objects, etc.) in which information is structurally defined. Data structures also include data that can be considered as data structures by combining data with functions, classes, methods, etc., written in any programming language.

[0118] The communication unit is implemented by the communication IF 991. The communication unit implements the function of communicating with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0119] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0120] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java®.

[0121] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0122] The functions realized by the components described herein may be implemented in a circuit or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the described functions. A processor, including transistors and other circuits, is considered a circuit or processing circuitry. A processor may be a programmed processor that executes a program stored in memory. In this specification, circuitry, unit, and means are hardware programmed to realize or perform the described functions. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to realize or perform the described functions. If such hardware is a processor that is considered a type of circuitry, then such circuitry, means, or unit is a combination of hardware and software used to constitute such hardware and / or processor.

[0123] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0124] (Note) The matters described in each of the above embodiments are noted below. (Note 1): To be described later.

[0125] 1...System 10...Terminal device 120...Communication unit 13...Input device 14...Output device 15...Memory 16...Storage 19...Processor 20...Server 22...Communication IF 23...Input / Output IF 25...Memory 26...Storage 29...Processor Claim tree Main claim: CL1 (A) Acquire an image in which a predetermined area has been photographed and present the image to the user. (B) Receive a request for an explanation about the image. (C) Sensing values ​​measured for the predetermined area are stored in a database, and in response to the receipt of a request for an explanation, retrieve the sensing values ​​measured when the image was taken and related to the requested explanation from the database. (D) Input the image for which an explanation is requested and the sensing values ​​obtained by the search into the LLM and output an explanatory text. (E) Present the output explanatory text to the user. Subclaims (Direction 1) Image preprocessing CL2: (D1) Remove any attached information from the image and input the image with the attached information removed into the LLM. ┗CL3: (D1-1) Use the image with attached information as input data, and input the image into a trained model that has been trained with the specifications for the attached information as ground truth data to output an image with the attached information removed. Input the image with the attached information removed and the retrieved sensing values ​​into the LLM and output a descriptive text. ┗CL4: (D1-2) The position where attached information is attached to the image is set in advance, and based on the set position, remove the attached information from the image, input the image with the attached information removed and the retrieved sensing values ​​into the LLM and output a descriptive text. (Direction 2) Specific example CL5: (A1) The image is related to surgery, and the sensing values ​​are various sensing values ​​measured during surgery. CL6: (A2) Images are captured by surveillance cameras, and sensing values ​​are various sensing values ​​measured in the monitored space. (Direction 3) Database CL7: (C1) Captured images are also stored in the database. ┗CL8: (C1-1) The databases storing images and sensing values ​​are different. CL9: (C2) The administrator of at least one of the sensing values ​​is different from the administrator of the other sensing values. CL10: Information processing device CL11: Method CL12: System

Claims

1. A program for execution on a computer comprising a processor and memory, the program causing the processor to perform the following steps: acquire an image of a predetermined region and present the image to a user; accept a request for an explanation of the image; retrieve sensing values ​​measured for the predetermined region from a database, in response to the acceptance of the request for an explanation, the sensing values ​​measured when the image was taken and relating to the requested explanation from the database; input the image for which an explanation has been requested and the sensing values ​​acquired by the search into a Large Language Model (LLM) and output an explanatory text; and present the output explanatory text to the user.

2. The program according to claim 1, wherein the step of outputting the explanatory text involves deleting any supplementary information attached to the image for which the explanation was requested, inputting the image from which the supplementary information has been deleted and the sensing value obtained by the search into the LLM, and outputting the explanatory text.

3. The program according to claim 2, wherein the step of outputting the explanatory text is to input the image with the attached supplementary information as input data to a trained model which has been trained with the specifications for the supplementary information as correct data, thereby outputting the image from which the supplementary information has been removed, inputting the image from which the supplementary information has been removed and the sensing value obtained by the search into the LLM, and outputting the explanatory text.

4. The program according to claim 2, wherein the step of outputting the explanatory text is characterized in that the position to which the supplementary information is attached to the image is predetermined, the supplementary information is deleted from the image based on the predetermined position, the image from which the supplementary information has been deleted and the sensing value obtained by the search are input to the LLM, and the explanatory text is output.

5. The program according to any one of claims 1 to 4, wherein the image is an image related to surgery, and the sensing value is various sensing values ​​measured during surgery.

6. The program according to any one of claims 1 to 4, wherein the image is an image captured by a surveillance camera, and the sensing value is various sensing values ​​measured in the space to be monitored.

7. The program according to any one of claims 1 to 4, wherein the captured image is also stored in the database.

8. The database for storing the image and the sensing value is a different program according to claim 7.

9. The program according to any one of claims 1 to 4, wherein the administrator of at least one of the sensing values ​​is different from the administrator of the other sensing values.

10. An information processing device comprising a control unit and a storage unit, wherein the control unit performs the steps of: acquiring an image of a predetermined area and presenting the image to a user; receiving a request for an explanation of the image; storing the captured image and sensing values ​​measured for the predetermined area in a database, and in response to receiving a request for an explanation, searching the database for the sensing values ​​measured when the image was captured and related to the requested explanation; inputting the image for which an explanation has been requested and the sensing values ​​obtained by the search into an LLM and outputting an explanatory text; and presenting the outputted explanatory text to a user.

11. A method to be performed on a computer comprising a processor and memory, wherein the processor performs the steps of: acquiring an image of a predetermined area and presenting the image to a user; receiving a request for an explanation of the image; having the captured image and sensing values ​​measured for the predetermined area stored in a database, and in response to the receipt of the request for an explanation, searching the database for the sensing values ​​measured when the image was captured and relating to the requested explanation; inputting the image for which an explanation has been requested and the sensing values ​​obtained by the search into an LLM and outputting an explanatory text; and presenting the outputted explanatory text to the user.

12. A system comprising: means for acquiring an image of a predetermined area and presenting the image to a user; means for receiving a request for an explanation of the image; means for storing the captured image and sensing values ​​measured for the predetermined area in a database, and in response to receiving a request for an explanation, searching the database for the sensing values ​​measured when the image was captured and related to the requested explanation; means for inputting the image for which an explanation has been requested and the sensing values ​​obtained by the search into an LLM and outputting an explanatory text; and means for presenting the outputted explanatory text to a user.

Images