Information processing equipment, methods, programs and systems

Abstract

To obtain the desired information with higher accuracy while minimizing effort. [Solution] A program to be executed on a computer having a processor and memory, the program causes the processor to perform the following steps: acquire an image of a predetermined area and present the image to the user; accept a request for an explanation of the image; have sensing values ​​measured for the predetermined area stored in a database and, in response to the acceptance of a request for an explanation, search the database for sensing values ​​measured when the image was taken and related to the requested explanation; input the image for which an explanation is requested and the sensing values ​​obtained by the search into the LLM and output an explanatory text; and present the output explanatory text to the user.

Description

Technical Field

[0001] This disclosure relates to an information processing apparatus, method, program, and system.

Background Art

[0002] Patent Document 1 discloses an information retrieval apparatus that can output information highly relevant to a person and their actions based on the results of sensing the person and their actions.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, in order to output information highly relevant to a person and their actions based on the results of sensing the person and their actions, a general-purpose knowledge database in which language information and various types of information are stored in association is searched. However, in Patent Document 1, since a general-purpose knowledge database in which language information and various types of information are stored in association is required, it is laborious.

[0005] An object of this disclosure is to provide information desired by a user with higher accuracy while avoiding labor.

Means for Solving the Problems

[0006] To solve the above problems, a program according to one aspect of this disclosure is a program to be executed on a computer having a processor and memory. The program causes the processor to perform the following steps: acquire an image of a predetermined area and present the image to the user; accept a request for an explanation of the image; retrieve sensing values ​​measured for the predetermined area from a database, which is stored therein, in response to the acceptance of a request for an explanation, the sensing values ​​measured when the image was taken and relating to the requested explanation; input the image for which an explanation is requested and the sensing values ​​obtained by the search into an LLM (Large Language Model) and output an explanatory text; and present the output explanatory text to the user. [Effects of the Invention]

[0007] According to this disclosure, users can be provided with the information they desire, with less effort and greater accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the overall configuration of System 1. [Figure 2] This figure shows an example of the functional configuration of terminal device 10. [Figure 3] This figure shows an example of the functional configuration of server 20. [Figure 4] This diagram shows the data structure of the user information table 2021. [Figure 5] This diagram shows the data structure of the image information table 2022. [Figure 6] This diagram shows the data structure of the sensing value table 2023. [Figure 7] This is a flowchart illustrating an example of how Server 20 operates. [Figure 8] This is a schematic diagram illustrating an example of the display screen of display 141. [Figure 9] This is a schematic diagram illustrating an example of the display screen of display 141. [Figure 10] A block diagram showing the basic hardware configuration of Computer 90. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0010] Furthermore, in the following description, "processor" refers to one or more processors. At least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

[0011] Furthermore, at least one processor may be a broad-sense processor, such as a hardware circuit that performs some or all of the processing (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0012] Furthermore, in the following explanation, we may use expressions such as "xxx table" to describe information that yields an output for a given input. This information can be data with any structure, or it can be a learning model such as a neural network that generates an output for a given input. Therefore, "xxx table" can be referred to as "xxx information."

[0013] In the following description, the configuration of each table is an example. One table may be divided into two or more tables, or all or part of two or more tables may be combined into one table.

[0014] In the following description, the "program" may be used as the subject to describe the process. However, since the program performs the defined process by being executed by the processor, appropriately using the storage unit and / or the interface unit, etc., the subject of the process may be the processor (or a device such as a controller having that processor).

[0015] The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (e.g., non-temporary) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0016] In the following description, an identification number is used as the identification information for various objects, but identification information of other types (e.g., an identifier including letters or symbols) may be adopted.

[0017] In the following description, when describing elements of the same type without distinction, reference signs (or common signs among the reference signs) are used, and when describing elements of the same type by distinction, the identification numbers (or reference signs) of the elements may be used.

[0018] In the following description, the control lines and information lines indicate those considered necessary for the description, and not necessarily all control lines and information lines are shown on the product. All components may be interconnected.

[0019] Each information processing device consists of a computer equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by said hardware configuration will be described later. For each of the terminal device 10 and the server 20, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0020] (Summary) The system according to this embodiment has a function for outputting a descriptive text about an image. The system stores, for example, images and sensing values ​​associated with each other in a database. When the system receives a request from a user for a description of an image, for example, it uses RAG (Retrieval-Augmented Generation) to search the database for the sensing values ​​associated with the image for which a description has been requested. The system inputs, for example, the image and the sensing values ​​obtained through the search into an LLM (Looking and Reading Machine), and has the LLM output a descriptive text about the image. The system presents the descriptive text output by the LLM to the user. This allows the user to obtain the desired information with greater accuracy while minimizing effort.

[0021] Images include, for example, images taken at a predetermined time and place. Specifically, for example, images include images taken in an operating room at a predetermined time, i.e., images related to surgery. Images also include, for example, images taken by a surveillance camera installed in a predetermined monitoring area at a predetermined time. Sensing values ​​include, for example, values ​​measured by a predetermined device at a predetermined time and place. Specifically, for example, sensing values ​​include temperature, heart rate, blood pressure, body temperature, operating parameters of devices used in surgery (signals from the operating pedal of an endoscope), etc., measured by a measuring device installed in an operating room (or on the patient) at a predetermined time. Sensing values ​​also include, for example, temperature, humidity, sound volume, etc., measured by a measuring device installed in a predetermined monitoring area at a predetermined time.

[0022] <1. Composition> The configuration of the information processing system will now be described. Figure 1 is a block diagram showing an example of the overall configuration of System 1. System 1 shown in Figure 1 includes, for example, a terminal device 10, a server 20, a camera 30, a sensor 31, a first database 32, a second database 33, and an LLM system 34. The terminal device 10, the server 20, the camera 30, the sensor 31, the first database 32, the second database 33, and the LLM system 34 communicate with each other via, for example, a network 80.

[0023] Figure 1 shows an example where System 1 includes one terminal device 10, but the number of terminal devices 10 included in System 1 is not limited to one. System 1 may include two or more terminal devices 10.

[0024] Figure 1 shows an example where System 1 includes one LLM system 34, but the number of LLM systems 34 included in System 1 is not limited to one. System 1 may include two or more LLM systems 34.

[0025] Figure 1 shows an example where server 20 is independent of the LLM system 34, but server 20 may also include the functions of the LLM system 34. In other words, server 20 may store LLM.

[0026] In this embodiment, a collection of multiple devices may be treated as a single server. The method of allocating the multiple functions required to implement the server 20 according to this embodiment to one or more hardware can be appropriately determined in view of the processing capacity of each hardware and / or the specifications required for the server 20.

[0027] The terminal device 10 shown in Figure 1 is an information processing device operated by the user who is viewing the image. The terminal device 10 can be implemented as, for example, a stationary PC (Personal Computer), a laptop PC, etc. The terminal device 10 may also be implemented as a mobile device such as a smartphone or tablet.

[0028] The terminal device 10 comprises a communication interface 12, an input device 13, an output device 14, memory 15, storage 16, and a processor 19. The input device 13 is a device for receiving input operations from the user (e.g., a touch panel, touchpad, mouse or other pointing device, keyboard, etc.). The output device 14 is a device for presenting information to the user (display, speaker, etc.).

[0029] Server 20 is an information processing device that, for example, uses RAG to provide a service that displays descriptive text about an image. Specifically, when Server 20 receives a request from a user for a description of an image, it searches the second database 33 for the sensing values ​​associated with the image for which a description has been requested. Server 20 inputs the image, instructions for creating a description of the image, and the retrieved sensing values ​​as prompts to the LLM system 34. Server 20 causes the LLM system 34 to output the description of the image. Server 20 then presents the description of the image output by the LLM system 34 to the user. In this way, Server 20 uses RAG by searching for sensing values ​​from the second database 33 and inputting the retrieved sensing values ​​as prompts to the LLM system 34.

[0030] The server 20 is an information processing device implemented by, for example, a computer connected to the network 80. As shown in Figure 1, the server 20 includes a communication IF 22, an input / output IF 23, memory 25, storage 26, and a processor 29. The input / output IF 23 functions as an interface to an input device for receiving input operations from the user and an output device for outputting information to the user.

[0031] Camera 30 is a device that receives light using a photodetector and outputs it as an imaging signal. Camera 30 is installed in a predetermined space, for example, and captures images in a direction corresponding to the operator's input within that space. Alternatively, Camera 30 may be installed in a predetermined space, for example, and capture images in a pre-set direction within that space.

[0032] Sensor 31 is, for example, a device installed in a predetermined space that measures a preset value. Alternatively, sensor 31 is a device installed on an object placed in a predetermined space that measures a preset value. The preset values ​​may include, for example, temperature, humidity, volume, heart rate, blood pressure, body temperature, or operating parameters of a device used in surgery.

[0033] The first database 32 is an information processing device that stores information about images. The first database 32 includes, for example, a communication interface, an input / output interface, memory, storage, and a processor. The first database 32 stores, for example, an image information table 2022. The image information table 2022 will be described later. The images managed in the image information table 2022 may be, for example, DICOM (Digital Imaging and Communications in Medicine).

[0034] In the example shown in Figure 1, the first database 32 is built outside of the server 20. "External" indicates, for example, that it is managed by an administrator separate from the server 20. Note that the first database 32 may also be built within the server 20.

[0035] The second database 33 is an information processing device that stores information related to sensing values. The second database 33 includes, for example, a communication interface, an input / output interface, memory, storage, and a processor. The second database 33 stores, for example, a sensing value table 2023. The sensing value table 2023 will be described later.

[0036] In the example shown in Figure 1, the second database 33 is built outside the server 20. However, the second database 33 may also be built within the server 20. Figure 1 also shows the case where there is only one second database 33. The second database 33 may be provided for each different sensing value. If there is a different second database 33 for each sensing value, each second database 33 may be managed by a different administrator.

[0037] The LLM system 34 is a system in which a large-scale artificial intelligence model (LLM), used, for example, in the field of natural language processing (NLP), has been built. By learning from large amounts of text data (web pages, books, articles, etc.), the LLM can understand patterns in human language and effectively perform natural language generation (NLG) tasks.

[0038] LLM is used in many NLP tasks, such as generating responses to specific questions, automatically generating text, summarizing text, translation, and sentiment analysis. LLM can also be used in a variety of applications, including education, entertainment, customer service, and product development. One example of LLM is multimodal LLM. Examples of LLM include the following: • GPT-4 (registered trademark) (OpenAI Inc.) • Gemini (registered trademark) (Google Inc.) • StableLM (StableAI Inc.) • Llama 3.2 (Meta Inc.)

[0039] The LLM system 34 receives instructions from the server 20 to create an image, sensing values, and a description of the image, and instructs the LLM to output a response based on the input image, sensing values, and instructions for creating the description of the image. The LLM system 34 then sends the response output from the LLM to the server 20.

[0040] Each information processing device consists of a computer equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by said hardware configuration will be described later. For each of the terminal device 10, server 20, and LLM system 34, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0041] <1.1 Terminal Device Configuration> Figure 2 shows an example of the functional configuration of the terminal device 10. As shown in Figure 2, the terminal device 10 comprises a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a storage unit 180, and a control unit 190. Each block included in the terminal device 10 is electrically connected, for example, by a bus.

[0042] The communication unit 120 performs processing such as modulation and demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190 and transmits it to an external source (for example, the server 20). The communication unit 120 performs reception processing on the signal received from an external source and outputs it to the control unit 190.

[0043] The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 includes a keyboard 131 and a mouse 132. The input device 13 converts the instructions input by the user into electrical signals and outputs the electrical signals to the control unit 190. The input device 13 may also include a receiving port that accepts electrical signals input from, for example, an external input device.

[0044] The output device 14 is a device for presenting information to the user operating the terminal device 10. The output device 14 is implemented, for example, by a display 141. The display 141 displays data according to the control of the control unit 190. The display 141 is implemented, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0045] The audio processing unit 17 performs, for example, digital-to-analog conversion processing of the audio signal. The audio processing unit 17 converts the signal received from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is implemented, for example, by an audio processing processor. The microphone 171 receives an audio input and provides the audio signal corresponding to that audio input to the audio processing unit 17. The speaker 172 converts the audio signal received from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0046] Camera 160 is a device that receives light using a photodetector and outputs it as a shooting signal.

[0047] The location information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. A GPS module is a receiving device used in a satellite positioning system. In a satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10, which is equipped with a GPS module, is detected based on the received signals. The location information sensor 150 may also detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0048] The storage unit 180 is implemented by, for example, memory 15 and storage 16, and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0049] User information 181 includes, for example, information about the user who uses the terminal device 10. User information includes, for example, the user's name, age, address, date of birth, contact information, etc.

[0050] The control unit 190 is realized when the processor 19 reads a program stored in the memory unit 180 and executes instructions contained in the program. The control unit 190 controls the operation of the terminal device 10. By operating according to the program, the control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193.

[0051] The operation reception unit 191 processes instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from the keyboard 131, mouse 132, etc.

[0052] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives voice signals input from the microphone 171 and converted into digital signals by the voice processing unit 17. The operation reception unit 191 obtains instructions from the user by, for example, analyzing the received voice signals and extracting predetermined nouns.

[0053] The transmitting / receiving unit 192 performs processing to enable the terminal device 10 to send and receive data with an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 sends information input by the user or instructions from the user to the server 20. The transmitting / receiving unit 192 also receives information provided by the server 20.

[0054] The presentation control unit 193 controls the output device 14 to present information provided by the server 20 to the user. Specifically, for example, the presentation control unit 193 displays information related to the summary transmitted from the server 20 on the display 141. The presentation control unit 193 also outputs information transmitted from the server 20 through the speaker 172.

[0055] <1.2 Functional Configuration of the Server> Figure 3 shows an example of the functional configuration of server 20. As shown in Figure 3, server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

[0056] The communications unit 201 performs processing to enable the server 20 to communicate with external devices.

[0057] The storage unit 202 includes, for example, a user information table 2021. However, the tables stored in the storage unit 202 are not limited to the user information table 2021.

[0058] User Information Table 2021 is a table that stores information about users. Further details will be provided later.

[0059] The control unit 203 is realized when the processor 29 reads a program stored in the memory unit 202 and executes instructions contained in the program. The program includes applications such as web browser applications. The program includes programming languages ​​such as JavaScript® that are executed on the web browser application stored in the terminal device 10. By operating according to the program, the control unit 203 performs the functions indicated as the reception control module 2031, the transmission control module 2032, the presentation control module 2033, the search control module 2034, the image processing module 2035, and the LLM processing control module 2036.

[0060] The receive control module 2031 controls the process by which the server 20 receives signals from external devices according to a communication protocol.

[0061] The transmission control module 2032 controls the process by which the server 20 transmits signals to external devices according to a communication protocol.

[0062] The presentation control module 2033 controls the process of presenting information to the user. Specifically, for example, the presentation control module 2033 presents an image to the user. The presentation control module 2033 also presents the explanatory text sent from the LLM system 34 to the user. More specifically, for example, the presentation control module 2033 presents the explanatory text about the image, which was output as a response from the LLM system 34, to the user.

[0063] The search control module 2034 receives a request for an image description and controls the process of retrieving sensing values ​​from the second database 33 based on the received request. Specifically, for example, the search control module 2034 receives an instruction from the user to create an image description as a request for an image description. In response to the receipt of the description request, the search control module 2034 identifies additional information attached to the image. Additional information includes, for example, metadata of the data to be attached. For example, additional information attached to an image includes information about the date and time the image was taken, the location where the image was taken, the equipment used to take the image, the operator who operated the equipment, etc.

[0064] The search control module 2034 searches the second database 33 for sensing values ​​with additional information that partially or completely matches the identified additional information, based on the identified additional information and the timing of the user's request for explanation. For example, the additional information attached to a sensing value includes information about the date and time the sensing value was measured, the location where the sensing value was measured, the sensor that measured the sensing value, and the equipment to which the sensor is attached. As a result, the search control module 2034 searches the second database 33 for sensing values ​​that were measured when the image for which the user's explanation is requested was taken, and that were measured at the location where the image was taken.

[0065] The image processing module 2035 controls the process of deleting supplementary information displayed in an image. Supplementary information is, for example, information attached to an image, and includes information about the object represented in the image (name, ID, etc.). Supplementary information also includes information about the state of the object represented in the image (parameters, etc.). Supplementary information is not an image captured by the camera 30. Specifically, for example, the image processing module 2035 takes an image with supplementary information attached as input data and inputs the image into a trained model that has been trained using specifications for supplementary information as ground truth data. The image processing module 2035 removes the supplementary information from the image by having the trained model output an image from which the supplementary information has been removed. If the position where supplementary information is attached in the image is predetermined, the image processing module 2035 may delete the supplementary information from the image based on the predetermined position.

[0066] The LLM processing control module 2036 inputs an image, sensing values, and instructions for creating a description of the image to the LLM, and outputs a description of the image. The image can be a still image or a video. Specifically, the LLM processing control module 2036 inputs the image for which an explanation is requested, the sensing values ​​obtained through the search, and instructions for creating a description of the image to the LLM. In other words, the LLM processing control module 2036 inputs the instructions for creating a description of the image and the sensing values ​​obtained through the search as prompts to the LLM, and also inputs the image for which an explanation is requested to the LLM. As a result, the LLM processing control module 2036 causes the LLM to output a description of the image that takes the sensing values ​​into account.

[0067] The LLM processing control module 2036 may input instructions to the LLM, for example, instructions input by the user to request an image description. Alternatively, the LLM processing control module 2036 may create instructions to input to the LLM based on, for example, instructions to request an image description. The LLM processing control module 2036 may, for example, create instructions to output an image description that takes sensing values ​​into account, based on instructions to request an image description.

[0068] <2 Data Structure> Figures 4 to 6 show the data structure of the tables stored by server 20. Note that Figures 4 to 6 are examples and do not exclude data that is not shown. Also, even if data is listed in the same table, it may be stored in separate memory areas in storage unit 202.

[0069] Figure 4 shows the data structure of the user information table 2021. The user information table 2021 shown in Figure 4 is a table that uses User ID as the key and has columns for Name, Age, Gender, Date of Birth, and Contact Information.

[0070] User ID is an item that stores an identifier to uniquely identify the user. Name is an item that stores the user's name. Age is an item that stores the user's age. Gender is an item that stores the user's gender. Date of Birth is an item that stores the user's date of birth. Contact information is an item that stores the contact information (e.g., phone number, email address, etc.) of the terminal device 10 that the user possesses.

[0071] Figure 5 shows the data structure of the image information table 2022. The image information table 2022 shown in Figure 5 is a table that uses the image ID as the key and contains the date and time, image data, and an ID related to the acquisition location.

[0072] The Image ID is an item that stores an identifier to uniquely identify the image. The Date and Time is an item that stores the date and time the image was created. The Image Data is an item that stores the image data. The Location ID is an item that stores an identifier to uniquely identify the location where the image was taken. The Date and Time and the Location ID are additional information associated with the image.

[0073] Figure 6 shows the data structure of the sensing value table 2023. The sensing value table 2023 shown in Figure 6 is a table that uses the sensing value ID as the key and contains the year, month, and time, an ID related to the acquisition location, and the sensing value.

[0074] The Sensing Value ID is an item that stores an identifier to uniquely identify the sensing value. The Date and Time is an item that stores the date and time the sensing value was generated. The Location ID is an item that stores an identifier to uniquely identify the location where the sensing value was obtained. The Sensing Value is an item that stores the sensing value. The Date and Time and the Location ID are additional information associated with the sensing value.

[0075] <3 operations> This section describes the operation of server 20 when using RAG to display descriptive text about images.

[0076] Figure 7 is a flowchart illustrating an example of the operation of server 20 when presenting descriptive text about an image. In the explanation of Figure 7, server 20 provides the service of presenting descriptive text about an image, for example, as SaaS (Software as a Service). In this case, the target image is, for example, an image related to surgery (an image taken with an endoscope). The image is taken, for example, by an endoscopic device installed in an operating room and stored in the image information table 2022. The user is, for example, a medical professional such as a doctor or nurse.

[0077] First, a user who wants to obtain a description of an image accesses server 20. The user logs in to the service provided by server 20 that displays the description of the image, for example, by entering their user ID and password.

[0078] In step S11, the server 20 presents an image to the user. Specifically, for example, the user operates the terminal device 10 and selects an image (a video related to surgery) that they wish to view. The operation reception unit 191 accepts the image selection input by the user. The transmission / reception unit 192 sends the image ID of the video selected by the user to the server 20. The reception control module 2031 receives the image ID sent from the terminal device 10. In other words, the server 20 accepts the video selection by the user. The presentation control module 2033 reads the video data stored in the image information table 2022 of the first database 32 based on the received image ID. The presentation control module 2033 presents the video data to the user. The presentation control unit 193 controls the display 141 to display the video based on the video data to the user.

[0079] The user views the video displayed on the display 141. The user operates the terminal device 10 and inputs instructions to create a descriptive text for a predetermined frame of the video displayed on the terminal device 10. The predetermined frame of the video is, for example, an image with a predetermined timestamp. An example of instructions to create a descriptive text for an image is, "This is an endoscopic image of laparoscopic surgery. Please describe the surgical situation." The operation reception unit 191 receives the instructions to create a descriptive text for an image as a request for an explanation of the image. The transmission / reception unit 192 sends the instructions to create a descriptive text for an image, the image ID of the video relating to the image for which an explanation is requested, and the timestamp of the video at the time the instructions were received to the server 20.

[0080] In step S12, the server 20 receives a request for an explanation of the image. Specifically, the receiving control module 2031 receives an instruction from the terminal device 10 to create an explanation of the image, the image ID of the video relating to the image for which an explanation is requested, and the timestamp of the video at the time the instruction was received. In other words, the server 20 receives a request from the user for an explanation of the image.

[0081] In step S13, the server 20 searches for sensing values ​​(various sensing values ​​measured during surgery). Specifically, for example, the search control module 2034 performs a vector search based on the instruction and targets sensing values ​​for the search. The search control module 2034 identifies additional information (date and time and ID related to the acquisition location) associated with the received image ID in the image information table 2022 of the first database 32. The search control module 2034 searches the sensing value table 2023 of the second database 33 for sensing values ​​to which the identified additional information (date and time and ID related to the acquisition location) and additional information including the video timestamp are attached. In other words, in response to the receipt of a request for explanation, the search control module 2034 searches the second database 33 for sensing values ​​measured at the timing of the video timestamp and measured in the operating room where the surgery is performed. Furthermore, in response to a request for explanation, the search control module 2034 retrieves sensing values ​​measured at the timing of the video's timestamp and for patients undergoing surgery from the second database 33.

[0082] In step S14, the server 20 deletes the ancillary information. Specifically, the image processing module 2035 reads the video identified by the received image ID from the image information table 2022 of the first database 32. The image processing module 2035 inputs the read video into, for example, a trained model in the memory unit 202. Ancillary information displayed in the video, such as the patient's name, patient ID, and gender, is deleted by the trained model. Ancillary information displayed in the video, such as the operating parameters of the surgical instrument arms, is also deleted by the trained model. The image processing module 2035 retrieves the video from which the ancillary information has been deleted, which is output from the trained model.

[0083] In step S15, the server 20 obtains a description of the video. Specifically, the LLM processing control module 2036 inputs the instruction received in step S12 to create a description of the image, the sensing values ​​retrieved in step S13, and the video from which the accompanying information has been removed in step S14 into the LLM. In other words, the LLM processing control module 2036 inputs the instruction received in step S12 to create a description of the image, the sensing values ​​retrieved in step S13, and the video from which the accompanying information has been removed in step S14 into the LLM as prompts.

[0084] In this case, the LLM processing control module 2036 may create an instruction to be input to the LLM, for example, an instruction to request an image description, and an instruction to output an explanation that takes sensing values ​​into account.

[0085] The LLM processing control module 2036 retrieves the explanatory text output from the LLM. An example of the explanatory text is: "This image is an endoscopic image of laparoscopic surgery. A needle holder to hold the needle and Versatile Grasping Forceps to grasp the tissue are being used for suturing. At this time, the patient's temperature is 37.0°C, heart rate is 80 beats / min, systolic blood pressure is 115 mmHg, and diastolic blood pressure is 75 mmHg. The surgery is progressing smoothly."

[0086] In step S16, the server 20 presents the descriptive text to the user. Specifically, the presentation control module 2033 reads the descriptive text output from the LLM in step S15. The presentation control module 2033 presents the descriptive text about the image to the user. The presentation control unit 193 controls the display 141 to display the descriptive text about the image to the user.

[0087] <4 Screen Examples> Figures 8 and 9 are schematic diagrams illustrating examples of the display screen of the display 141 of the terminal device 10. Figure 8 illustrates the case where the image is, for example, an image related to surgery (an image taken with an endoscope).

[0088] The screen shown in Figure 8 includes a first region 1411 and a second region 1412. The first region 1411 is, for example, a region for displaying images related to surgery. The images displayed in the first region 1411 are, for example, images obtained from image data read from the image information table 2022 by the presentation control module 2033.

[0089] The image displayed in the first region 1411 includes supplementary information A1, A2, A3, and A4. Supplementary information A1 is information related to the operation of the endoscope's operating pedal. Supplementary information A2 is information related to the surgery. Supplementary information A3 is information indicating the date and time the image was taken. Supplementary information A4 is information indicating the position of the endoscope. Supplementary information A1, A2, A3, and A4 are information within the image, but they are not information generated by the camera 30 taking the image.

[0090] The second area 1412 is an area that displays instructions, for example, for creating a descriptive text about an image displayed in the first area 1411, which has been entered by the user. In the screen shown in Figure 8, this area includes a button 1413. Button 1413 is a button for entering a request for a description of the image. In other words, button 1413 is a button for sending the prompt displayed in the second area 1412 to the server 20.

[0091] In step S11, when the server 20 accepts the selection of an image, the presentation control module 2033 reads the selected image from the image information table 2022. The presentation control module 2033 then presents the read image to the user. The presentation control unit 193 displays the image in the first area 1411.

[0092] The user checks the image displayed in the first area 1411 and enters instructions to create a description of the image. The instructions to create the description of the image are displayed in the second area 1412. The user presses button 1413. When button 1413 is pressed, the operation reception unit 191 receives a request for a description of the image. The transmission / reception unit 192 sends the request for a description of the image to the server 20.

[0093] Figure 9 is a schematic diagram showing another example of the display screen of the display 141 of the terminal device 10. In Figure 9, the example is that the image is, for example, an image related to surgery (an image taken with an endoscope).

[0094] The screen shown in Figure 9 includes the third region 1414, the fourth region 1415, and the fifth region 1416. The third region 1414 is, for example, a region that displays images related to surgery. The fourth region 1415 is, for example, a region that displays instructions entered by the user to create a descriptive text about the image displayed in the third region 1414. The fifth region 1416 displays, for example, a descriptive text about the image displayed in the third region 1414, which is output by the LLM system 34.

[0095] In step S16, the presentation control module 2033 reads the explanatory text output from the LLM in step S15. The presentation control unit 193 displays the explanatory text about the image in the fifth region 1416.

[0096] Improving the accuracy of the output information from the LLM can be difficult by modifying the LLM itself. Therefore, a method (Retrieval-Augmented Generation: RAG) is being used to improve the accuracy of the output information from the LLM by inputting input information associated with external information into the LLM, without modifying the LLM itself.

[0097] As described above, in this embodiment, the search control module 2034 receives a request for an image description and controls the process of searching for sensing values ​​from the second database 33. The LLM processing control module 2036 inputs the image and sensing values ​​into the LLM and outputs a description of the image. Therefore, according to this embodiment, the information desired by the user can be provided with higher accuracy while minimizing effort.

[0098] In the above embodiment, the image and sensing value are associated with each other using additional information already associated with the image and sensing value (IDs regarding time and acquisition location), and then input into the LLM. Therefore, it is not necessary to pre-associate information for linking the image and sensing value (primary key) with the image and sensing value, thus streamlining information processing.

[0099] Furthermore, in the above embodiment, the image from which the supplementary information has been removed is input to the LLM. Therefore, according to this embodiment, the information desired by the user can be provided with higher accuracy.

[0100] <Variation> In the above embodiment, the example described was one in which the image is an image related to surgery and the sensing values ​​are various sensing values ​​measured during surgery. However, the image may be an image captured by a surveillance camera, and the sensing values ​​may be various sensing values ​​measured in the space being monitored.

[0101] Furthermore, the above embodiment described an example in which one database (second database 33) stores the sensing values. However, the sensing values ​​may be recorded in multiple databases. For example, the sensing values ​​may be recorded in different databases for each type of sensing value.

[0102] Furthermore, the administrator of at least one of the sensing values ​​may be different from the administrator of the other sensing values.

[0103] Furthermore, in the above embodiment, the case in which instructions for creating a descriptive text about an image are input via the input device 13 was described as an example. However, the instructions for creating a descriptive text about an image may also be input via the voice processing unit 17.

[0104] Furthermore, the above embodiment described an example where the server presents a descriptive text about an image. However, the server may also present a descriptive text about video. The system stores, for example, associated video and sensing values ​​in a database. When the server receives a request from a user for a description of a video, for example, it retrieves information about the video from the database. The server retrieves, for example, an image of the video for which a description is requested at the moment the request is entered. The server uses RAG, as described above, to search the database for sensing values ​​associated with the retrieved image for the video for which a description is requested. The server inputs, for example, the image and the sensing values ​​obtained through the search into the LLM, and has the LLM output a descriptive text about the video at the time the user requested the description. The system presents the descriptive text output from the LLM to the user.

[0105] Furthermore, in the above embodiment, the example described was when the search control module 2034 performs a vector search based on an instruction and searches for the sensing value using additional information (IDs for time and acquisition location) already associated with the image and sensing value. However, the search control module 2034 may also search for the sensing value using a multimodal LLM. The multimodal LLM may be stored in the server 20 or an external service may be used. For example, when the search control module 2034 receives a request from the user for a description of an image, it inputs the image and the instruction into the multimodal LLM. The multimodal LLM outputs a response about the image. Based on the response output from the multimodal LLM, the search control module 2034 performs a vector search, keyword search, hybrid search, and / or semantic search to search for the sensing value from the database. The search control module 2034 inputs the image, the searched sensing value, and an instruction to create a description of the image as prompts into the LLM, and causes the LLM to output a description of the image. The system displays the explanatory text output from the LLM to the user.

[0106] <Basic Computer Hardware Configuration> Figure 10 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 comprises at least a processor 901, main memory 902, auxiliary storage 903, and a communication interface IF991. These are electrically connected to each other by a communication bus 921.

[0107] The processor 901 is hardware for executing the instruction set written in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0108] Main memory 902 is used to temporarily store programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0109] Auxiliary storage device 903 refers to a storage device for saving data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.

[0110] The IF991 communication interface is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. A network consists of various mobile communication systems, such as the internet, LANs, and wireless base stations. For example, a network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via a wired connection, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0111] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0112] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 (Figure 10) is described below. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0113] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0114] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0115] The memory unit is implemented by the main memory 902 and the auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform operations such as adding, updating, and deleting data stored in the memory unit according to the various programs.

[0116] A database, specifically a relational database, is used to manage and link together tabular data sets called masters, which are structurally defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and master has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit can instruct the processor 901 to add, delete, or update records in specific tables and masters stored in the memory unit, according to various programs. Furthermore, by storing data, various programs, and various databases in the memory unit, the information processing device and information processing system related to this disclosure can be considered to have been manufactured.

[0117] Furthermore, the databases and masters in this disclosure may include any data structures (lists, dictionaries, associative arrays, objects, etc.) in which information is structurally defined. Data structures also include data that can be considered as data structures by combining data with functions, classes, methods, etc., written in any programming language.

[0118] The communication unit is implemented by the communication IF991. The communication unit provides the functionality to communicate with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0119] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, either partially or entirely, by designing them as integrated circuits, for example. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0120] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java (registered trademark).

[0121] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0122] The functions realized by the components described herein may be implemented in a circuit or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor is considered to be a circuit or processing circuitry, including transistors and other circuits. A processor may be a programmed processor that executes a program stored in memory. In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein. If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0123] While several embodiments of this disclosure have been described above, these embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications are permitted without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0124] <Note> The details described in each of the above embodiments are noted below. (Note 1) A program for execution on a computer having a processor and memory, the program causing the processor to perform the following steps: acquire an image of a predetermined area and present the image to the user; accept a request for an explanation of the image; retrieve sensing values ​​measured for the predetermined area from a database stored in the database in response to the request for an explanation, the sensing values ​​measured when the image was taken and relating to the requested explanation; input the image for which an explanation is requested and the sensing values ​​obtained through the search into an LLM (Large Language Model) and output an explanatory text; and present the output explanatory text to the user. (Note 2) The step of outputting the explanatory text involves removing any attached information from the image for which an explanation is requested, inputting the image from which the attached information has been removed and the sensing values ​​obtained through the search into the LLM, and outputting the explanatory text, as described in Appendix 1 of the program. (Note 3) The step of outputting the explanatory text involves the program described in Appendix 2, which takes an image with attached supplementary information as input data, and inputs the image into a trained model that has been trained using the specifications for the supplementary information as ground truth data, thereby outputting an image with the supplementary information removed, inputting the image with the supplementary information removed and the sensing values ​​obtained through the search into the LLM, and outputting the explanatory text. (Note 4) The step of outputting the explanatory text involves a program described in Appendix 2, in which the location where supplementary information is attached to an image is predetermined, the supplementary information is removed from the image based on the predetermined location, the image from which the supplementary information has been removed and the sensing values ​​obtained by the search are input to the LLM, and the explanatory text is output. (Note 5) The image is a surgical image, and the sensing values ​​are various sensing values ​​measured during surgery, as described in one of the programs in Appendix 1 to Appendix 4. (Note 6) The program is one of the following, where the image is captured by a surveillance camera and the sensing values ​​are various sensing values ​​measured in the monitored space. (Note 7) The database also stores the captured images, as described in one of the programs in Appendix 1 to Appendix 6. (Note 8) The databases for storing images and sensing values ​​are each different programs described in Appendix 7. (Note 9) The program described in one of the appendices 1 through 8, where at least one of the sensing values ​​is managed by a different program than the other sensing values. (Note 10) An information processing device comprising a control unit and a storage unit, wherein the control unit performs the steps of: acquiring an image of a predetermined area and presenting the image to a user; receiving a request for an explanation about the image; storing sensing values ​​measured for the predetermined area in a database, and in response to receiving a request for an explanation, searching the database for sensing values ​​measured when the image was taken and related to the requested explanation; inputting the image for which an explanation is requested and the sensing values ​​obtained by the search into an LLM and outputting an explanatory text; and presenting the output explanatory text to the user. (Note 11) A method to be performed on a computer having a processor and memory, wherein the processor performs the steps of: acquiring an image of a predetermined area and presenting the image to a user; receiving a request for an explanation of the image; having sensing values ​​measured for the predetermined area stored in a database, and in response to receiving a request for an explanation, searching the database for sensing values ​​measured when the image was taken and relating to the requested explanation; inputting the image for which an explanation is requested and the sensing values ​​obtained by the search into an LLM and outputting an explanatory text; and presenting the output explanatory text to the user. (Note 12) A system comprising: means for acquiring an image of a predetermined area and presenting the image to the user; means for receiving a request for an explanation about the image; means for storing sensing values ​​measured for the predetermined area in a database and, in response to receiving a request for an explanation, searching the database for sensing values ​​measured when the image was taken and related to the requested explanation; means for inputting the image for which an explanation is requested and the sensing values ​​obtained through the search into an LLM and outputting an explanatory text; and means for presenting the output explanatory text to the user. [Explanation of symbols]

[0125] 1... System 10…Terminal device 120... Communications Department 13…Input device 14…Output device 15…Memory 16…Storage 19… Processor 20... Server 22...Communication IF 23…Input / Output Interface 25…Memory 2 hours… storage 29… Processor

Claims

1. A program to be executed by a computer having a processor and memory, wherein the program is to be executed by the processor, The steps include acquiring an image of a predetermined area and presenting the image to the user, A step of receiving a request for explanation about the aforementioned image, The sensing values ​​measured for the predetermined area are stored in a database, and in response to the receipt of the request for the explanation, the sensing values ​​measured when the image was taken and relating to the requested explanation are retrieved from the database. The steps include inputting the image for which the above explanation was requested and the sensing values ​​obtained by the above search into an LLM (Large Language Model) and outputting an explanatory text, The steps include presenting the outputted explanatory text to the user, A program that executes the command.

2. The program according to claim 1, wherein the step of outputting the explanatory text involves deleting any supplementary information attached to the image for which the explanation was requested, inputting the image from which the supplementary information has been deleted and the sensing value obtained by the search into the LLM, and outputting the explanatory text.

3. The program according to claim 2, wherein the step of outputting the explanatory text is to input the image with the attached supplementary information as input data to a trained model which has been trained with the specifications for the supplementary information as correct data, thereby outputting the image from which the supplementary information has been removed, inputting the image from which the supplementary information has been removed and the sensing value obtained by the search into the LLM, and outputting the explanatory text.

4. The program according to claim 2, wherein the step of outputting the explanatory text is characterized in that the position to which the supplementary information is attached to the image is predetermined, the supplementary information is deleted from the image based on the predetermined position, the image from which the supplementary information has been deleted and the sensing value obtained by the search are input to the LLM, and the explanatory text is output.

5. The aforementioned image is an image related to surgery. The program according to any one of claims 1 to 4, wherein the sensing value is a variety of sensing values ​​measured during surgery.

6. The aforementioned image is an image captured by a surveillance camera. The program according to any one of claims 1 to 4, wherein the sensing value is a variety of sensing values ​​measured in the space to be monitored.

7. The program according to any one of claims 1 to 4, wherein the database further stores the captured images.

8. The database for storing the image and the sensing value is a different program according to claim 7.

9. The program according to any one of claims 1 to 4, wherein the administrator of at least one of the sensing values ​​is different from the administrator of the other sensing values.

10. An information processing apparatus comprising a control unit and a storage unit, wherein the control unit is The steps include acquiring an image of a predetermined area and presenting the image to the user, A step of receiving a request for explanation about the aforementioned image, The sensing values ​​measured for the predetermined area are stored in a database, and in response to the receipt of the request for the explanation, the sensing values ​​measured when the image was taken and relating to the requested explanation are retrieved from the database. The steps include inputting the image for which the above explanation was requested and the sensing values ​​obtained by the above search into the LLM and outputting an explanatory text, The steps include presenting the outputted explanatory text to the user, An information processing device that performs the following actions.

11. A method to be performed on a computer comprising a processor and memory, wherein the processor The steps include acquiring an image of a predetermined area and presenting the image to the user, A step of receiving a request for explanation about the aforementioned image, The sensing values ​​measured for the predetermined area are stored in a database, and in response to the receipt of the request for the explanation, the sensing values ​​measured when the image was taken and relating to the requested explanation are retrieved from the database. The steps include inputting the image for which the above explanation was requested and the sensing values ​​obtained by the above search into the LLM and outputting an explanatory text, The steps include presenting the outputted explanatory text to the user, How to do it.

12. A means for acquiring an image of a predetermined area and presenting the image to the user, A means for receiving requests for explanations regarding the aforementioned image, The sensing values ​​measured for the predetermined area are stored in a database, and in response to the receipt of a request for explanation, the means retrieves the sensing values ​​measured when the image was taken and relating to the requested explanation from the database. A means for inputting the image for which the above explanation was requested and the sensing values ​​obtained by the above search into the LLM and outputting an explanatory text, A means for presenting the outputted explanatory text to the user, A system that is equipped with [the following].

Images