Endoscope, training data candidate image acquisition device, training data candidate image acquisition method, and program

By identifying similar image groups and calculating the differences in inference results within the endoscopic system, suitable medical images are selected as training data. This solves the problem that machine learning is not good at recognizing images in existing technologies, and improves data collection efficiency and the reliability of inference models.

CN118984670APending Publication Date: 2024-11-19OLYMPUS CORPORATION(JP)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280094286.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-30
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing technologies struggle to efficiently select medical images that machine learning is not good at recognizing as training data, resulting in low data collection efficiency.

Method used

Through the similar image group determination unit, reasoning unit, reasoning result difference calculation unit, and learning image selection unit in the endoscope system, the machine learning model is used to compare and reason about the content of continuous video images and select images to be used in the learning process.

Benefits of technology

It enables the reliable selection of medical images that machine learning is not good at recognizing, improves the efficiency and quality of training data collection, and enhances the reliability of the inference model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984670A_ABST
    Figure CN118984670A_ABST
Patent Text Reader

Abstract

Provided are an endoscope, a training data candidate image acquisition device, a training data candidate image acquisition method, and a program, which are capable of reliably selecting a medical image that is not good at recognition by machine learning prepared in advance. When acquiring a training data candidate image for acquiring an image based on an endoscope for learning an inference model for an endoscope, the contents of endoscope images captured continuously over time are compared, and a similar image group in which the same part is observed even if a non-similar image is included within a predetermined number of frames is determined (S3). In the present invention, machine learning is used to infer a specific object image included in an captured endoscopic image (S1), for each image determined to be included in a group of similar images, a difference in inference results (S5) is calculated, and an image to be used in learning is selected on the basis of the calculated difference in inference results (S7).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an endoscope used when generating an inference model for the medical field through learning, a training data candidate image acquisition device for acquiring training data candidate images, a training data candidate image acquisition method, and a program. Background Art

[0002] In the medical field, various systems have been developed to capture medical images of various anatomical structures of each patient in order to classify and evaluate disease conditions in image diagnosis. As these capturing systems, for example, endoscope systems, CT (computed tomography) systems, MRI (magnetic resonance imaging) systems, X-ray systems, ultrasound systems, PET (positron emission tomography) systems, etc. are known. In addition, the following method is known: machine learning is performed on annotation data of medical personnel such as doctors, thereby realizing a lesion detection function based on the so-called computer-aided detection / diagnosis, namely CADe / x (Computer-Aided Detection / Diagnosis).

[0003] However, as described above, in order to improve the performance of a machine learning-based recognizer, a large amount of data is generally required. Therefore, in a system where the recognizer needs to perform machine learning, it is predicted that the amount of data to be processed will increase. However, a large amount of data requires a huge recording capacity, and the network line is occupied during the transmission of the data, and each data confirmation takes a lot of labor and time. Therefore, it is predicted that "efficient data collection" will be needed in the future. As this "efficient data collection", for example, consider collecting only "medical images that machine learning is not good at recognizing" as useful data. Here, as a technology for selecting useful medical images from a large number of medical images, for example, the following technology is disclosed in Patent Document 1: from multiple medical images, only medical images that capture the desired part are efficiently transmitted.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent No. 5048286 Summary of the invention

[0007] Problems to be solved by the invention

[0008] The technology described in the above-mentioned patent document 1 is nothing more than a technology for efficiently transmitting medical images that have been taken of a desired part. Therefore, it is considered difficult to select "medical images that machine learning is not good at recognizing" using the technology disclosed in patent document 1. That is, in the existing technology, it is difficult to collect only "medical images that machine learning is not good at recognizing" as useful data.

[0009] The present invention was completed in view of such a situation, and its object is to provide an endoscope, a training data candidate image acquisition device, a training data candidate image acquisition method and a program that can reliably select pre-prepared medical images that machine learning is not good at recognizing.

[0010] Means for solving problems

[0011] In order to achieve the above-mentioned purpose, the endoscope of the first invention comprises: a similar image group determination unit, which compares the contents of endoscopic images obtained by continuous video recording in time, and determines a similar image group that observes the same part within a specified number of frames in a manner that includes non-similar images; an inference unit, which uses machine learning to infer specific object images contained in the endoscopic images obtained by video recording; an inference result difference calculation unit, which calculates the difference in inference results of the inference unit for each image contained in the similar image group determined by the similar image group determination unit; and a learning image selection unit, which selects an image to be used in learning based on the difference in the inference results calculated by the inference result difference calculation unit.

[0012] An endoscope according to a second invention is the first invention, wherein the similar image group includes a result of determining an image group corresponding to a sudden viewpoint position shift according to similarity between images before and after the endoscopic image captured continuously in time.

[0013] An endoscope according to a third aspect of the present invention is the endoscope according to the first aspect of the present invention, and includes an imaging information acquisition unit capable of acquiring imaging information corresponding to each image selected by the learning image selection unit.

[0014] An endoscope according to a fourth invention is the endoscope according to the first invention, wherein the similar image group determination unit converts the image content into numerical values ​​that can track a pattern of the same object in the endoscopic image, and determines whether the images are similar using the numerical values.

[0015] An endoscope according to a fifth aspect of the present invention is characterized in that in the first aspect of the present invention, the learning image selection unit removes images with extremely poor conditions.

[0016] The training data candidate image acquisition device of the sixth invention is used to acquire images for learning an inference model, wherein the training data candidate image acquisition device comprises: a similar image group determination unit, which compares the contents of images obtained by continuous video recording in time, and determines a similar image group that observes the same part within a specified number of frames in a manner that includes non-similar images; an inference unit, which uses machine learning to infer specific object images contained in the video recorded images; an inference result difference calculation unit, which calculates the difference in inference results of the inference unit for each image contained in the similar image group determined by the similar image group determination unit; and a learning image selection unit, which selects images to be used in learning based on the difference in the inference results calculated by the inference result difference calculation unit.

[0017] The training data candidate image acquisition method of the seventh invention is used to obtain images for learning an inference model, wherein the contents of images obtained by continuous video recording in time are compared, a group of similar images that observe the same part within a specified number of frames is determined even if non-similar images are included, a machine learning model is used to infer images of specific objects contained in the images obtained by video recording, and the difference in inference results inferred using the machine learning model is calculated for each image determined to be included in the similar image group, and an image to be used in learning is selected based on the calculated difference in inference results.

[0018] The program of the eighth invention enables a computer that obtains candidate images of training data for learning an inference model to perform the following processing: compare the contents of images obtained by continuous video recording in time, determine a group of similar images that observe the same part within a specified number of frames even if they include non-similar images, use a machine learning model to infer images of specific objects contained in the images obtained by video recording, calculate the difference in inference results inferred using the machine learning model for each image determined to be included in the group of similar images, and select an image to be used in learning based on the calculated difference in inference results.

[0019] The training data candidate image acquisition device of the 9th invention comprises: an inference unit, which uses a machine learning model to infer an input image; an inference result change calculation unit, which calculates the change in the inference result of the identified image pair; and a learning image selection unit, which selects a training data candidate image to be used in learning based on the change calculated by the inference result change calculation unit.

[0020] A training data candidate image acquisition device according to a tenth invention is the training data candidate image acquisition device in the ninth invention, wherein the training data candidate image acquisition device includes an image recognition unit configured to determine that there is substantially no change in an image due to temporally continuous input.

[0021] The training data candidate image acquisition device of the 11th invention is the 10th invention, wherein the image recognition unit determines that the input has substantially no change on the image based on at least one of the movement amount of the corresponding point and the change amount of the brightness, saturation, and contrast of the image.

[0022] A training data candidate image acquisition device according to a twelfth invention is the device according to the ninth invention, wherein the input image is a medical image.

[0023] A training data candidate image acquisition device according to a 13th invention is the device according to the 9th invention, wherein the inference unit performs inference on at least one of classification, detection, and region extraction of the input image.

[0024] A training data candidate image acquisition device according to a fourteenth invention is characterized in that, in the ninth invention, the inference unit outputs the reliability of the inference.

[0025] A training data candidate image acquisition device according to a fifteenth invention is the device according to the ninth invention, wherein the inference result change calculation unit uses a differential value of at least one of the inference result and the reliability.

[0026] A training data candidate image acquisition device according to a sixteenth invention is the ninth invention, wherein the learning image selection unit selects a training data candidate image to be used in the learning when the amount of change exceeds a predetermined value.

[0027] In the training data candidate image acquisition method of the 17th invention, a machine learning model is used to infer the input image, the change in the inference result of the identified image pair is calculated, and the training data candidate image used in learning is selected based on the calculated change.

[0028] The program of the 18th invention enables a computer that obtains training data candidate images for learning an inference model to perform the following processing: use a machine learning model to infer an input image, calculate the change in the inference result of the identified image pair, and select a training data candidate image to be used in learning based on the calculated change.

[0029] The training data candidate image acquisition device of the 19th invention comprises: an inference unit, which uses a machine learning model to infer an input image; an inference result change calculation unit, which calculates the change in the inference result of a pair of images continuous in time; and a learning image selection unit, which selects a training data candidate image to be used in learning based on the change calculated by the inference result change calculation unit.

[0030] A training data candidate image acquisition device according to a 20th invention is the training data candidate image acquisition device according to the 19th invention, wherein the input image is a medical image.

[0031] A training data candidate image acquisition device according to a 21st invention is the device according to the 19th invention, wherein the inference unit infers at least one of classification, detection, and region extraction of the input image.

[0032] A training data candidate image acquisition device according to a 22nd invention is characterized in that, in the 19th invention, the inference unit outputs the reliability of the inference.

[0033] A training data candidate image acquisition device according to a 23rd invention is the one according to the 19th invention, wherein the inference result change calculation unit uses a differential value of at least one of the inference result and the reliability.

[0034] A training data candidate image acquisition device according to a 24th invention is the training data candidate image acquisition device according to the 19th invention, wherein the learning image selection unit selects a training data candidate image to be used in the learning when the amount of change exceeds a predetermined value.

[0035] In the training data candidate image acquisition method of the 25th invention, a machine learning model is used to infer the input image, and the change in the inference result of a pair of images continuous in time is calculated. Based on the calculated change, the training data candidate image used in learning is selected.

[0036] The program of the 26th invention enables a computer that obtains candidate images of training data for learning an inference model to perform the following processing: use a machine learning model to infer an input image, calculate the change in the inference results of a pair of images that are consecutive in time, and select candidate images of training data to be used in learning based on the calculated change.

[0037] Effects of the Invention

[0038] According to the present invention, it is possible to provide an endoscope, a training data candidate image acquisition device, a training data candidate image acquisition method and a program, which can reliably select pre-prepared medical images that machine learning is not good at recognizing. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a block diagram showing a main electrical configuration of an endoscope system according to one embodiment of the present invention.

[0040] Figure 2 This is a flowchart showing an imaging operation in the endoscope system according to one embodiment of the present invention.

[0041] Figure 3 This is a flowchart showing the operation of similar image group determination in the endoscope system according to one embodiment of the present invention.

[0042] Figure 4 It is a diagram for explaining determination of a similar image group in the endoscope system according to one embodiment of the present invention.

[0043] Figure 5 This is a flowchart showing a modified example of the operation of similar image group determination in the endoscope system according to one embodiment of the present invention.

[0044] Figure 6 It is a diagram for explaining selection of a similar image group in the endoscope system according to one embodiment of the present invention.

[0045] Figure 7 This is a diagram for explaining the flow of data between blocks in the endoscope system according to one embodiment of the present invention.

[0046] Figure 8 This is a diagram showing an example of the structure of data in a recording unit of an endoscope system according to an embodiment of the present invention.

[0047] Fig. 9 It is a diagram showing a modified example of the structure of data in the recording unit of the endoscope system according to one embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described by assuming that it is applied to a device such as an endoscope, which obtains candidate images that become training data for learning when using image data based on camera signals to create or improve an inference model and maintain performance. Equipment such as an endoscope is used by experts such as physicians, and can approach an object, illuminate it as needed, and display the image data obtained by continuous camera shooting as an image on a display device. When visually viewing the image, or with the assistance of the inference model, the expert pays attention to the part of concern and observes it. Therefore, when observing the same object, multiple similar image frames can be obtained from the image frames obtained continuously. The degree of image change can be known by digitizing the content of the image obtained by continuous camera shooting in time and comparing the numerical value of each image, or digitizing the difference between the image data, etc. An embodiment of the present invention has a similar image group determination unit that determines a similar image group whose image change is less than or equal to a specified value, and can determine whether the same object is being observed.

[0049] In addition, in one embodiment of the present invention, there is an inference unit that inputs each image frame obtained in sequence into a machine learning model for inference, and a determination unit that determines the difference between the inference results of each image determined by the similar image determination unit to be included in the similar image group. According to this structure, although the same object is observed, when there is a difference in the inference result, if the cause of the difference is analyzed, it is possible to analyze the situation that the inference unit is not good at. That is, if the image used in learning is selected based on the difference in the inference result, the selected image becomes a clue when generating an effective improved inference model. Therefore, in one embodiment of the present invention, a learning image selection unit is provided.

[0050] Next, use Figures 1 to 9 An embodiment in which the present invention is applied to an endoscope system will be described. The endoscope system functions as a training data candidate image acquisition device (endoscope) for acquiring images for learning an inference model. Figure 1 1 is a block diagram showing the main electrical structure of the endoscope system. The endoscope system is composed of an endoscope device 10 and a learning device 30. However, the blocks in the endoscope device 10 do not need to be set in an integrated device, and can also be divided into multiple devices. For example, the camera unit 11 and other blocks can also be set in different devices. In this case, the parts other than the camera unit 11 can also be connected to an internal network in a hospital or the like.

[0051] In addition, the endoscope system can also cooperate with in-hospital systems, electronic medical record systems, etc. In the case where the images recorded in the recording unit 18 described later are subject to confidentiality obligations, etc., they may not be output from the endoscope device 10 to, for example, the learning device 30. In addition, when the learning device 30 is configured in the hospital, and the exchange of data is permitted in the contract, etc., the learning device 30 can also be set as a system in the hospital. However, in the following description, it is assumed that an organization outside the hospital uses the learning device to develop an inference model, but it is not limited to this. In addition, each block in the learning device 30 can also be divided into multiple devices, and the learning unit 34 can also be configured in the server.

[0052] The endoscope device 10 has an imaging unit 11, an image processing unit 12, a display unit 12a, an inference unit 13, a difference determination unit 14, an image determination unit 15, a condition determination unit 16, an information association unit 17, a recording unit 18, a communication unit 19, and a control unit 20. As described above, each block in the endoscope device 10 can also be separated from the endoscope device 10 and used as another device. It is assumed that it will be developed as a so-called IoT device, and an example of carrying the same hardware as the terminal device is shown. However, if the endoscope device 10 cooperates with the learning device 30 to share the role, there are also functional blocks that can be omitted. In order to insert into a cavity or a tubular object to observe the inside, the endoscope device 10 has a portion for an insertion portion composed of a long cylindrical tube, but regarding the insertion portion, there are flexible insertion portions and non-flexible insertion portions. In addition, the imaging unit 11 is often provided at the front end of the insertion portion.

[0053] The imaging unit 11 is assumed to include an imaging unit, a light source unit, an operation unit, and a treatment unit. The imaging unit includes an optical lens for imaging, an imaging element, an imaging control circuit, and the like. In addition, in the case of having an automatic focus adjustment function, it includes a focus detection circuit, an automatic focus adjustment device, and the like. The optical lens forms an optical image of an object. The imaging element is arranged near the position where the optical image is formed, converts the optical image into an image signal, and outputs the image signal to the image processing unit 12 after performing AD conversion on the image signal. After receiving an instruction to start imaging from the control unit 20, the imaging control circuit performs readout control of the image signal from the imaging element at a prescribed rate.

[0054] In addition, for easy observation, the light source unit in the camera unit 11 performs illumination for brightening the walls of the digestive tract in the body cavity. The light source unit includes light sources such as laser light sources, LED light sources, xenon lamps, halogen lamps, and has an optical lens for illumination. The detection characteristics of the tissue vary according to the wavelength of the illumination light, so the light source unit may also have a function of changing the wavelength of the light source, and may also change the image processing by a known method according to the wavelength change. Detection does not necessarily have to be performed by the physician visually. In addition, the operating unit in the camera unit 11 has an operating component for performing shooting instructions for endoscopic static images, shooting start and shooting end instructions for endoscopic dynamic images; an action unit that performs actions associated with the operation; a treatment unit; a function execution unit, etc. In addition, there may also be an operating component for performing switching instructions for the focus of the endoscopic image. The changes caused by the above-mentioned operations can become parameters of image changes.

[0055] Furthermore, the operating unit in the camera unit 11 has an operating angle knob for bending the front end of the insertion portion of the endoscope. In addition, there is a function execution unit for executing the function of supplying air / water into the body cavity through a flexible tube, and the function of sucking air and liquid. In addition, the treatment unit has a function execution unit such as a treatment instrument such as a biopsy forceps for collecting a part of the tissue for biological examination, and can also have a treatment instrument such as a snare and a high-frequency knife for removing the affected part such as a polyp. These function execution units such as treatment instruments (which can also be roughly classified as operating units) can be operated by operating components that cause the function execution units to act. With the above-mentioned operation, the shape of the affected part sometimes changes, or bleeding occurs. In addition, when heat is used during the operation, water vapor, smoke, water splashes, etc. may be generated. In addition, during the treatment, there is also a clamp left in the body. These changes during the operation change and the changes in the state of the object accompanied by the operation can become parameters of image change. When the operation changes, changes in the brightness, color saturation, and contrast of the image may occur.

[0056] The image processing unit 12 has an image processing circuit, and receives an image signal from the camera unit 11, and performs various image processing such as development processing according to instructions from the control unit 20. The image processing unit 12 outputs the image data after image processing to the display unit 12a, the inference unit 13, the image determination unit 15, and the communication unit 19. As the image processing in the image processing unit 12, in addition to adjusting the color tone and brightness of the image, it is also possible to perform contrast emphasis, edge emphasis and other emphasis processing for improving visual recognition, grayscale processing for natural grayscale, and further, HDR (High Dynamic Range) processing for improving image quality using multiple image frames, multiple synthesis processing such as super-resolution processing, etc. The image processing unit 12 functions as an image processing unit (image processing circuit) that processes image frame information into visually recognizable information. The change when the operation is changed becomes a parameter of image change. In addition, the image processing unit 12 can also entrust the above-mentioned function to the image processing unit 32 in the learning device 30, and omit it from the endoscope device 10. However, when the endoscope apparatus 10 is required to be independent as an IoT device, if the image processing unit 12 is provided in the endoscope apparatus 10 , images can be transmitted to the outside, and the degree of freedom can be increased.

[0057] The display unit 12a includes a display device such as a display / monitor and a display control circuit. The display unit 12a inputs the image data processed by the image processing unit 12 according to the control signal of the control unit 20, and displays the endoscopic image, etc. In addition, the endoscopic image, etc. superimposed with the inference result of the inference unit 13 can also be displayed.

[0058] The inference unit 13 has an inference engine, which inputs image data of the image frame from the camera unit 11 and performs inference. An inference model for inferring its suggestions is set in the inference engine. The inference engine can be composed of hardware, and can also be composed of software. In addition, the inference unit 13 can also have a forward propagation type neural network, etc., and the forward propagation type neural network is described in the learning unit 34. The data input to the inference engine is not limited to the image data of the image frame, and the status information (related information, auxiliary information) when the frame image is obtained can also be input for inference. By using the status information for inference, a more reliable inference result can be obtained. In addition, the inference is performed for each frame, but it is not limited to each frame. The inference can also be performed every several frames according to the requirements of the visual recognizability of the inference result. In addition, several consecutive frames can be input to the inference unit for inference.

[0059] The reasoning unit 13 inputs image data and performs reasoning using a reasoning model generated by machine learning, thereby, for example, inferring suggestions for assisting doctors in making diagnoses. In addition, the reasoning unit 13 can also determine what the object (affected part, organ, etc.) captured in the image obtained by the camera unit 11 is, determine its position, determine the range of the affected part, etc., divide (segment) the image, classify it, and perform reasoning such as good or bad judgment. The reasoning unit 13 performs reasoning according to the instructions from the control unit 20 and outputs the reasoning result to the control unit 20. In addition, when performing reasoning, the reasoning unit 13 calculates the reliability of the reasoning and outputs the calculated reliability value to the difference judgment unit 14.

[0060] The use method (display method, etc.) of the inference result changes according to the inference specification of the inference unit 13, and the detection method of the difference in the inference result can also adopt various methods. For example, in the case of inference quality judgment, the 0.× display changes with time, in the case of inference position determination, the coordinates of the specified position in the screen change with time, and in the case of inference segmentation (the range occupied by the object), the shape and area of ​​each segment in the screen change with time, so these can be used as changes in the inference result.

[0061] In addition, the image input to the inference unit 13 is an image from the imaging unit 11, and here it is assumed that it is a medical image such as an endoscope. However, the medical image is not limited to an endoscopic image obtained by an endoscopic device, and may be an image obtained by medical equipment such as an X-ray device, an MRI device, an ultrasonic diagnostic device, a dermatology camera, and a dental camera. When an endoscopic image is used as an input image, in this embodiment, it is possible to cope with a sudden change in the field of view in the endoscope.

[0062] Furthermore, the input image input to the inference unit 13 is not limited to medical images. For example, sudden changes in the field of view may also occur in industrial endoscopes, cameras mounted on robot vehicles and drones. In addition, endoscopes are devices that are inserted and used from a narrow insertion hole. Therefore, the front end portion of the tube with a thinner outer diameter is easy to swing, and this swing greatly affects the acquisition of the image. When the object is at a close distance, the change is particularly large. The up and down, left and right shaking of the front end portion becomes a change in the composition, and the swing of the insertion direction of the front end portion causes a change in the size of the object. Sometimes unnecessary content is captured due to the up and down, left and right changes, and sometimes the size of the object changes due to changes in relative distance. In addition, when the image is rotated due to torsion, when the camera unit is not facing the observed object but observing it at an angle, etc., the image may be deformed. These changes in conditions also become parameters that can change the image, and in other words, sometimes the image changes greatly in a compound way. This change is not limited to endoscopes, and this change may occur in many inspection devices that use images.

[0063] The inference unit 13 functions as an inference unit (inference engine) as follows (for example, see Figure 2 and Figure 5 S1 Figure 7 The inference unit 44 is configured to infer a specific object image contained in the image obtained by the camera (which may also be an endoscopic image) using a machine learning model. In addition, the inference unit 13 functions as an inference unit (inference engine) that infers the input image using a machine learning model (for example, referring to Figure 2 and Figure 5 S1 Figure 7 The above-mentioned input image is a medical image. The above-mentioned inference unit infers at least one of the classification, detection and region extraction of the input image (for example, referring to Figure 2 and Figure 5 In addition, the inference unit outputs the reliability of the inference (for example, referring to Figure 2 and Figure 5 of S1).

[0064] The difference determination unit 14 may also include a difference determination circuit, and may also be implemented by executing a program by a processor including a CPU or the like. The difference determination unit 14 calculates the difference in the inference result of the inference unit 13 (the difference has been explained by way of example). That is, the inference unit 13 performs inference for each frame and outputs the inference result, and therefore, the difference determination unit 14 calculates the difference in the inference result. When calculating the difference (change), the differential value of the inference result may also be used. In addition, when calculating the difference (change), the change in the value representing the reliability calculated by the inference unit 13 may also be calculated. That is, the inference unit 13 performs inference every time image data is input, and calculates the reliability of the inference result, and therefore, the difference may also be determined based on the change in the reliability value. In addition, as described later, the image determination unit 15 determines whether the image belongs to a similar image group, and therefore, the difference determination unit 14 determines the difference for each image included in the similar image group.

[0065] The difference determination unit 14 functions as an inference result difference calculation unit that calculates the difference in the inference result of the inference unit (for example, referring to Figure 2 and Figure 5 As calculation of the difference in the inference result, for example, at least one of a change in position, a change in size, a change in the inferred range, etc. of the specific object may be calculated.

[0066] The image determination unit 15 may also include an image determination circuit, and may also be implemented by executing a program by a processor including a CPU. The image determination unit 15 determines whether the image data inputted from the image processing unit 12 in a time sequence belongs to a similar image group. When a doctor or the like inserts the front end of an endoscope device equipped with an imaging element into a body cavity, the imaging element acquires an endoscopic image. When the front end approaches a specific object such as an affected part, the doctor or the like acquires more images near the affected part in order to observe the affected part carefully. The image determination unit 15 may also determine whether it is an image group including the specific object. In addition, the specific object is an inspection object observed by an endoscope or the like, and the specific object image means a portion of the image of the specific object in the screen. Various objects are captured in the endoscopic image, but among these objects, the object determined as a detection object as a specification of the AI ​​(inference unit) is called a specific object. For example, as a feature of the above-mentioned endoscopic image, in many cases, the object in the acquired image changes due to a slight swing of the front end, the position of the specific object in the image changes suddenly, and redundant content enters the image. Therefore, the position and size of the object change drastically.

[0067] The image determination unit 15 digitizes the content of the image to determine whether it is a similar image, and furthermore, determines whether the continuous input has substantially no change on the image. The numerical value is a value that can track the pattern of the same object in the image. As described above, the numerical value can be changed according to the time series detection result, etc. In addition, it is also possible to compare images that are adjacent in time (or the inference results of the images), etc. The numerical value used to represent the similarity of the images can be appropriately determined by a known method. Here, it is assumed that the numerical values ​​of composition, color, brightness, etc., the change in the size of the object, whether the object is located in the screen, etc., can also be determined by the numerical value.

[0068] When determining a similar image group, the image determination unit 15 digitizes the contents of the image to make the determination. However, images obtained by continuous shooting in time, that is, a series of image groups, may also include non-similar images that cannot be said to be numerically similar. Figure 6 As will be described later, the tip of the endoscope has minute movements, making it difficult to focus, and the object is easily deviated from the image. Therefore, when observing the same part within a specified number of frames, the similar image group may also include non-similar images in the middle of the observation. Figure 3 In addition, the image determination unit 15 can, for example, calculate the movement amount of the corresponding points to determine that the two input images are substantially unchanged, or can determine that the two images are substantially unchanged based on at least one of the changes in brightness, saturation, and contrast of the images.

[0069] The image determination unit 15 functions as a similar image group determination unit that compares the contents of images (which may be endoscopic images) obtained by continuous imaging in time and determines a similar image group (for example, referring to the similar image group) that observes the same part within a predetermined number of frames, including non-similar images. Figure 2 and Figure 5 S3 Figure 3 , Figure 7 The image recognition unit 43). In addition, the similar image group determination unit digitizes the similar image group into a numerical value that can track the pattern of the same object in the image of the endoscope image, and uses the numerical value to determine whether it is a similar image (for example, referring to Figure 3). When an observer performs an endoscopic examination, in order to fully confirm the object with the naked eye, he stares at the same object and confirms a specific object while changing the observation method. Therefore, the number of frames corresponding to the time required for such observation is a specified number of frames. In addition, as described above, when observing with an endoscope, the front end cannot be fixed in space and moves slightly, so it is difficult to stare and the observed image easily deviates from the object. As a similar image group, images that may contain training image candidates that are important for diagnosis and are equivalent to the above-mentioned tricky images and missed images (for example, images of observing the same part, etc.) are included. Therefore, non-similar images that exist in the image group that may contain similar images are also included and determined as a similar image group.

[0070] The above-mentioned similar image group may also include the result of determining the image group corresponding to the sudden change in viewpoint position movement according to the similarity of the images before and after the endoscopic images obtained by continuous shooting in time. In addition, the similar image group may also include the result of determining the following image group according to the similarity of the images before and after the endoscopic images obtained by continuous shooting in time, which is a sudden change and corresponds to the jitter of the visual image caused by the unconscious eye movement called saccade (in the case of humans, it is corrected by the brain and therefore not known). The above-mentioned image equivalent to saccade is used Figure 4 and Figure 6 The image change corresponding to the saccade mentioned here conceptually represents the sudden change of the image due to the change of the relative positional relationship between the front end of the endoscope and the object, that is, the positional relationship between the object at a very close distance and the light source provided at the front end of the endoscope.

[0071] The above-mentioned image changes corresponding to scanning are related to the particularity of inspection using an endoscope or the like. That is, even if the observation is to be performed in the same manner, when there is movement of the object or the camera unit, although it is a relatively small relative change, it becomes a large image change. That is, even if the object is captured in the picture, its size and position in the picture are prone to sudden changes, and the distribution of exposure and brightness is also prone to sudden changes. Moreover, such images are not aimed and taken, and are generated or not generated depending on the situation, so it is also difficult to become an image of intended training data.

[0072] In the endoscopic image, it is easy to cause a sudden change in the image similar to the above-mentioned image change corresponding to the sweep. The object is easy to move, and the object is immersed in body fluids or cleaning fluids, and the operation of endoscopic observation such as suction and gas supply is performed, and special light observation is performed, staining is performed, and operation is performed in cooperation with treatment instruments. In order to observe the object, various methods are often used together. Sometimes they occur simultaneously and change in a complex manner, and unintentional (or unconscious) sudden changes in the image are very likely to occur. In response to this situation, it is preferable to correctly confirm the object.

[0073] Thus, regarding the image change (sudden change) corresponding to the scan, there are many parameters that become the cause, and it may occur in a complex manner. Therefore, in machine learning, it is difficult to prepare patterns of all complex causes in most cases. In the case of similarity determination using pattern matching and similarity determination by digitizing the image, the images at the time of these sudden changes may be determined to be not similar images. That is, they may be determined to be non-similar images with different contents. In addition, pattern matching refers to processing the numerical data of each pixel or each coordinate in the screen in two dimensions, so it can be said that the data is numerically compared.

[0074] When similar images are acquired continuously in time, even if they are simply set as non-similar images, there is a high possibility that they contain important images. Therefore, in the present embodiment, non-similar images that meet the above conditions are treated as images constituting a group of similar images. That is, in the acquisition of candidate images of training data for acquiring endoscopic images for learning an inference model for an endoscope, the contents of endoscopic images acquired by continuous imaging in time are compared, and a similar image group is determined including non-similar images for a group of similar images that observe the same part within a predetermined number of frames. The content of an image refers to the content (image feature) that makes the image characteristic, such as the shape, pattern, shadow, color, size, position (rotation, deformation, etc., but these can also be corrected) of the object in the image acquired by imaging.

[0075] The status determination unit 16 may also include a status determination circuit, and may also be implemented by executing a program by a processor including a CPU. The status determination unit 16 determines information related to the usage status of the endoscope device 10 used by a physician or the like. Therefore, the status determination unit 16 may also determine, based on the image data obtained by the image processing unit 12, whether the positional relationship between the front end portion provided with the imaging element and the wall surface in the body cavity is directly opposite or inclined. In addition, it may also be determined whether the focus is on, how much the depth is, etc. In addition, the usage status of the light source unit in the imaging unit 11 may also be determined. For example, if the wavelength of the light source light is known, it may be determined whether narrow-band light observation (NBI: Narrow Band Imaging) is being performed. Furthermore, the usage status of the treatment unit in the imaging unit 11 may also be determined. For example, if a physician or the like performs a water injection operation, the effect of the water injection appears in the image, and if a suction operation is performed, the effect of the suction appears.

[0076] The determination result of the situation determination unit 16 is output to the control unit 20. As described later, the situation determination result at this time is associated with the image data. In a case where the front end portion is not facing the affected part, etc. but is in a tilted position, it is difficult to see the affected part, etc., and it is easy to become difficult to infer. In addition, in the case of a water injection operation, the screen is affected by the water, and it is difficult to see the affected part, etc., and it is easy to become difficult to infer. If such situation information exists, as long as the image is inferred together with the situation information, the reliability of the inference of finding the affected part can be improved. The situation determination unit 16 functions as a camera information acquisition unit that can acquire camera information corresponding to each image selected by the learning image selection unit (for example, refer to Figure 5 S6).

[0077] The information association unit 17 may include an information association circuit, and may be implemented by executing a program on a processor including a CPU. The information association unit 17 associates at least one of the status information determined by the status determination unit 16, the similar image information determined by the image determination unit 15, and the information related to the difference in the reliability value of the inference determined by the difference determination unit 14 with the image data processed by the image processing unit 12. The image data after the information is associated by the information association unit 17 is recorded in the recording unit 18. In addition, regarding the image data determined by the difference determination unit 14 to have a difference of more than a predetermined value in the reliability of the inference, the communication unit 19 is notified of this fact. As described later, the communication unit 19 sends the notified image data to the learning device 30, and uses it as training data when re-learning.

[0078] The recording unit 18 has an electrically rewritable nonvolatile memory and / or a volatile memory. The recording unit 18 stores a program for causing the CPU in the control unit 20 to operate. In addition, the recording unit 18 stores the model information of the endoscope device 10, various characteristic values ​​in the endoscope 10, adjustment values, etc. Furthermore, the recording unit 18 may also record image data processed by the image processing unit 12 and associated with information by the information association unit 17. The details of the data structure recorded in the recording unit 18 are described in detail in the following. Figure 8 and Fig. 9 In addition, in the recording unit 18, it is also possible not to record Figure 8 and Fig. 9 All of the information shown is shared with the recording unit 33 in the learning device 30.

[0079] As described above, the recording unit 18 records the image data of the original (raw) dynamic image obtained by the camera unit 11 and processed by the image processing unit 12, and the image data after the information is associated by the information association unit 17. The original dynamic image is recorded in the recording unit 18 in order to improve the inference model based on the recorded content later. Therefore, it is not necessary to be the selected image data itself, and it can be simply found by searching the image data. In other words, it is also possible to directly record the dynamic image during the original inspection and select a specific frame from the recorded image. For example, it can also be information used to indicate a specific frame, such as an image at the beginning of the inspection, an image indicating the detection of a specific object, etc. Such information is used to indicate the number of frames from a specific time, an image of a few minutes and seconds, etc. Figure 8 and Fig. 9 This will be described later.

[0080] In addition, when relearning is performed to improve the inference model, when customization is performed for a specific situation, when versatility is further improved, or when a new inference model is created, there are various specifications and directions for the creation of inference models. Therefore, corresponding to the specifications and directions of the inference model, it is preferred to record the necessary related data together. For example, when creating an inference model that matches a specific profile (gender, age group, race, medical history, etc.), or when improving such an inference model, it is easy to select the best training data candidate by organizing and recording the test results of what kind of profile. In addition, data representing various backgrounds when the data was acquired also plays a role in actual data application, and it is also possible to record information such as which hospital and which doctor conducted the examination, the presence or absence of personal information, various contracts such as informed consent forms, and the presence or absence of conditions of use.

[0081] As described above, when an image is recorded in the recording unit 18, the associated information may be recorded in an associated manner. Figure 4In the example, when the cross (×) as the reliability judgment result is set as learning data, it may be possible to refer to the information of the circle (○) for annotation, so the coordinates and / or judgment results can also be recorded as reference values. Figure 4 In the example, the reliability judgment is 0××0. When 0 and 0 are judged the same, the image of × in the middle can be converted into training data by referring to the image of 0. After obtaining the dynamic image, the annotator or robot can simply make appropriate annotations on the image of × by referring to the related information (the judgment result of reliability).

[0082] The communication unit 19 has a communication circuit (transceiver circuit) and communicates with the communication unit 31 in the learning device 30. The communication unit 19 selects necessary information from the information recorded in the recording unit 18 and sends it, thereby reducing the load of communication and recording. In addition, by not sending unnecessary data, security can be improved. For example, images with different inference results obtained by the camera unit 11 are sent to the learning device 30 for learning. However, in the absence of restrictions or according to a contract, all information (including images) can be sent. At this time, the communication unit 19 can also send image data with different inference results of each image included in the similar image group after the information is associated by the information association unit 17 to the communication unit 31 in the learning device 30. In addition, communication can also be used when replacing the inference model, and the communication unit 19 receives the inference model generated by the learning unit 34 in the learning device 30 through the communication unit 31. In addition, each request signal and the like are sent and received, and the information and the like that meet the conditions are sent and received according to the request signal.

[0083] The control unit 20 is a processor having a CPU (Central Processing Unit) and its peripheral circuits and memory. The processor may be one or may be composed of multiple chips. The CPU controls each unit in the endoscope device 10 according to the program stored in the memory, thereby executing the entire endoscope device 10. Each unit in the endoscope device 10 is implemented by being controlled by the CPU in a software manner. The difference determination unit 14, the image determination unit 15, the condition determination unit 16, and the information association unit 17 may all or part be implemented by the processor in the control unit 20. The above-mentioned processor may also implement all or part of the similar image group determination unit, the image recognition unit, the inference result difference calculation unit, the inference result change calculation unit, the learning image selection unit, and the shooting information acquisition unit. Similarly, the imaging unit 11, the image processing unit 12, and the inference unit 13 may also all or part of their functions be implemented by the processor in the control unit 20. Furthermore, the control unit 20 may operate in cooperation with the control unit 35 in the learning device 30 so that the endoscope device 10 and the learning device 30 operate integrally.

[0084] The control unit 20 selects an image to be used for relearning (for example, a reference image) using the difference result calculated by the difference determination unit 14 for the inference result (which may be reliability, etc.) inferred by the inference unit 13, with respect to the image determined by the image determination unit 15 as an image constituting a part of the similar image group. Figure 2 S7, Figure 5 S8). For example, when the reliability changes from a high-reliability state to a low-reliability state, generally speaking, there is a high possibility that the image with high reliability is produced using the same image as the image when producing the inference model, while on the contrary, there is a high possibility that the image with low reliability is not used when producing the inference model. However, regarding such images included in the similar image group, even if they are non-similar images, they may be images of the same object. Such images are tricky images for the inference model, and it is hoped that reliable inference can be performed on such tricky images. Therefore, in the present embodiment, when such inference results (which may also be reliability, etc.) change, the image at this time is selected and set as a candidate image for learning. When producing training data, experts such as doctors can determine whether the candidate image can be used as training data. That is, this is because, when not selected, it takes labor and time to adopt it as training data, and it takes time to improve it, but if selected, the candidate is the training data, and sometimes it can be immediately applied to the improvement of the inference model.

[0085] The control unit 20 functions as a learning image selection unit that selects an image to be used for relearning based on the difference in the inference result calculated by the inference result difference calculation unit (for example, refer to Figure 2 S7, Figure 5 S8, Figure 7 The image selection unit 46 for learning is used. The image selection based on the difference in the inference result means, for example, selecting the changed image for relearning when there is a difference in the inference result, such as when the reliability changes by more than a predetermined value, when the size of a specific object changes suddenly, when the position of the object in the image changes suddenly by more than a predetermined value, etc. Here, the selection may be a selection for recording the image frame itself, but is not limited to this, and may also be a method for selecting information such as an image frame that can be immediately retrieved.

[0086] The above-mentioned learning image selection unit can also remove images that do not meet specific conditions and have extremely poor conditions. Regarding the specific condition, for example, there is a case where the image is too difficult to be visually recognized even after image correction, etc. As described above, from the images included in the similar image group, the image used in the re-learning is selected based on the difference in the inference result of the inference result calculation unit. Therefore, images that are not similar to the images observing the same part at first glance are sometimes also selected as candidate images for learning. However, it can be said that the possibility of extremely poor conditions, such as images in which the entire screen becomes completely white due to water injection operations, etc., and images in which the entire screen becomes completely black due to the positional relationship of the light source, originally containing the specific object set as the target is very low.

[0087] Furthermore, the longer the time for photographing the object is, the more likely it is that the image will be of poor condition. That is, when (1) the exposure time is long, (2) the light source is turned on for a long time in order to obtain a sufficient amount of light in the body cavity, (3) the gain in the imaging element or circuit is high, etc., the image may be extremely poor. This is because when the exposure time etc. becomes longer, the relative positional relationship between the specific object and the imaging unit 11 changes accordingly, and thus the image shakes, etc., resulting in a poor image.

[0088] Therefore, the learning image selection unit removes such extremely poor images that are unlikely to contain specific objects from the selection objects. This can be determined based on the contrast of the image data obtained by the camera unit 11, the amount of image jitter, etc. In addition, even if the image quality is not directly determined, it can be determined as an image with extremely poor conditions based on the shooting conditions, etc. In addition, when multiple frames of images are synthesized by HDR (High Dynamic Range) shooting, depth synthesis shooting, etc., it is easy to become an image with extremely poor conditions. Therefore, in this case, it can also be determined as an image with extremely poor conditions based on the shooting conditions.

[0089] The control unit 20 also functions as a learning image selection unit that selects a candidate training data image to be used in learning based on the amount of change calculated by the inference result change calculation unit (for example, referring to Figure 2 S7, Figure 5 S8, Figure 7 The learning image selection unit 46 selects the training data candidate image used in the learning when the amount of change exceeds the specified value. That is, when the amount of change is greater than the specified value, although it is a substantially identical recognition image, it is different from the previous image and may not have been used in the learning before, so it is selected as the learning candidate image.

[0090] The learning device 30 is assumed to be a part responsible for the creation and improvement of the inference model, and is assumed to be located outside the hospital (examination facility). The learning device 30 has a communication unit 31 to generate data transmission and reception through communication, and also has an image processing unit 32, a recording unit 33, a learning unit 34, and a control unit 35 to efficiently perform advanced and highly professional learning. For example, when the recording unit 33 is located outside, communication and the like may become a constraint during learning, but the learning device 30 may also cooperate with a recording unit located in another place through communication. The learning device 30 may also be configured on a server, etc., in which case it is connected to the endoscope device 10 via a communication network such as the Internet. In addition, the learning device 30 is connected to a plurality of devices, receives a plurality of training data from these devices, performs learning using the training data, and generates an inference model. In addition, training data candidates such as image data may also be received, and the learning device 30 annotates and creates training data, and generates an inference model using the training data.

[0091] The communication unit 31 has a communication circuit (transceiver circuit) and communicates with the communication unit 19 in the endoscope device 10. As described above, the communication unit 31 receives image data in which the inference results of each image included in the similar image group are different. In addition, the communication unit 31 sends the inference model generated by the learning unit 34 to the communication unit 19. Furthermore, the communication unit 31 sends the inference model generated by re-learning using the image data for re-learning, which is selected from the endoscope device 10 based on the difference in inference results, to the communication unit 19. In addition, as described above, in the case where the recording unit 33 is configured externally, communication and the like may become a constraint during learning, but it is also possible to cooperate with a recording unit located in another place through the communication unit 31.

[0092] The image processing unit 32 has an image processing circuit, which inputs image data from the endoscope device 10, and performs various image processing such as development processing according to instructions from the control unit 35. The image processing may be the same processing as that in the image processing unit 12, and the processing content may be appropriately changed. Here, the processed image data may be recorded in the recording unit 33, and may also be displayed on a display device, etc. Furthermore, image processing may also be performed when learning is performed. For example, the image size may be changed, and emphasis processing may be performed for ease of processing, etc., and image processing may be performed on the learning image recorded in the recording unit 33, thereby facilitating learning. In addition, the image processing unit 32 may intentionally generate an image that is difficult to judge, and use the image for testing when the learning unit 34 performs learning.

[0093] The recording unit 33 has an electrically rewritable non-volatile memory and / or a volatile memory. The recording unit 33 stores a program for causing the CPU in the control unit 35 to operate. In addition, the recording unit 33 also stores various characteristic values, adjustment values, etc. in the learning device 30. Furthermore, the recording unit 33 records the training data for learning sent from the endoscope device 10 (including image data for relearning selected based on the difference in the inference result). The details of the data structure recorded in the recording unit 33 are described in detail in the following manner. Figure 8 and Fig. 9 In addition, in the recording unit 33, it is also possible not to record Figure 8 and Fig. 9 All of the information shown is shared with the recording unit 18 in the endoscope device 10.

[0094] The learning unit 34 has an inference engine, and performs machine learning such as deep learning using the training data for learning recorded in the recording unit 33 to generate an inference model. Relearning is performed when a predetermined number of image data for relearning selected from the endoscope device 10 based on the difference in inference results are accumulated. By performing relearning, an inference model that can perform highly reliable inference even for difficult images, missed images, etc. can be generated.

[0095] Here, deep learning is explained. "Deep learning" is obtained by multi-layering the process of "machine learning" using neural networks. The "forward propagation neural network" that sends information from front to back to make judgments is a representative neural network. In addition, the reasoning unit 13 also has a forward propagation neural network. In the simplest structure, the forward propagation neural network (for example Figure 6The inference engine 13A also has the same structure) has an input layer composed of N1 neurons, an intermediate layer composed of N2 neurons given by parameters, and an output layer composed of N3 neurons corresponding to the number of categories to be judged. The input layer and the intermediate layer, and each neuron in the intermediate layer and the output layer are connected by connection weights, and bias values ​​are added to the intermediate layer and the output layer, thereby making it easy to form a logic gate.

[0096] The neural network can perform simple discrimination and can be 3 layers. However, by making the intermediate layers multiple, it is also possible to learn the combination of multiple feature quantities in the process of machine learning. In recent years, from the perspective of the time spent on learning, judgment accuracy, and energy consumption, neural networks with 9 to 152 layers have become practical. In addition, the following "convolutional neural network" can also be used: a process called "convolution" that compresses the feature quantities of an image is performed, and it operates with minimal processing, which is strong in pattern recognition. In addition, the following "recursive neural network" (fully connected recursive neural network) can also be used: it processes more complex information, corresponds to information analysis whose meaning changes according to order and sequence, and allows information to flow in both directions.

[0097] In order to realize these technologies, the general-purpose arithmetic processing circuits that have existed in the past, such as CPU and FPGA (Field Programmable Gate Array), can also be used. However, it is not limited to this. Most of the processing of neural networks is matrix multiplication. Therefore, processors called GPU (Graphic Processing Unit) and Tensor Processing Unit (TPU) dedicated to matrix calculations can also be used. In recent years, this kind of artificial intelligence (AI) dedicated hardware "neural network processing unit (NPU)" is sometimes designed to be integrated and assembled with other circuits such as CPU to become part of the processing circuit.

[0098] In addition, as a method of machine learning, there are methods such as support vector machines and support vector regression. Regarding learning here, there are methods for calculating the weights, filter coefficients, and biases of the identifier, and there are also methods using logistic regression processing. In the case of making a machine determine something, people need to teach the machine how to determine. In this embodiment, a method of deriving the determination of an image through machine learning is adopted, but in addition, a rule-based method that adapts to the rules obtained by people through empirical rules / heuristic algorithms can also be used.

[0099] The control unit 35 is a processor having a CPU (Central Processing Unit) and its peripheral circuits and memory. The processor may be one or may be composed of a plurality of chips. The CPU controls each part in the learning device 30 according to the program stored in the memory, thereby executing the entire learning device 30. Each part in the learning device 30 is implemented by being controlled by the CPU in a software manner. In addition, the control unit 35 may also cooperate with the control unit 20 in the endoscope device 10 to enable the endoscope device 10 and the learning device 30 to operate as a whole.

[0100] Next, use Figure 8 The structure of the data recorded in the recording unit 50 is described. The recording unit 50 can be applied to the recording unit 18 in the endoscope device 10 and / or the recording unit 33 in the learning device 30, and the recording unit 50 has an electrically rewritable nonvolatile memory. Figure 8 The recording unit 50 does not store all the data shown, but the data may be recorded in two recording units. That is, the data required by the endoscope device 10 and the learning device 30 may be recorded separately. The recording unit 50 has two recording areas, namely, the inspection image recording unit 51 and the application information recording unit 52.

[0101] The examination image recording unit 51 is an area for recording examination moving images in the same manner as recording medical records, and records examination moving images A51a and examination moving images B51b. Figure 8 In the example, only two dynamic images A and B are shown as dynamic images for examination, but of course three or more dynamic images for examination can be recorded. Sometimes it is important to record as evidence for the diagnosis of the patient, and sometimes a static image equivalent to one frame of a dynamic image is recorded as a report, but in Figure 8 The display of static images is omitted in FIG. In the inspection image recording unit 51, not only images but also inspection results and the like may be recorded.

[0102] In addition, an application information recording unit 52 is provided in the recording unit 50. When the dynamic images recorded in the inspection image recording unit 51 for use in the evidence report are used as training data, for example, a processing procedure (such as an informed consent form) is required for this purpose, and depending on the dynamic images, doctors and patients may not agree to the application. In addition, when learning, it is preferable to consider various related information during the inspection to perform the learning. Therefore, such an application information recording unit 52 for recording application conditions, etc., can be provided to organize and record information on which dynamic image can be used for what purpose. In this case, the dynamic image application information folder and the training data folder can also be recorded separately. In addition, the application information recording unit does not necessarily need to be located in the same recording device.

[0103] exist Figure 8 In the example shown, dynamic image application information A52 and dynamic image application information B53 are set in the application information recording unit 52. Figure 8 In the figure, only two pieces of application information, application information A and application information B, are shown as dynamic image application information. However, depending on the number of dynamic images to be inspected, more than three pieces of application information can be recorded. The information recorded in dynamic image application information A53 and dynamic image application information B56 is the same, so here, the information recorded in dynamic image application information A53 will be mainly described.

[0104] The original dynamic image information 53a is information indicating which dynamic image the information corresponds to. For example, dynamic image application information A53 corresponds to the inspection dynamic image A51a, and dynamic image application information B56 corresponds to the inspection dynamic image B51b. The acquisition time information 53b is information such as the inspection date and time, the inspection agency, the name of the doctor in charge of the inspection, and the model name of the device used. The inference model type 53c indicates the type of inference model used by the endoscope device 10 used in the inspection. When the inference model used is unknown, it is unknown which inference model to use for relearning.

[0105] The application condition information 53d indicates under what conditions the dynamic image can be applied. For example, when the scope of application is determined by an informed consent form, the meaning is recorded. The profile information 53e is information such as the gender, age, ID, and examination results of the subject (patient) being examined. The inference result information 53f is information related to the results inferred according to each frame of the image.

[0106] In addition, a first training data folder 54 and a second training data folder 55 are provided in the application information recording unit 52. The first and second training data folders 54 and 55 record images selected as training data candidates by the control unit 20 based on the difference in the inference result using the inspection dynamic image A51a, that is, the first training data candidate 54a and the second training data candidate 55a. Regarding the candidate image, the candidate image data itself may be recorded, and in addition, information that can specify the candidate image data from the images recorded in the inspection image recording unit 51 (for example, the image of the frame number, or the image taken at the time and minute, etc.) may also be recorded. In addition, the associated information (the first associated information 54b and the second associated information 55b) associated by the information associating unit 17 is recorded. As the first associated information 54b and the second associated information 55b, for example, status information, similar image information, information related to the difference in reliability values, etc. are recorded.

[0107] The dynamic image application information record B56 also records original dynamic image information 56a, acquisition time information 56b, inference model type 56c, application condition information 56d, introduction information 56e, inference result information 56f, first training data folder 57, first training data candidate 57a, first related information 57b, second training data folder 58, second training data candidate 58a, and second related information 58b. However, these information are the same as the information in the dynamic image application information recording unit A55, so detailed description is omitted.

[0108] Next, use Fig. 9 A modified example of the structure of the data recorded in the recording unit 50 will be described. That is, Fig. 9 This is a modified example of the recording method of the data organized and recorded in the recording unit 50, and the dynamic image application information A and the dynamic image application information B are recorded in the dynamic image recording unit A61 and the dynamic image recording unit B65. The data is managed in a unified manner, so there is no need to find the recording location of the dynamic image when applying the information, but the file size may be too large.

[0109] That is, in this modification, the inspection image 61a is recorded in the moving image recording unit A61, and the inspection image 65a is recorded in the moving image recording unit B65. Figure 8 The dynamic images uniformly recorded in the inspection image recording unit 51 are the same. In addition, in this variant, the acquisition time information 62a, application condition information 62b, inference model type 62c, brief information 62d, inference result information 62e, the first training data folder 63 (including the first training data candidate 63a, the first related information 63b), and the second training data folder 64 (including the second training data candidate 64a, the second related information 64b) are recorded in the dynamic image recording unit A61. Similarly, the acquisition time information 66a, application condition information 66b, inference model type 66c, brief information 66d, inference result information 66e, the first training data folder 67 (including the first training data candidate 67a, the first related information 67b), and the second training data folder 68 (including the second training data candidate 68a, the second related information 68b) are recorded in the dynamic image recording unit A65. In addition, in Fig. 9 In the recording section, only two moving image recording sections, namely the moving image recording section A61 and the moving image recording section B65, are depicted. However, the recording area may be appropriately increased according to the number of inspection moving images.

[0110] like Figure 8 , Fig. 9As shown, various applications can be performed regarding which recording unit is placed in which recording device. In addition, regarding which folder the data is placed in and which recording unit records the data, appropriate applications can be made according to the current situation and environment. In addition, the training data folders can also be divided according to the characteristics of the training data.

[0111] Next, in describing the operation of one embodiment of the present invention, first, a method of selecting an image to be used for relearning from a similar image group will be described. Figure 6 (a)(b) is an example of when the distal end portion of the endoscope device 10 is inserted into the body cavity of the patient, and shows an example of the image obtained at this time. The blackened images (P1, P2, P11, P12) are the images when the endoscope device 10 is inserted into the patient's body cavity. In addition, the blackened images (P8, P9, P18, P19) are the images when the endoscope device 10 is pulled out. In these transition images P1, P2, P8, P9, no specific object such as the affected part is captured. In addition, the oblique line shadow images (P5, P6, P15, P16) are observation images when a doctor or the like finds a specific object such as an affected part and observes it carefully. Intermediate images P3, P4, P7 are obtained between the transition images P1, P2, P8, P9 and the observation images P5, P6. Similarly, intermediate images P13, P14, P17 are obtained between the transition images P11, P12, P18, P19 and the observation images P15, P16.

[0112] In addition, images P5 to P7 and P15 to P17 within the dotted frame Is are images of the same object, are similar to each other, and belong to a similar image group. It is sufficient to use a known method to determine whether they are similar images. For example, the image determination unit 15 can also calculate feature quantities from the images, and if the feature quantities of the two images are within a specified range, they are determined to be similar images. In addition, the image determination unit 15 can also determine whether they are similar images based on the composition, color distribution, etc. of the image.

[0113] In addition, the image data obtained by the imaging element of the imaging unit 11 is input to the input layer of the inference engine 13A of the inference unit 13 after image processing. The inference engine 13A performs inference through a neural network and outputs the inference result from the output layer. The inference engine 13A, for example, infers whether there is a specific object image such as an affected part in the endoscopic image. Based on the inference result, it can be indicated that images P5, P6, and P16 are specific object images. Figure 6 In (a), a thick frame is used to surround images P5 and P6. Figure 6 In (b), the image P16 is surrounded by a thick frame to indicate that it is a specific object image, but the best method can be appropriately selected for the display method. In addition, the inference engine 13A calculates the reliability of the inference result. Figure 6In (a), when the reliability value is higher than the specified value, it is indicated by a circle (⊥), and when the reliability value is lower than the specified value, it is indicated by a cross (×).

[0114] Furthermore, as described above, the status determination unit 16 determines the usage status of the endoscope device 10. Figure 6 In the example shown, the situation determination unit 16 obtains angle information such as whether the distal end portion of the endoscope device 10 faces the wall surface in the body cavity from the front, that is, directly facing or inclined. Figure 6 (a) shows the angle information obtained. Figure 6 In FIG. 1 , only the angle information is shown. However, the information is not limited to this information, and other information may be obtained to select an image to be used in relearning from the similar image group.

[0115] exist Figure 6 In the example shown in (a), in images P5 and P6, the inference engine 13A infers that they are images of specific objects such as the affected part, and the reliability is also high, and they are in a state of facing the specific object. Figure 6 In the example shown in (b), in the image P16, the inference engine 13A infers that it is an image of a specific object such as an affected part, and its reliability is also high, indicating that the patient is facing the specific object.

[0116] The reliability of the inference results of images P5, P6, and P16 containing specific objects is high, so it can be said that the possibility of containing specific objects such as affected parts is high. However, in the similar image group Is, images that are judged to have low reliability (for example, images P7, P15, and P17) may also contain specific objects such as affected parts. Figure 6 In the example shown in (b), the image P15 includes a specific object such as an affected part, but the reliability of the inference is low and it is displayed as not including the specific object. Figure 6 In the examples shown in (a) and (b), when the endoscope device 10 is facing a specific object such as an affected part, the reliability of the inference is high. However, on the other hand, when the endoscope device 10 is tilted relative to the specific object, the reliability of the inference becomes low. That is, when the front end of the endoscope is tilted relative to the wall surface in the body cavity, the inference model set in the inference engine 13A does not seem to be able to perform inference with high reliability. In this case, an image (difficult image) with a different inference result can be used for relearning to generate an inference model with higher reliability.

[0117] Thus, when the image from the endoscope device 10 is input into the inference model for inference, there are the following particularities in the endoscope, etc., and therefore, there is a problem unique to the endoscope device that it is difficult to produce an inference model with high reliability. That is, (1) the front end of the endoscope device is not fixed in space, so the direction of the front end shakes and the direction of the image that can be obtained changes sharply (a sudden change in the image equivalent to a sweep). In this way, the line of sight of the image sometimes jumps out. (2) The distance between the front end of the endoscope and the wall surface in the body cavity is originally at a close distance (at an extremely close distance), so if the front end moves slightly, the observed object changes significantly. (3) By causing the light source and the camera element used to illuminate the body cavity to move simultaneously, the brightness changes suddenly and the image changes suddenly. In addition, changes in the brightness, saturation, and contrast of the image may occur.

[0118] As described above, the front end of the endoscope has slight movements, so it is difficult to stare at it and has the characteristic of being easily deviated from the object. At this time, there may be important images (training image candidates) equivalent to tricky images and missed images. By learning such images that become training candidates as training data, images containing specific objects such as affected parts can also be extracted as training candidates. Therefore, in this embodiment, an image that becomes a candidate for training data is selected from a group of similar images that are similar to images of specific objects such as affected parts. In addition, the similar image group may also include non-similar images obtained while observing the same part. In addition, when selecting candidates for training data from the similar image group, the difference in inference results of adjacent images is used.

[0119] Next, use Figure 2 The flowchart shown here explains the imaging operation in this embodiment. The CPU of the control unit 20 in the endoscope device 10 controls each unit in the endoscope device 10 according to a program stored in a memory such as the recording unit 18, thereby realizing this flowchart.

[0120] exist Figure 2 When the action of the process shown starts, first, imaging and reasoning are performed (S1). Here, the control unit 11 instructs the imaging element and the imaging control circuit in the imaging unit 11 to start the imaging action. In the imaging action, the imaging signal of one screen (one frame) is read out sequentially at a time interval of a predetermined frame rate. The imaging action is repeatedly performed until it is determined in step S9 that the imaging action is completed. When the imaging action starts, the imaging element in the imaging unit 11 outputs an image signal, and the image processing unit 12 performs image processing on the image signal to make it visually recognizable information. The image data subjected to the image processing is output to the display unit 12a and displayed as an endoscopic image.

[0121] In step S1, the inference unit 13 inputs the image data to which the image processing has been performed and performs inference. That is, the inference unit 13 uses the machine learning model to perform inference on the specific object image contained in the endoscopic image obtained by taking the image. In other words, the inference unit 13 uses the machine learning model to perform inference on the input image. The inference result is superimposed on the endoscopic image in the image processing unit 12, and the image is displayed on the display unit 12a. The inference is performed, for example, to display a suggestion that assists the diagnosis to the user of the endoscope device 10, such as a physician. When a specific object such as an affected part is included in the endoscopic image, the display of the meaning may also be performed. In addition, the inference of the suggestion related to the endoscopic operation for finding the specific object may also be performed. In addition, the inference unit 13 may also perform inference on at least one of the classification, detection and area extraction of the input image. That is, if an affected part exists in the input image, the affected part may be detected, and the type of the affected part, the detection position, and the range of the affected part may be inferred. When performing these inferences, the inference unit 13 calculates the reliability of the inference.

[0122] After the imaging and reasoning are performed, the similar image group is then determined (S3). Here, the image determination unit 15 compares the contents of the endoscopic images obtained by continuous imaging in time, and determines the similar image group that observes the same part within a predetermined number of frames even if it includes non-similar images. Figure 6 ), in this embodiment, an image for relearning is selected from the images in the similar image group Is. In this step, it is determined whether the input image is a similar image. Specifically, when the image data of one frame is output from the image processing unit 12, the image determination unit 15 compares the feature amount of the previous frame and the current frame to determine whether it is a similar image. In addition, parameters other than the feature amount can also be used to determine whether it belongs to the similar image group.

[0123] As described above, the images of the endoscope are continuous, and even if the same part is detected, they are easy to change. Through simple pattern matching and similarity determination when the images are digitized, it may be determined that they are not similar images. That is, they may be determined as non-similar images with different contents. However, when similar images are obtained continuously in time, even if they are simply set as non-similar images, there is a high possibility that important images are included. Therefore, here, non-similar images that meet the conditions are processed as images constituting a group of similar images. That is, in the acquisition of training data candidate images for acquiring endoscopic images for learning inference models for endoscopes, the contents of endoscopic images obtained by continuous photography in time are compared, and for similar image groups observing the same part within a specified number of frames, they are determined as similar image groups in a manner that non-similar images are included. The content of the image refers to the content (image feature) that makes the image characteristic, such as the shape, pattern, shadow, color, size, position (rotation, deformation, etc., but these can also be corrected) of the object of the image obtained by photography.

[0124] As described above, when determining a similar image group, sometimes non-similar images are also included in the similar image group. Figure 4 This point will be explained. Figure 4 The following are examples of images acquired by the camera unit 11. Images P21, P22, and P29 are Figure 6 The transition images of the same endoscope device 10 as (a) and (b) when plugging and unplugging are shown. Images P23 and P24 are intermediate images. Images P25 and P28 are images when the endoscope tip is facing a specific target part such as an affected part. In images P25 and P28, the specific target part such as the affected part is clearly captured, and the reliability of the inference is also high.

[0125] Images P26 and P27 between images P25 and P28 are images equivalent to saccades (unconscious eye movements), which appear as slight image movements. Saccades are unintentional changes in the visual field, and image changes equivalent to saccades refer to sudden changes in the image depending on the positional relationship between an object at a very close distance and the light source and the object. Figure 6As described in the description of , the image change equivalent to a glance is related to the particularity of the inspection using an endoscope or the like. Thus, Figures P26 and P27 are images equivalent to a glance, which become unintended images, unexpected images, and sometimes take various forms of changes. Therefore, images equivalent to glances are sometimes not necessarily included in the training data during learning, and it is easy to become difficult to reason. Here, it is manifested as a change in quality. Therefore, images P26 and P28 are easily judged to be dissimilar to images P25 and P26. However, even if there is an image with quality change, as long as the image quality is not extremely deteriorated, a specific object such as an affected part is captured, which may be an object that you hope not to miss according to the specifications of the inference model. Therefore, in this embodiment, non-similar images in the similar image group are also processed as a similar image group. The detailed actions for similar image group determination in step S3 are used Figure 3 This will be described later.

[0126] After the similar image group determination is performed, it is then determined whether there is a difference in the inference result within the image group (S5). As described above, the image acquired by the camera unit 11 and processed by the image processing unit 12 is inferred by the inference unit 13. In this step, it is determined whether there is a difference in the inference result within the similar image group. That is, the difference determination unit 14 calculates the difference in the inference result of the inference unit for each image included in the similar image group determined by the image determination unit 15.

[0127] In addition, in step S5, reliability can also be used to determine the difference in the inference result. In addition, in addition to reliability, for example, when the detection range and position of a specific object such as an affected part are deviated, it can also be determined that there is a difference in the inference result. In addition, when a specific object such as an affected part appears or does not appear, or the appearance state changes, it can also be determined that there is a difference in the inference result. Furthermore, it can also be determined by how much the difference in the value of reliability changes. In addition, even if the image changes suddenly, the inference result will not be disordered, just as the visual image is corrected in the brain even if there is an unconscious movement of the eyes of a person, that is, scanning, so that it can become an inference model that is easy to use and friendly to people who use an endoscopic device for inspection. Regarding this sudden change of the image, there are many parameters that become the cause, and it may occur in a complex manner. Therefore, when machine learning, it is difficult to prepare all the patterns in most cases.

[0128] In addition, as factors that may cause reliability to change, there are the following situations (1) to (5). That is, there are situations where (1) structural emphasis processing that affects image quality is performed, (2) image processing such as TXI (Texture and Color Enhancement Imaging) is changed, (3) the light source is changed in NBI (Narrow Band Imaging) and RDI (Red dichromatic Imaging), (4) treatment instruments such as forceps are removed and inserted, and (5) magnified observation is performed (cap pressing, immersion method shooting with the hood filled with water). In addition, in addition to simply judging reliability as high (0) or low (×), the reliability judgment method can also be changed "according to the mode."

[0129] When the judgment result in step S5 is that there is a difference in the inference result within the image group, a learning image is selected (S7). In this step, the control unit 20 selects an image to be used in learning (which may also include relearning) based on the difference in the inference result calculated by the difference judgment unit 14. In other words, the control unit 20 selects a candidate image of training data to be used in learning based on the amount of change calculated by the difference judgment unit 14. As described above, in the case where the images in the similar image group have a difference in the inference result, that is, when it is judged to be low in reliability, a specific object such as an affected part may also be photographed. Therefore, the control unit 20 selects such an image for relearning. Of course, the selected image may include an image in which a specific object such as an affected part is not photographed. However, such an inappropriate image can be excluded when annotating in order to produce training data. Compared with excluding images, by selecting difficult images and missed images for relearning, a better inference model can be generated.

[0130] The selected image itself or the information that can be used to retrieve the selected image is organized in a manner that can use these images and information to immediately improve the inference model, recorded in the recording unit 18, and sent to the learning device 30 through the communication unit 19. In addition, whenever an image is selected, it can be sent to the learning device 30 through the communication unit 19, recorded in the recording unit 33, and re-learning is performed at the time when a specified number of images for re-learning are collected to generate an inference model. In addition, when selecting learning images in step S7, images with extremely poor image quality can be removed. Images that are not suitable for learning can also be defined as having poor image quality. In addition, in the case where visual recognizability is not improved even if image processing is performed, in the case where there is no difference in the level of the pixel signal of the two-dimensional configuration (for example, the entire screen is all white or all black) and cannot be recognized as an image, etc., in the case where the image is too poor, it is not suitable for learning.

[0131] After the learning image is selected in step S7, or the judgment result in step S5 is that there is no difference in the inference result within the image group, it is then determined whether to end (S9). When the physician or the like performs the end operation of the endoscopic examination, it is determined to be ended. If the judgment result is not the end, the process returns to step S1. On the other hand, if it is the end, the process ends after the end processing is performed.

[0132] Next, use Figure 3 The flowchart shown in FIG. 1 is for step S3 (refer to Figure 2 ) is used to illustrate the action of determining similar image groups.

[0133] When the process of similar image group determination begins, first, the image features are temporarily recorded (S11). Here, the image processing unit 16 or the image determination unit 15 calculates the feature quantity of the image obtained by the camera unit 11. As described above, the feature quantity of the image can be calculated by a known method. Here, the calculated image feature quantity is temporarily recorded in a memory provided inside the recording unit 18, the control unit 20, etc. The history of the image feature quantity is temporarily recorded in the memory, that is, the image feature quantity is temporarily recorded in a time sequence.

[0134] After temporarily recording the image feature, it is then determined whether the image feature is not similar to the previous frame (S13). Here, the difference determination unit 14 compares the image feature of the latest image frame with the image feature of the previous image frame based on the image feature temporarily recorded in step S11 to determine whether the image features are similar. In this case, if the image feature of the two images is outside the specified range, they are determined to be not similar.

[0135] If the result of the determination in step S13 is that the image feature is not similar to the previous frame, then it is determined whether the feature is similar to the image before the previous frame (S15). Here, the difference determination unit 14 compares the image feature of the latest image frame with the image feature of the image before the previous frame based on the image feature temporarily recorded in step S11, and determines whether the feature is similar to the image before the previous frame. As the image before the previous frame, how many frames ago to compare can be determined according to the design concept, and can also be appropriately changed according to the state of the image.

[0136] In step S15, the number of frames before the previous frame can be any number of frames corresponding to the time when a doctor or other expert performs an endoscopic examination or the like, when a specific object may be missed, or when a difficult image may be generated. For example, the number of frames corresponding to the time until hand tremor converges, the time until the microvibration of the front end converges, and the time until the jitter of the front end during the endoscope operation converges, or the number of frames corresponding to the time until the operator's hand jitter converges. In addition, in the images obtained during a specific treatment period, sometimes only the image of the treatment looks different. Such images can be processed as the same similar image group. If the type of treatment is known, it can be detected, and the "number of frames immediately before" here is determined based on the time (average time, etc.) spent on the treatment, and the image before the previous frame is determined. Similarly, if it is possible to determine the situation of bleeding accompanying the treatment, the situation of water vapor generation, etc., the number of frames before the previous frame can be determined based on the situation.

[0137] When the judgment result in step S13 is that the features of the image of the latest frame and the image of the previous frame are similar, or when the judgment result in step S15 is that the features of the image of the latest frame and the images before the previous frame are similar, it is set as a similar image (S17). The judgment results of steps S13 and S15 are that the latest image is similar to the previous frame or the frame before the previous frame, so it is set as a similar image. It can be set as a similar image by pattern matching and similarity determination when the image is digitized. However, in the judgment based on pattern matching, etc., it may be determined that it is not a similar image (non-similar), but when similar images are obtained continuously in time, even if it is simply set as a non-similar image, the possibility of containing an important image is high. Therefore, as described above, non-similar images that meet the conditions are processed as images that constitute a group of similar images.

[0138] If the result of determination in step S15 is that the features are not similar to the images before the previous frame, or if the images are set as similar image groups in step S17, the process of determining the similar image group ends and returns to the original process. In the process of determining the similar image group, even if the result of determination of the feature amount of the image is not similar, if the features are similar to the frame images before the previous frame, the similar image group is determined. That is, the similar image group sometimes includes non-similar images.

[0139] Thus, in one embodiment of the present invention, images are acquired by continuously photographing in time, and the machine learning model is used to infer the acquired images (S1). In addition, the content of the acquired images is digitized, the digits are compared, and the images whose changes are below or below a specified value are determined as a similar image group (S3). The difference between the inference results of each image included in the determined similar image group is calculated to determine whether there is a difference (S5). The image used in learning is selected based on the difference in the inference result (S7). Thus, in this embodiment, candidate images for training data for learning an inference model can be efficiently acquired. That is, in a similar image group, the inference results are sometimes different, such as images with high inference results and images with low inference results. However, when the inference result is low, the input image may be a tricky image for the learning device, and therefore, such an image is selected as a candidate image for training data. When performing annotation, etc., training data can be selected from the image.

[0140] Next, use Figure 5 The flowchart shown is Figure 2 In this modification, only Figure 2 In the flowchart shown, step S6 is added and step S7 is replaced by step S8, and therefore, these differences will be described.

[0141] exist Figure 5 The process shown starts with Figure 2 In the case of the illustrated flow, imaging and inference are performed ( S1 ), similar image group determination is performed ( S3 ), and it is determined whether there is a difference in inference results within the image group ( S5 ).

[0142] If the result of determination in step S5 is that there is a difference in the inference results within the image group, a status difference determination is performed (S6). Here, the status determination unit 16 performs a difference determination on the usage status of the endoscope device 10, etc. Regarding the information obtained for the status determination, for example, Figure 4 , Figure 6Angle information such as that shown in (a) and (b) (angle information of the wall surface in the body cavity and the front end of the endoscope). The angle information can be determined based on changes in the image, etc., or it can be detected by a built-in sensor. It can also be detected by the uniformity and distribution of the lighting. It is also possible to directly record sensor data, the results of detection of the brightness distribution of the illumination light, etc. In addition, as information used in condition determination, there is focus information in the imaging unit, depth information, light source unit information in the imaging unit 11, treatment information such as water injection operation and suction operation, etc. In the case of inferring a specific object such as an affected part, the reliability of the inference is improved when these condition information is present in addition to the image data. Therefore, when preparing training data, the difference in these conditions is determined, and the information association unit 17 is able to associate the condition difference information with the image data, and use the associated image data for re-learning.

[0143] After the situation difference is determined, a learning image is then selected (S8). In this step, the control unit 20 selects an image to be used in learning (which may also include relearning) based on the difference in the inference result calculated by the difference determination unit 14. That is, similar to step S7, in step S5, an image is selected from the similar image group for which the inference result is determined to be different. In this case, the situation difference information obtained in step S7 is organized, that is, recorded in association with the selected image. Based on the recorded information, customized learning can be performed for each situation.

[0144] After the learning image is selected in step S8, or the judgment result in step S5 is that there is no difference in the inference result within the image group, it is then determined whether to end (S9). When the physician or the like performs an end operation of the endoscopic examination, it is determined to be ended. If the judgment result is not the end, the process returns to step S1. On the other hand, if it is the end, the process ends after the end processing is performed.

[0145] Thus, in a modified example of an embodiment of the present invention, a state difference determination is performed (see S6), and the state difference determination information is obtained and associated with the image data (see S8). If the image data is used for relearning, it is possible to perform inference corresponding to the state change during the endoscopic examination, which can improve the reliability of the inference.

[0146] Next, use Figure 7 , the data flow in one embodiment of the present invention is centered for explanation. Figure 7 , the data flow between the imaging unit 41, the developing unit 42, the image recognition unit 43, the inference unit 44, the inference result change calculation unit 45, the learning image selection unit 46, and the recording unit 47 is shown. As described later, these units correspond to Figure 1Each part of the present invention may realize all or part of its functions through a processor, or may be realized through hardware and software.

[0147] The camera unit 41 corresponds to Figure 1 The imaging unit 11 obtains an image of a specific object such as an affected part using RAW data and outputs it to the developing unit 42. As described above, the image data output from the imaging unit 11 is continuous in time, and when the latest RAW data 2 is output, the RAW data 1 immediately before it is an image pair, and further, when the next RAW data 3 is output after the latest RAW data 2 is output, the RAW data 2 and the RAW data 3 are an image pair. That is, images continuous in time are set as a pair.

[0148] The developing section 42 performs development processing on the RAW data output from the imaging section 41. Figure 1 In the example, the image processing unit 12 performs the function of the developing unit 42. The image data after the developing process is output to the image recognition unit 43. The image recognition unit 43 determines whether there is a large change between the image pairs, and extracts one or more image pairs that are determined (recognized) as having no large change. The extracted image pairs are output to the inference unit 44. Figure 1 In the example, the image determination unit 15 functions as the image recognition unit 43 .

[0149] The image recognition unit 43 functions as an image recognition unit that determines whether the image is substantially unchanged due to the continuous input in time (for example, see Figure 2 and Figure 5 S3 Figure 3 , Figure 1 Here, regarding the input pair of images, for example, when a plurality of images such as the first image, the second image, the third image, etc. are sequentially input to the image recognition unit, the first image and the second image become a pair of images, and the second image and the third image become a pair of images. In addition, the image recognition unit determines that the input pair of images is substantially unchanged based on at least one of the movement amount of the corresponding point and the change amount of the brightness, saturation, and contrast of the image.

[0150] The inference unit 44 inputs the image pair from the image recognition unit 43, performs inference on each image, and outputs the inference result of one pair (inference result pair) to the inference result change calculation unit 45. When outputting the inference result, the reliability of the inference result is also output to the inference result change calculation unit 45. Figure 1 Inference unit 13 performs the function of inference unit 44. Figure 1 In the example, the inference unit 13 does not directly input the image pair from the image recognition unit 43 , but may receive information that the image pair is from the control unit 20 .

[0151] The inference result change calculation unit 45 calculates the change amount of the inference result and its reliability of the image pair based on the inference result pair output from the inference unit 44. That is, the inference result in the latest image is compared with the inference result in the previous image, and the change amount is calculated. In addition, the reliability of the two inference results is compared and the change amount is calculated. The calculated change amount is output to the learning image selection unit 46. Figure 1 In the example, the difference determination unit 14 functions as the inference result change calculation unit 45 .

[0152] The inference result variation calculation unit 45 functions as an inference result variation calculation unit that calculates the variation amount of the inference result of the recognized image pair (for example, see Figure 2 and Figure 5 S5, Figure 1 The identified image pair refers to, for example, two images (image pair) inputted continuously in time and having substantially no change. The inference result change calculation unit uses the differential value of at least one of the inference result and the reliability (for example, reference Figure 2 and Figure 5 S5, Figure 1 The inference result change calculation unit 45 functions as an inference result change calculation unit that calculates the amount of change in the inference result of a temporally continuous image pair (for example, see Figure 2 and Figure 5 S5, Figure 1 The difference determination unit 14).

[0153] In addition, in the Figure 2 In step S5, it is determined whether there is a difference in the inference results within the image group. This process is equivalent to the inference result change calculation unit 45 calculating the change amount of the inference results of the recognized image pair. As the difference in the inference results, for example, when the value of the reliability of the inference is different by more than a predetermined value, it can be determined that there is a difference in the inference results. In other words, it can also be said that the inference result change output unit 45 calculates the change amount of the inference results of the image pair that is continuous in time.

[0154] When the amount of change calculated by the inference result change calculation unit 45 is large, the learning image selection unit 46 selects the image pair as an image to be used in learning (learning image candidate) and outputs it to the recording unit 47 as the selected learning image candidate. When the amount of change of the inference result (or reliability) is large, it is likely that the image is not good at machine learning, so it is selected as a learning candidate image. In fact, when preparing training data, experts such as doctors can determine whether to use it in learning. In addition, Figure 1 In the example shown in FIG. 4 , the control unit 20 functions as the learning image selection unit 46 .

[0155] The recording unit 47 is an electrically rewritable nonvolatile memory, and sequentially records the candidate learning images input from the learning image selection unit 46. Training data for learning is created from the candidate learning images recorded in the recording unit 47, and an inference model is generated by machine learning or the like. As described above, in the case of an image that is not suitable for learning, the image can be excluded, and training data can be created using an appropriate image. Figure 1 In the example, the recording unit 18 and / or the recording unit 33 functions as the recording unit 47 .

[0156] In this way, Figure 7 As shown, in this embodiment, the images obtained from the camera unit 41 that are continuous in time are set as a pair, and the image recognition unit 43 determines that the image pair is the image pair, and the learning candidate is selected based on the change amount calculated by the inference result conversion calculation unit 45. Figure 7 In the description of , two images that are continuous in time are described. However, for example, regarding images A, B, and C that are continuous in time, when the inference results of each are A=C, but A≠B and B≠C, only image B may be selected (extracted), or three images A, B, and C may be selected. That is, when the amount of change in the inference result of the image pair is greater than a specified value, both sides of the image pair may be selected, or an image may be selected based on the comparison result of the image pair with the previous and next pairs.

[0157] As described above, a training data candidate image acquisition device according to one embodiment of the present invention and its variant example acquires training data candidate images based on images acquired by an endoscope for learning an inference model for an endoscope. In addition, the training data candidate image acquisition device includes a similar image group determination unit (e.g., referring to Figure 1 The image determination unit 15, Figure 2 and Figure 5 S3 Figure 3 , Figure 7 The similar image group determination unit compares the contents of the endoscopic images obtained by continuous imaging in time, and determines the similar image group that observes the same part within a predetermined number of frames in a manner that includes non-similar images. In addition, the training data candidate image acquisition device includes: an inference unit (for example, referring to Figure 1 Reasoning Department 13, Figure 2 and Figure 5 S1 Figure 7 An inference unit 44, etc.) that uses a machine learning model to infer a specific object image contained in the endoscopic image obtained by taking a picture; and an inference result difference calculation unit (e.g., referring to Figure 1 The difference determination unit 14, Figure 2 and Figure 5 S5, Figure 7 The inference result variation calculation unit 45, etc., which calculates the difference of the inference result of the inference unit for each image included in the similar image group determined by the similar image group determination unit. In addition, the training data candidate image acquisition device further includes a learning image selection unit (e.g., referring to Figure 1 The control unit 20, Figure 2 S7, Figure 5 S8, Figure 7 The learning image selection unit 46 or the like) selects an image to be used in the relearning based on the difference in the inference result calculated by the inference result difference calculation unit. Therefore, in the present embodiment, medical images that machine learning is not good at recognizing can be reliably selected. That is, in the image group determined to be a similar image group, the image to be used in the relearning is selected based on the difference in the inference result of the inference unit. Therefore, by relearning using a difficult image, the inference model can be improved.

[0158] In addition, a training data candidate image acquisition device according to one embodiment of the present invention and its modified example includes: an inference unit (for example, referring to Figure 1 Reasoning Department 13, Figure 2 and Figure 5 S1 Figure 7 The reasoning unit 44, etc.) uses the machine learning model to reason about the input image; the reasoning result change calculation unit (for example, referring to Figure 1 The image determination unit 15, Figure 2 and Figure 5 S3 Figure 3 , Figure 7 an image recognition unit 43, etc.) which calculates the amount of change in the inference result of the identified image pair; and a learning image selection unit (e.g., referring to Figure 1 The control unit 20, Figure 2 S7, Figure 5 S8, Figure 7 The learning image selection unit 46 of the inference unit selects candidate training data images to be used in learning based on the amount of change calculated by the inference result change calculation unit. Therefore, in this embodiment, medical images that machine learning is not good at recognizing can be reliably selected. That is, between images that are determined to be recognized image pairs, images to be used in learning are selected based on the difference in the inference results of the inference unit. Therefore, by relearning using tricky images, the inference model can be improved.

[0159] In addition, a training data candidate image acquisition device according to one embodiment of the present invention and its modified example includes: an inference unit (for example, referring to Figure 1 Reasoning Department 13, Figure 2 and Figure 5 S1 Figure 7The reasoning unit 44, etc.) uses the machine learning model to reason about the input image; the reasoning result change calculation unit (for example, referring to Figure 1 The image determination unit 15, Figure 2 and Figure 5 S3 Figure 3 , Figure 7 an image recognition unit 43, etc.) which calculates the amount of change in the inference result of a temporally continuous image pair; and a learning image selection unit (e.g., referring to Figure 1 The control unit 20, Figure 2 S7, Figure 5 S8, Figure 7 The learning image selection unit 46 of the inference unit selects candidate training data images to be used in learning based on the amount of change calculated by the inference result change calculation unit. Therefore, in this embodiment, medical images that machine learning is not good at recognizing can be reliably selected. That is, between images that are determined to be a temporally continuous image pair, an image to be used in learning is selected based on the difference in the inference results of the inference unit. Therefore, by relearning using difficult images, the inference model can be improved.

[0160] In addition, in one embodiment of the present invention and its modified example, the description is given on the premise that the image processing unit 12, the inference unit 13, the difference determination unit 14, the image determination unit 15, the condition determination unit 16, the information association unit 17, the recording unit 18, etc. are provided in addition to the camera unit 11 in the endoscope device 10. However, other structures, such as a display unit, etc., may also be provided in the endoscope device 10. In addition, it is not necessary to provide all blocks in the endoscope device 10, and they may be appropriately dispersed. In this case, data communication can be performed between the devices through a communication unit, etc. In addition, external blocks such as the learning unit 34 in the learning device 30 may also be configured.

[0161] In one embodiment of the present invention and its variant, the control units 20 and 35 are described as devices composed of a CPU, a memory, etc. However, in addition to being constituted by a CPU and a program in a software manner, a part or all of each unit may be constituted by a hardware circuit, and a hardware structure such as a gate circuit generated by a program language described by Verilog, VHDL (Verilog Hardware Description Language), etc. may be used. In addition, a hardware structure using software such as a DSP (Digital Signal Processor) may be used. Of course, these may be appropriately combined.

[0162] In addition, the control units 20 and 35 are not limited to the CPU, and any element that functions as a controller can be used. The processing of the above-mentioned parts can also be performed by one or more processors configured as hardware. For example, each part can be a processor configured as an electronic circuit, or each circuit part in a processor configured by an integrated circuit such as an FPGA (Field Programmable Gate Array). Alternatively, a processor configured by one or more CPUs reads and executes a computer program recorded in a recording medium, thereby also performing the functions of each part.

[0163] In one embodiment and its variant of the present invention, the endoscope device 10 and the learning device 30 are described as having blocks for performing respective functions. However, they do not need to be provided in an integrated device, and for example, the above-mentioned parts may be dispersed if connected via a communication network such as the Internet.

[0164] In addition, when describing one embodiment of the present invention and its variant, in order to easily describe the inspection scene of the endoscope, the description is based on the endoscope. However, when observing using the image data of the object, as long as it is a device that performs some kind of reasoning, it can be widely used. Even a built-in camera or a civilian camera in a portable terminal is sometimes used to make judgments as a medical device in recent years. In this case, due to the instability of holding, a situation described as a scanning image micro-motion may occur. In addition, in cameras mounted on cars, drones, robots, etc., when the positional relationship with the object is unstable, the same image micro-motion is likely to occur. In these devices, the invention of the present application can be applied.

[0165] In addition, in each embodiment of the present invention, the judgment based on logic is mainly described, and a part of the judgment is made by reasoning using machine learning. As for whether to make a judgment based on logic or a judgment based on reasoning, in this embodiment, either one is appropriately selected for use. In addition, in the judgment process, a hybrid judgment can also be made by partially utilizing the advantages of each.

[0166] In addition, in recent years, artificial intelligence that can uniformly determine various judgment criteria is often used, and uniformly improving the branches of the flowchart shown here is of course also within the scope of the present invention. For such control, if the user can input good or bad, the user's preferences are learned, and the embodiment shown in this application can be customized in a direction suitable for the user.

[0167] In addition, the control mainly described by the flowchart in the technology described in this specification can be set by the program in most cases, and sometimes it is saved in the recording medium or recording unit. Regarding the recording method for the recording medium or recording unit, it can be recorded when the product leaves the factory, or it can be recorded using the released recording medium, or it can be downloaded through the Internet.

[0168] In addition, in one embodiment of the present invention, a flowchart is used to illustrate the actions in this embodiment, but the order of the processing steps can be changed, and any steps can be omitted and added. Furthermore, the specific processing content in each step can be changed.

[0169] Furthermore, even if the process flow in the claims, the specification, and the drawings is described using words expressing a sequence such as “first” or “next” for the sake of convenience, it does not necessarily mean that the process must be performed in this order unless otherwise specified.

[0170] The present invention is not limited to the above-mentioned embodiments, and the structural elements can be deformed and concretized in the implementation stage without departing from the scope of its purpose. In addition, various inventions can be formed by appropriate combinations of multiple structural elements disclosed in the above-mentioned embodiments. For example, some structural elements of all the structural elements shown in the embodiments can be deleted. Furthermore, the structural elements of different embodiments can also be appropriately combined.

[0171] Description of Reference Numerals

[0172] 10: Endoscope device; 11: Camera unit; 12: Image processing unit; 13: Inference unit; 14: Difference determination unit; 15: Image determination unit; 16: Status determination unit; 17: Information association unit; 18: Recording unit; 19: Communication unit; 20: Control unit; 30: Learning device; 31: Communication unit; 32: Image processing unit; 33: Recording unit; 34: Learning unit; 35: Control unit.

Claims

1. An endoscope, characterized in that: The endoscope has: a similar image group determination unit that compares the contents of endoscopic images obtained by continuous imaging in time and determines a similar image group that observes the same part within a predetermined number of frames in a manner that includes non-similar images; an inference unit that uses machine learning to infer a specific object image included in the captured endoscopic image; an inference result difference calculation unit that calculates a difference in the inference result of the inference unit for each image included in the similar image group determined by the similar image group determination unit; as well as The learning image selection unit selects an image to be used for learning based on the difference of the inference result calculated by the inference result difference calculation unit.

2. The endoscope according to claim 1, characterized in that The similar image group includes a result of determining an image group corresponding to a sudden viewpoint position shift based on similarity between images before and after the endoscopic image captured continuously in time.

3. The endoscope according to claim 1, characterized in that The endoscope includes an imaging information acquisition unit capable of acquiring imaging information corresponding to each image selected by the learning image selection unit.

4. The endoscope according to claim 1, characterized in that The similar image group determination unit converts the content of the image into a numerical value capable of tracking a pattern of the same object in the endoscopic image, and determines whether the image is similar using the numerical value.

5. The endoscope according to claim 1, characterized in that The learning image selection unit removes images with extremely poor conditions.

6. A training data candidate image acquisition device for acquiring images for inference model learning, characterized in that: The training data candidate image acquisition device comprises: a similar image group determination unit that compares the contents of images captured continuously in time and determines a similar image group that observes the same part within a predetermined number of frames so as to include non-similar images; an inference unit that uses machine learning to infer an image of a specific object included in the captured image; an inference result difference calculation unit that calculates a difference in the inference result of the inference unit for each image included in the similar image group determined by the similar image group determination unit; as well as The learning image selection unit selects an image to be used for learning based on the difference of the inference result calculated by the inference result difference calculation unit.

7. A method for obtaining candidate images for training data, which is used to obtain images for learning an inference model, characterized in that: Comparing the contents of images captured continuously in time, determining a group of similar images that observe the same part within a predetermined number of frames even if they include dissimilar images, Use machine learning models to infer specific object images contained in captured images. For each image determined to be included in the similar image group, calculating the difference in inference results inferred using the machine learning model, An image to be used in learning is selected based on the calculated difference in the inference result.

8. A program, characterized in that The program causes a computer that obtains candidate images of training data for learning an inference model to execute the following processing: Comparing the contents of images captured continuously in time, determining a group of similar images that observe the same part within a predetermined number of frames even if they include dissimilar images, Use machine learning models to infer specific object images contained in captured images. For each image determined to be included in the similar image group, calculating the difference in inference results inferred using the machine learning model, An image to be used in learning is selected based on the calculated difference in the inference result.

9. A training data candidate image acquisition device, characterized in that: The training data candidate image acquisition device comprises: The inference unit uses the machine learning model to perform inference on the input image; an inference result change calculation unit that calculates a change amount of the inference result of the recognized image pair; as well as A learning image selection unit selects a training data candidate image to be used in learning, based on the amount of change calculated by the inference result change calculation unit.

10. The training data candidate image acquisition device according to claim 9, characterized in that: The training data candidate image acquisition device includes an image recognition unit configured to determine that there is substantially no change in an image due to temporally continuous input.

11. The training data candidate image acquisition device according to claim 10, characterized in that: The image recognition unit determines that the input has substantially no effect on the image based on at least one of an amount of movement of the corresponding point and an amount of change in brightness, saturation, and contrast of the image.

12. The training data candidate image acquisition device according to claim 9, characterized in that: The input image is a medical image.

13. The training data candidate image acquisition device according to claim 9, characterized in that: The inference unit performs inference on at least one of classification, detection, and region extraction of the input image.

14. The training data candidate image acquisition device according to claim 9, characterized in that: The inference unit outputs the reliability of the inference.

15. The training data candidate image acquisition device according to claim 9, characterized in that: The inference result change calculation unit uses a differential value of at least one of the inference result and the reliability.

16. The training data candidate image acquisition device according to claim 9, characterized in that: The learning image selection unit selects a training data candidate image to be used in the learning when the amount of change exceeds a predetermined value.

17. A method for obtaining candidate images for training data, characterized in that: Use a machine learning model to perform inference on the input image, Calculate the change in the inference result of the identified image pair, Based on the calculated amount of change, a candidate image of training data to be used in learning is selected.

18. A program, characterized in that The program causes a computer that obtains candidate images of training data for learning an inference model to execute the following processing: Use a machine learning model to perform inference on the input image, Calculate the change in the inference result of the identified image pair, Based on the calculated amount of change, a candidate image of training data to be used in learning is selected.

19. A training data candidate image acquisition device, characterized in that: The training data candidate image acquisition device comprises: The inference unit uses the machine learning model to perform inference on the input image; an inference result variation calculation unit that calculates a variation of an inference result of a temporally continuous image pair; as well as A learning image selection unit selects a training data candidate image to be used in learning, based on the amount of change calculated by the inference result change calculation unit.

20. The training data candidate image acquisition device according to claim 19, characterized in that: The input image is a medical image.

21. The training data candidate image acquisition device according to claim 19, characterized in that: The inference unit performs inference on at least one of classification, detection, and region extraction of the input image.

22. The training data candidate image acquisition device according to claim 19, characterized in that: The inference unit outputs the reliability of the inference.

23. The training data candidate image acquisition device according to claim 19, characterized in that: The inference result change calculation unit uses a differential value of at least one of the inference result and the reliability.

24. The training data candidate image acquisition device according to claim 19, characterized in that: The learning image selection unit selects a training data candidate image to be used in the learning when the amount of change exceeds a predetermined value.

25. A method for obtaining candidate images for training data, characterized in that: Use a machine learning model to perform inference on the input image, Calculate the change in the inference results of temporally consecutive image pairs, Based on the calculated amount of change, a candidate image of training data to be used in learning is selected.

26. A program, characterized in that The program causes a computer that obtains candidate images of training data for learning an inference model to execute the following processing: Use a machine learning model to perform inference on the input image, Calculate the change in the inference results of temporally consecutive image pairs, Based on the calculated amount of change, a candidate image of training data to be used in learning is selected.