Image recording system, image recording method and recording medium

Abstract

The image recording system (1) includes an acquisition section (110), a recording candidate RAW image group extraction section (120), and a RAW image recording section (130). The acquisition section acquires a time-series RAW image group including a plurality of RAW images along a time series in a first time interval. The recording candidate RAW image group extraction section extracts a recording candidate RAW image group from the time-series RAW image group, the recording candidate RAW image group being included in a second time interval that is a part of the first time interval. The RAW image recording section records at least one RAW image included in the recording candidate RAW image group as a RAW image to be recorded, that is, a recording target RAW image.

Description

Technical Field

[0001] This invention relates to image recording systems, image recording methods, and recording media. Background Technology

[0002] Techniques for detecting lesions from medical images using image recognition through machine learning are known. Patent Document 1 discloses a technique that uses a learning image of a diagnostic subject and information representing the abnormality of the diagnostic subject pre-assigned to the learning image to perform machine learning, thereby generating a neural network that identifies abnormal regions in endoscopic images.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2018 / 008593 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] Image processing of medical images is performed based on the default settings of the medical device or the user's settings. Therefore, the learned model obtained through machine learning is affected by the settings of the medical device when generating the medical images used as training images. Therefore, a method is considered where RAW images, as training images, are acquired as unprocessed images while maintaining the signal from the imaging element. However, when RAW images are used as training images, there is a problem that the amount of accumulated RAW image data becomes very large.

[0008] Methods for solving problems

[0009] One aspect of this disclosure relates to an image recording system comprising: an acquisition unit that acquires a time-series RAW image set, the time-series RAW image set comprising a plurality of RAW images along a time sequence in a first time interval; a recording candidate RAW image set extraction unit that extracts a recording candidate RAW image set from the time-series RAW image set, the recording candidate RAW image set being contained in a second time interval that is part of the first time interval; and a RAW image recording unit that records at least one RAW image contained in the recording candidate RAW image set as the RAW image to be recorded, i.e., the recording target RAW image.

[0010] Other aspects of this disclosure relate to an image recording method, comprising the following steps: obtaining a time-series RAW image group, the time-series RAW image group comprising a plurality of RAW images along a time sequence in a first time interval; extracting a recording candidate RAW image group from the time-series RAW image group, the recording candidate RAW image group comprising a second time interval as part of the first time interval; and recording at least one RAW image contained in the recording candidate RAW image group as the RAW image to be recorded, i.e., the recording target RAW image. Attached Figure Description

[0011] Figure 1 This is an example of the structure of a shooting system.

[0012] Figure 2 It is the first structural example of an image recording system.

[0013] Figure 3 This diagram illustrates the processing performed by the processing unit in the first structural example.

[0014] Figure 4 This is a flowchart of the processing performed by the processing unit in the first structural example.

[0015] Figure 5 This diagram illustrates the processing performed by the processing unit in the first structural example.

[0016] Figure 6 This is a flowchart of the processing performed by the processing unit in the second structural example.

[0017] Figure 7 This is the third structural example of an image recording system.

[0018] Figure 8 This is the fourth structural example of an image recording system.

[0019] Figure 9 This diagram illustrates the processing performed by the processing unit in the fourth structural example. Detailed Implementation

[0020] The following describes this embodiment. Furthermore, the embodiments described below do not unduly limit the scope of the claims. Moreover, the structures described in this embodiment are not all essential structural elements of the present invention.

[0021] 1. Camera system

[0022] Figure 1 This is a structural example of an imaging system 10 for capturing and recording medical images. The imaging system 10 includes an endoscope system 5 and an image recording system 1.

[0023] Endoscopic system 5 is a system for capturing medical images. Medical images are images of the inside of the body captured by medical endoscopes, such as video endoscopes for the digestive tract or rigid endoscopes for surgery. Figure 1 The diagram illustrates a video endoscope as an example. The endoscope system 5 includes an endoscope body 2, a control device 3, and a monitor 4.

[0024] The endoscope 2 is inserted into the body cavity of the subject by a slender insertion part 201 to capture images of the subject's internal cavity and outputs an image signal as a medical image of the internal cavity. The endoscope 2 includes the insertion part 201, a holding part 204 disposed at the base end of the insertion part 201, and a universal cable 203 extending from the holding part 204. A connector is provided at the base end of the universal cable 203, which is connected to the control device 3. An imaging module 202 is provided at the front end of the insertion part 201, and the imaging module 202 outputs a RAW image signal as an image signal of the medical image. The imaging module 202 captures multiple medical images in a time sequence by means of dynamic image capture, etc. In the case of dynamic image capture, each frame of the dynamic image corresponds to a medical image.

[0025] The control device 3 controls the entire endoscope system 5 and processes the image signal output from the imaging module 202, outputting the processing result. Specifically, the control device 3 generates a color image by developing the RAW image and displays the color image on the monitor 4. Additionally, the control device 3 outputs multiple RAW images captured in a time sequence to the image recording system 1. The image recording system 1 records a portion of the multiple RAW images sent from the endoscope system 5 as learning images. Details of the image recording system 1 will be described later. Furthermore, in Figure 1 In this system, the image recording system 1 becomes an external system of the control device 3, but the image recording system 1 can also be combined with the control device 3.

[0026] RAW images are images before they are developed into color images; they directly record the pixel values ​​corresponding to each pixel of the image sensor. For example, in the case of a primary-color Bayer type image sensor, the image sensor is configured with R pixels, G pixels, and B pixels, but in a RAW image, each pixel value is recorded as is, regardless of its color. A color image is an image obtained by interpolating and compressing the RAW image according to the Bayer type configuration, with each pixel having an RGB pixel value. Because RAW images are unprocessed, uncompressed data, each frame has a larger data size than a color image.

[0027] Color images undergo image processing suitable for endoscopic observation. Since this image processing is not necessarily suitable for machine learning, when color images are used as training images, machine learning is performed under the influence of the image processing already applied to them. On the other hand, RAW images, being undeveloped images, possess a high degree of flexibility in image processing. Therefore, by accumulating RAW images as training images, it is possible to process these RAW images into images suitable for machine learning and then use them for machine learning. This can lead to an improvement in image recognition accuracy. Based on the above reasons, there is a high demand for RAW image data as training data for image recognition utilizing machine learning, but the sheer volume of RAW image data presents a significant challenge.

[0028] 2. Example of the first structure of an image recording system

[0029] Figure 2 This is a first structural example of image recording system 1. Furthermore, in Figure 2 The diagram also shows a training data generation system 400 and a learning device 500, but when the image recording system 1 records images, the training data generation system 400 and the learning device 500 do not need to be connected to the image recording system 1.

[0030] Image recording system 1 includes an acquisition unit 110, a candidate RAW image group extraction unit 120, and a RAW image recording unit 130. For example... Figure 3 As shown, the acquisition unit 110 acquires a temporal RAW image group GIM1 containing multiple RAW images along a time sequence within a first time interval TP1. The recording candidate RAW image group extraction unit 120 extracts a recording candidate RAW image group GIM2 contained in a second time interval TP2, which is part of the first time interval TP1, from the temporal RAW image group GIM1. The RAW image recording unit 130 records at least one RAW image contained in the recording candidate RAW image group GIM2 as the recording target RAW image to be recorded.

[0031] According to this embodiment, only a portion of the RAW images in the temporal RAW image group GIM1 acquired by the acquisition unit 110 is recorded. Therefore, compared to recording the entire temporal RAW image group GIM1, the amount of recorded data can be reduced. Thus, both reducing the amount of recorded data for learning images and acquiring RAW images for learning can be achieved. Furthermore, according to this embodiment, the candidate RAW image group GIM2 contained in the second time interval TP2 is extracted. Therefore, compared to recording only one RAW image at a specific timing, multiple RAW images with a certain temporal width become recording candidates, thus enabling the recording of RAW images suitable for machine learning.

[0032] The following is a detailed description of the first structural example. The image recording system 1 includes a processing unit 100 and a storage unit 200. The processing unit 100 includes an acquisition unit 110, a recording candidate RAW image group extraction unit 120, a RAW image recording unit 130, and a RAW image recording importance determination unit 140.

[0033] Image recording system 1 may be, for example, an information processing device such as a PC (Personal Computer). Alternatively, image recording system 1 may be a system in which a terminal device and an information processing device are connected via a network. For example, the terminal device may include a storage unit 200, and the information processing device may include a processing unit 100. Alternatively, image recording system 1 may be a cloud system in which multiple information processing devices are connected via a network.

[0034] The storage unit 200 stores the timing RAW image group GIM1 acquired by the acquisition unit 110. The storage unit 200 is a storage device such as a semiconductor memory or a hard disk drive. The semiconductor memory is, for example, a volatile memory such as RAM, or a non-volatile memory such as EEPROM.

[0035] The processing unit 100 is a processor. The processor can also be an integrated circuit device such as a CPU, microcomputer, DSP, ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array). The processing unit 100 may contain one or more processors. Alternatively, the processing unit 100 as a processor can be, for example, a processing circuit or processing device composed of one or more circuit components, or it can be a circuit device with one or more circuit components mounted on a substrate.

[0036] The operation of the processing unit 100 can also be implemented through software processing. That is, the storage unit 200 stores a program describing all or part of the operations of the acquisition unit 110, the candidate RAW image group extraction unit 120, the RAW image recording unit 130, and the RAW image recording importance determination unit 140 included in the processing unit 100. The processor executes the program stored in the storage unit 200, thereby implementing the operation of the processing unit 100. The program can be stored in an information storage medium that is a computer-readable medium. The information storage medium can be, for example, an optical disc, a memory card, an HDD, or a semiconductor memory. A computer is a device that includes an input device, a processing unit, a storage unit, and an output unit.

[0037] Figure 4This is a flowchart of the processing performed by the processing unit 100 in the first structural example. In step S1, the acquisition unit 110 acquires RAW images. Specifically, the acquisition unit 110 is the data interface of the processing unit 100, receives the time-series RAW image group GIM1 transmitted from the endoscope system 5, and temporarily stores the time-series RAW image group GIM1 in the storage unit 200. Alternatively, the time-series RAW image group GIM1 transmitted from the endoscope system 5 may be temporarily stored in the storage unit 200, and the acquisition unit 110 may read the time-series RAW image group GIM1 from the storage unit 200.

[0038] like Figure 3 As shown, the time-series RAW image group GIM1 consists of multiple RAW images captured sequentially within a first time interval TP1. The first time interval TP1 can be any time interval. For example, when the endoscope system 5 captures a moving image, and the acquisition unit 110 acquires and stores the moving image in the storage unit 200, the capture time of one moving image corresponds to the first time interval TP1. Alternatively, when the candidate RAW image group extraction unit 120 extracts the candidate RAW image group GIM2, the time interval processed by one extraction process corresponds to the first time interval TP1.

[0039] In step S2, the candidate RAW image group extraction unit 120 determines the second time interval TP2. Specifically, in step S3, the candidate RAW image group extraction unit 120 analyzes the RAW images contained in the temporal RAW image group GIM1, and determines the second time interval TP2 based on the analysis result in step S2.

[0040] Figure 5 This diagram illustrates the processing performed by the processing unit 100. Figure 5 An example is shown of the recording candidate RAW image group extraction unit 120 detecting lesions from RAW images. Figure 5 The "extraction time interval" is equivalent to Figure 3 The second time interval TP2 is defined as the period from the time t1 when the lesion appears to the time t2 when the lesion disappears. In other words, the time interval during which the lesion is captured in the RAW image is the second time interval TP2.

[0041] The candidate RAW image group extraction unit 120 detects the appearance and disappearance of lesions, for example, by analyzing time-series RAW images. In this case, the time interval between the start time t1 and the end time t2 is determined and becomes the second time interval TP2. Alternatively, the candidate RAW image group extraction unit 120 analyzes each RAW image contained in the time-series RAW image group GIM1 one by one to determine whether a lesion has been captured in each RAW image. In this case, the time interval corresponding to the RAW image determined to have captured a lesion ultimately becomes the second time interval TP2.

[0042] Furthermore, while the analysis of RAW images is illustrated as an example of lesion detection, the analysis method is not limited to this. Other examples of analysis methods will be described later. Also, the second time interval TP2 can be determined based on the system information of the endoscope system 5. This example will be explained in the second structural example.

[0043] exist Figure 4 In step S4, the RAW image recording importance determination unit 140 determines the importance of the RAW image. Importance refers to the importance in machine learning using the RAW image. Importance, for example, indicates whether an object was captured for image recognition based on machine learning, or whether the object was captured using a shooting method that contributes to high-precision learning.

[0044] like Figure 5 As shown, let the RAW images contained in the candidate RAW image group GIM2 be IM1 to IMn. n is an integer of 2 or higher. The RAW image recording importance determination unit 140 determines the importance of each RAW image from IM1 to IMn. Figure 5 In the example, importance is categorized into two levels: "high" and "low," with each RAW image being classified as either "high" or "low." The RAW image recording importance determination unit 140 determines importance, for example, based on whether the entire lesion is captured in the RAW image, or the proportion of the lesion area in the RAW image. However, the method for determining importance is not limited to the methods described above. Examples of other determination methods will be described later.

[0045] The RAW image recording importance determination unit 140 outputs RAW images with high importance, i.e., those determined to be "high" in importance, as the recording target RAW images. Figure 5 Examples are shown where IM1, IM5, IM6, ... are determined to be RAW images to be recorded. Furthermore, importance can be classified into three or more levels, or it can be a continuous value. In this case, the RAW image recording importance determination unit 140 outputs RAW images with an importance level of the threshold or higher as RAW images to be recorded.

[0046] exist Figure 4 In step S5, the RAW image recording unit 130 determines whether it is necessary to record a RAW image. That is, if there is no RAW image that was determined to be a recording target RAW image in step S4, the RAW image recording unit 130 determines that it is not necessary to record a RAW image and ends the processing of this flowchart. If there is a RAW image that was determined to be a recording target RAW image in step S4, the RAW image recording unit 130 determines that it is necessary to record a RAW image and records the recording target RAW image into the storage unit 200.

[0047] For example, the RAW image recording unit 130 records the target RAW image in the storage unit 200 by deleting RAW images other than the target RAW image from the time-series RAW image group GIM1 temporarily stored in the storage unit 200. Alternatively, the RAW image recording unit 130 may also delete the time-series RAW image group GIM1 temporarily stored in the storage unit 200, so that the storage unit 200 can newly store the target RAW image.

[0048] The training data generation system 400 includes a processor 410, a storage unit 420, a display unit 430, and an operation unit 440. RAW images recorded in the storage unit 200 of the image recording system 1 are sent to the training data generation system 400 and stored in the storage unit 420.

[0049] The processor 410 reads a RAW image from the storage unit 420, performs development processing on the RAW image to generate a display image, and displays the display image on the display unit 430. The processor 410 includes a training label acquisition unit and a training label recording unit. The user inputs a training label corresponding to the display image using the operation unit 440. The training label acquisition unit acquires the input training label. The training label recording unit associates the training label with the RAW image as a training image and records the training image in the storage unit 420.

[0050] The learning device 500 includes a storage unit 520 and a processor 510. Training images stored in the storage unit 420 of the training data generation system 400 are sent to the learning device 500 and stored in the storage unit 520 as training data 521.

[0051] The storage unit 520 stores a machine learning model 522. The processor 510 uses training data 521 to perform machine learning on the learning model 522. The machine learning-adapted learning model 522 is then transmitted as a learned model to the endoscope system for image recognition within the endoscope system.

[0052] In the above implementation, the recorded object RAW image and the training label associated with the recorded object RAW image are used as training images for machine learning.

[0053] According to this embodiment, the image recording system 1 records RAW images, which are used for machine learning, thereby improving the accuracy of machine learning. The image recording system 1 selects which RAW images should be recorded, thus achieving a balance between reducing the amount of training images and improving the accuracy of machine learning.

[0054] As a method to reduce the amount of data in the RAW images to be recorded, methods such as recording only RAW images at specific time intervals during scene transitions or only RAW images at specific time intervals during shutter operations are also considered. However, in these methods, only RAW images at specific time intervals are recorded. Therefore, there are issues such as not recording RAW images suitable for machine learning, or not recording RAW images before and after that specific time interval.

[0055] In this embodiment, the image recording system 1 includes a RAW image recording importance determination unit 140 that selects the target RAW image from the candidate RAW image group GIM2. The RAW image recording unit 130 records the target RAW image selected by the RAW image recording importance determination unit 140. Specifically, the RAW image recording importance determination unit 140 selects the target RAW image by performing an importance-based determination on each RAW image in the candidate RAW image group GIM2.

[0056] In this way, a candidate RAW image group GIM2 is extracted, and the RAW images with high importance are recorded, thus recording RAW images suitable for machine learning. Furthermore, by extracting the candidate RAW image group GIM2 contained in the second time interval TP2, multiple RAW images containing information about temporal variations can be recorded. Therefore, for example, an image recognition system that takes into account the temporal variations of images can be generated through machine learning.

[0057] Furthermore, in this embodiment, the processing time of the RAW image recording importance determination unit 140 for processing a single RAW image is longer than the processing time of the candidate RAW image group extraction unit 120 for processing a single RAW image. Specifically, the processing time of the RAW image recording importance determination unit 140 in determining the importance of a single RAW image is longer than the processing time of the candidate RAW image group extraction unit 120 in determining whether to include a single RAW image as a recording candidate.

[0058] In this way, by extracting candidate RAW image group GIM2 from time-series RAW image group GIM1 through a less demanding process, the number of objects requiring importance assessment with a heavy processing load is reduced. This reduces the overall processing load of selecting RAW images for recording.

[0059] As a low-load processing method performed by the candidate RAW image group extraction unit 120, it is envisioned to base judgment on recognition results input from an external image recognition system. Image recognition, for example, is the detection of lesions, abnormal mucous membranes, or organs. Alternatively, as a low-load processing method, it is envisioned to base judgment on parameters calculated under relatively low load, such as image brightness. Alternatively, as a low-load processing method, as described in the second structural example, it is envisioned to base judgment on system information input from the endoscope system 5. System information includes the type of illumination light, information related to the procedure, or information related to user operation.

[0060] The RAW image recording importance determination unit 140 performs a heavy processing task, for example, it is assumed that the RAW image recording importance determination unit 140 determines the importance based on its own image recognition. Image recognition includes, for example, the analysis of the type of lesion, the size of the lesion, or the treatment status.

[0061] In addition, in this embodiment, the acquisition unit 110 acquires multiple medical images captured by the endoscope system 5 in a time sequence as multiple RAW images.

[0062] In machine learning using medical images, it is envisioned that various image extensions corresponding to image recognition objects be performed on medical images. That is, given various image recognition objects such as objects with features related to their convex or concave structures, blood vessel movement, color, or texture, image extensions corresponding to these features are performed. To enable such various image extensions, it is desirable to accumulate RAW images suitable for flexible image processing. According to this embodiment, RAW images can be accumulated for machine learning using medical images while reducing the amount of recorded data.

[0063] 3. Various implementation methods of the first structural example

[0064] This section describes a specific example of the method used by the candidate RAW image group extraction unit 120 to extract the candidate RAW image group GIM2. Any one of the following methods can be used, or a combination of two or more methods can be employed.

[0065] In the first method, the candidate RAW image group extraction unit 120 detects the state of the subject in the RAW images contained in the time-series RAW image group GIM1, and extracts and records the candidate RAW image group GIM2 based on the state of the subject. For example, the following first to third examples can be envisioned.

[0066] In the first example, the candidate RAW image group extraction unit 120 records the appearance and disappearance of lesions in the time-series RAW image group GIM1, and sets the interval from appearance to disappearance as the second time interval TP2. The appearance and disappearance of lesions are detected, for example, through image recognition processing such as AI processing. A lesion is tissue that has undergone pathological changes, such as a polyp or cancer.

[0067] In the second example, the candidate RAW image group extraction unit 120 records the appearance and disappearance of abnormal mucosa in the time-series RAW image group GIM1, and sets the interval from appearance to disappearance as the second time interval TP2. The appearance and disappearance of abnormal mucosa are detected, for example, by image recognition processing such as AI processing. Abnormal mucosa is an abnormal mucosa, such as mucosa that has developed inflammation, bleeding, or atrophy.

[0068] In the third example, the candidate RAW image group extraction unit 120 determines the organs captured in the temporal RAW image group GIM1, and sets the time interval in which the organs are captured as objects of machine learning as the second time interval TP2. For example, organs are detected through image recognition processing such as AI processing.

[0069] According to this embodiment, before selecting RAW images based on importance, candidate RAW images can be roughly extracted based on the state of the subject being photographed. That is, RAW images of subjects that were not captured for machine learning image recognition, or RAW images of subjects captured in a state unsuitable for machine learning, can be excluded, and all other RAW images can be recorded as candidates.

[0070] In the second method, the candidate RAW image group extraction unit 120 directly or indirectly detects the brightness of the RAW images contained in the time-series RAW image group GIM1, and extracts the candidate RAW image group GIM2 based on the brightness of the RAW images.

[0071] Specifically, the candidate RAW image group extraction unit 120 detects the brightness of the RAW images and sets the time interval of brightly captured RAW images as the second time interval TP2. Brightness is an index value representing the luminance of an image, such as the average luminance value of the entire image. Directly detecting brightness means calculating the brightness based on the RAW image. Indirectly detecting brightness means calculating the brightness based on information other than the RAW image. For example, the brightness can also be calculated based on the illumination amount of the endoscope system 5. This example is also an example of system information in the second structural example.

[0072] According to this embodiment, before selecting RAW images based on importance, candidate RAW images can be roughly extracted based on their brightness. That is, RAW images that are too dark or too bright, or otherwise unsuitable for machine learning, can be excluded, and all other RAW images can be recorded as candidates.

[0073] In the third method, the candidate RAW image group extraction unit 120 detects the processing actions of the endoscope system 5 and extracts the candidate RAW image group GIM2 based on the processing actions of the endoscope system 5.

[0074] Specifically, the candidate RAW image group extraction unit 120 records the time interval from the appearance to the disappearance of the processing device in the RAW image, and sets this time interval as a second time interval TP2. The appearance to disappearance of the processing device is detected by image recognition processing. Image recognition processing includes, for example, contour detection of the processing device, detection of high-brightness areas, or detection of specific color areas.

[0075] According to this embodiment, candidate RAW images can be roughly extracted based on the processing action before selecting RAW images based on importance. That is, the probability of capturing the learning object for machine learning is high during the time interval of the processing action, so by extracting RAW images of that time interval, the recorded object RAW image can be extracted.

[0076] In the fourth method, the candidate RAW image group extraction unit 120 detects the movement of the insertion unit 201 of the endoscope system 5, and extracts the candidate RAW image group GIM2 based on the movement of the insertion unit 201. For example, the following first and second examples can be envisioned.

[0077] In the first example, the interval during which the candidate RAW image group extraction unit 120 detects that the front end of the lens is stationary is recorded, and this interval is set as the second time interval TP2. The stationarity of the front end of the lens is detected by performing motion detection on the RAW images. That is, if the amount of motion detected from the RAW image is less than a predetermined amount, the front end of the lens is detected as stationary.

[0078] In the second example, the candidate RAW image extraction unit 120 detects the insertion / removal direction of the insertion unit 201 and sets the appropriate removal time interval as the second time interval TP2. The insertion / removal direction is detected by motion detection of the RAW images. That is, if motion converging towards the vanishing point is detected from the RAW images, it is detected that the insertion unit 201 has been removed. Since examinations such as those of the colon are performed while the insertion / removal is being performed, RAW images within the removal time interval are used as recording candidates.

[0079] According to this embodiment, before selecting RAW images based on importance, candidate RAW images can be roughly extracted based on the movement of the insertion unit 201. That is, when the insertion unit 201 performs a specific movement, since the probability of capturing a learning object for machine learning is high, the recorded object RAW image can be extracted by extracting RAW images that fall within the time interval of the specific movement of the insertion unit 201.

[0080] Next, a specific example of the method by which the RAW image recording importance determination unit 140 selects the RAW image to be recorded will be described. Any one of the following methods can be used, or a combination of two or more methods can be used.

[0081] In the first method, the RAW image recording importance determination unit 140 analyzes the characteristics of lesions in the RAW image and sets an importance level based on the characteristics of the lesions. The characteristics of the lesions are detected, for example, through image recognition processing such as AI processing. For example, consider the following first and second examples.

[0082] In the first example, the RAW image recording importance determination unit 140 sets the importance based on the type of lesion. For example, when a rare case is detected, the RAW image recording importance determination unit 140 sets the importance of the RAW image to high.

[0083] In the second example, the RAW image recording importance determination unit 140 analyzes the size of the lesion and sets its importance based on that size. For example, when the RAW image recording importance determination unit 140 detects a small lesion that is easily missed, it sets the importance of that RAW image to be high.

[0084] In the second method, the RAW image recording importance determination unit 140 analyzes the handling status of the endoscope system 5 and sets an importance level based on the handling status. The handling status is detected by image recognition processing. Image recognition processing includes, for example, detection of the contour of the handling instrument, detection of high-brightness areas, or detection of specific color areas. Alternatively, the handling status can also be detected based on system information from the endoscope system 5. For example, the handling status can also be detected based on the on / off operation information of the electrosurgical scalpel. This example is also an example of system information in the second structural example. As examples of the second method, the following first and second examples are envisioned.

[0085] In the first example, the RAW image recording importance determination unit 140 determines whether the RAW image was captured before, during, or after processing, and sets the importance of the RAW image based on the determination result.

[0086] In the second example, the RAW image recording importance determination unit 140 determines the type of processing device captured in the RAW image and sets the importance of the RAW image according to the type of processing device.

[0087] In the third method, the RAW image recording importance determination unit 140 determines whether the RAW image is of a region that has already been photographed. If it is a newly captured RAW image, the importance of the RAW image is set to high. For example, by using image recognition processing such as AI processing, the parts captured in the RAW image are detected, and by comparing the determination results with those of already captured images, it is determined whether the parts have already been photographed.

[0088] Next, a specific example of the method by which the RAW image recording unit 130 records the RAW image of the recording object will be described.

[0089] The RAW image recording unit 130 records RAW images deemed to be of high importance, and does not record other RAW images. Alternatively, RAW images not deemed of high importance can be recorded as compressed data. This method will be described later.

[0090] 4. Example of the second structure

[0091] In the second structural example, the candidate RAW image group extraction unit 120 extracts the candidate RAW image group GIM2 based on the system information of the endoscope system 5. Alternatively, the first and second structural examples can be combined. That is, the candidate RAW image group extraction unit 120 can also extract the candidate RAW image group GIM2 based on both the RAW image analysis results and the system information analysis results.

[0092] The hardware structure of the second structural example and Figure 1 and Figure 2 The first structural example shown is the same. Figure 6 This is a flowchart of the processing performed by the processing unit 100 in the second structural example. Steps S14 to S16 and Figure 4 Steps S4 to S6 of the first structural example shown are the same, so the explanation is omitted.

[0093] In step S11, the acquisition unit 110 acquires RAW images and system information of the endoscope system 5. Specifically, the acquisition unit 110 is a data interface of the processing unit 100, receives the time-series RAW image group GIM1 and system information transmitted from the endoscope system 5, temporarily stores the time-series RAW image group GIM1 in the storage unit 200, and outputs the system information to the recording candidate RAW image group extraction unit 120. The system information is information related to the control, operation, or status of the endoscope system 5, and is transmitted via signals or data. The system information, for example, is time-series information, similar to that of the RAW images, and the system information for each RAW image captured is correlated with the RAW image and input to the acquisition unit 110.

[0094] In step S12, the candidate RAW image group extraction unit 120 determines the second time interval TP2. Specifically, in step S13, the candidate RAW image group extraction unit 120 analyzes the system information of the endoscope system 5 when the sequential RAW image group GIM1 was captured, and based on the analysis result, determines the second time interval TP2 in step S12 and extracts the candidate RAW image group GIM2. A specific example of the system information will be described later.

[0095] According to this embodiment, before selecting RAW images based on importance, candidate RAW images can be roughly extracted based on the system information of the endoscope system 5. That is, since the probability of capturing the learning object for machine learning is high under a specific control, operation, or state of the endoscope system 5, the recorded object RAW image can be extracted by extracting RAW images of the time interval that constitutes that specific control, operation, or state.

[0096] This section describes a specific example of the method used by the candidate RAW image group extraction unit 120 to extract the candidate RAW image group GIM2. Any one of the following methods can be used, or a combination of two or more methods can be employed.

[0097] In the first method, the candidate RAW image group extraction unit 120 detects the type of light source when capturing sequential RAW image group GIM1, and sets the time interval for capturing images using the desired type of light source as the second time interval TP2. The type of light source is based on the type of spectrum of white light or special light, or the type of light-emitting element such as xenon lamp, LED, or laser diode. System information indicating the type of light source includes, for example, a control signal for the illumination light mode.

[0098] In the second method, the candidate RAW image group extraction unit 120 records the shooting magnification of the detected lens, and sets the time interval for shooting at the desired magnification as the second time interval TP2. System information representing the shooting magnification includes, for example, control signals for zoom operation.

[0099] In the third method, the candidate RAW image group extraction unit 120 detects release or freeze operations, and sets the time interval before and after the timing of the release or freeze operation as the second time interval TP2. The release or freeze operation is, for example, set to... Figure 1 The button on the grip section 204 is used to perform the operation, and the signal of the button operation is equivalent to system information.

[0100] In the fourth method, the user makes an input to the endoscope system 5 indicating that a RAW image should be recorded, and the candidate RAW image group extraction unit 120 sets a second time interval TP2 based on this input information. This input is, for example, via a setting in... Figure 1The input can be made through the operating part of the grip unit 204, the control device 3 or the monitor 4, or through voice input, etc.

[0101] 5. Example of a third structure

[0102] Figure 7 This is a third structural example of image recording system 1. In Figure 7 In this embodiment, the acquisition unit 110 includes an image conversion unit 112. Furthermore, descriptions of the structural elements explained in the first and second structural examples are appropriately omitted.

[0103] The image conversion unit 112 performs conversion processing on the RAW images contained in the temporal RAW image group GIM1 to generate converted images. The candidate RAW image group extraction unit 120 extracts the candidate RAW image group GIM2 based on the converted images.

[0104] Conversion processing is the process of converting a RAW image into a color image, also known as developing. In addition, conversion processing can also include white balance adjustment, noise reduction, grayscale conversion, color conversion, or emphasis processing.

[0105] According to this embodiment, a RAW image can be converted into an image suitable for processing to extract the candidate RAW image group GIM2, and the candidate RAW image group GIM2 can be extracted from the converted image.

[0106] 6. Example of the fourth structure

[0107] In the fourth structural example, RAW images that were not selected as recording objects are recorded as compressed data. Figure 8 This is the fourth structural example of image recording system 1. In Figure 8 In this embodiment, the RAW image recording unit 130 includes a compression unit 114. Additionally, the processor 510 of the learning device 500 includes an image expansion unit 512. Furthermore, descriptions of structural elements described in the first and second structural examples are appropriately omitted.

[0108] like Figure 9 As shown, the compression unit 114 converts RAW images in the candidate RAW image group GIM2 that were not selected by the RAW image recording importance determination unit 140 into compressed data. That is, RAW images determined by the RAW image recording importance determination unit 140 to have "low" importance are converted into compressed data. The RAW image recording unit 130 records this compressed data and the RAW images selected by the RAW image recording importance determination unit 140 in the storage unit 200. Figure 9 The example shown is RAW recording of IM1, IMn, etc., and compressed recording of IM2, etc.

[0109] Additionally, the compression unit 114 can also convert RAW images in the temporal RAW image group GIM1 that were not selected by the RAW image recording importance determination unit 140 into compressed data, and the RAW image recording unit 130 records this compressed data in the storage unit 200. That is, the compression unit 114 not only converts RAW images in the candidate RAW image group GIM2 that were not selected by the RAW image recording importance determination unit 140 into compressed data, but also converts RAW images that were not extracted as candidate RAW image group GIM2 into compressed data.

[0110] Compressed data refers to the data generated from a RAW image after development and compression, essentially the compressed data of a color image. The amount of compressed data generated from a single RAW image is less than the amount of data from the original RAW image.

[0111] According to this embodiment, in addition to high-importance RAW images, compressed data obtained by compressing other RAW images is also recorded. Therefore, machine learning can be performed using RAW images and compressed data. Machine learning requires a large number of images, but by adding compressed data, a large number of images are ensured, and by using RAW images in a portion of the data, high-precision learning is possible.

[0112] The recorded RAW image is used for machine learning after being converted into an expanded image through image processing. Because the recorded RAW image offers high flexibility in image processing, it is easily converted into the desired expanded image. Compressed data is used for machine learning without being converted into an expanded image. That is, the image expansion unit 512 of the learning device 500 converts the recorded RAW image into an expanded image through image expansion processing. The image expansion unit 512 does not perform image expansion processing on the compressed data. Image expansion processing includes processes such as development processing, white balance processing, noise reduction, grayscale conversion processing, color conversion processing, or emphasis processing. The processor 510 uses the expanded image and compressed data as learning images to perform machine learning for the learning model 522.

[0113] The embodiments and their variations have been described above. However, this disclosure is not directly limited to the embodiments and their variations. During implementation, the constituent elements can be modified and specified without departing from the spirit of the invention. Furthermore, the multiple constituent elements disclosed in the above embodiments and variations can be appropriately combined. For example, several constituent elements can be deleted from all the constituent elements described in the embodiments and variations. Moreover, the constituent elements described in different embodiments and variations can be appropriately combined. Thus, various modifications and applications can be made without departing from the spirit of the invention. Additionally, in the specification or drawings, for terms described at least once together with different terms that are more general or synonymous, they can be replaced with different terms at any position in the specification or drawings.

[0114] Label Explanation

[0115] 1 Image recording system, 2 Endoscope body, 3 Control device, 4 Monitor, 5 Endoscope system, 10 Imaging system, 100 Processing unit, 110 Acquisition unit, 112 Image conversion unit, 114 Compression unit, 120 Recording candidate RAW image group extraction unit, 130 RAW image recording unit, 140 RAW image recording importance judgment unit, 200 Storage unit, 201 Insertion unit, 202 Imaging module, 203 Universal cable, 204 Holding unit, 300 Display unit, 400 Training data generation system, 410 Processor, 420 Storage unit, 430 Display unit, 440 Operation unit, 500 Learning device, 510 Processor, 512 Image expansion unit, 520 Storage unit, 521 Training data, 522 Learning model, GIM1 Temporal RAW image group, GIM2 Recording candidate RAW image group, IM1~IMn RAW images, TP1 First time interval, TP2 Second time interval.

Claims

1. An image recording system, characterized by The image recording system includes: an acquisition unit that acquires a time-series RAW image group including a plurality of RAW images along a time series in a first time interval; a recording candidate RAW image group extraction unit that extracts a recording candidate RAW image group from the time-series RAW image group, the recording candidate RAW image group being included in a second time interval that is a part of the first time interval; a RAW image recording importance determination unit that selects a recording target RAW image in accordance with an importance determined for each RAW image of the recording candidate RAW image group, the importance indicating whether an object of image recognition based on machine learning is photographed or whether the object of image recognition is photographed in a photographing mode that contributes to learning of high precision; and a RAW image recording unit that performs RAW recording of the recording target RAW image for machine learning selected by the RAW image recording importance determination unit, the RAW image recording unit converts RAW images in the recording candidate RAW image group or the time-series RAW image group that are not selected by the RAW image recording importance determination unit into compressed data, and records the compressed data.

2. The image recording system according to claim 1, wherein the recording target RAW image is used for machine learning as a training image together with a training label associated with the recording target RAW image.

3. The image recording system according to claim 1, wherein processing time of the RAW image recording importance determination unit for processing of one RAW image is longer than processing time of the recording candidate RAW image group extraction unit for processing of one RAW image.

4. The image recording system according to claim 1, wherein the RAW image recording importance determination unit selects the recording target RAW image by performing importance-based determination for each RAW image of the recording candidate RAW image group.

5. The image recording system according to claim 1, wherein the acquisition unit includes an image conversion unit that performs conversion processing on RAW images included in the time-series RAW image group, and generates converted images, the recording candidate RAW image group extraction unit extracts the recording candidate RAW image group in accordance with the converted images.

6. The image recording system according to claim 1, wherein the recording candidate RAW image group extraction unit detects a state of a photographing object of RAW images included in the time-series RAW image group, and extracts the recording candidate RAW image group in accordance with the state of the photographing object.

7. The image recording system according to claim 1, wherein the recording candidate RAW image group extraction unit directly or indirectly detects brightness of RAW images included in the time-series RAW image group, and extracts the recording candidate RAW image group in accordance with the brightness of the RAW images.

8. The image recording system according to claim 1, wherein The acquisition unit acquires a plurality of RAW images photographed by an endoscope system in a time series as the plurality of RAW images.

9. The image recording system according to claim 1, wherein The acquisition unit acquires the time-series RAW image group photographed by an endoscope system, The recording-candidate-RAW-image-group extraction unit detects a treatment action of the endoscope system, and extracts the recording-candidate-RAW-image-group in accordance with the treatment action.

10. The image recording system according to claim 1, wherein The acquisition unit acquires the time-series RAW image group photographed by an endoscope system, The recording-candidate-RAW-image-group extraction unit detects a movement of an insertion section of the endoscope system, and extracts the recording-candidate-RAW-image-group in accordance with the movement of the insertion section.

11. The image recording system according to claim 1, wherein The acquisition unit acquires the time-series RAW image group photographed by an endoscope system and system information of the endoscope system, The recording-candidate-RAW-image-group extraction unit extracts the recording-candidate-RAW-image-group in accordance with the system information.

12. An image recording method characterized by, comprises the steps of: acquiring a time-series RAW image group including a plurality of RAW images along a time series in a first time interval; extracting a recording-candidate-RAW-image-group from the time-series RAW image group, the recording-candidate-RAW-image-group being included in a second time interval that is a part of the first time interval; selecting a recording-target-RAW-image in accordance with an importance degree determined for each RAW image of the recording-candidate-RAW-image-group, the importance degree indicating whether an object of image recognition based on machine learning is photographed or whether the object of image recognition is photographed in a photographing mode that contributes to learning of high accuracy; and performing RAW recording of the recording-target-RAW-image for machine learning selected by the RAW-image-recording-importance-degree-determination unit, converting RAW images in the recording-candidate-RAW-image-group or the time-series RAW image group that are not selected as the recording-target-RAW-image into compressed data, and recording the compressed data.

13. The image recording method according to claim 12, wherein The recording-target-RAW-image is a RAW image used for machine learning together with a training label associated therewith as a training image.

14. The image recording method according to claim 12, wherein The recording-target-RAW-image is an image used for machine learning after being converted into an expanded image by an image expansion process, The compressed data is an image used for machine learning without being converted into the expanded image.

15. A computer-readable information storage medium storing a program, characterized by comprising: The program causes a computer to execute the steps of: acquiring a time-series RAW image group including a plurality of RAW images along a time series in a first time interval; extracting a recording-candidate-RAW-image-group from the time-series RAW image group, the recording-candidate-RAW-image-group being included in a second time interval that is a part of the first time interval; selecting a RAW image to be recorded in accordance with an importance degree of determination for each RAW image of the recording candidate RAW image group, the importance degree indicating whether an object of image recognition based on machine learning is photographed or whether the object of image recognition is photographed in a photographing mode that contributes to learning of high precision; and performing RAW recording of the recording object RAW image for machine learning selected by the RAW image recording importance degree determination section, converting RAW images in the recording candidate RAW image group or the time-series RAW image group that are not selected as the recording object RAW image into compressed data, and recording the compressed data.

Images