Information processing system, endoscope system, and method for operating information processing system

Abstract

An information processing system (10) comprises a processing unit (4) for detecting a target object from an image for detection. The processing unit (4) divides the image for detection into a first grid cell group. The target object is overlapped on a second grid cell group among the first grid cell group. At this time, the processing unit (4) generates bounding boxes in second grid cells included in the second grid cell group. The processing unit (4) uses the bounding boxes generated in the second grid cells to surround the target object area positioned in the second grid cells, and causes the display unit (3) to display the position and shape of the target object using a group of a plurality of bounding boxes overlapped on the image for detection.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an endoscope system, and Method for operating an information processing system etc. [Background technology]

[0002] Non-Patent Document 1 discloses YOLO (You Only Look Once) as an object detection method using machine learning. YOLO is a method for detecting specific objects that has been trained in advance, and outputs a bounding box that surrounds the detected object. This bounding box is superimposed on an image, thereby displaying the bounding box that surrounds the detected object.

[0003] Non-Patent Document 2 discloses U-net as a semantic segmentation technique using machine learning. Semantic segmentation is a technique for dividing an image into regions that belong to each category. U-net classifies each pixel in the image into categories. Pixels classified into a category that indicates a specific object are displayed in a specific color, and the region in the image where the specific object exists is displayed as filled in with the specific color.

[0004] Non-Patent Document 3 discloses BodyPix as a method for assigning attributes to images using machine learning. BodyPix divides an image into grid cells and assigns an attribute to each grid cell. For example, BodyPix determines whether each grid cell belongs to a person or background, and whether a grid cell belonging to a person belongs to a body part, such as a face or an arm. Then, the grid cells are displayed in different colors according to the attribute, thereby displaying the human body parts and the background in a grid. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] "Object Detection YOLO", [online], [searched September 9, 2020], Internet<URL:https: / / www.renom.jp / ja / notebooks / tutorial / image_processing / yolo / notebook.html> [Non-patent document 2] "U-Net: Semantic Segmentation Method Using Deep Learning", [online], [Retrieved September 9, 2020], Internet<URL:https: / / blog.negativemind.com / 2019 / 03 / 15 / semantic-segmentation-by-u-net / > [Non-patent document 3] "[Updated] BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js", [online], November 18, 2020, [Searched on September 9, 2020], Internet<URL:https: / / blog.tensorflow.org / 2019 / 11 / updated-bodypix-2.html> Summary of the Invention [Problem to be solved by the invention]

[0006] YOLO is capable of extremely fast detection, making it possible to display detection results in real time for videos and other applications. On the other hand, YOLO can only enclose an object of interest in a single rectangular bounding box, so it cannot determine the contour shape of the object of interest. U-net determines the object of interest on a pixel-by-pixel basis and colors the image differently, so it can represent the contour shape of the object of interest. On the other hand, U-net requires a long calculation time, so it cannot maintain real-time performance for videos and other applications. BodyPix determines the attributes of grid cells, which are coarser than pixels, so it operates faster than semantic segmentation methods such as U-net. On the other hand, because the object of interest is represented by a collection of coarse grid cells, it cannot represent the contour shape of the object of interest in detail.

[0007] As described above, the conventional technology has a problem in that when displaying an object of interest using machine learning, it is not possible to maintain real-time performance while displaying the contour shape of the object of interest. [Means for solving the problem]

[0008] One aspect of the present disclosure relates to an information processing system that includes a processing unit that performs object detection of an object from a detection image, wherein the processing unit divides the detection image into a first group of grid cells, and when the object is located overlapping a second group of grid cells among the first group of grid cells, generates a bounding box in a second grid cell included in the second group of grid cells, surrounds a portion of the object located in the second grid cell with the bounding box generated in the second grid cell, and displays the position and shape of the object on a display unit by superimposing the set of multiple bounding boxes on the detection image.

[0009] Another aspect of the present disclosure relates to an endoscope system including the information processing system described above and an imaging device that captures the detection image.

[0010] Furthermore, yet another aspect of the present disclosure relates to an information processing method for performing object detection of a target object from a detection image, which divides the detection image into a first group of grid cells, and when the target object is located so as to overlap a second group of grid cells among the first group of grid cells, generates a bounding box in a second grid cell included in the second group of grid cells, surrounds a portion of the target object located in the second grid cell with the bounding box generated in the second grid cell, and displays the position and shape of the target object on a display unit by superimposing the set of multiple bounding boxes on the detection image.

[0011] Furthermore, still another aspect of the present disclosure relates to an annotation data generation method that inputs mask data indicating the position and shape of an object in a training image, divides the mask data into a group of grid cells, and when multiple grid cells of the group of grid cells overlap the object, generates a bounding box in each of the grid cells, and uses the collection of the generated multiple bounding boxes as an annotation of the object. [Brief explanation of the drawings]

[0012] [Figure 1] 1 shows an example of the configuration of an information processing system and an endoscope system. [Figure 2] 10 is a flowchart of a process performed by a processing unit. [Figure 3] FIG. 3 is an explanatory diagram of a process performed by a processing unit. [Figure 4] FIG. 3 is an explanatory diagram of a process performed by a processing unit. [Figure 5] FIG. 3 is an explanatory diagram of a process performed by a processing unit. [Figure 6] Display example when using BodyPix. [Figure 7] 10 is a display example when object detection according to the present embodiment is used. [Figure 8] First example of bounding box BBX. [Figure 9] Second example of bounding box BBX. [Figure 10] Third example of bounding box BBX. [Figure 11] An example of training images and mask data for laparoscopic cholecystectomy. [Figure 12] An example of annotation data. [Figure 13] 10 is a flowchart of a learning process. [Figure 14] 1 is a flowchart of an annotation data generation method. [Figure 15] FIG. 1 is an explanatory diagram of a method for generating annotation data. DETAILED DESCRIPTION OF THE INVENTION

[0013] The present embodiment will be described below. Note that the present embodiment described below does not unduly limit the content described in the claims. Furthermore, not all of the configurations described in the present embodiment are necessarily essential components of the present disclosure.

[0014] 1. Information processing systems, endoscope systems 1 shows an example of the configuration of an information processing system 10 according to this embodiment and an endoscope system 100 including the information processing system 10. The endoscope system 100 includes a processor unit 1, an endoscope 2, and a display unit 3. The endoscope system 100 may further include an operation unit 9.

[0015] An imaging device is provided at the tip of the endoscope 2, and this tip is inserted into the abdominal cavity. The imaging device consists of an objective optical system that forms an image of the subject and an image sensor that captures the formed image. The imaging device captures images of the abdominal cavity, and the imaging data is sent from the endoscope 2 to the processor unit 1.

[0016] The processor unit 1 is a device that performs various processes in the endoscope system 100. For example, the processor unit 1 controls the endoscope system 100 and performs image processing, etc. The processor unit 1 includes an imaging data receiving unit 8 that receives imaging data from the endoscope 2, and an information processing system 10 that detects an object from the imaging data using a trained model.

[0017] The imaging data receiving unit 8 is, for example, a connector to which the cable of the endoscope 2 is connected, or an interface circuit that receives imaging data.

[0018] The information processing system 10 includes a memory unit 7 that stores a trained model, and a processing unit 4 that detects an object from an image based on the trained model stored in the memory unit 7.

[0019] The storage unit 7 is, for example, a storage device such as a semiconductor memory, a hard disk drive, or an optical disk drive. A trained model is stored in advance in the storage unit 7. Alternatively, a trained model may be input to the information processing system 10 from an external device such as a server via a network, and the trained model may be stored in the storage unit 7.

[0020] The processing unit 4 includes a detection unit 5 that detects an object from an image by inference using a trained model, and an output unit 6 that superimposes information indicating the object on the image based on the detection result and displays it on the display unit 3. As an object detection algorithm, various algorithms called detection, such as YOLO (You Only Look Once) or SSD (Single Shot multibox Detector), are used. However, the generation of candidate frames, the generation and display of bounding boxes, and the training data used during learning are different from conventional methods. Details of these will be described later.

[0021] Various types of hardware can be envisioned for executing inference using a trained model. For example, the detection unit 5 is a general-purpose processor such as a CPU, MPU, GPU, or DSP. In this case, the storage unit 7 stores a program in which an inference algorithm is written and parameters used in the inference algorithm as a trained model. Alternatively, the detection unit 5 may be a dedicated processor such as an ASIC or FPGA in which the inference algorithm is implemented as hardware. In this case, the storage unit 7 stores parameters used in the inference algorithm as a trained model. A neural network can be applied as the inference algorithm. In this case, the parameters are weight coefficients for the node connections in the neural network.

[0022] The display unit 3 is a monitor that displays the image output from the output unit 6, and is a display device such as a liquid crystal display or an organic EL display.

[0023] The operation unit 9 is a device that allows the operator to operate the endoscope system 100. For example, the operation unit 9 is a button, a dial, a foot switch, a touch panel, or the like. As will be described later, the output unit 6 may change the display mode of the object based on input information from the operation unit 9.

[0024] In the above description, the information processing system 10 is included in the processor unit 1, but part or all of the information processing system 10 may be provided outside the processor unit 1. For example, the storage unit 7 and the detection unit 5 may be realized by an external processing device such as a PC or a server. In this case, the imaging data receiving unit 8 transmits the imaging data to the external processing device via a network or the like. The external processing device transmits information indicating the detected object to the output unit 6 via a network or the like. The output unit 6 superimposes the received information on an image and displays it on the display unit 3.

[0025] Fig. 2 shows a flowchart of the processing performed by the processing unit 4. Figs. 3 to 5 show explanatory diagrams of the processing performed by the processing unit 4.

[0026] As shown in Fig. 2, in step S1, an image is input to the detection unit 5. Specifically, the endoscope 2 captures video of the inside of the abdominal cavity, and frame images of the video are input in real time to the detection unit 5. Steps S1 to S5 shown in Fig. 2 are performed for each frame image.

[0027] In step S2, the detection unit 5 divides the image into grid cells GCA. As shown in FIG. 3, the image is divided by a grid with horizontal spacing X and vertical spacing Y, and each divided area becomes a grid cell GCA. One grid cell GCA has a horizontal side length X and a vertical side length Y. The horizontal direction corresponds to the horizontal scanning direction of the image, and the vertical direction corresponds to the vertical scanning direction of the image. The lengths X and Y are expressed, for example, in terms of the number of pixels. The multiple grid cells GCA obtained by dividing the entire image are also referred to as a first grid cell group, and each of these is also referred to as a first grid cell.

[0028] As shown in FIG. 4, assume that an object 50 is captured in a portion of the image. The object 50 is an object that the detection unit 5 is intended to detect. While FIG. 4 shows an example in which the object 50 is directly captured in the image, as will be described later, the object 50 may be an object that is not directly captured in the image but is included within the angle of view of the image. In FIG. 4, a plurality of grid cells GCB of the first grid cell group overlap the object 50. Such a plurality of grid cells GCB that overlap the object 50 is also referred to as a second grid cell group, and each of these is also referred to as a second grid cell. In FIG. 4, the second grid cell group is indicated by hatching. The second grid cell group is included in the first grid cell group, and the grid cells themselves are the same. That is, the grid cell GCB also has a horizontal side length X and a vertical side length Y.

[0029] As shown in step S3 of FIG. 2, the detection unit 5 generates a candidate frame for each grid cell. At this time, the detection unit 5 generates a candidate frame for a grid cell GCB that overlaps the object 50. The detection unit 5 also generates multiple candidate frames for one grid cell GCB. The candidate frame is generated so as to fit within the grid cell GCB, i.e., so as not to overlap with adjacent grid cells. Note that the candidate frame is not limited to this, and details of the candidate frame will be described later.

[0030] In step S4, the detection unit 5 determines a bounding box for each grid cell. The detection unit 5 generates multiple candidate bounding box frames, calculates a reliability score for each of the multiple candidate bounding box frames, and determines a bounding box from among the multiple candidate bounding box frames based on the reliability score. Specifically, the detection unit 5 determines the candidate frame with the highest reliability score as the bounding box. As shown in FIG. 5, a bounding box BBX is generated for each grid cell GCB that overlaps the object 50. That is, while conventional object detection generates one bounding box for one object 50, in this embodiment, multiple bounding boxes BBX are generated for one object 50, and the object 50 is covered by a collection of the multiple bounding boxes BBX.

[0031] The bounding box BBX is a rectangle that includes the portion of the object 50 that overlaps with the grid cell GCB. The confidence score is a score that indicates the likelihood of the candidate frame as a bounding box BBX. In other words, a candidate frame that properly includes the portion of the object 50 that overlaps with the grid cell GCB has a high confidence score.

[0032] In step S5, the output unit 6 superimposes the bounding box BBX determined in the grid cell GCB on the image and outputs the superimposed image to the display unit 3. The output unit 6 may superimpose only the frame of the bounding box BBX on the image, or may superimpose a rectangle with the interior of the bounding box BBX filled in on the image. The output unit 6 may also superimpose the bounding box BBX on the image by alpha-blending the bounding box BBX and the image. As described above, multiple bounding boxes BBX are generated for one object 50, and therefore the object 50 is displayed as if it were covered by the collection of the multiple bounding boxes BBX.

[0033] As described above, the information processing system 10 of this embodiment includes a processing unit 4 that performs object detection of a target object from a detection image. The detection unit 5 of the processing unit 4 divides the detection image into a first grid cell group. When the target object 50 is located so as to overlap a second grid cell group among the first grid cell group, the detection unit 5 generates a bounding box BBX in a second grid cell GCB included in the second grid cell group. The output unit 6 of the processing unit 4 surrounds the portion of the target object 50 located in the second grid cell GCB with the bounding box BBX generated in the second grid cell GCB, and displays the position and shape of the target object 50 as a collection of multiple bounding boxes BBX superimposed on the detection image on the display unit 3.

[0034] Here, the detection image refers to an image input to the processing unit 4 as a target for object detection, and is a biological image captured by the endoscope 2. In FIG. 1, the image input from the endoscope 2 to the processing unit 4 via the image data receiving unit 8 is the detection image. Alternatively, the image captured by the endoscope 2 may be recorded in the storage unit 7 or the like, and the recorded image may be input to the processing unit 4 when it is played back. In this case, the image input from the storage unit 7 to the processing unit 4 is the detection image.

[0035] According to this embodiment, an object detection method that generates a bounding box for a detected object is used, enabling high-speed processing and maintaining real-time performance for video. Furthermore, since the position and shape of the object 50 are displayed as a collection of multiple bounding boxes BBX, the position and shape of the object 50 can be expressed in more detail than in conventional object detection that surrounds the object 50 with a single bounding box or in BodyPix, which paints each grid cell differently. In this way, according to this embodiment, both real-time performance and display of the position and shape can be achieved.

[0036] An example of a display using BodyPix is ​​shown in Fig. 6. As shown in the upper part of Fig. 6, an image including an object 50 is divided into a plurality of grid cells GCC, and among the plurality of grid cells GCC, grid cells GCC determined to include the object 50 are displayed in color. In Fig. 6, grid cells GCC determined to include the object 50 are indicated by hatching.

[0037] In BodyPix, colors are assigned in units of grid cells GCC, so the position and shape of the object 50 cannot be expressed in more detail than the grid cells GCC. According to this embodiment, as described in FIG. 5, the processing unit 4 can generate a bounding box BBX that is smaller than the grid cells GCB, so that the position and shape of the object 50 can be expressed in more detail even if the object 50 has a complex shape that includes curved portions.

[0038] As shown in the lower part of FIG. 6 , when BodyPix is ​​applied to an object 51 having an elongated shape, such as a blood vessel, bile duct, ureter, or nerve, the elongated shape can only be represented by the width of the grid cell GCC. Furthermore, the position of the object 51 fluctuates due to the pulsation of the living body or the operation of the endoscope 2, but fluctuations smaller than the width of the grid cell GCC are not reflected in the display, and therefore the position of the object 50 is not displayed appropriately. According to this embodiment, as shown in FIG. 7 , the processing unit 4 can generate a bounding box BBX that is narrower than the grid cell GCB, thereby representing the elongated shape of the object 51. Furthermore, even if the position of the object 51 fluctuates, the fluctuation in the position of the object 50 can be captured in detail by using a bounding box BBX that is narrower than the grid cell GCB.

[0039] The output unit 6 may superimpose the bounding box BBX on the detection image with an opacity that corresponds to the reliability score. In α blending, when the blending ratio of the bounding box BBX is α and the blending ratio of the detection image is 1-α, α corresponds to the opacity. The higher the reliability score of the bounding box BBX, the higher the opacity of the bounding box BBX is, as determined by the output unit 6.

[0040] In this way, the position and shape of the object 50 can be expressed in more detail. For example, it is assumed that the reliability score of the bounding box BBX at the outline of the object 50 will decrease. In this case, the bounding box BBX placed at the outline of the object 50 will be displayed with lower opacity and lighter color than the bounding box BBX placed inside the object 50. This allows the shape of the object 50 to be displayed so as to appear closer to its actual shape.

[0041] 2. About bounding boxes and candidate frames FIG. 8 shows a first example of a bounding box BBX.

[0042] Assume that the first second grid cell GCB1 and the second second grid cell GCB2 are adjacent to the second grid cell GCB in the horizontal direction, and the third second grid cell GCB3 and the fourth second grid cell GCB4 are adjacent to the second grid cell GCB in the vertical direction. Furthermore, the length between the anchor ANKs of the first second grid cell GCB1 and the second second grid cell GCB2 is defined as XA, and the length between the anchor ANKs of the third second grid cell GCB3 and the fourth second grid cell GCB4 is defined as YA. In this case, the detection unit 5 generates a bounding box BBX whose horizontal side length x is smaller than XA, whose vertical side length y is smaller than YA, and which does not include the anchor ANKs of the first to fourth second grid cells GCB1 to GCB4.

[0043] The anchor ANK is a representative point used by the detection unit 5 when generating a candidate frame, and is, for example, the center point of each grid cell. That is, the detection unit 5 generates the candidate frame based on the anchor ANK of the second grid cell GCB. The center of the candidate frame and the anchor ANK do not have to coincide. The lengths x, y, XA, and YA are expressed, for example, in terms of the number of pixels. "The bounding box BBX does not include the anchor ANK of grid cells GCB1 to GCB4" means that the anchor ANK of grid cells GCB1 to GCB4 does not exist within the rectangular area enclosed by the bounding box BBX.

[0044] According to this embodiment, the size x × y of the bounding box BBX is limited by XA × YA. That is, it is limited by the distance between the anchors of adjacent grid cells. Furthermore, the position of the bounding box BBX is limited so as not to exceed the anchors of adjacent grid cells. As a result, multiple bounding boxes BBX are generated for objects larger than a grid cell, and the object 50 is represented by a collection of these multiple bounding boxes BBX.

[0045] FIG. 9 shows a second example of a bounding box BBX.

[0046] The length of the horizontal side of the second grid cell GCB is X, and the length of the vertical side is Y. In this case, the length x of the horizontal side of the bounding box BBX is less than or equal to X, and the length y of the vertical side is less than or equal to Y.

[0047] According to this embodiment, a bounding box BBX smaller than a grid cell GCB is generated, so the position and shape of the object 50 can be expressed in detail by a bounding box BBX smaller than a grid cell GCB.

[0048] The detection unit 5 may generate a bounding box BBX that satisfies at least one of x≦X and y≦Y. That is, the length x of the horizontal side of the bounding box BBX may be less than or equal to X, and the length y of the vertical side may be less than YA in Fig. 8. Alternatively, the length x of the horizontal side of the bounding box BBX may be less than XA in Fig. 8, and the length y of the vertical side may be less than Y.

[0049] A candidate frame for determining the bounding box BBX shown in FIG. 8 or 9 will be described.

[0050] The detection unit 5 determines a bounding box BBX from among a plurality of candidate frames. At this time, the plurality of candidate frames include candidate frames that satisfy at least one of the following conditions: the length x of the horizontal side of the candidate frame is smaller than X, or the length y of the vertical side of the candidate frame is smaller than Y. As described in FIG. 9, X is the length of the horizontal side of the grid cell GCB, and Y is the length of the vertical side of the grid cell GCB. Note that it is not necessary for all of the plurality of candidate frames to satisfy the above condition; it is sufficient if some of the plurality of candidate frames satisfy the above condition.

[0051] In this way, the detection unit 5 can determine as the bounding box BBX a candidate frame that satisfies at least one of the following conditions: the length x of the horizontal side of the candidate frame is smaller than X, or the length y of the vertical side of the candidate frame is smaller than Y. By selecting such a bounding box BBX, it becomes possible to express the position and shape more precisely than the grid cell GCB.

[0052] Furthermore, the detection unit 5 may generate a plurality of candidate frames as follows: the length of the horizontal side of each candidate frame is xc, and the length of the vertical side is yc, where xc is smaller than XA in Fig. 8, yc is smaller than YA in Fig. 8, and each candidate frame does not include the anchors ANK of the first to fourth second grid cells GCB1 to GCB4.

[0053] In this way, the detection unit 5 determines the bounding box BBX from among the plurality of candidate frames, thereby generating a bounding box BBX that meets the conditions described in Fig. 8. That is, the length x of the horizontal side of the bounding box BBX is smaller than XA, the length y of the vertical side is smaller than YA, and the bounding box BBX does not include the anchors ANK of the first to fourth second grid cells GCB1 to GCB4.

[0054] Furthermore, the detection unit 5 may generate a plurality of candidate frames as follows: each candidate frame has a horizontal side length xc that is equal to or less than X and a vertical side length yc that is less than YA, or each candidate frame has a horizontal side length xc that is less than XA and a vertical side length yc that is equal to or less than Y.

[0055] In this way, the detection unit 5 determines the bounding box BBX from among the plurality of candidate frames, and it is possible to generate a bounding box BBX that satisfies the following conditions: The horizontal side length x of the bounding box BBX is equal to or less than X and the vertical side length y is smaller than YA, or the horizontal side length x of the bounding box BBX is smaller than XA and the vertical side length y is equal to or less than Y.

[0056] Furthermore, the detection unit 5 may generate a plurality of candidate frames as follows: Each candidate frame has a horizontal side length xc that is equal to or less than X, and a vertical side length yc that is equal to or less than Y.

[0057] In this way, the detection unit 5 determines the bounding box BBX from among the multiple candidate frames, thereby generating a bounding box BBX that meets the conditions described in Fig. 9. That is, the length x of the horizontal side of the bounding box BBX is smaller than X, and the length y of the vertical side is smaller than Y.

[0058] FIG. 10 shows a third example of a bounding box BBX.

[0059] Two adjacent second grid cells in the second grid cell group are designated as GCBa and GCBb. A bounding box BBXa generated in one of the second grid cells, GCBa, and a bounding box BBXb generated in the other second grid cell, GCBb, do not overlap each other.

[0060] If overlapping of bounding boxes were allowed, a large bounding box could be generated, which could result in a coarse representation of the position and shape. According to this embodiment, the position and shape of the object 50 are represented by a collection of multiple bounding boxes that do not overlap with each other, so the position and shape of the object 50 can be represented in detail.

[0061] A candidate frame for determining the bounding box BBX shown in FIG. 10 will be described below.

[0062] The detection unit 5 determines a set of non-overlapping candidate frames from among a plurality of candidate frames generated in one of two adjacent second grid cells GCBa, GCBb, one of which is a second grid cell GCBa, and a plurality of candidate frames generated in the other second grid cell GCBb, as the bounding boxes BBXa, BBXb of the one and the other second grid cells GCBa, GCBb.

[0063] In Figure 10, candidate frame CAFa is generated in second grid cell GCBa, and candidate frames CAFb1 and CAFb2 are generated in second grid cell GCBb. In reality, multiple candidate frames are generated in second grid cell GCBa. In the example of Figure 10, candidate frames CAFa and CAFb1 overlap. The detection unit 5 determines the pair of candidate frames CAFa and CAFb2, which do not overlap each other, as bounding boxes BBXa and BBXb.

[0064] In this way, it is possible to generate bounding boxes BBXa and BBXb generated in two adjacent second grid cells GCBa and GCBb so that they do not overlap with each other.

[0065] The horizontal length xc and vertical length yc of the candidate frame described above in Figures 8 to 10 are integer multiples of the unit length a. The horizontal length x and vertical length y of the bounding box BBX determined from such a candidate frame are also integer multiples of the unit length a.

[0066] When n and m are integers greater than or equal to 1, xc = n × a and yc = m × a. The unit length a is preset as a unit of side length and is smaller than the side lengths X and Y of the grid cell. More specifically, the unit length a is smaller than X / 2 and Y / 2.

[0067] 8 or 9, the size of the candidate frame xc × yc is limited, so n and m are equal to or smaller than finite integers. This limits the number of candidate frames generated by the detection unit 5 to a finite number, thereby reducing the processing load of object detection.

[0068] 3. Learning process and annotation data generation method Next, a learning process for realizing object detection according to this embodiment will be described. Object detection according to this embodiment can be applied to, for example, laparoscopic cholecystectomy. Below, the learning process will be described using laparoscopic cholecystectomy as an example. However, the application of the object detection and learning process according to this embodiment is not limited to laparoscopic cholecystectomy. In other words, this embodiment can be applied when machine learning is performed based on training data in which an operator has annotated images, and the trained model detects objects from the images.

[0069] Figure 11 shows an example of training images and mask data for laparoscopic cholecystectomy. Training images are images used in the training process and are captured in advance using an endoscope system. Training images are, for example, frame images from a video of the abdominal cavity.

[0070] The training image includes images of the liver KZ, the gallbladder TNN, and treatment tools TL1 and TL2. The common bile duct, cystic duct, Rouviere's groove, and the lower edge of S4 are included within the field of view of this training image as targets for object detection. The annotation worker identifies the common bile duct, cystic duct, Rouviere's groove, and the lower edge of S4 from the training image and assigns mask data to each. After the mask data is assigned, the training image is annotated with mask data TGA indicating the common bile duct, mask data TGB indicating the cystic duct, mask data TGC indicating the Rouviere's groove, and mask data TGD indicating the lower edge of S4. For example, the worker designates an area such as the common bile duct using a pointing device such as a mouse or a touch panel.

[0071] As shown in Fig. 12, the mask data is converted into a set of multiple bounding boxes, and this set of bounding boxes is used as annotation data. The conversion method will be described later as an annotation data generation method. Fig. 12 shows the mask data TGA as an example, but the same method is used to generate bounding boxes for the mask data TGB to TGD.

[0072] As shown in FIG. 12, the training image is divided into a first grid cell group, which is a plurality of first grid cells GCA, in the same manner as the detection image. In FIG. 12, a plurality of grid cells GCE of the first grid cell group overlap with the mask data TGA. Such a plurality of grid cells GCE overlapping with the mask data TGA will be referred to as a third grid cell group, and each of these will be referred to as a third grid cell. In FIG. 12, the third grid cell group is indicated by hatching. The third grid cell group is included in the first grid cell group, and the grid cells themselves are the same. That is, the third grid cell GCE also has a horizontal side length X and a vertical side length Y.

[0073] A bounding box BBXT is assigned to each of the third grid cells GCE that overlap the mask data TGA. In conventional object detection, one bounding box is assigned to one piece of mask data TGA, but in this embodiment, multiple bounding boxes BBXT are generated for one piece of mask data TGA, and the mask data TGA is covered by a collection of these multiple bounding boxes BBXT.

[0074] FIG. 13 is a flowchart of the learning process.

[0075] The learning process is performed by a learning device. The learning device includes a processing unit, a memory unit, an operation unit, and a display unit. The learning device is, for example, an information processing device such as a PC. The processing unit is a processor such as a CPU. The processing unit performs machine learning on the learning model to generate a trained model. The memory unit is a storage device such as a semiconductor memory or a hard disk drive. The operation unit is various operation input devices such as a mouse, a touch panel, or a keyboard. The display unit is a display device such as a liquid crystal display. Note that the information processing system 10 in Figure 1 may also function as the learning device. In this case, the processing unit 4, memory unit 7, operation unit 9, and display unit 3 correspond to the processing unit, memory unit, operation unit, and display unit of the learning device, respectively.

[0076] As shown in FIG. 13, in step S11, the processing unit of the learning device reads out training data from the storage unit. For example, one or more training images and corresponding annotation data are read out for one inference. The annotation data is a set of bounding boxes as described in FIG. 12. The data indicating the bounding boxes may be, for example, data indicating the position coordinates and size of the bounding boxes. Note that mask data and training images may be stored in the storage unit as training data. In this case, the processing unit converts the mask data into bounding boxes, and the bounding boxes are used as annotation data for learning.

[0077] In step S12, the processing unit estimates the position and shape of the object from the training image and outputs the result. That is, the processing unit inputs the training image to the neural network. The processing unit executes inference processing using the neural network and outputs a set of bounding boxes that indicate the position and shape of the object.

[0078] In step S13, the processing unit compares the estimated bounding box with the bounding box of the annotation data and calculates the error based on the result. That is, the processing unit calculates the error between the bounding box output from the neural network and the bounding box of the training data.

[0079] In step S14, the processing unit adjusts the model parameters of the learning model so as to reduce the error. That is, the processing unit adjusts the weighting coefficients between nodes in the neural network based on the error calculated in step S13.

[0080] In step S15, the processing unit determines whether or not the parameter adjustment has been completed a specified number of times. If the parameter adjustment has not been completed the specified number of times, the processing unit executes steps S11 to S15 again. If the parameter adjustment has been completed the specified number of times, the processing unit ends the learning process as shown in step S16. Alternatively, the processing unit determines whether or not the error calculated in step S13 has become equal to or less than a specified value. If the error has not become equal to or less than the specified value, the processing unit executes steps S11 to S15 again. If the error has become equal to or less than the specified value, the processing unit ends the learning process as shown in step S16.

[0081] A trained model is obtained through the above training process and stored in the storage unit 7 of FIG. 1. The training image is divided into a first grid cell group, and an object in the training image is positioned so as to overlap a third grid cell group among the first grid cell group. At this time, the trained model is trained using training data annotated from the training image, and is trained to output the position and shape of the object as a set of multiple bounding boxes generated from the third grid cell group. As described in FIG. 13, the annotation represents the position and shape of the object as a set of multiple bounding boxes by surrounding the portion of the object located in the third grid cell GCE with a bounding box BBXT generated in the third grid cell GCE included in the third grid cell group. Specifically, the position and shape of the object are represented by mask data, and the position and shape of the mask data are represented by a set of multiple bounding boxes by surrounding the portion of the mask data located in the third grid cell GCE.

[0082] According to this embodiment, the learning process is performed using annotations that represent the position and shape of an object in a learning image using a collection of multiple bounding boxes, and object detection using this trained model makes it possible to display the position and shape of an object in a detection image represented by a collection of multiple bounding boxes.

[0083] The following effects can be expected in laparoscopic cholecystectomy. Specifically, the common bile duct, cystic duct, Rouviere's groove, and lower edge of S4 are landmarks in laparoscopic cholecystectomy, but their positions and shapes are not clearly displayed in images. Specifically, the common bile duct and cystic duct are covered by organs or tissues, while the Rouviere's groove and lower edge of S4 are exposed and visible, but their boundaries are unclear. For example, a physician with extensive tacit knowledge of laparoscopic cholecystectomy annotates the landmarks. The positions and shapes of the landmarks identified by the physician with tacit knowledge are generated as mask data. Then, machine learning is performed using this mask data as training data, enabling the detection and display of landmarks whose positions and shapes are clarified based on the tacit knowledge. In the object detection of this embodiment, the positions and shapes of landmarks can be represented by a collection of bounding boxes rather than a single rectangle. This allows doctors and other medical professionals to be provided with detailed information about the position and shape of unclear landmarks while ensuring real-time performance, which is important in surgery.

[0084] Next, an annotation data generation method for converting mask data into a bounding box will be described. Fig. 14 is a flowchart of the annotation data generation method. Fig. 15 is an explanatory diagram of the annotation data generation method. The annotation data generation method may be executed by, for example, an information processing device different from the learning device, or may be executed as preprocessing of the learning process in the learning device.

[0085] As shown in Fig. 14, mask data is input in step S21. The mask data is data that indicates the area of ​​the object in the learning image, and is data in which each pixel is assigned a value of 1 or 0. 1 indicates the object, and 0 indicates a non-object. The following explanation will be given using the mask data TGA in Fig. 11 as an example.

[0086] In step S22, the mask data TGA is converted into a set of axa squares, where a is the unit length of the candidate frame and bounding box. As shown in the top diagram of FIG. 15, the mask data TGA is divided into axa grids. In the top diagram, pixels in the mask data TGA marked with "1" are indicated by hatching. As shown in the middle diagram of FIG. 15, axa squares that overlap with pixels marked with "1" in the mask data TGA are extracted. In the middle diagram, the extracted axa squares are indicated by solid lines.

[0087] In step S23, the mask data TGA converted into a set of a×a squares is divided into a first grid cell group. The middle diagram in Fig. 15 shows an example in which 3×3 a×a squares correspond to the first grid cell GCA. However, the number of a×a squares included in one grid cell may be arbitrary.

[0088] In step S24, a bounding box is generated for each grid cell. As shown in the middle and bottom diagrams of FIG. 15, a bounding box BBXT is generated in the third grid cell GCE so as to encompass a set of a×a squares converted from the mask data TGA. Since the bounding box BBXT encompasses a set of a×a squares, it naturally encompasses the portion of the mask data TGA that is included in the third grid cell GCE. By generating a bounding box BBXT for each third grid cell GCE in the third grid cell group, the mask data TGA is covered by multiple bounding boxes BBXT.

[0089] In step S25, the bounding boxes BBXT are output as annotation data of the object indicated by the mask data TGA.

[0090] The annotation data generation method described above inputs mask data TGA that indicates the position and shape of an object in a learning image, and divides the mask data TGA into a group of grid cells. When multiple grid cells GCE of the group of grid cells overlap the object, the annotation data generation method generates a bounding box BBXT in each grid cell of the multiple grid cells GCE. The annotation data generation method uses the collection of the generated multiple bounding boxes BBXT as an annotation of the object.

[0091] According to this embodiment, annotation data is generated that encloses one object in multiple bounding boxes. Machine learning is performed using this annotation data, and object detection using the trained model enables the display of the position and shape of the object in the detection image represented by a set of multiple bounding boxes.

[0092] More specifically, the length of the horizontal side of a grid cell GCE included in the grid cell group is defined as X, the length of the vertical side is defined as Y, and a is a unit length smaller than X and Y. In this case, the annotation data generation method converts the mask data TGA into a set of a×a squares. The annotation data generation method selects squares belonging to each grid cell of the plurality of grid cells GCE from the set of a×a squares, and generates a bounding box BBXT containing the selected squares as the bounding box BBXT for each grid cell.

[0093] In machine learning using annotation data, an inference model infers a bounding box of unit length a from a training image. According to this embodiment, the annotation data generation method generates a bounding box of unit length a, so that during error evaluation, the inferred bounding box of unit length a is compared with the bounding box of unit length a in the annotation data. Comparing bounding boxes of the same unit length a simplifies error evaluation.

[0094] Although the present disclosure has been described above with reference to the present embodiment and its modifications, it is not limited to the embodiments and modifications as they are. In the implementation stage, the components may be modified and embodied without departing from the spirit of the present disclosure. Furthermore, multiple components disclosed in the above-described embodiments and modifications may be combined as appropriate. For example, some components may be omitted from all components described in the embodiments and modifications. Furthermore, components described in different embodiments and modifications may be combined as appropriate. In this manner, various modifications and applications are possible without departing from the spirit of the present disclosure. Furthermore, a term described at least once in the specification or drawings together with a different term having a broader or equivalent meaning may be replaced with that different term anywhere in the specification or drawings. [Explanation of symbols]

[0095] 1 processor unit, 2 endoscope, 3 display unit, 4 processing unit, 5 detection unit, 6 output unit, 7 memory unit, 8 imaging data receiving unit, 9 operation unit, 10 information processing system, 50, 51 object, 100 endoscope system, ANK anchor, BBX, BBXT, BBXa, BBXb bounding box, CAFa, CAFb1, CAFb2 candidate frame, GCA first grid cell, GCB, GCBa, GCBb second grid cell, GCB1 first second grid cell, GCB2 second second grid cell, GCB3 third second grid cell, GCB4 fourth second grid cell, GCE third grid cell, TGA, TGB, TGC, TGD mask data, a unit length

Claims

1. a detection unit that performs object detection of a target object from the detection image; an output unit; Including, In the object detection, the detection unit Dividing the detection image into a first group of grid cells; generating a plurality of candidate frames for each second grid cell included in a second grid cell group that overlaps the object among the first grid cells; For each of the plurality of candidate frames, a reliability score is calculated such that a candidate frame that includes a portion of the object that overlaps with the second grid cell is high; determining a candidate box having the highest confidence score from among the plurality of candidate boxes as a bounding box for the second grid cell; Each candidate frame is: a first second grid cell and a second second grid cell are adjacent to the second grid cell in the horizontal direction, and a third second grid cell and a fourth second grid cell are adjacent to the second grid cell in the vertical direction, and when a length between anchors of the first and second second grid cells is XA and a length between anchors of the third and fourth second grid cells is YA, a length xc of a horizontal side is smaller than XA, a length yc of a vertical side is smaller than YA, and the anchors of the first to fourth second grid cells are not included; The plurality of candidate frames include: the candidate frame includes a candidate frame that satisfies at least one of the following: when the length of a horizontal side of the second grid cell is X and the length of a vertical side of the second grid cell is Y, the length xc of the horizontal side of the candidate frame is smaller than X, or the length yc of the vertical side of the candidate frame is smaller than Y; The output unit An information processing system characterized in that the bounding boxes generated for each of the second grid cells by the detection unit are superimposed on the detection image and displayed on a display unit, thereby superimposing the position and shape of the object on the detection image using a collection of multiple bounding boxes and displaying it on the display unit.

2. In claim 1, The plurality of candidate frames include: An information processing system comprising a candidate frame that satisfies at least one of the following: a horizontal side length xc of the candidate frame is greater than the X; or a vertical side length yc of the candidate frame is greater than the Y.

3. In claim 1, The detection unit An information processing system characterized in that the plurality of candidate frames are generated based on the anchor of the second grid cell.

4. In claim 1, The output unit An information processing system characterized in that the bounding box is superimposed on the detection image with an opacity according to the reliability score.

5. In claim 1, The detection unit an information processing system characterized in that, from among the plurality of candidate frames generated in one of two adjacent second grid cells in the second grid cell group and the plurality of candidate frames generated in the other second grid cell, a pair of candidate frames that do not overlap with each other is determined as the bounding box of the one and the other second grid cell.

6. In claim 1, The plurality of candidate frames include: An information processing system comprising a candidate frame in which xc is equal to or smaller than X and yc is equal to or smaller than Y.

7. In claim 1, An information processing system, wherein the xc and yc are integer multiples of a unit length a.

8. The information processing system according to claim 1; an imaging device that captures the detection image; An endoscope system comprising:

9. 1. A method for operating an information processing system including a detection unit that performs object detection of a target object from a detection image, and an output unit that displays a result of the object detection on a display unit, The detection unit, in the object detection, Dividing the detection image into a first group of grid cells; generating a plurality of candidate frames for each second grid cell included in a second grid cell group that overlaps the object among the first grid cells; For each of the plurality of candidate frames, a reliability score is calculated such that a candidate frame that includes a portion of the object that overlaps with the second grid cell is high; determining a candidate box having the highest confidence score from among the plurality of candidate boxes as a bounding box for the second grid cell; Each candidate frame is: a first second grid cell and a second second grid cell are adjacent to the second grid cell in the horizontal direction, and a third second grid cell and a fourth second grid cell are adjacent to the second grid cell in the vertical direction, and when a length between anchors of the first and second second grid cells is XA and a length between anchors of the third and fourth second grid cells is YA, a length xc of a horizontal side is smaller than XA, a length yc of a vertical side is smaller than YA, and the anchors of the first to fourth second grid cells are not included; The plurality of candidate frames include: the candidate frame satisfies at least one of the following: when the length of a horizontal side of the second grid cell is X and the length of a vertical side of the second grid cell is Y, the length xc of the horizontal side of the candidate frame is smaller than X, or the length yc of the vertical side of the candidate frame is smaller than Y; The output unit, in the display, A method for operating an information processing system, characterized in that the bounding boxes generated for each of the second grid cells by the object detection are superimposed on the detection image and displayed on the display unit, thereby superimposing the position and shape of the object on the detection image using a collection of multiple bounding boxes and displaying it on the display unit.

Images