Determining the type and location of a target object

Abstract

In one embodiment, a method 100 is described. The method is computer-implemented. The method includes receiving 102 imaging data of an area including a radiographic device couch and an indication of a specified number, type, and location of at least one couch accessory associated with use of the radiographic device couch by a subject. The method further includes determining 104 the type and location of at least one target object within the area using an instance segmentation model for processing the imaging data. Determining the type of the at least one target object includes determining whether the at least one target object is of the type of couch accessory specified by the indication. The method further includes comparing 106 the determined location of at least one target object determined to be of the type of couch accessory specified by the indication with the indicated specified location of the couch accessory. The method further includes 108 indicating that the couch accessory is misplaced in response to determining that the determined location of the at least one target object does not correspond to the indicated specified location of the couch accessory.

Description

[Technical Field]

[0001] The present invention relates to a method, machine-readable medium and apparatus for determining the type and location of a target object. [Background technology]

[0002] Various accessories may be used in connection with a patient support, known as a "couch," for a radiological imaging device, such as a computed tomography (CT) scanner. These accessories may include a headrest, foot extensions, knee supports, an interventional control box, etc. Different types of radiological imaging devices and radiological imaging protocols may use corresponding accessories. For example, various types of brain scans require the use of different headrests appropriate for the clinical scenario. Similarly, providing a knee support can ensure that the lumbar spine is at the correct angle during imaging. In general, proper use of couch accessories can assist in patient imaging. For example, proper use of couch accessories can contribute to better image quality and / or lead to reduced radiation (e.g., X-ray) dose. Furthermore, proper use of couch accessories can improve the consistency of operation among different radiologists. Summary of the Invention [Problem to be solved by the invention]

[0003] Aspects or embodiments described herein relate to improving patient imaging outcomes and may prevent one or more problems associated with improper or inconsistent use of accessories used in conjunction with couches for radiographic imaging devices. [Means for solving the problem]

[0004] In a first aspect, a method is described. The method is computer-implemented. The method includes receiving imaging data of an area including a radiographic device couch. The method further includes receiving an indication of a designated number, type, and location of at least one couch accessory associated with use of the radiographic device couch by a subject. The method further includes determining the type and location of at least one target object in the area using an instance segmentation model for processing the imaging data. Determining the type of the at least one target object includes determining whether the at least one target object is of the type of couch accessory designated by the indication. The method further includes comparing the determined location of at least one target object determined to be of the type of couch accessory designated by the indication with an indicated designated location of the couch accessory. In response to determining that the determined location of the at least one target object does not correspond to the indicated designated location of the couch accessory, the method further includes indicating that the couch accessory is misplaced.

[0005] In some embodiments, the designated position of the at least one couch accessory is based on the subject's need for the at least one couch accessory to be properly positioned to support the subject.

[0006] In some embodiments, the received imaging data includes depth information of the area. For example, such imaging data may be provided by a range sensor, examples of which may be a stereo or 3D camera or radio frequency radar.

[0007] In some embodiments, the determining step includes determining that multiple target objects are present in the region, and in response to this determination, the method includes determining a couch accessory type corresponding to each of the multiple target objects. In response to determining that multiple target objects of the same type are present, the method further includes selecting one target object from the multiple target objects of the same type that has a highest predicted probability by an instance segmentation model.

[0008] In some embodiments, the method further comprises determining a distance between adjacent target objects of different types, and in response to determining that the distance is less than a threshold, the method further comprises selecting from the adjacent target objects one target object having a highest predicted probability of being of a different type by the instance segmentation model.

[0009] In some embodiments, the method further comprises comparing successive frames from the imaging data to determine whether the position of target objects in the region has changed between successive frames and / or whether the number and / or type of target objects in the region has changed between successive frames.

[0010] In some embodiments, in response to determining that the target object is present in both consecutive frames, the method includes comparing a predicted probability of the target object in a most recent one of the consecutive frames to a prediction threshold. In response to determining that the predicted probability exceeds the prediction threshold, the method includes providing an indication that the target object is present in the most recent frame, or in response to determining that the predicted probability does not exceed the prediction threshold, the method includes determining a distance between the target objects in the consecutive frames. In response to determining that the distance between the target objects in the consecutive frames is less than a distance threshold, the method includes providing an indication that the target object is present in the most recent frame, or in response to determining that the distance between the target objects in the consecutive frames is not less than the distance threshold, the method includes providing an indication that the target object is not present in the most recent frame.

[0011] In some embodiments, in response to determining that the target object is present in the most recent frame of the successive frames but not in the earliest frame of the successive frames, the method includes comparing a predicted probability of the target object in the most recent frame to a prediction threshold, and in response to determining that the predicted probability exceeds the prediction threshold, the method includes providing an indication that the target object is present in the most recent frame, or in response to determining that the predicted probability does not exceed the prediction threshold, the method includes providing an indication that the target object is not present in the most recent frame.

[0012] In some embodiments, in response to determining that the target object is present in an earliest one of the successive frames but not in a most recent one of the successive frames, the method includes determining whether the target object is located at a boundary in the most recent frame. In response to determining that at least a portion of the target object is located at a boundary, the method includes providing an indication that the target object is not present in the most recent frame, or in response to determining that the target object is not located at a boundary, the method includes providing an indication that the target object is present in the most recent frame. In some embodiments, the method includes determining a type of each target object in the region and tracking each target object in the successive frames of imaging data.

[0013] In some embodiments, the method includes causing a display to display the detected type and / or location of each target object against a representation of the radiographic device couch.

[0014] In some embodiments, the method includes causing the display to indicate that the couch accessory is misplaced if the determined position of at least one target object does not correspond to the indicated specified position of the couch accessory.

[0015] In some embodiments, the instance segmentation model is implemented by a mask region-based convolutional neural network (R-CNN) trained using a dataset prepared with multiple images of different radiography couch environments annotated according to at least one predetermined annotation principle.

[0016] In some embodiments, the different radiographic device couch environments (settings) include at least one of: at least one different combination of position, number and / or type of couch accessories in at least one of the plurality of images; at least one different background in at least one of the plurality of images; at least one partially obstructed couch accessory in at least one of the plurality of images; and the presence of a subject using the radiographic device couch in at least one of the plurality of images.

[0017] In some embodiments, the at least one predetermined annotation principle includes at least one of: a predetermined minimum accuracy in annotating couch accessories; not annotating fully occluded couch accessories; not annotating portions of couch accessories that experience at least a specified level of shadowing in the image; and ignoring couch accessories in the image that are below a predetermined exposure level.

[0018] In a second aspect, a tangible machine-readable medium is described that includes instructions that, when executed on at least one processor, cause the at least one processor to perform the method of the first aspect or a related embodiment.

[0019] In a third aspect, an apparatus is described. The apparatus includes a processing circuit. The processing circuit implements the method of the first aspect or a related embodiment. The processing circuit includes a receiving module configured to receive imaging data of an area including a radiographic device couch. The receiving module is further configured to receive an indication of a specified number, type, and location of at least one couch accessory associated with use of the radiographic device couch by a subject. The processing circuit also includes a determining module configured to determine the type and location of at least one target object within the area using an instance segmentation model to process the imaging data, where determining the type of the at least one target object includes determining whether the at least one target object is a couch accessory of the type specified by the indication. The processing circuit also includes a comparing module configured to compare the determined location of at least one target object determined to be a couch accessory of the type specified by the indication with the indicated specified location of the couch accessory. The processing circuitry further includes an indication module configured to indicate that the couch accessory is misplaced in response to determining that the determined position of the at least one target object does not correspond to the indicated designated position of the couch accessory.

[0020] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 illustrates a method for improving patient imaging results according to one embodiment. [Figure 2] FIG. 2 is a schematic diagram of a system for improving patient imaging results, according to one embodiment. [Figure 3a] FIG. 3a illustrates a method for improving patient imaging results according to one embodiment. [Figure 3b] FIG. 3b illustrates a method for improving patient imaging results according to one embodiment. [Figure 4]FIG. 4 is a schematic diagram of a machine-readable medium for improving patient imaging results, according to one embodiment. [Figure 5] FIG. 5 is a schematic diagram of an apparatus for improving patient imaging results according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Exemplary embodiments of the present invention will now be described, by way of example only, with reference to the drawings in which:

[0023] 1 illustrates a method 100 (e.g., a computer-implemented method) that can be used to improve the results of performing (radiation) imaging of a patient using a radiation imaging device. Method 100 can be performed by a computer, such as a user's computing device, or by a server or cloud-based service (e.g., communicatively coupled to a user device). Method 100 can be used to provide specific information that can be used by an operator, such as a clinician or radiologist, to determine whether accessories used in combination with a radiation imaging device couch are being used properly when performing radiation imaging.

[0024] Method 100 includes, in block 102, receiving imaging data of an area (e.g., a "region of interest") including a radiographic device couch. In block 102, method 100 further includes receiving an indication of a specified number, type, and location of at least one couch accessory associated with use of the radiographic device couch by the subject. The received image data may be acquired by a patient, an operator such as a radiologist or other clinical staff, and / or an imaging device such as a camera positioned to acquire an image (e.g., a color image) of the area including the couch, which may include accessories to be used in combination with the couch. Alternatively, the received imaging data may be acquired by a distance sensor, in which case the imaging data includes depth information of the area. The camera or distance sensor may be positioned in a room including the radiographic device and, in some cases, may be used by an operator to (visually) monitor the patient during a radiographic procedure. Thus, different radiographic device couch environments (i.e., different rooms with different couches, radiographic devices, and / or equipment layouts) may have cameras or distance sensors at different positions within the environment (e.g., having cameras located on the ceiling), causing the view of the couch and / or patient to differ from environment to environment. For example, the cameras may be configured to acquire images corresponding to a side view, a top view (which may be acquired in the case of a ceiling-located camera), or a fluoroscopic view of the couch and / or patient. This means that there are many different possible views to be analyzed, which may be a complex task due to the different possible views and / or different characteristics and layouts of the environments.

[0025] The received instructions for the specified number, type, and location of at least one couch accessory may be entered manually or automatically. For example, an operator may enter details of the number, type, and location of at least one couch accessory expected to be needed by the subject (e.g., patient). Such details may be entered at a user interface, a user computer, etc. Alternatively, the details may be determined automatically, e.g., generated by a computer based on specific information such as patient details (e.g., height, weight, medical condition, etc.). The received instructions may indicate where at least one type of couch accessory should be placed relative to the subject (i.e., under, over, or around the subject), e.g., in a manner personalized to the subject.

[0026] Method 100 further includes determining the type and location of (e.g., at least one) target object within the region using an instance segmentation model for processing the imaging data at block 104. Determining the type of target object at block 104 includes determining whether the target object is a couch accessory type specified by the instruction information.

[0027] Thus, the target object type may refer to a predetermined type of accessory associated with the radiography device couch. Examples of such predetermined types of accessories include a headrest (pillow), a foot extension, a knee brace, an interventional control box, etc. The instance segmentation model can distinguish between different predetermined types of accessories and determine the location of at least one target object within the region. When trained as described below, the instance segmentation model can distinguish and recognize different types of couch accessories.

[0028] Method 100 further includes, at block 106, comparing the determined location of at least one target object determined to be of the type of couch accessory specified by the instruction information with the indicated specified location of the couch accessory.

[0029] In response to determining that the determined location of at least one target object does not correspond to the indicated designated location of the couch accessories, method 100 includes indicating that the couch accessories are incorrectly positioned at block 108. The indication may include, for example, a warning message to prompt staff to correct the number, type, and / or location of the couch accessories.

[0030] Thus, in some cases, method 100 can enable a determination that a target object within the region is, for example, one of a headrest, a foot extension, a knee brace, an intervention control box, etc. Method 100 can also determine the location of the target object, for example, within the region or relative to the couch and / or the patient. If it is determined that at least one target object is not in a specified location, method 100 can provide instructions to alert an operator so that the position of the at least one target object can be corrected. Method 100 can track at least one target object within the region and identify the type of the target object (i.e., a predetermined type of accessory) while acquiring imaging data, which can be live camera footage.

[0031] Method 100 performs instance segmentation of couch accessories based on received image data (e.g., acquired by a camera, a range sensor, etc.). As previously described, the type and location of the target object (or couch accessory) can be determined by implementing an instance segmentation model. In some cases, the instance segmentation model can be used to determine the (3D) contour of the target object (e.g., a perimeter or other outer profile corresponding to the shape). In some embodiments, when a new accessory is detected within the field of view of the camera or range sensor, the accessory can be tracked (and its "type" identified) until it leaves the field of view of the camera. Even if the accessory is partially or completely occluded during the tracking process, the instance segmentation model can still determine that a particular accessory type is at a particular location due to the tracking function.

[0032] In some embodiments, the designated position of at least one couch accessory is based on the subject's need for the at least one couch accessory to be properly positioned to support the subject. In this manner, the designated position can be personalized to the subject.

[0033] In some embodiments, method 100 can determine the type and location of couch accessories initially placed on the couch before the patient lies on the couch. In some embodiments, method 100 can determine the type and location of couch accessories when the patient initially lies on the couch and then couch accessories are placed above, around, or below the patient. If accessories are misused or forgotten, method 100 can facilitate providing instructional information (e.g., warning messages) to enable staff to correct the number, type, and / or location of couch accessories.

[0034] Thus, implementation of certain methods described herein may help an operator ensure that the accessories used are in accordance with the specified radiographic imaging protocol. Accordingly, method 100 may lead to a reduction in errors due to the use of incorrect couch accessories, which may result in reduced workload, shorter radiographic imaging times, and / or improved radiographic image quality.

[0035] In some embodiments, the instance segmentation model is implemented by a Mask Region-based Convolutional Neural Network (Mask R-CNN). Mask R-CNN may be trained using a dataset prepared using multiple images (e.g., camera images) of different radiography couch environments annotated according to at least one predefined annotation principle, which will be described in detail later. The instance segmentation model may be capable of recognizing and tracking target objects even when different possible views and / or different features and layouts of the environment are present in the images.

[0036] Mask R-CNN is an example of an instance segmentation algorithm in deep learning that can distinguish (identify) different object types. In combination with an instance segmentation model, certain embodiments described herein implement an object tracking algorithm for tracking couch accessories.

[0037] The training of the instance segmentation model will now be described according to one embodiment. As previously described, different radiography device couch environments (i.e., rooms) have different configurations. To the extent that the instance segmentation model can determine the type of couch accessories, a dataset was prepared to take into account various possible radiography device couch configurations that may be used worldwide by different operators.

[0038] Four different types of couch accessories were used to prepare the dataset. The first type of couch accessory is a headrest for supporting the patient's head on the couch (there are three different subtypes of headrests). The second type of couch accessory is a foot extension (one subtype) for supporting the patient's feet. The third type of couch accessory is a knee brace (one subtype) for supporting the patient's legs and / or knees. The fourth type of couch accessory is an interventional control box (one subtype) for controlling interventional equipment used on the patient, for example, during a radiological imaging procedure. Each of these couch accessories has a unique shape, which means that the instance segmentation model can distinguish between these different couch accessories.

[0039] In some embodiments, the different radiographic device couch settings used to provide the dataset include camera imaging data (i.e., “multiple images”) captured by a camera in at least one of the following couch configurations (first through fourth):

[0040] For a first couch configuration, at least one image of the plurality of images in the dataset has at least one different combination of couch accessory locations, numbers, and / or types. For example, couch accessories may be positioned in different locations and captured by the camera. In other examples, not all different types of couch accessories are present in the image. In other examples, all types of couch accessories are present in the image.

[0041] For the second couch configuration, at least one different background may be used in at least one of the multiple images, for example, different objects (i.e., non-target objects) may be present in the field of view and / or may be visible in the field of view because the camera is in a different position.

[0042] For a third couch configuration, at least one partially occluded couch accessory may be present in at least one of the plurality of images. For example, a couch accessory may be at least partially occluded when it is being introduced into or removed from the environment, being handled by an operator, being at least partially occluded by a patient, and / or being at least partially occluded by equipment such as other couch accessories.

[0043] In the fourth couch configuration, a subject (e.g., a patient) using a couch of the radiographic device may be present in at least one of the images. For example, the subject may be sitting or lying on the couch, and at least one couch accessory may be positioned above, around, or below the subject.

[0044] The annotation tool that implements (or at least enables a trainee to facilitate implementation of) at least one predefined annotation principle can be embodied in a computer user interface (not shown) that displays camera images and enables a trainee to input (e.g., "annotate") the type of couch accessory for each of those images. For example, the instance segmentation algorithm can identify specific objects in the images, and the trainee can select a couch accessory type from a predetermined set of couch accessory types (e.g., from a previously entered list of couch accessories present in multiple images) to train the model.

[0045] In some embodiments, the at least one predetermined annotation principle includes at least one of the following (first to fourth) predetermined annotation principles:

[0046] The first predetermined annotation principle may refer to a predetermined minimum accuracy of the couch accessory annotation. For example, by using a human trainer, the annotation of the couch accessory (i.e., the selection of the "type" of the couch accessory) may be considered to be highly accurate. In other words, the trainer may be considered to annotate as accurately as possible.

[0047] A second predefined annotation principle may indicate that fully occluded couch accessories should not be annotated. For example, occluded couch accessories may cause errors in training an instance segmentation model. However, partially occluded couch accessories may still be annotated. For example, unenclosed / exposed parts of couch accessories (i.e., unoccluded parts) may be annotated.

[0048] A third predetermined annotation principle may indicate that portions (or the entirety) of a couch accessory in an image that receive at least a specified level of shadowing should not be annotated. For example, shadowing can cause errors in training an instance segmentation model. In some examples, for the same couch accessory, in portions of the image that are not expected to receive heavy shadowing (e.g., below a threshold), the accessory may be annotated in those portions. However, if the shadowing is too heavy (e.g., above a threshold) in a portion of the image, the accessory will not be annotated in that portion. In this way, even if a portion of an accessory receives heavy shadowing, the entire accessory need not be left unmarked (i.e., unannotated) by the trainer.

[0049] A fourth predetermined annotation principle may refer to ignoring couch accessories in an image below a predefined exposure level, for example, an exposure level below 10% (e.g., due to insufficient light in the environment) may be deemed to produce unacceptable errors when used to train an instance segmentation model.

[0050] In this way, by using the different couch configurations described above in combination with the at least one predefined annotation principle, the images can be varied sufficiently to allow the instance segmentation model to be adequately trained in different possible environments, which can enable the instance segmentation model to accurately determine the type of couch accessory in the imaging data, thereby facilitating the object tracking procedure.

[0051] The dataset used to train the instance segmentation model included 1,455 images for training and 200 images for validation. Additionally, an augmentation process was applied to the images used to assist in training the instance segmentation model to account for possible errors and / or variations in image quality expected to be obtained in the field (due to non-optimal lighting and camera settings). In other words, it is expected that different cameras, lighting devices, and environments used by end users to monitor radiology imaging procedures may lead to some variability in image quality. By using an augmentation method to account for this potential variability, the instance segmentation model may still be able to determine the type of couch accessory in an image even when the image quality is suboptimal or not as expected.

[0052] Examples of augmentation methods used to create a dataset include the following augmentation methods: In some embodiments, a selection of these augmentation methods can be used to train an instance segmentation model to generate a dataset that identifies couch accessory types with a minimum predictive accuracy (e.g., greater than 99%).

[0053] The enhancement methods include applying an affine transformation to an image; adding noise sampled from a Gaussian distribution element-by-element (i.e., pixel-by-pixel) to an image; multiplying all pixels in an image by a specific value, thereby darkening or brightening the image; degrading image quality by JPEG compression; setting rectangular regions in an image (i.e., the pixel values ​​within these rectangular regions) to zero; alpha-blending two image sources using an alpha / opacity value; blurring an image by calculating a simple average over a neighborhood within the image (e.g., applying a blur filter); adjusting the contrast of an image; horizontally flipping or mirroring an input image; vertically flipping or mirroring an input image; converting a snowless landscape to a snowy landscape; and adding clouds to an image.

[0054] This level of training (i.e., the differences between different images with different couch configurations and / or augmentation methods, and the number of images used in the dataset) was found to result in reliable and consistent identification of couch accessory types and facilitate tracking of appropriate couch accessories.

[0055] FIG. 2 illustrates a system 200 for implementing certain methods described herein. Images acquired using the system 200 can be used to train and / or validate the models described above. However, the system 200 can also be used by an end user (e.g., an operator) to improve the results of imaging a patient using a radiographic device 202. While the end user can train a model (e.g., using their own couch attachment), the actual system 200 used by the end user may differ from the system 200 used to train and / or validate the model. For example, a manufacturer or installer of the radiographic device 202 can train a model by using images from multiple different environments (e.g., using different system 200 configurations), while the end user can use the system 200 as shown in FIG. 2.

[0056] The radiological imaging device 202 further includes a couch 204 supporting a plurality of couch accessories 206a-206d (four in this case, but any number would be acceptable). These couch accessories 206a-206d have different shapes in FIG. 2 and can be recognized using the instance segmentation model. When the radiological imaging device 202 is in use, a patient (not shown) lies on the couch 204, and the couch accessories 206a-206d required by the patient can be appropriately positioned to support the patient. Because the couch accessories 206a-206d can be positioned below the patient, the couch accessories 206a-206d may be at least partially occluded. However, the instance segmentation model trained as described above has been found to identify and track the couch accessories 206a-206d with sufficient accuracy.

[0057] In this embodiment, system 200 includes a camera 208 (i.e., an "imager") for acquiring imaging data, a computing device 210 (e.g., including a processing unit and optional memory), and a display 212 for displaying the imaging data. In this embodiment, computing device 210 is configured to receive image data from camera 208, which is configured to capture an image of area 214 shown in FIG. 2.

[0058] In another example, the system 200 can include a distance sensor as an embodiment of an imaging device, configured to provide depth information of the region 214. In other words, the region 214 can be a (3D) image (captured by the camera 208) of the radiation imaging device 202, the couch 214, the couch accessories 206a-206d, and / or any other object or subject within the region 214. The computing device 210 is communicatively coupled to the display 212 for displaying imaging data thereon, which may include, if relevant, information indicating the locations of all identified couch accessories 206a-206d and / or any other information, such as warning messages regarding missing and / or misplaced couch accessories 206a-206d.

[0059] In some embodiments, computing device 210 may be implemented by a user's computer. In some embodiments, computing device 210 may be implemented by a server or cloud-based computing service. The processing unit of computing device 210 may perform certain methods described herein (e.g., perform method 100, cause camera 208 to acquire image data, and / or cause display 212 to display image data including information about which couch accessories 206a-206d are needed and / or whether any couch accessories 206a-206d are missing or in the wrong position). If present, memory of computing device 210 (or other memory not forming part of computing device 210) may store the instance partitioning model itself, any instructions for implementing the instance partitioning model, and / or instructions for implementing tracking of couch accessories.

[0060] In some embodiments in which an instance segmentation model is being trained, computing device 210 accesses memory (e.g., of computing device 210 or other memory not provided by computing device 210) containing a training dataset and performs a training procedure (e.g., receives input from a trainee via a user interface (e.g., provided by display 212)).

[0061] 3a and 3b illustrate a method 300 (e.g., a computer-implemented method) that can be used to improve the results of imaging a patient using a radiological imaging device. Method 300 may be implemented by a computer, such as a user's computing device (e.g., computing device 210), or a server or cloud-based service (e.g., communicatively coupled to a user device). Method 300 includes certain blocks for implementing method 100. Certain blocks may be omitted and / or performed in a different order than illustrated by FIG. 3. For example, portions of method 300 may be implemented when training an instance segmentation model, but those portions of method 300 may not be implemented by an end user and, therefore, may be omitted when implemented by an end user. Where appropriate, reference is made to method 100 and system 200 described above.

[0062] The method 300 is initialized at block 302 .

[0063] At block 304 of method 300, an instance segmentation model is constructed (i.e., trained and validated as described above).

[0064] At block 306 of method 300, a first frame (i.e., a first image of the imaging data, which image data may include 3D information) is acquired (e.g., by having the camera 208 acquire the first frame, which is received by the computing device 210 either directly from the camera 208 or from a memory for storing image data).

[0065] In block 308 of the method 300, the instance segmentation model is used to determine the type of couch accessories 206a-206d and the location of the couch accessories 206a-206d in the first frame. Determining the location of the couch accessories may include determining where the couch accessories are located in the frame and / or determining segments of the outline (e.g., profile or perimeter) of the couch accessories.

[0066] The selection of which couch accessories 206a-206d (hereinafter referred to as "target objects" since any number and type of couch accessories may be present within a particular field of view) to track and the tracking of the target objects will now be described.

[0067] In some embodiments, method 300 includes, in response to determining that multiple target objects are present in region 214, determining the type of couch accessory corresponding to each of these target objects, and in response to determining, at block 310, that multiple target objects of the same type are present (i.e., "Yes" in FIG. 3a), method 300 further includes, at block 312, selecting one target object from the multiple target objects of the same type that has the highest predicted probability according to the instance segmentation model. In other words, if the number of target objects of the same type exceeds one, method 300 may select the target object with the highest predicted probability (and discard any remaining target objects with lower predicted probabilities). In this manner, method 300 enables tracking one type of target object (i.e., one target object of the same type) at a time.

[0068] In some embodiments, the instance segmentation model can assign a predicted probability to each detected target object based on how confident the model is that the target object is indeed one of a predetermined type of target object.

[0069] In some embodiments, method 300 includes determining the distance between adjacent target objects of different types at block 314. It may be beneficial to have depth image data of region 214, which allows computing device 210 to output a more accurate distance measure between adjacent target objects. In response to determining that the distance is less than a threshold at block 316 (i.e., "Yes" in FIG. 3a), method 300 further includes selecting, at block 318, one target object from the adjacent target objects that has the highest predicted probability of being of a different type by the instance segmentation model. In other words, if the distance between the adjacent target objects is less than a threshold, method 300 selects the type of couch accessory with the highest predicted probability (e.g., to reduce the couch accessory identification and / or tracking error rate). Method 300 then proceeds to block 320.

[0070] If the distance is greater than or equal to the threshold at block 316 (ie, "No" in FIG. 3a), the method 300 proceeds to block 320.

[0071] In some embodiments, the threshold may be a number of pixels (which translates to the physical distance between the edges of two adjacent target objects), such as 50 pixels or some other number, depending on the resolution of the camera 208. As is true for all 2D and 3D imaging data examples, the threshold may be selected according to the accuracy of the instance segmentation model's ability to distinguish between physically close target objects. For example, the threshold may be dynamically adjusted depending on the predicted probability of the target objects.

[0072] In block 320 of method 300, the target object identified and retained in the first frame is retained (e.g., for purposes of target object tracking) if it has a predicted probability above a certain threshold (e.g., above 99.9% or other suitable value).

[0073] At block 322 of method 300, a prediction result is obtained for the first frame: a prediction of the location and identity (i.e., type) of the target object in the first frame. This information can be used to update a memory accessible to computing device 210. For example, the memory can hold a previous prediction result (e.g., labeled "last_r"), and block 322 causes this previous prediction result to be updated with the prediction result obtained for the first frame.

[0074] At block 324 of method 300, a second frame (i.e., a second image of the imaging data) is acquired (e.g., by having the camera or range sensor 208 acquire the second frame and transmit it to the computing device 210). The functionality of block 324 corresponds to the functionality provided by block 306. The same functionality of blocks 306 through 318 used (at block 322) to obtain the prediction result for the first frame is used to obtain the prediction result for the second frame. In other words, the functionality of blocks 306, 308, 310, 312, 314, 316, and 318 (applied to the first frame) corresponds to the functionality of blocks 324, 326, 328, 330, 332, 334, and 336 (applied to the second frame), respectively.

[0075] At block 338, a prediction result for the second frame is obtained, as described below. Thus, in some embodiments, method 300 includes at block 338 comparing successive frames (i.e., the first and second frames) from the imaging data to determine whether the positions of target objects in the region have changed between successive frames and / or whether the number and / or type of target objects in the region have changed between successive frames.

[0076] Block 338 is shown in more detail in Figure 3b and described below.

[0077] In some embodiments, in response to determining at block 340 that the target object is present in both consecutive frames, method 300 includes at block 342 comparing the predicted probability of the target object in the most recent of the consecutive frames (i.e., the second frame) to a prediction threshold (e.g., 99.9%) determined by an instance segmentation model.

[0078] In response to determining that the predicted probability exceeds the prediction threshold (i.e., "yes"), method 300 includes providing an indication that a target object is present in the most recent frame at block 344. In some embodiments, providing the indication includes saving information about the detected target object (e.g., labeled "this_r") for use in prediction results for the second frame.

[0079] However, in response to determining that the predicted probability does not exceed the predicted threshold (i.e., "No"), the method 300 includes, at block 346, determining the distance between the target objects in successive frames (e.g., the distance between the centers of the target objects in pixels between successive frames, or other distance measure).

[0080] In response to determining at block 348 that the distance between the target objects in successive frames is less than the distance (or pixel) threshold (i.e., "yes"), method 300 includes providing an indication that a target object is present in the most recent frame at block 350. In some embodiments, providing the indication includes saving information about the detected target object (e.g., labeled "this_r") for use in prediction results for the second frame.

[0081] In response to determining that the distance between the target objects in successive frames is not below the distance threshold (i.e., "NO"), method 300 includes providing an indication that the target object is not present in the most recent frame at block 352. For example, the indication may cause the target object to be discarded and not used or ignored in the prediction result for the second frame.

[0082] In some embodiments, in response to determining at block 354 that a target object is present in the most recent (i.e., second) of the consecutive frames but not in the earliest (i.e., first) of the consecutive frames, method 300 includes at block 356 comparing the predicted probability of the target object in the most recent frame to a prediction threshold (e.g., similar to block 342).

[0083] In response to determining that the predicted probability exceeds the predicted threshold, method 300 includes, at block 358, providing an indication that the target object is present in the most recent frame (i.e., "yes," as in block 344).

[0084] In response to determining that the predicted probability does not exceed the predicted threshold (i.e., "No"), method 300 includes, in block 360, providing an indication that the target object is not present in the most recent frame (e.g., similar to block 352).

[0085] In some embodiments, in response to determining at block 362 that the target object is present in the earliest (first) one of the consecutive frames but not in the latest (second) one of the consecutive frames, method 300 includes determining whether the target object is located at a boundary (e.g., whether the object may at least partially cross the boundary) in the latest frame at block 364. For example, the target object may be at least partially occluded by leaving the region (e.g., by being removed by the operator) or by being placed under the patient.

[0086] In response to determining that at least a portion of the target object is located on the boundary (i.e., "yes"), method 300 includes, in block 366, providing an indication that the target object is not present in the most recent frame (e.g., similar to block 352).

[0087] In response to determining that the target object is not located at a boundary, the method 300 includes providing an indication that the target object is present in the most recent frame at block 368 (eg, similar to block 344).

[0088] As a result of performing block 338, another prediction result is obtained for the second frame (i.e., "this_r") in block 370. Thus, the prediction results obtained in blocks 322 and 370 correspond to the identification and tracking of the type of target object between the first and second (consecutive) frames, respectively.

[0089] At block 372, the method 300 outputs a prediction result for the second frame, which may be used as described below.

[0090] At block 374, the method 300 updates the memory so that the prediction result for the second frame replaces the prediction result for the first frame (i.e., because the prediction result for the first frame is no longer valid).

[0091] At block 376, the method empties the memory holding the prediction results for the second frame (ie, in anticipation that the next frame may be acquired by the camera 208).

[0092] At block 378, the method 300 determines whether there is a next frame (i.e., whether the camera 208 continues to acquire imaging data). If yes, the method 300 returns to block 324 and repeats the subsequent blocks of the method 300. If no, the method 300 ends at block 380.

[0093] Thus, in some embodiments, the method 300 includes determining the type of each target object in the region and tracking each target object in successive frames of imaging data.

[0094] In some embodiments, in response to outputting the prediction results in block 372, the method 300 includes, in block 382, ​​causing a display (e.g., display 212) to display the detected type and / or location of each target object in association with a representation (e.g., a displayed image) of the radiation imaging device couch 204.

[0095] At block 384 of method 300, the type and / or location of each target object in the region is compared to an expected (anticipated) object configuration for the region. The expected object configuration may correspond to a specified number, type, and location of at least one couch accessory associated with use of the radiography device couch by the subject. In some embodiments, in response to determining that at least one target object in the region does not correspond to the expected object configuration and / or that at least one target object is missing from the region, method 300 at block 386 includes causing a display to indicate that if the determined location of at least one target object does not correspond to the indicated specified location of the couch accessory and / or if at least one target object does not correspond to the expected object configuration, the couch accessory is misplaced and / or at least one target object is missing from the region.

[0096] In some embodiments, the expected object configuration includes a specified number, type, and / or location of each target object associated with the subject (e.g., patient) who will be using the couch. In other words, an operator may input the expected object configuration for the patient into computing device 210. If computing device 210 (performing method 300) detects that a target object (e.g., couch accessory) is missing or misplaced, block 386 causes the display to indicate this, as appropriate, for example, via a warning message.

[0097] 4 schematically illustrates a tangible, machine-readable medium 400 having stored thereon instructions 402 that, when executed by at least one processor 404, cause the at least one processor 404 to perform a particular method described herein, such as method 100 or 300. In this embodiment, the instructions 402 include instructions 406 for performing the function of block 102 of method 100. The instructions 402 further include instructions 408 for performing the function of block 104 of method 100. The instructions 402 further include instructions 410 for performing the function of block 106. The instructions 402 further include instructions 412 for performing the function of block 108 of method 100. The instructions 402 may include other instructions for performing the function of any of the blocks of method 300.

[0098] FIG. 5 illustrates an apparatus 500 that may be used to implement certain methods described herein, such as methods 100 or 300. The apparatus 500 may have modules with functionality corresponding to certain features described in connection with the system 200 of FIG. 2, such as the computing device 210 of the system 200 of FIG. 2. In this embodiment, the apparatus 500 includes a receiving module 504 configured to perform the functionality of block 102 of the method 100. The apparatus 500 further includes a determining module 506 configured to perform the functionality of block 104 of the method 100. The apparatus 500 further includes a comparing module 508 configured to perform the functionality of block 106 of the method 100. The apparatus 500 further includes an indicating module 510 configured to perform the functionality of block 108 of the method 100. The receiving module 504, the determining module 506, the comparing module 508, and / or the indicating module 510 (or any additional modules) may also be used to perform the functionality of any of the blocks of the method 300.

[0099] In some cases, any of the above-described modules (e.g., receiving module 504, determining module 506, comparing module 508, and / or indicating module 510) may include at least one dedicated processor (e.g., an application specific integrated circuit (ASIC) and / or a field programmable gate array (FPGA) etc.) to perform the functions of that module. For example, the at least one dedicated processor may be programmed to perform the functions described above or configured to access a memory storing instructions and execute such instructions to perform the functions described above.

[0100] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive, i.e., the invention is not limited to the disclosed embodiments.

[0101] One or more features described in one embodiment may be combined with or substituted for features described in other embodiments. For example, methods 100, 300 of Figure 1 or Figures 3a-3b may be modified based on features described in connection with system 200 (see Figure 2), machine-readable medium 400, and / or apparatus 500, and vice versa.

[0102] This disclosure includes subject matter defined by the following numbered paragraphs: Paragraph 1. A computer-implemented method comprising: - receiving imaging data of an area including a radiological imaging device couch; - determining a type and location of at least one target object within the region using an instance segmentation model for processing the imaging data, wherein determining the type of the at least one target object includes determining whether the at least one target object is an accessory of a predetermined type associated with a radiological imaging device couch; A method comprising: Paragraph 2. The method of Paragraph 1, wherein the determining step includes determining that a plurality of target objects are present in the region, and in response thereto, the method includes determining a predetermined type for each of the target objects from the plurality, and in response to determining that a plurality of target objects of the same predetermined type are present, the method further includes selecting, from the plurality of target objects of the same predetermined type, a target object having a highest predicted probability according to an instance segmentation model. Paragraph 3. The method of paragraph 1 or 2, further comprising determining a distance between adjacent target objects of different types, and in response to determining that the distance is less than a threshold, the method further comprises selecting from the adjacent target objects one target object having a highest predicted probability of being of a different type according to the instance segmentation model. Paragraph 4. The method of any preceding paragraph, further comprising the step of comparing successive frames from the imaging data to determine whether the position of target objects has changed in the region between successive frames and / or whether the number and / or type of target objects has changed in the region between successive frames. Paragraph 5. The method of Paragraph 4, wherein in response to determining that the target object is present in both of the consecutive frames, the method: - comparing a predicted probability of a target object in a most recent of consecutive frames with a prediction threshold; and - in response to determining that the predicted probability is above a prediction threshold, the method includes providing an indication that the target object is present in the most recent frame; or - in response to determining that the predicted probability does not exceed a predicted threshold, the method includes determining a distance between target objects in successive frames; - in response to determining that the distance between the target object in successive frames is less than a distance threshold, the method comprises providing an indication that the target object is present in the most recent frame; or - in response to determining that the distance between the target objects in successive frames is not below a distance threshold, the method comprising providing an indication that the target object is not present in the most recent frame; method. Paragraph 6. The method of Paragraph 4, wherein in response to determining that the target object is present in the latest of the successive frames but not in the earliest of the successive frames, the method comprises: - comparing the predicted probability of a target object in the most recent frame with a prediction threshold; and - in response to determining that the predicted probability is above a prediction threshold, the method comprises providing an indication that the target object is present in the most recent frame; or - in response to determining that the predicted probability does not exceed the prediction threshold, the method includes providing an indication that the target object is not present in the most recent frame; method. Paragraph 7. The method of paragraph 4, responsive to determining that the target object is present in an earliest one of the successive frames but not in a latest one of the successive frames, - determining whether the target object is located at a boundary in the latest frame; and - in response to determining that at least a portion of the target object is located at the boundary, the method comprises providing an indication that the target object is not present in the latest frame; or - in response to determining that the target object is not located at a boundary, the method comprising providing an indication that the target object is present in the most recent frame; method. Paragraph 8. The method of any preceding paragraph, comprising determining a type of each target object within the region, and tracking each target object in successive frames of imaging data. Paragraph 9. The method of any preceding paragraph, further comprising the step of causing a display to display the detected type and / or position of each target object relative to a representation of the radiographic device couch. Paragraph 10. The method of Paragraph 9, comprising the step of comparing the type and / or location of each target object in the region with an expected object configuration for the region, and in response to determining that at least one target object in the region does not correspond to the expected object configuration or that at least one target object is missing in the region, the method comprises the step of causing a display to indicate that at least one target object does not correspond to the expected object configuration and / or that at least one target object is missing in the region. Paragraph 11. The method of Paragraph 10, wherein the expected object configuration includes a specified number, type and / or location of each target object associated with the subject using the couch. Paragraph 12. Any of the preceding methods, wherein the instance segmentation model is implemented by a mask region-based convolutional neural network (R-CNN) trained using a dataset prepared with a plurality of images of different radiography device couch environments annotated according to at least one predetermined annotation principle. Paragraph 13. A tangible, machine-readable medium comprising instructions that, when executed on at least one processor, cause the at least one processor to perform the method of any preceding paragraph. Paragraph 14. An apparatus including a processing circuit, the processing circuit comprising: - a receiving module configured to receive imaging data of an area including a radiological imaging device couch; - a determination module configured to determine a type and a location of at least one target object within the region using an instance segmentation model for processing the imaging data, wherein determining the type of the at least one target object includes determining whether the at least one target object is an accessory of a predetermined type associated with a radiology imaging device couch; An apparatus having: Paragraph 15. The method of paragraphs 1 to 13 or the apparatus of paragraph 14, wherein the imaging data includes depth information of the region.

[0103] Embodiments of the present disclosure may be provided as a method, a system, or a combination of machine-readable instructions and processing circuitry. Such machine-readable instructions may be contained on a non-transitory machine (e.g., computer) readable storage medium (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-readable program code.

[0104] The present disclosure is described with reference to flowcharts and block diagrams of methods, apparatuses, and systems according to embodiments of the present disclosure. Although the flowcharts show a specific order of execution, the order of execution may differ from that shown. Blocks described in connection with one flowchart may be combined with those of other flowcharts. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented by machine-readable instructions.

[0105] The machine-readable instructions may be executed by, for example, a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to implement the functions described in the description and figures. In particular, a processor or processing circuit, or modules thereof, may execute machine-readable instructions. Thus, the functional modules of system 200 and / or device 500 (e.g., computing device 210, receiving module 504, and / or decision module 506) and other devices described herein may be implemented by a processor that executes machine-readable instructions stored in a memory or operates according to instructions embedded in logic circuitry. The term "processor" should be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, programmable gate array, etc. All of the methods and functional modules may be performed by a single processor or divided among multiple processors.

[0106] Such machine-readable instructions may be stored on a computer-readable storage device capable of directing a computer or other programmable data processing apparatus to operate in a particular mode.

[0107] Such machine-readable instructions may be loaded into a computer or other programmable data processing apparatus such that the computer or other programmable data processing apparatus performs a sequence of operations to produce a computer-implemented process. In this manner, the instructions executing on the computer or other programmable apparatus implement the functions specified in the flowchart and / or block diagram blocks.

[0108] Furthermore, the teachings herein may be implemented in the form of a computer program product, the computer program product being stored on a storage medium and including a plurality of instructions for causing a computer device to perform the methods described in the embodiments of the present disclosure.

[0109] Elements or steps described in connection with one embodiment may be combined with or substituted for elements or steps described in connection with other embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the singular does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program may be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, or in other forms, such as via the Internet or other wired or wireless communication systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. receiving imaging data of an area including a radiographic device couch and an indication of a specified number, type, and location of at least one couch accessory associated with use of the radiographic device couch by a subject; determining a type and location of at least one target object in the region using an instance segmentation model for processing the imaging data, wherein determining the type of the at least one target object includes determining whether the at least one target object is a couch accessory type specified by the instruction information; comparing the determined location of the at least one target object determined to be of the type of couch accessory specified by the instruction information with an indicated designated location of the couch accessory; In response to determining that the determined position of the at least one target object does not correspond to the indicated designated position of the couch accessory, indicating that the couch accessory is misplaced; comparing successive frames from the imaging data to determine whether the target object is present in a most recent one of the successive frames based on whether the position of the target object has changed in the region between the successive frames and / or whether the number and / or type of the target objects has changed in the region between the successive frames; 10. A computer-implemented method comprising:

2. 10. The computer-implemented method of claim 1, wherein the designated position of the at least one couch accessory is based on the subject's need for the at least one couch accessory to be properly positioned to support the subject.

3. 3. The computer-implemented method of claim 1, wherein the determining step includes determining that multiple target objects exist within the region, and in response to this determination, the computer-implemented method includes determining a type of couch accessory corresponding to each of the multiple target objects, and in response to determining that multiple target objects of the same type exist, the computer-implemented method further includes selecting one target object from the multiple target objects of the same type that has a highest predicted probability by the instance segmentation model.

4. 4. The computer-implemented method of claim 1, further comprising determining a distance between adjacent target objects of different types, and in response to determining that the distance is less than a threshold, selecting from the adjacent target objects one target object that has a highest predicted probability of being of a different type according to the instance segmentation model.

5. In response to determining that the target object is present in both of the consecutive frames, the computer-implemented method includes comparing a predicted probability of the target object in a most recent frame of the consecutive frames to a prediction threshold; In response to determining that the predicted probability exceeds the prediction threshold, the computer-implemented method includes providing an indication that the target object is present in the most recent frame; or In response to determining that the predicted probability does not exceed the predicted threshold, the computer-implemented method includes determining a distance between the target objects in the successive frames; In response to determining that the distance between the target objects in the successive frames is less than a distance threshold, the computer-implemented method includes providing an indication that the target object is present in the most recent frame; or In response to determining that the distance between the target objects in the successive frames is not less than the distance threshold, the computer-implemented method includes providing an indication that the target object is not present in the most recent frame. The computer-implemented method of claim 1 .

6. In response to determining that the target object is present in a most recent frame of the successive frames but not in an earliest frame of the successive frames, the computer-implemented method includes comparing a predicted probability of the target object in the most recent frame to a prediction threshold; In response to determining that the predicted probability exceeds the prediction threshold, the computer-implemented method includes providing an indication that the target object is present in the most recent frame; or In response to determining that the predicted probability does not exceed the prediction threshold, the computer-implemented method includes providing an indication that the target object is not present in the most recent frame. The computer-implemented method of claim 1 .

7. In response to determining that the target object is present in an earliest one of the successive frames but not in a latest one of the successive frames, the computer-implemented method includes determining whether the target object is located at a boundary in the latest frame; In response to determining that at least a portion of the target object is located at the boundary, the computer-implemented method includes providing an indication that the target object is not present in the most recent frame; or In response to determining that the target object is not located at the boundary, the computer-implemented method includes providing an indication that the target object is present in the most recent frame. The computer-implemented method of claim 1 .

8. 8. A computer-implemented method according to claim 1, comprising determining the type of each target object within the region and tracking each target object in successive frames of the imaging data.

9. 9. A computer-implemented method according to any one of claims 1 to 8, further comprising causing a display to display the detected type and / or position of each target object relative to a representation of the radiographic device couch.

10. 10. The computer-implemented method of claim 9, further comprising causing the display to indicate that the couch accessory is misplaced if the determined position of the at least one target object does not correspond to the indicated specified position of the couch accessory.

11. 11. The computer-implemented method of claim 1, wherein the instance segmentation model is implemented by a mask region-based convolutional neural network trained using a dataset prepared with a plurality of images of different radiography device couch environments annotated according to at least one predetermined annotation principle.

12. 12. A tangible, machine-readable medium comprising instructions that, when executed on at least one processor, cause the at least one processor to perform the computer-implemented method of any one of claims 1 to 11.

13. 1. An apparatus comprising a processing circuit, the processing circuit comprising: a receiving module for receiving imaging data of an area including a radiographic device couch and an indication of a specified number, type, and location of at least one couch accessory associated with use of the radiographic device couch by a subject; a determination module that determines a type and location of at least one target object within the region using an instance segmentation model for processing the imaging data, wherein determining the type of the at least one target object includes determining whether the at least one target object is a couch accessory of the type specified by the instruction information; and a comparison module that compares the determined location of the at least one target object determined to be a couch accessory of the type specified by the instruction information with an indicated specified location of the couch accessory; an indication module that indicates that the couch accessory is misplaced in response to determining that the determined position of the at least one target object does not correspond to an indicated designated position of the couch accessory; a tracking module that compares successive frames from the imaging data to determine whether the target object is present in a most recent one of the successive frames based on whether the position of the target object has changed in the region between the successive frames and / or whether the number and / or type of the target objects has changed in the region between the successive frames; An apparatus having:

14. The computer-implemented method of claim 1 , wherein the imaging data includes depth information for the region.

15. The device of claim 13, wherein the imaging data includes depth information of the region.

Images