Method and system for processing images

Abstract

One aspect of the disclosure relates to a method and system for processing an image. A system performs a method for processing an image of machine-readable code. The method includes receiving an image of a machine-readable code including encoding information, wherein the machine-readable code is at least partially occluded by a substance having a dominant color; generating an adjusted image by adjusting a color space of the image based on the dominant color; binarizing at least a machine-readable code region of the image, wherein the machine-readable code region of the image depicts a machine-readable code; and decoding the binarized machine readable code region to determine encoded information. Other apparatus and methods are also described.

Description

[0001] This application is a divisional application of the invention patent application entitled “Method and system for processing images” with application date of October 14, 2019 and application number 201980082995.1.

[0002] Related applications

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 745,577, filed on October 15, 2018, and entitled “METHODS AND SYSTEMS FOR PROCESSING ANIMAGE,” which is incorporated herein by reference in its entirety. Technical Field

[0004] The subject matter disclosed herein relates generally to the technical field of special-purpose machines that facilitate image processing, including computerized variations of the software configurations of such special-purpose machines and improvements to such variations, and to techniques for improving such special-purpose machines. Background Art

[0005] One common way to package item information is to associate the item with a unique visual graphic (such as a machine-readable code). For example, a machine-readable code associated with a particular item may include identifying information about the item, descriptive information about the item, or both, and the machine-readable code may be used to distinguish the associated item from other (e.g., similar) items.

[0006] Typically, barcodes and other graphics containing data can be machine-readable to provide a faster and more accurate way to interpret the information represented by the machine-readable code. For example, the machine-readable code can be read and interpreted by a specialized optical scanner. As another example, the machine-readable code can be read and interpreted using image processing techniques.

[0007] However, conventional image processing techniques for reading visual machine-readable codes may produce inaccurate or incomplete results if the image does not clearly depict the machine-readable code. For example, in some instances, the machine-readable code may be partially covered or obscured. For example, when a 2D machine-readable code (e.g., a QR code) is stained with a substance, conventional image processing techniques may have difficulty accurately processing the 2D machine-readable code because the substance may make it more difficult to distinguish between different shaded elements (e.g., blocks) in the patterned matrix of the 2D machine-readable code. Summary of the Invention

[0008] One aspect of the present invention relates to a computer-implemented method for processing an image of a machine-readable code attached to an item of a surgical system, the surgical system comprising a display and one or more processors, the method comprising: accessing, by the one or more processors, an image depicting the machine-readable code; determining, by the one or more processors, that at least one corner point of the machine-readable code is occluded, spurious, or blurred such that a machine-readable code region of the image is not localizable; providing, by the one or more processors, the image as input to a trained neural network; receiving, by the one or more processors, a cleaned image of the machine-readable code as output from the trained neural network; locating, by the one or more processors, a machine-readable code region of the cleaned image of the machine-readable code; decoding, by the one or more processors, encoded information contained within the machine-readable code region of the cleaned image; outputting, by the one or more processors, the decoded information; and displaying, on the display, patient information generated by the surgical system based on the decoded information. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Some example embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings.

[0010] Figure 1 is a schematic diagram illustrating a system for processing an image according to some example embodiments.

[0011] Figure 2 is a flowchart illustrating the operation of a system when performing a method of processing an image according to some example embodiments.

[0012] Figures 3A-3H is a diagram illustrating a method according to some example embodiments. Figure 2 A method for imaging and processing images of machine-readable codes.

[0013] Figure 4 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium and performing any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0014] Example methods (e.g., processes or algorithms) facilitate image processing, including image processing of machine-readable codes soiled by substances (e.g., blood), and example systems (e.g., special-purpose machines configured by specialized software) are configured to facilitate such image processing. The examples represent possible variations only. Unless otherwise expressly stated, structures (e.g., structural components, such as modules) are optional and can be combined or subdivided, and operations (e.g., in processes, algorithms, or other functions) can change order or be combined or subdivided. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various example embodiments. However, it will be apparent to those skilled in the art that the present subject matter can be practiced without these specific details.

[0015] In some example embodiments, a method for processing an image of a machine-readable code includes: receiving an image of a machine-readable code including encoded information, wherein the machine-readable code is at least partially obscured by a substance having a dominant color; generating an adjusted image by adjusting a color space of the image based on the dominant color; and binarizing at least a machine-readable code region of the image, wherein the machine-readable code region of the image depicts the machine-readable code. The method may also include capturing the image of the machine-readable code with an optical sensor, decoding the binarized machine-readable code region to determine the encoded information, or both.

[0016] In certain example embodiments, a system for processing an image of a machine-readable code includes one or more processors configured to (e.g., at least) receive an image of a machine-readable code including encoded information, wherein the machine-readable code is at least partially obscured by a substance having a dominant color; generate an adjusted image by adjusting a color space of the image based on the dominant color; and binarize at least a machine-readable code region of the image, wherein the machine-readable code region of the image depicts the machine-readable code. The one or more processors may also be configured to decode the binarized machine-readable code region to determine the encoded information. In some variations, the system includes an optical sensor configured to capture an image of the machine-readable code.

[0017] In various example embodiments, the received or captured image is a color image, and the image may be adjusted at least in part by adjusting the color space of the color image to a grayscale representation (e.g., by isolating a color channel associated with or similar to the dominant color of the substance). The machine-readable code region may be located in the image using techniques such as corner detection techniques, edge detection techniques, other suitable computer vision techniques, or any suitable combination thereof. Additionally, additional image processing (e.g., binarization, with or without one or more color thresholding processes) may be performed to further process (e.g., "clean" the machine-readable code region of the image for interpretation (e.g., decoding)).

[0018] The methods and systems described herein can be used in various applications, such as processing an image of a machine-readable code associated with (e.g., attached to, representing, or otherwise corresponding to) a surgical textile, wherein the machine-readable code may be at least partially obscured by one or more bodily fluids (e.g., blood). For example, the dominant color of the substance on the machine-readable code may be red, and the image of the machine-readable code can be adjusted by isolating the red channel of the image within the color space of the image. The machine-readable code may include any suitable coded information that can provide useful information to a user (e.g., a unique identifier of the associated surgical textile, a type of the associated surgical textile, or both). For example, in response to determining the coded information of the machine-readable code and determining an identifier of the surgical textile associated with the machine-readable code (e.g., upon these determinations), a textile counter index can be incremented. The value of the textile counter index can then be presented as an output on a display, via an audio device, or both.

[0019] In some example embodiments, a system includes:

[0020] one or more processors; and

[0021] A memory storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:

[0022] accessing an image depicting a machine-readable code, the machine-readable code being at least partially obscured in the image by a substance having a dominant color;

[0023] generating an adjusted version of the image by adjusting a color space of the image based on a dominant color of a substance that at least partially obscures the machine-readable code; and

[0024] At least one region of the adjusted version of the image is binarized, the region depicting the machine-readable code.

[0025] In certain example embodiments, a method includes:

[0026] accessing, by one or more processors of the machine, an image depicting a machine-readable code, the machine-readable code being at least partially obscured in the image by a substance having a dominant color;

[0027] generating, by one or more processors of the machine, an adjusted version of the image by adjusting a color space of the image based on a dominant color of a substance that at least partially obscures the machine-readable code; and

[0028] At least one region of the adjusted version of the image is binarized, by one or more processors of the machine, the region depicting the machine-readable code.

[0029] In various example embodiments, a machine-readable medium includes instructions that, when executed by one or more processors of a machine, cause the machine to perform operations including:

[0030] accessing an image depicting a machine-readable code, the machine-readable code being at least partially obscured in the image by a substance having a dominant color;

[0031] generating an adjusted version of the image by adjusting a color space of the image based on a dominant color of a substance that at least partially obscures the machine-readable code; and

[0032] At least one region of the adjusted version of the image is binarized, the region depicting the machine-readable code.

[0033] In general, the methods and systems described herein can be used to process an image of one or more machine-readable codes. That is, the image depicts one or more machine-readable codes that can be read by an optical device. Examples of such optical machine-readable codes include barcodes (e.g., linear barcodes or other one-dimensional (1D) barcodes, or two-dimensional (2D) barcodes, such as QR codes) and / or other suitable graphics that carry coded information in an optically readable form (e.g., in the form of a patterned matrix of black elements and white elements or other optically contrasting elements). Such machine-readable codes can be used in various applications to provide information related to one or more items associated with (e.g., attached to, representing, or otherwise corresponding to) the machine-readable code. For example, during surgery and other medical procedures, surgical textiles (e.g., surgical sponges or other items that can be used to absorb various liquids (including patient blood)) may include machine-readable codes so that each machine-readable code can be associated with a specific surgical textile. In some cases, the machine-readable code can be depicted (e.g., printed, woven, etc.) on a label sewn or otherwise attached to the surgical textile. In some cases, the machine-readable code can be depicted in the surgical textile itself. The machine-readable code may include encoded information about the associated surgical textile, such as its manufacturer, its type, its material, its size, its identifier (e.g., a serial number that is unique to the surgical textile with respect to other surgical textiles), or any suitable combination thereof. Thus, the machine-readable code may be scanned (e.g., imaged) and interpreted through a decoding process (such as through computer vision techniques) to generate and utilize the encoded information contained therein.

[0034] In some cases, machine-readable codes associated with surgical textiles are scanned both before and after a medical procedure (e.g., a surgical procedure) to track surgical textiles and identify any surgical textiles that may have been inadvertently retained within a patient. Such scans can provide a "before" count and an "after" count of surgical textiles. The difference between the "before" and "after" counts can prompt medical personnel to locate any apparently missing textiles, perform a recount, perform an X-ray scan of the patient, or perform other risk mitigation.

[0035] However, during medical procedures, machine-readable codes on surgical textiles may become stained with blood or otherwise become soiled (e.g., by other bodily fluids). In cases where the machine-readable code includes dark elements and light elements (e.g., light and dark segments of a matrix), any dark matter (e.g., blood) may at least partially obscure the machine-readable code and interfere with or even prevent accurate scanning of the machine-readable code. With respect to the counting of surgical textiles described above, such potential erroneous scanning of the machine-readable code may result in uncertainty as to whether the count is correct (e.g., an "after" count). An erroneous count may cause medical personnel to erroneously conclude that surgical textiles remain in the patient's body, or worse, to erroneously conclude that all surgical textiles were counted and all textiles were removed from the patient's body. The methods and systems described herein are capable of processing images of machine-readable codes and robustly preventing errors due to occlusion of the machine-readable code.

[0036] Furthermore, while some example embodiments of the methods and systems described herein may be used to track surgical textiles within the same medical procedure or medical session, other example embodiments may additionally or alternatively be used to track surgical textiles between different medical procedures or medical sessions. Surgical textiles may inadvertently travel between different medical sessions (e.g., on or with a nurse or other personnel moving between different rooms). This may result in inaccurate counting of textiles at the source session, the destination session, or both, such as due to inadvertent double counting of traveling surgical textiles at their source session, their destination session, or both. Therefore, in some example embodiments, the methods and systems described herein may be used to identify textiles used during different medical procedures and to improve accuracy in tracking surgical textiles, counting surgical textiles, or both.

[0037] The methods and systems described herein may be used in a variety of environments, including in a hospital or clinic environment (e.g., an operating room), a military environment (e.g., a battlefield), or other suitable medical treatment environments. The methods described herein may be computer-implemented and executed at least in part by one or more processors. Figure 1 As shown, the methods discussed herein can be performed at least in part by a computer device, such as a mobile device 150 (e.g., a tablet computer, a smartphone, etc.), configured to capture images of one or more surgical textiles in an operating room or other medical environment and process the resulting images. Furthermore, the methods discussed herein can be performed by one or more processors separate from the mobile device 150 (e.g., on-site in the operating room or remotely outside the operating room).

[0038] Method for processing an image of a machine-readable code

[0039] like Figure 2 As shown, according to some example embodiments, a method 200 for processing an image of a machine-readable code includes receiving an image of a machine-readable code including encoded information (at operation 210), wherein the machine-readable code is at least partially obscured by a substance having a dominant color. The method 200 also includes generating an adjusted image by adjusting the color space of the image based on the dominant color (at operation 220). The method 200 also includes binarizing at least a machine-readable code region of the image (at operation 230), wherein the machine-readable code region of the image depicts the machine-readable code. The method 200 further includes decoding the binarized machine-readable code region to determine the encoded information (at operation 250). In some variations, the method 200 also includes capturing an image of the machine-readable code (at operation 208). In certain variations, Figure 2 The operations (eg, steps) depicted in the drawings may be performed in an order different from that depicted.

[0040] like Figure 2 As shown, some example embodiments of method 200 include capturing an image of a machine-readable code (at operation 208) or generating or otherwise obtaining at least one image of a machine-readable code. The one or more images of the machine-readable code may be stored in a database on a suitable data storage medium (e.g., locally or remotely). Thus, the receiving of the image of the machine-readable code in operation 210 may include receiving the image from a memory or other suitable storage device. For example, the image may have been previously acquired and stored in a storage medium. Each image may depict the entire surgical textile associated with (e.g., attached to) the machine-readable code, or only a portion of the surgical textile associated with the machine-readable code. The surgical textile may be, for example, a surgical sponge, a surgical dressing, a towel, or other suitable textile.

[0041] Each image can be a single still image or an image frame from a video feed and can include an area (e.g., a machine-readable code area) that depicts a corresponding machine-readable code (e.g., within the camera's field of view). The camera can be in a handheld device or a mobile device (e.g., a tablet computer). The camera can be mounted to a support (such as a table) or can be an overhead camera. The image can be an optical image that captures color characteristics, having component values for each pixel in a color space (e.g., RGB, CMYK, etc.). The image can be stored in a memory or suitable data storage module (e.g., locally or remotely) and processed. Processing the image can include normalizing the color characteristics of the image based on a set of one or more optical references (e.g., color references). The color references can represent, for example, one or more shades of red (e.g., a grid of boxes of different shades of red). Normalizing the image can include utilizing the color references to compensate for variations in lighting conditions throughout a medical procedure (e.g., a surgical procedure), artificially matching the lighting conditions in the image to a template image, artificially matching the lighting conditions in the image to a model of fluid component concentrations that is dependent on the lighting conditions, or any suitable combination thereof. For example, normalizing an image can include identifying a color reference captured in the image, determining an assigned color value associated with the identified color reference, and adjusting the image so that the color value of the color reference in the image substantially matches the assigned color value associated with the color reference. For example, the assigned color value can be determined by looking up the color reference in a database (e.g., identified by a code, a position within a set of color references, a position relative to a known feature of a channel, or any suitable combination thereof). Adjusting the image can include, for example, adjusting exposure, contrast, saturation, temperature, hue, or any suitable combination thereof.

[0042] Image preprocessing

[0043] Method 200 may include generating an adjusted image (at operation 220), such as by adjusting the color space of the image based on the dominant color of the substance that at least partially obscures the machine-readable code depicted in the image. In some example embodiments, color conversion by isolating a color (e.g., a color channel similar to the dominant color of the substance that at least partially obscures the machine-readable code) can mitigate the effect of the substance obscuring one or more features (e.g., elements or segments) of the machine-readable code. For example, blood tends to absorb less red light (e.g., light with wavelengths within the range of 620nm-740nm or a portion of this range, such as 635nm-700nm)) and reflect more red light than other colors of light (e.g., light with wavelengths outside the range of 620nm-740nm). In the example case where the machine-readable code includes white elements and black elements, any white elements stained with blood may be misinterpreted as black elements. Assuming, for example, that an image has red, green, and blue (RGB) color components, white pixels have substantially the same red (R) value as red pixels, even though white pixels additionally have green (G) and blue (B) values. Thus, isolating the red (R) channel of the image (e.g., by removing the green (G) and blue (B) channels of the image) causes any blood-stained white elements of the machine-readable code to become recognizable as the original white elements of the machine-readable code, and the blood-stained white elements thus become disambiguated from the black elements of the machine-readable code. Thus, the red channel of the image can be isolated (e.g., and retained in the adjusted image) such that, in the adjusted image, any white features of the machine-readable code that are occluded by blood will appear similar to unobstructed white features of the machine-readable code (e.g., elements that are not occluded by blood).

[0044] In other example embodiments, one or more other color channels, such as any one or more color channels of a predefined color space (e.g., RGB, XYZ, CIE-LAB, YCrCb, CMYK, etc.), are isolated (e.g., and retained in the adjusted image). Additionally or alternatively, other color mappings may be used to preprocess the image. For example, some variations of method 200 apply a linear or nonlinear equation (e.g., a predefined equation) that maps from an existing color space (e.g., the RGB or other color space of the optical sensor used to capture the image) to another color space. In certain example embodiments, method 200 applies a mapping learned from data using a machine learning technique (such as SYM-regression, a neural network, K-nearest neighbor, locally weighted linear regression, decision tree regression, or any suitable combination thereof).

[0045] In various example embodiments, method 200 also includes adjusting the image in other suitable ways. In some cases, material on the machine-readable code (such as blood) can problematically interfere with texture information, produce false textures, or both, and the location of the machine-readable code region of the image (at operation 240) can be based on corner detection, edge detection, or both. If material on the machine-readable code obscures corners, makes corners appear less clear, produces false corners (e.g., due to glare, clots, etc.), or any combination thereof, method 200 can also include reducing high-frequency noise and preserving or restoring high-frequency signals to maintain the sharpness of edges and corners (e.g., by improving the signal-to-noise ratio). For example, high-frequency noise can be reduced using a suitable smoothing algorithm (e.g., Gaussian blur, median blur, or any suitable combination thereof). Additionally, preserving or restoring one or more high-frequency signals can be achieved, for example, using a suitable deblurring algorithm (e.g., unsharp filtering, a suitable optimization-based algorithm, etc.).

[0046] As another example, the method 200 can reduce high frequency noise while preserving high frequency signals by applying a bilateral filter to the image at a suitable threshold. Additionally or alternatively, a trained neural network or other machine learning model can take as input an image of a machine readable code and output a suitably pre-processed or adjusted image, wherein the model can be trained using manually processed (e.g., manually "cleaned") images processed in a desired manner.

[0047] Locating machine-readable code

[0048] Method 200 (at operation 230) may include locating an area of an image depicting a machine-readable code (e.g., locating a machine-readable code area of the image). Locating the area may involve estimating a position of the machine-readable code within the image, a size of the machine-readable code, a perimeter of the machine-readable code, an orientation of the machine-readable code, any other suitable physical aspect of the machine-readable code, or any suitable combination thereof. In some example embodiments, one or more suitable computer vision techniques are used to find one or more salient features of the machine-readable code, such as corners, straight edges near corners, edge orientations at 90 degrees to each other, aligned edges, certain spatial frequency bands, a bimodal color distribution, or any suitable combination thereof. As an example, method 200 may include finding an L-shaped finder pattern associated with the machine-readable code, an alternating black and white sequential pattern in the image associated with the machine-readable code, or both.

[0049] In some example embodiments, a neural network model (e.g., Fast-RCNN, YOLO, SSD, or other architectures suitable for object detection and localization tasks) or other machine learning model is trained to predict the location of a machine-readable code from an original image or a pre-processed image processed in a manner similar to that described above (e.g., as part of locating the machine-readable code, such as operation 230 in method 200). For example, such a neural network can be trained with a sufficient number of occluded (e.g., blood-stained) images of the machine-readable code to extract features that are robust to occlusion.

[0050] In some example embodiments, a corner detection algorithm (e.g., a Harris corner detection algorithm) is used to obtain a heat map with heat map values for coordinates within an image (e.g., as part of locating a machine-readable code, such as operation 230 in method 200). For example, the heat map value for a coordinate can be generated by analyzing the rate of change (e.g., the rate of change in brightness) within a sliding window as the sliding window slides around the coordinate. For example, the amount of change can be approximated as a quadratic function of the sliding window offset, and the lowest rate of change in any direction can be found and used as the heat map value. Thus, a high heat map value means that the sliding window will change a lot in any direction (e.g., a corner), while a low heat map value means that at least one direction (e.g., a long edge) will not significantly change the window. If the heat map value for a coordinate in the image is above a first predetermined upper threshold, the coordinate can be determined to be a corner coordinate (e.g., a corner of the machine-readable code). Additionally, if the heat map value for a coordinate in the image is above a second predetermined lower threshold (which is lower than the first predetermined threshold), the coordinate can be determined to be a rectangular coordinate (e.g., a coordinate along a side, top, or bottom edge). Any corner coordinates and rectangular coordinates that are spatially close to the corner coordinates can be considered as a set of coordinates of interest. In various example embodiments, fewer thresholds or more thresholds can be used to appropriately classify the coordinates.

[0051] Outlier coordinates (e.g., coordinates that may not be from or correspond to a machine-readable code) can be removed from the coordinates of interest (e.g., as part of locating the machine-readable code, such as in operation 230 of method 200), such as based on the median, the interquartile range, another suitable statistical metric, or any suitable combination thereof. After removing the outlier coordinates, a tightly rotated rectangle can be fit around the remaining coordinates of interest (e.g., under the assumption that machine-readable codes are generally rectangular) as an initial estimate of the location of the machine-readable code, the area of the machine-readable code, the perimeter of the machine-readable code, the orientation of the machine-readable code, or any suitable combination thereof.

[0052] In some example embodiments, the initial estimates of the machine-readable code attributes described above may subsequently be adjusted (e.g., as part of locating the machine-readable code, such as at operation 230 in method 200). For example, the estimated area of the machine-readable code or the boundaries of the machine-readable code may be modified to have an aspect ratio that is the same as or similar to a known aspect ratio (e.g., length to width ratio) of a depicted (e.g., imaged) machine-readable code. Additionally or alternatively, the estimated orientation may be refined by performing a Hough transform within the rectangle and taking the median orientation of the resulting Hough lines (e.g., after rotating some of the Hough lines by 90 degrees to account for the fact that some lines will be perpendicular to any machine-readable code line orientation).

[0053] It should be understood that the above-described application of the Harris corner detection algorithm (e.g., as part of locating a machine-readable code, such as operation 230 in method 200) may be modified for different machine-readable code shapes. For example, other suitable shapes may be fitted around the coordinates of interest, which may depend on the shape of the depicted (e.g., imaged) machine-readable code, if known (e.g., a circle for a circular machine-readable code, a triangle for a triangular machine-readable code, a pentagon for a pentagonal machine-readable code, etc.).

[0054] In some example embodiments, after estimating one possible location of a machine-readable code region within an image, multiple potential machine-readable code locations may be estimated by slightly scaling the estimated machine-readable code location (e.g., expanding or contracting the estimated perimeter of the machine-readable code region) (e.g., as part of locating the machine-readable code, such as operation 230 in method 200). These multiple potential machine-readable code locations may be passed to subsequent processes (e.g., binarization or decoding as described below) with the goal that at least one estimated machine-readable code location will result in a successful decode.

[0055] Binarization

[0056] In certain example embodiments, method 200 (at operation 240) includes further image processing, such as binarizing the region located in at least operation 230 (e.g., the machine-readable code region of the image). As used herein, "binarization" refers to converting an image or a region thereof into only two colors (e.g., a light color, such as white, and a dark color, such as black). Binarization of the located region converts the machine-readable code region of the image into a binary image (e.g., black and white, rather than a grayscale representation with at least three different grayscale shades (e.g., black, gray, and white)), which can have the effect of removing any residual darkening caused by substances (e.g., blood) in the image of the machine-readable code. The located region can be further processed based on local information of the region or otherwise specific to information of the region (such as its color histogram, its orientation histogram, or both). For example, the region can be binarized at least in part using Otsu thresholding (e.g., based on the color histogram of the region). In other example embodiments, posterization or other methods of quantizing color information into more than two resulting colors are used instead of binarization.

[0057] As another example, binarizing a region (e.g., a machine-readable code region) may include fitting a grid shaped like the machine-readable code to an edge map of the region, generating a median color histogram of the resulting grid blocks, and applying Otsu thresholding on the generated median color histogram. Otsu thresholding can be used to determine which grid blocks correspond to light-colored elements in the machine-readable code and which grid blocks correspond to dark-colored elements in the machine-readable code. The lines of the grid can be linear or parameterized by nonlinear equations so that, for example, the grid can be fitted to image regions depicting machine-readable codes that are not composed of straight lines, machine-readable codes that have been bent or distorted, or both. As another example, in some example embodiments, a neural network or other suitable machine learning model can be trained to output a predicted grid shaped like the machine-readable code based on the original image or a pre-processed image processed in a manner similar to those described above. For example, such a neural network can be trained with a sufficient number of manually curated grids.

[0058] Although the above examples are described as using Otsu thresholding, it should be understood that any suitable thresholding technique may be applied to binarize at least one region of an image (eg, a machine-readable code region).

[0059] decoding

[0060] Given an image or a portion thereof (e.g., at least a region depicting a machine-readable code) that has been processed (e.g., "cleaned") as described herein, certain example embodiments of method 200 (at operation 250) include decoding at least the region depicting the machine-readable code. This may be performed by decoding the binarized machine-readable code region of the image to determine the encoded information present in the machine-readable code. Any suitable technique for processing (e.g., reading and decoding) machine-readable code may be applied to obtain the information (e.g., a string of text characters, such as alphanumeric characters) encoded in the machine-readable code. For example, such an algorithm may use vertical and horizontal scan lines to look for an L-shaped finding pattern, an alternating timing pattern, or both in the machine-readable code, and then use the resulting position and scale information to evaluate each element (e.g., a content block) of the machine-readable code. Decoding and error correction methods may then be used to convert the elements (e.g., content blocks) of the machine-readable code into decoded data (e.g., a decoded string).

[0061] In the event that multiple potential locations for the machine-readable code have been estimated or guessed, decoding of the machine-readable code can be performed for each of the potential locations, and the results can be compared with each other. If at least one of the potential locations for the machine-readable code results in a successful decode, a decoded string can be returned (e.g., output for subsequent use). In addition, in some example embodiments, the return of the decoded string can be further conditioned on sufficient consistency between the potential locations. For example, if no two guesses result in conflicting decoded strings, or if a suitable number (e.g., a majority) of guesses result in the same, common decoded string, then the decoded string can be returned.

[0062] Using decoded information

[0063] After returning the decoded string from the machine-readable code, the information in the decoded string can be utilized in any suitable manner, which can depend on the type of encoded information. For example, method 200 can include incrementing a textile counter index based on the decoded information (e.g., incrementing the textile counter index if it is determined that the scanned and decoded machine-readable code is different from other scanned and decoded machine-readable codes). In some example embodiments, multiple textile counter indices can be maintained (e.g., a first index for a surgical sponge, a second index for a chux, etc.). For example, the encoded information can include, in addition to a unique identifier (e.g., a serial number), a type of textile (e.g., a textile type), such that only the corresponding textile counter index for that textile type is incremented in response to the machine-readable code being scanned and decoded.

[0064] Additionally, method 200 may further include outputting the textile counter index, such as by displaying the textile counter index on a display (e.g., as a count of textiles of that type), or outputting the textile counter index (e.g., a count of textiles of that type) as an audible count via an audio device (e.g., a speaker).

[0065] System for processing images of machine-readable codes

[0066] like Figure 1 As shown, according to some example embodiments, a system 100 for processing an image of a machine-readable code includes at least one processor 152 and a memory 154 having instructions stored therein. The processor 152 is configured to execute the stored instructions such that it is configured to: receive an image of a machine-readable code including encoded information, wherein the machine-readable code is at least partially obscured by a substance having a dominant color; generate an adjusted image by adjusting the color space of the image based on the dominant color; binarize at least one region of the image (e.g., a machine-readable code region), wherein the region of the image depicts a machine-readable code; and decode the binarized region of the image to determine the encoded information. In certain example embodiments, the system 100 may be configured to substantially perform the method 200 described in more detail above. The following description will now be made with respect to Figure 4 An example of system 100 is further described.

[0067] like Figure 1As further shown in FIG, system 100 may include a camera 156 configured to obtain (e.g., capture or otherwise generate) one or more images of the machine-readable code, and system 100 may include a display 158 (e.g., a display screen) configured to display the one or more images of the machine-readable code. In some example embodiments, some or all of system 100 may be in an integrated device (e.g., mobile device 150) and placed near a patient during a surgical procedure (e.g., in an operating room) to evaluate patient fluids contained in (e.g., absorbed by) surgical textiles. For example, system 100 may at least partially include a handheld or mobile electronic computing device (e.g., mobile device 150) that may be configured to execute a local fluid analysis application. Such a handheld or mobile electronic computing device may be, for example, or include, a tablet computer, a laptop computer, a mobile smartphone, or any suitable combination thereof, which may include camera 156, processor 152, and display 158. However, in other example embodiments, some or all of the system components may be separated into discrete, interconnected devices. For example, the camera 156, the display 158, or both can be located substantially near the patient during the surgical procedure (e.g., in the operating room), while the processor 152 can be located at a remote location (e.g., separate from the camera 156 or display 158 in the operating room, or outside the operating room) and communicate with the camera 156 and display 158 via a wired connection or a wireless connection or other network.

[0068] In general, one or more processors 152 can be configured to execute instructions stored in memory 154, such that when the processor 152 executes the instructions, the processor 152 performs various aspects of the methods described herein. The instructions can be executed by a computer-executable component within a user computer or other user device (e.g., a mobile device, a wristband, a smartphone, or any suitable combination thereof) that is integrated with an application, applet, host, server, network, website, communication service, communication interface, hardware, firmware, software, or any suitable combination thereof. The instructions can be stored in memory or another computer-readable medium (such as RAM, ROM, flash memory, EEPROM, optical disk (e.g., CD or DVD), hard drive, floppy disk drive, or any other suitable device).

[0069] As described above, one or more processors 152 can be integrated into a handheld or mobile device (e.g., mobile device 150). In other example embodiments, one or more processors 152 are incorporated into a computing device or system (such as a cloud-based computer system, a mainframe computer system, a grid computer system, or other suitable computer system). Additionally or alternatively, one or more processors 152 can be incorporated into a remote server that receives images of surgical textiles, reconstructs such images (e.g., as described above), analyzes such images (e.g., as described above), and transmits quantifications of one or more aspects of fluid in the surgical textile to another computing device, which can have a display for displaying the quantifications to a user. Figure 4 Examples of the one or more processors 152 are further described.

[0070] The system 100 may also include an optical sensor (e.g., in the camera 156) that operates to generate one or more images (such as a set of one or more still images or as part of a video feed). The camera 156 may include at least one optical image sensor (e.g., a CCD, CMOS, etc. that captures a color optical digital image whose pixels have red, green, and blue (RGB) color components), other suitable optical components, or both. For example, the camera 156 may include a single image sensor paired with suitable corresponding optical devices, filters (e.g., a color filter array such as a Bayer pattern filter), or both. As another example, the camera 156 may include multiple image sensors paired with suitable corresponding optical devices (such as at least one prism or diffractive surface) to separate white light into separate color channels (e.g., RGB), where each color channel is detected by a corresponding image sensor. According to various example embodiments, the camera 156 includes any suitable image sensor and other optical components to enable the camera 156 to generate an image.

[0071] The camera 156 can be configured to transmit images to the processor 152 for analysis, to a database storing the images, or to both. As previously described, the camera 156 can be integrated into the same device (e.g., the mobile device 150) as one or more of the other components of the system 100, or the camera 156 can be a separate component that transmits image data to the other components.

[0072] The system 100 may also include a display 158 (e.g., a display screen) that operates to display or otherwise communicate (e.g., present) to a user (e.g., a doctor or nurse) some or all of the information generated by the system 100 (including, but not limited to, patient information, images of surgical textiles, quantitative indicators characterizing fluid in surgical textiles, or any suitable combination thereof). The display 158 may include a screen on a handheld or mobile device, a computer monitor, a television screen, a projector screen, or other suitable display.

[0073] In some example embodiments, the display 158 is configured to display a user interface (e.g., a graphical user interface (GUI)) that enables a user to interact with the displayed information. For example, the user interface may enable a user to manipulate an image (e.g., zoom, crop, rotate, etc.) or manually define an area in the image that depicts at least a machine-readable code. As another example, the user interface may enable a user to select display options (e.g., font, color, language, etc.), select content to be displayed (e.g., patient information, quantitative indicators or other fluid-related information, alarms, etc.), or both. In some such example embodiments, the display 158 is user-interactive and includes a resistive or capacitive touch screen that responds to skin, stylus, or other user contact. In other such example embodiments, the display 158 is user-interactive via a cursor controlled by a mouse, keyboard, or other input device.

[0074] In some example embodiments, system 100 includes an audio system that conveys information to the user. Display 158, the audio system, or both can provide (e.g., present) the current value of the textile counter index, which can help track the use of surgical textiles during the procedure.

[0075] Example

[0076] Figures 3A to 3H are images of 2D machine-readable codes attached to surgical textiles according to some example embodiments. These machine-readable codes are partially obscured to varying degrees by a red dye solution (e.g., water mixed with red food coloring) that simulates blood. For example, Figure 3A The machine-readable code depicted in is only lightly covered by the dye solution (mainly on the outer borders of the machine-readable code). Figures 3B to 3H The machine-readable code depicted in FIG is generally increasingly covered with the dye solution.

[0077] Use a machine-readable code reader application (e.g., configured to read QR codes) on a mobile device (e.g., mobile device 150) to attempt to scan and decode Figures 3A to 3H Each of the machine-readable codes depicted in the machine-readable code. The machine-readable code reader successfully decodes Figure 3AThe machine-readable code, but failed to decode Figures 3B to 3H machine-readable code.

[0078] A color (RGB) image is captured by a camera (e.g., camera 156) and converted to a grayscale representation by isolating and acquiring only the R (red) channel values. For each image, multiple guesses are generated at the region depicting the machine-readable code by applying a Harris corner detector (e.g., implementing the Harris corner detection algorithm), thresholding the heat map values to identify corner coordinates and rectangular coordinates, removing outliers in the corner coordinates and rectangular coordinates, and fitting a rectangle to the resulting remaining coordinates. The machine-readable code region of the image is further pre-processed using Otsu thresholding to binarize the localized region depicting the machine-readable code and then fed into the machine-readable code processing algorithm to search for successfully decoded strings. For each machine-readable code, if there is any successful attempt to decode any guess region among the multiple guesses at the machine-readable code region, and if no decoding results are conflicting, the decoded string is returned. As the process is repeated, for Figures 3A to 3H All machine readable codes shown in , achieved successful decoding of the machine readable code.

[0079] Any one or more components in the components described herein can be implemented using hardware alone or using a combination of hardware and software. For example, any component described herein can physically include the arrangement of one or more processors (for example, processor 152), which are configured to perform the operation described herein for the component. As another example, any component described herein can include software, hardware, or both, which configure the arrangement of one or more processors (for example, processor 152) to perform the operation described herein for the component. Therefore, the different components described herein can include and configure the different arrangements of processors at different time points, or include and configure the single arrangement of such processors at different time points. Each component described herein is an example of a device (means) for performing the operation described herein for the component. In addition, any two or more components described herein can be combined into a single component, and the function described herein for a single component can be broken down in multiple components. In addition, according to various example embodiments, the components described herein as being implemented in a single system or machine (for example, a single device) can be distributed across multiple systems or machines (for example, multiple devices).

[0080] Any of the systems or machines (e.g., devices) discussed herein may be, include, or otherwise be implemented in a special-purpose (e.g., specialized or otherwise non-conventional and non-general purpose) computer that has been modified to perform one or more of the functions described herein for the system or machine (e.g., configured or programmed by special-purpose software, such as one or more software modules of a special-purpose application, operating system, firmware, middleware, or other software program). For example, the following description of Figure 4 Discussion can realize the special-purpose computer system of any one or more methods in the method discussed herein, and therefore this type of special-purpose computer can be the device for performing any one or more methods in the method discussed herein.In the technical field of this type of special-purpose computer, compared with other special-purpose computers that lack the structure discussed herein or otherwise cannot perform the function discussed herein, it is technically improved to perform the special-purpose computer of the function discussed herein by the structure discussed herein particularly modified (for example, by special software configuration).Therefore, the special-purpose machine configured according to the system and method discussed herein provides the improvement to the technology of similar special-purpose machine.In addition, any two or more of the system discussed herein or the machine can be combined into a single system or machine, and the function described in this article for any single system or machine can be broken down in multiple systems or machines.

[0081] Figure 4 is a block diagram illustrating components of a machine 400 (e.g., mobile device 150) capable of reading instructions 424 from a machine-readable medium 422 (e.g., a non-transitory machine-readable medium, a machine-readable storage medium, a computer-readable storage medium, or any suitable combination thereof) and performing, in whole or in part, any one or more of the methodologies discussed herein, according to some example embodiments. Specifically, Figure 4 The machine 400 is shown in the example form of a computer system (e.g., a computer) in which instructions 424 (e.g., software, a program, an application, an applet, an application, or other executable code) for causing the machine 400 to perform any one or more of the methodologies discussed herein may be executed in whole or in part.

[0082] In some embodiments, the machine 400 may be a standalone device or may be communicatively coupled (e.g., networked) to other machines. In a networked deployment, the machine 400 may operate as a server or a client machine in a server-client network environment, or as a peer machine in a distributed (e.g., peer-to-peer) network environment. The machine 400 may be a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a cellular phone, a smartphone, a set-top box (STB), a personal digital assistant (PDA), a network appliance, a network router, a network switch, a network bridge, or any other machine capable of executing instructions 424, sequentially or otherwise, to perform actions specified by the machine. Further, while a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute instructions 424 to perform all or a portion of any one or more of the methodologies discussed herein.

[0083] The machine 400 includes a processor 402 (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any suitable combination thereof), a main memory 404, and a static memory 406, which are configured to communicate with each other via a bus 408. The processor 402 contains solid-state digital microcircuits (e.g., electronic, optical, or both) that can be temporarily or permanently configured by some or all of the instructions 424 so that the processor 402 can be configured to perform, in whole or in part, any one or more of the methods described herein. For example, a set of one or more microcircuits of the processor 402 can be configured to execute one or more modules (e.g., software modules) described herein. In some example embodiments, the processor 402 is a multi-core CPU (e.g., a dual-core CPU, a quad-core CPU, an 8-core CPU, or a 128-core CPU), wherein each of the multiple cores behaves as a separate processor capable of performing, in whole or in part, any one or more of the methods discussed herein. Although the benefits described herein may be provided by a machine 400 having at least a processor 402, these same benefits may be provided by a different kind of machine that does not include a processor (e.g., a purely mechanical system, a purely hydraulic system, or a hybrid mechanical-hydraulic system) if such a machine is configured to perform one or more of the methods described herein.

[0084] The machine 400 may also include a graphics display 410 (e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, a cathode ray tube (CRT), or any other display capable of displaying graphics or video). The machine 400 may also include an alphanumeric input device 412 (e.g., a keyboard or keypad), a pointer input device 414 (e.g., a mouse, a touchpad, a touch screen, a trackball, a joystick, a stylus, a motion sensor, an eye tracking device, a data glove, or other pointing tool), a data storage device 416, an audio generating device 418 (e.g., a sound card, an amplifier, a speaker, a headphone jack, or any suitable combination thereof), and a network interface device 420.

[0085] The data storage 416 (e.g., a data storage device) includes a machine-readable medium 422 (e.g., a tangible and non-transitory machine-readable storage medium) on which are stored instructions 424 embodying any one or more of the methods or functions described herein. The instructions 424 may also reside, completely or at least partially, within the main memory 404, within the static memory 406, within the processor 402 (e.g., within a cache memory of the processor), or any suitable combination thereof, before or during execution thereof by the machine 400. Thus, the main memory 404, the static memory 406, and the processor 402 may be considered machine-readable media (e.g., a tangible and non-transitory machine-readable medium). The instructions 424 may be transmitted or received over the network 490 via the network interface device 420. For example, the network interface device 420 may transmit the instructions 424 using any one or more transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)).

[0086] In some example embodiments, the machine 400 may be a portable computing device (e.g., a smartphone, tablet, or wearable device) and may have one or more additional input components 430 (e.g., sensors or meters). Examples of such input components 430 include an image input component (e.g., one or more cameras), an audio input component (e.g., one or more microphones), a directional input component (e.g., a compass), a position input component (e.g., a Global Positioning System (GPS) receiver), an orientation component (e.g., a gyroscope), a motion detection component (e.g., one or more accelerometers), an altitude detection component (e.g., an altimeter), a temperature input component (e.g., a thermometer), and a gas detection component (e.g., a gas sensor). Input data collected by any one or more of these input components 430 may be accessed and made available to any of the modules described herein (e.g., with appropriate privacy notices and protections, such as opt-in or opt-out consent implemented according to user preferences, applicable regulations, or any suitable combination thereof).

[0087] As used herein, the term "memory" refers to a machine-readable medium capable of storing data temporarily or permanently, and may be considered to include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. Although the machine-readable medium 422 is shown as a single medium in the example embodiment, the term "machine-readable medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store instructions. The term "machine-readable medium" should also be considered to include any medium or combination of multiple media capable of carrying (e.g., storing or transmitting) instructions 424 for execution by the machine 400, such that the instructions 424, when executed by one or more processors (e.g., processor 402) of the machine 400, cause the machine 400 to perform, in whole or in part, any one or more of the methods described herein. Thus, "machine-readable medium" refers to a single storage device or device, as well as a cloud-based storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, one or more tangible and non-transitory data storage repositories (e.g., data volumes) in the form of, for example, solid-state memory chips, optical disks, magnetic disks, or any suitable combination thereof.

[0088] As used herein, "non-transitory" machine-readable media specifically does not include propagated signals themselves. According to various example embodiments, instructions 424 for execution by machine 400 may be transmitted via a carrier medium (e.g., a machine-readable carrier medium). Examples of such carrier media include non-transitory carrier media (e.g., non-transitory machine-readable storage media, such as solid-state memory that can be physically moved from one place to another) and transitory carrier media (e.g., a carrier wave or other propagated signal that transmits instructions 424).

[0089] The various operations of the example methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily configured or permanently configured, such processors can constitute a processor-implemented module that operates to perform one or more operations or functions described herein. As used herein, a "processor-implemented module" refers to a hardware module, wherein the hardware includes one or more processors. Therefore, the operations described herein can be at least partially processor-implemented, hardware-implemented, or both, because a processor is an example of hardware, and at least some of the operations within any one or more of the methods discussed herein can be performed by one or more processor-implemented modules, hardware-implemented modules, or any suitable combination thereof.

[0090] In addition, such one or more processors can perform operations in a "cloud computing" environment or as a service (e.g., in a "software as a service" (SaaS) implementation). For example, at least some of the operations within any one or more of the methods discussed herein can be performed by a group of computers (e.g., as an example of a machine including a processor), where these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)). The execution of certain operations can be distributed among one or more processors (whether residing only within a single machine or deployed across multiple machines). In some example embodiments, one or more processors or hardware modules (e.g., processor-implemented modules) can be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, one or more processors or hardware modules can be distributed across multiple geographic locations.

[0091] Throughout this specification, plural instances can be implemented as the parts, operations or structures described as single instances. Although the individual operations of one or more methods are shown and described as separate operations, one or more operations in the individual operations can be performed simultaneously, and there is no requirement to perform the operations in the order shown. The structure and function thereof presented as separate parts and functions in the example configuration can be implemented as a combined structure or parts with combined functions. Similarly, the structure and function presented as a single component can be implemented as separate parts and functions. These and other variations, modifications, additions and improvements fall within the scope of the subject matter herein.

[0092] Some parts of the subject matter discussed herein can be presented in terms of an algorithmic representation or symbolic representation of data stored as a bit or binary digital signal in a memory (e.g., a computer memory or other machine memory) that operates. Such algorithmic representation or symbolic representation is a technical example used by those of ordinary skill in the art of data processing to convey the essence of their work to other persons of skill in the art. As used herein, an "algorithm" is a self-consistent sequence of operations or similar processes that result in a desired result. In this context, algorithms and operations involve the physical manipulation of physical quantities. Typically, but not necessarily, such quantities can take the form of electrical, magnetic, or optical signals that can be stored, accessed, transferred, combined, compared, or otherwise manipulated by a machine. Sometimes, primarily for commonly used reasons, it is convenient to use words such as "data," "content," "bit," "value," "element," "symbol," "character," "item," "number," "digit," etc. to refer to such signals. However, these words are merely convenient labels and are to be associated with appropriate physical quantities.

[0093] Unless otherwise specifically stated, discussions herein using words such as "access," "process," "detect," "calculate," "calculate," "determine," "generate," "present," "display," and the like refer to actions or processes that can be performed by a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities in one or more memories (e.g., volatile memory, non-volatile memory, or any suitable combination thereof), registers, or other machine components that receive, store, transmit, or display information. Furthermore, unless otherwise specifically stated, the terms "a" or "an" are used herein (as is common in patent documents) to include one or more than one instance. Finally, as used herein, the conjunction "or" refers to a non-exclusive "or" unless otherwise specifically stated.

[0094] The following enumerated descriptions describe various examples of the methods, machine-readable media, and systems (eg, machines, devices, or other apparatuses) discussed herein.

[0095] A first example provides a method comprising:

[0096] accessing, by one or more processors of the machine, an image depicting a machine-readable code, the machine-readable code being at least partially obscured in the image by a substance having a dominant color;

[0097] generating, by one or more processors of the machine, an adjusted version of the image by adjusting a color space of the image based on a dominant color of a substance that at least partially obscures the machine-readable code; and

[0098] At least one region of the adjusted version of the image is binarized, by one or more processors of the machine, the region depicting the machine-readable code.

[0099] A second example provides the method according to the first example, further comprising: capturing, by the optical sensor, an image depicting a machine-readable code, the machine-readable code being at least partially obscured by a substance having a dominant color.

[0100] A third example provides the method according to the first or second example, wherein a dominant color of the substance at least partially obscuring the machine-readable code is substantially red.

[0101] A fourth example provides a method according to any one of the first to third examples, wherein:

[0102] The image is a color image; and

[0103] Adjustment of the color space of an image includes converting the color space of a color image into a grayscale representation based on the dominant color of the substance.

[0104] A fifth example provides the method according to any one of the first to fourth examples, wherein binarizing at least the region of the image includes color thresholding a histogram of at least the region of the image.

[0105] A sixth example provides the method according to any one of the first to fifth examples, further comprising: locating an area depicting a machine-readable code in the adjusted version of the image.

[0106] A seventh example provides a method according to the sixth example, wherein locating an area depicting a machine-readable code in the adjusted version of the image comprises performing at least one of: corner detection on the adjusted version of the image or edge detection on the adjusted version of the image.

[0107] An eighth example provides a method according to any of the first to seventh examples, wherein the image depicts a machine-readable code attached to a surgical textile soiled with a substance having a dominant color.

[0108] A ninth example provides the method of any one of the first to eighth examples, wherein the machine-readable code represents encoded information including at least one of a type of the surgical textile or an identifier of the surgical textile.

[0109] The tenth example provides a method according to any one of the first to ninth examples, further including: determining the encoded information represented by the machine-readable code by decoding a binary area depicting the machine-readable code (for example, by decoding a binary area in which an image depicts the machine-readable code).

[0110] An eleventh example provides the method according to the tenth example, further comprising:

[0111] In response to determining the encoded information represented by the machine-readable code, a weave counter index is incremented.

[0112] A twelfth example provides a system (e.g., a computer system) comprising:

[0113] one or more processors; and

[0114] A memory storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including:

[0115] accessing an image depicting a machine-readable code, the machine-readable code being at least partially obscured in the image by a substance having a dominant color;

[0116] generating an adjusted version of the image by adjusting a color space of the image based on a dominant color of a substance that at least partially obscures the machine-readable code; and

[0117] At least one region of the adjusted version of the image is binarized, the region depicting the machine-readable code.

[0118] A thirteenth example provides the system according to the twelfth example, further comprising an optical sensor configured to capture an image depicting a machine-readable code that is at least partially obscured by a substance having a dominant color.

[0119] A fourteenth example provides the system according to the twelfth or thirteenth example, wherein a dominant color of the substance that at least partially obscures the machine-readable code is substantially red.

[0120] A fifteenth example provides a system according to any one of the twelfth to fourteenth examples, wherein:

[0121] The image is a color image; and

[0122] Adjustment of the color space of an image includes converting the color space of a color image into a grayscale representation based on the dominant color of the substance.

[0123] A sixteenth example provides the system according to any one of the twelfth to fifteenth examples, wherein binarizing at least the region of the image comprises color thresholding a histogram of at least the region of the image.

[0124] A seventeenth example provides the system of any of the twelfth to sixteenth examples, wherein the operations further comprise: locating an area depicting the machine-readable code in the adjusted version of the image.

[0125] An eighteenth example provides a system according to the seventeenth example, wherein locating an area depicting a machine-readable code in the adjusted version of the image comprises performing at least one of: corner detection on the adjusted version of the image or edge detection on the adjusted version of the image.

[0126] A nineteenth example provides the system of any of the twelfth to eighteenth examples, wherein the image depicts a machine-readable code attached to a surgical textile soiled with a substance having a dominant color.

[0127] A twentieth example provides the system according to any one of the twelfth to nineteenth examples, wherein the machine-readable code represents encoded information including at least one of a type of the surgical textile or an identifier of the surgical textile.

[0128] The twenty-first example provides a system according to any one of the twelfth to twentieth examples, wherein the operation further includes: determining the encoded information represented by the machine-readable code by decoding a binarized region depicting the machine-readable code (for example, by decoding a binarized region in which an image depicts the machine-readable code).

[0129] A twenty-second example provides the system according to any one of the twelfth to twenty-first examples, wherein the operations further comprise: incrementing a weave counter index in response to determining the encoded information represented by the machine-readable code.

[0130] The twenty-third example provides a machine-readable medium (e.g., a non-transitory machine-readable storage medium) including instructions that, when executed by one or more processors of a machine, cause the machine to perform operations including:

[0131] accessing an image depicting a machine-readable code, the machine-readable code being at least partially obscured in the image by a substance having a dominant color;

[0132] generating an adjusted version of the image by adjusting a color space of the image based on a dominant color of a substance that at least partially obscures the machine-readable code; and

[0133] At least one region of the adjusted version of the image is binarized, the region depicting the machine-readable code.

[0134] The twenty-fourth example provides a carrier medium carrying machine-readable instructions for controlling a machine to perform the operations (eg, method operations) performed in any one of the examples described above.

Claims

1. A computer-implemented method for processing an image of a machine-readable code attached to an item of a surgical system, the surgical system comprising a display and one or more processors, the method comprising: accessing, by the one or more processors, an image depicting machine-readable code; determining, with the one or more processors, that at least one corner point of the machine-readable code is occluded, spurious, or blurred such that a region of the machine-readable code in the image is not localizable; providing, with the one or more processors, the image as input to a trained neural network; receiving, with the one or more processors, a cleaned image of the machine-readable code as output from the trained neural network; locating, with one or more processors, a machine-readable code region of the cleaned image of the machine-readable code; decoding, with one or more processors, encoded information contained within the machine-readable code region of the cleaned image; outputting the decoded information using one or more processors; and Patient information generated by the surgical system based on the decoded information is displayed on the display.

2. The method of claim 1 , wherein the step of locating the machine-readable code region further comprises performing at least one of corner detection or edge detection by: applying a corner detection algorithm to the cleaned image to generate a heat map comprising a heat map value corresponding to each coordinate within the image; and Corners of a machine-readable code in a clean image are detected based on a heat map generated by a corner detection algorithm.

3. The method of claim 1 , further comprising, using the trained neural network: identifying a set of coordinates in the image as potentially corresponding to a machine-readable code; identifying at least one outlier coordinate in the set of coordinates that is unlikely to correspond to a machine-readable code; and The at least one outlier coordinate is removed to generate a clean set of coordinates.

4. The method according to claim 3, wherein the step of decoding the encoded information further comprises: identifying the set of clean coordinates; and A portion of the cleaned image corresponding to the set of cleaned coordinates is analyzed. The method of claim 1 , further comprising reducing high frequency noise of the image to generate the cleaned image.

6. The method of claim 5, further comprising applying at least one smoothing algorithm to reduce high frequency noise in the image. 7 . The method of claim 5 , further comprising applying a bilateral filter to the image to reduce high frequency noise of the image while preserving high frequency signals.

8. The method of claim 1, further comprising increasing a signal-to-noise ratio of the image to generate the cleaned image.

9. The method of claim 8, further comprising applying at least one deblurring algorithm to the image to increase the signal-to-noise ratio.

10. The method of claim 1 , wherein the step of determining that at least one corner point of the machine-readable code is occluded, falsified, or blurred further comprises: Applying a corner detection algorithm to the image; and At least one corner of the machine-readable code is identified as being occluded, spurious, or blurred by analyzing the output of the corner detection algorithm.

11. The method according to claim 1, wherein The trained neural network is trained based on manually cleaned images of machine-readable codes and / or images of masked machine-readable codes.

12. A computer-implemented method for processing an image of a machine-readable code attached to an item of a surgical system, the surgical system comprising a display and one or more processors, the method comprising: accessing, by the one or more processors, an image depicting machine-readable code; determining, by the one or more processors, that at least one corner point of the machine-readable code is occluded, aliased, or blurred; providing, by the one or more processors, the image as input to a trained neural network; receiving, by the one or more processors, a cleaned image of the machine-readable code as output from the trained neural network; applying a corner detection algorithm to the cleaned image to generate a heat map including a heat map value corresponding to each coordinate within the image; detecting corners of the machine-readable code in the clean image based on a heat map generated by a corner detection algorithm; decoding, with one or more processors, encoded information contained in the machine-readable code of the cleaned image; outputting the decoded information using one or more processors; and Patient information generated by the surgical system based on the decoded information is displayed on the display.

13. The method of claim 12, further comprising reducing high frequency noise of the image to generate the cleaned image. The method of claim 12 , further comprising increasing a signal-to-noise ratio of the image to generate the cleaned image.

15. The method of claim 12, further comprising, utilizing the trained neural network: identifying a set of coordinates in the image as potentially corresponding to a machine-readable code; identifying outlier coordinates in the set of coordinates that are unlikely to correspond to a machine-readable code; and Remove outlier coordinates to produce a clean set of coordinates.

16. A computer-implemented method for processing an image of a machine-readable code attached to an item of a surgical system, the surgical system comprising a display and one or more processors, the method comprising: accessing, by the one or more processors, an image depicting machine-readable code; determining, by the one or more processors, that at least one corner point of the machine-readable code is occluded, aliased, or blurred; providing, by the one or more processors, the image as input to a trained neural network; Processing at least one region of the image with the trained neural network to produce a cleaned image by: identifying a set of coordinates in the image that may correspond to a machine-readable code; removing outlier coordinates from the set of coordinates to generate a clean set of coordinates corresponding to the estimated location of the machine-readable code; receiving, by the one or more processors, a cleaned image of the machine-readable code as output from the trained neural network; decoding, with the one or more processors, encoded information contained in a machine-readable code of the cleaned image; outputting the decoded information with the one or more processors; and Patient information generated by the surgical system based on the decoded information is displayed on the display.

17. The method of claim 16, wherein the step of decoding the encoded information further comprises: identifying the set of clean coordinates; and A portion of the cleaned image corresponding to the set of cleaned coordinates is analyzed.

18. The method of claim 16, further comprising reducing high frequency noise of the image to generate the cleaned image.

19. The method of claim 18, further comprising applying at least one smoothing algorithm to reduce high frequency noise in the image.

20. The method of claim 16, wherein the step of locating the machine-readable code further comprises performing at least one of corner detection or edge detection by: applying a corner detection algorithm to the cleaned image to generate a heat map comprising a heat map value corresponding to each coordinate within the image; and Corners of a machine-readable code in a clean image are detected based on a heat map generated by a corner detection algorithm.

21. A computer-implemented method for processing an image of a machine-readable code attached to an item of a surgical system, the surgical system comprising a display and one or more processors, the method comprising: accessing, with the one or more processors, an image depicting machine-readable code; determining, with the one or more processors, that the machine-readable code is not localizable within the image based on at least a portion of the machine-readable code being obscured, occluded, smeared, or blurred; providing, with the one or more processors, the image as input to a trained neural network; Refine the image using a trained neural network to make machine-readable codes localizable; decoding information encoded within a machine-readable code region of the image using the trained neural network; and Fluid-related information is displayed on a display based on the decoded information.

22. A computer-implemented method for processing an image of a machine-readable code attached to an item of a surgical system, the surgical system comprising a display and one or more processors, the method comprising: accessing, by the one or more processors, a pre-processed image depicting a machine-readable code; providing, with the one or more processors, a pre-processed image as input to a trained neural network; generating, using the trained neural network, an adjusted image in which the orientation of the machine-readable code has been refined; decoding, using the trained neural network, information encoded within the machine-readable code region of the adjusted image; and Fluid-related information is displayed on a display based on the decoded information.

23. A computer-implemented method for processing images using a surgical system comprising a display and one or more processors, the method comprising: receiving, at the one or more processors, a pre-processed image depicting a machine-readable code attached to an item containing a fluid, the fluid comprising patient blood; providing, with the one or more processors, the pre-processed image as input to a trained neural network; receiving, with the one or more processors, a processed image of the machine-readable code as output from a trained neural network; decoding, with the one or more processors, information encoded within the machine-readable code, wherein the decoded information is unique to the article to which the machine-readable code is attached; and Fluid-related information is displayed on a display based on the decoded information.

Images