Continuous video-based product inspection

Abstract

Disclosed herein is a product identification system that includes one or more image capture devices that provide video data representing an enhanced security environment. The product identification system can determine the continuous presence of a specified product during a period of time, for example, during the performance of a mission in an enhanced security environment. The video data can be spatiotemporally sliced ​​according to multiple orientations relative to a time axis and two spatial axes. Multiple data streams are created, wherein the slices of each corresponding data stream have the same orientation. The data streams are processed according to one or more machine learning models, such as a convolutional neural network trained to determine whether a specified product is continuously present in the video data. An indication indicating the continuous presence of the specified product can be provided to a downstream system and used to indicate whether a function of the downstream system is to be performed.

Description

Technical Field

[0001] The present disclosure relates to the field of product authentication, and more particularly to video-based continuous product authentication. Background Art

[0002] Medical procedures can present many challenges to medical professionals. For example, depending on the procedure being performed on the patient, there may be many different products and instruments that are appropriate, and each of these products and instruments may have best practices associated with them. If a particular product is incorrectly selected or used, there may be serious consequences for all parties involved, including the patient, the medical professional who selects and / or uses the product, and the manufacturer of the product. Summary of the invention

[0003] Generally, the present disclosure describes techniques for performing continuous product authentication in an environment where medical procedures may be performed. Conventional systems may be able to identify objects in two-dimensional (2D) static images. However, one technical problem with such conventional systems is that they do not provide any method to continuously determine the presence and authenticity of a product over a period of time. For example, a static image does not necessarily include or otherwise convey information over a period of time, but rather provides at most a snapshot of a particular moment. Therefore, conventional systems may have difficulty using static images to capture information such as how a product is used over time, changes in the appearance of a product after use or application, changes in the way a product is used throughout a period of interest, or even whether the same product or the same type of product is used appropriately or even completely continuously throughout the period of time.

[0004] One example where the continuous identification technology described herein may be desirable is during a medical procedure. For example, two sterilized products, "Brand A" and "Brand B" sterilized products, may be used, each with a different application technology. A medical professional may decide to use the "Brand A" sterilized product. Being able to determine which product has been identified and is in continuous presence or use may be useful for automatically providing guidance materials to product-specific medical professionals. Therefore, being able to determine the continuous presence of an identified product can be used to prevent the presentation of erroneous guidance materials to medical professionals. In addition, the continuous presence of an identified product can also be automatically recorded to provide proof that a specified product has been used (or not used, as the case may be). In addition, other systems can also use the continuous presence of a product to determine whether the product is used in the correct manner. In addition, information collected from the continuous identification process can be provided to other systems for use. For example, the corresponding feature map extracted by the neural network layer of the system during the continuous identification process described herein can be provided to a downstream system.

[0005] As a technical solution to the aforementioned technical problems, some aspects of the present disclosure include a product identification training system that uses video data to train a machine learning model to identify products that can be used in an environment that requires an enhanced level of security. Examples of such environments include medical environments, such as operating rooms in hospitals, same-day surgery facilities, or other medical facilities. For example, a machine learning system can be used to train a neural network. A machine learning model can be trained to identify various types of products. Examples of such products include materials, medical devices, protective materials, etc. applied to a patient during surgery. A machine learning model can be trained to identify various aspects of a product. For example, a machine learning model can be trained to identify a container for a product. In addition, a machine learning model can be trained to identify the appearance of a product before it is applied to a patient, the appearance of a product when it is initially applied to a patient, and / or the appearance of a product that changes over time.

[0006] The trained machine learning model can be deployed for use by a supporting product identification system in environments with elevated safety concerns, such as hospitals, same-day surgery facilities, clinics, long-term care facilities, sterile processing environments, hazardous materials environments, or other such environments where product safety concerns may make continuous product identification desirable. As an example, an operating room in a hospital may be equipped with one or more cameras to monitor the preparation and performance of medical procedures. Video data from one or more cameras is provided to a product identification system. The product identification system can apply the trained machine learning model to the video data to determine whether a specified product is present in the video data. Other aspects of the product can also be identified from the video data. For example, dosage or quantity information, expiration dates, etc. can be identified from containers marked or coded with products. More information that can be identified from the video data includes the time of care provided, the duration of care provided (e.g., the length of time a wound dressing or catheter is used), patient position (e.g., appropriate angles to reduce the risk of ventilator-associated pneumonia, positions that may cause pressure sores), hand hygiene compliance, compliance with other best practices that may be established by the healthcare institution or specified in official guidelines (such as the use of alcohol-based patient skin disinfectants, incision cloths, antibiotics before skin incisions, control of metabolites (such as glucose), etc.).

[0007] If a specified product is identified within the video data, the product identification system can provide the identified aspects (e.g., presence of an indicator, quantity, dosage, expiration date, etc.) to a downstream system, which can use one or more of the identified aspects to perform further functions. For example, a downstream system may be able to identify activities associated with the product because these activities are performed by the end user of the product. In response to knowing that the identified product is being used and in response to the activities associated with the product, the downstream system can provide various types of feedback to the end user of the product specific to the identified product. For example, when identifying the product, the downstream instruction system can present information about a video monitor or other output device in the operating room. This information may include product-specific product usage information, product data, warnings, contraindications, etc. In addition, when identifying the identified product in the video data, the downstream system can use the identified aspects of the identified product to determine whether the activities performed by the medical professional are performed correctly.

[0008] The aspects described above and other aspects described below can provide technical improvements to conventional systems. For example, unlike the system described herein, conventional systems are generally unable to process video data as a three-dimensional (3D) spatiotemporal entity for identifying products, and are unable to determine whether a specified product is continuously present in a medical procedure environment based on the spatiotemporal analysis of video data. The spatiotemporal analysis techniques described herein can capture features corresponding to changes in product appearance over time, with more advantages than conventional systems. The spatiotemporal analysis techniques described herein can also be used to capture features of motion and behavior of user interaction with a product, regardless of its appearance, and use such features to identify and / or identify a product. For example, due to the design of the product, different product designs may result in certain motions or behaviors. The techniques described herein can capture such motions and behaviors and use them to help identify products. In addition, the techniques described herein can capture past context while enabling the system to make decisions about product authenticity at various times.

[0009] In addition, the aspects described above and other aspects described below can provide one or more practical applications, such as continuously identifying products and providing an indication of authenticity and continuing the presence of the product to a downstream system, which can confirm when the identified product is used and whether the identified product is being used correctly. In addition, the downstream system can provide real-time feedback on the use of the product by medical professionals.

[0010] In one or more aspects of the embodiments, a product identification system includes at least one image capture device configured to capture video data representing an enhanced security environment; a machine learning model trained to identify a specified product; and at least one computing device including a memory and one or more processors connected to the memory, wherein the memory includes instructions that cause the one or more processors to continuously perform operations, the operations including: determining a three-dimensional data window from the video data, the three-dimensional data window having a first spatial axis, a second spatial axis, and a time axis; generating at least one data stream from the three-dimensional data window including data that changes along the time axis of the three-dimensional data window, at least one data stream having multiple slices of the three-dimensional data window; determining the continuous presence of a specified product in the enhanced security environment based at least in part on the at least one data stream and the machine learning model; and providing an indication of the continuous presence of the specified product to a downstream system.

[0011] In one or more additional aspects of the embodiments, a method includes capturing video data representing an enhanced security environment via at least one video capture device; continuously performing operations during at least a portion of one or more actions taken relative to the enhanced security environment, the operations including: determining a three-dimensional data window from the video data, the three-dimensional data window having a first spatial axis, a second spatial axis, and a time axis; generating at least one data stream from the three-dimensional data window including data that changes along a time axis of the three-dimensional data window, at least one data stream having multiple slices of the three-dimensional data window; determining the continuous presence of a specified product in the enhanced security environment based at least in part on the at least one data stream and a machine learning model trained to identify the specified product; and providing an indication identifier representing the continuous presence of the specified product to a downstream system.

[0012] In aspects of one or more additional embodiments, a non-transitory computer-readable medium includes instructions that, when executed, cause one or more processors of a product identification system to continuously perform operations during at least a portion of one or more tasks in an enhanced security environment, the operations comprising: receiving video data representing the enhanced security environment; determining a three-dimensional data window having a first spatial axis, a second spatial axis, and a time axis from the video data; generating at least one data stream from the three-dimensional data window, at least one data stream including data that changes along a time axis of the three-dimensional data window, at least one data stream having multiple slices of the three-dimensional data window; determining the continuous presence of a specified product in the enhanced security environment based at least in part on the multiple data streams and a machine learning model; and providing an indication to a downstream system indicating the continuous presence of the specified product.

[0013] The details of one or more examples of the disclosure are shown in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram illustrating an example medical procedure environment including a product authentication system in accordance with various techniques of the present disclosure.

[0015] Figure 2A is a diagram depicting an example product identification training system operating in accordance with the techniques of this disclosure.

[0016] Figure 2B is a diagram depicting an example product authentication system operating in accordance with the techniques of this disclosure.

[0017] Figure 3 is a block diagram depicting an example of video data provided by the techniques of the present disclosure Figure 1 , Figure 2A and Figure 2B The illustrated exemplary embodiments of the product identification training system and the product identification system are processed.

[0018] Figure 4 is a block diagram depicting an example of video data provided by the techniques of the present disclosure Figure 1 , Figure 2A and Figure 2B The illustrative examples of the product identification training system and product identification system shown are divided into two-dimensional slices.

[0019] Figure 5 is a block diagram depicting an example of a data slice provided by the techniques of the present disclosure Figure 1 , Figure 2A and Figure 2B The convolutional neural network and neural network fusion layers in the exemplary examples of the product identification system and product identification training system shown are processed.

[0020] Figure 6 is a diagram depicting an example data slice provided by a convolutional neural network using the present invention's techniques Figure 1 , Figure 2A and Figure 2B A block diagram of processes performed by an illustrative example of a product identification system and a product identification training system is shown.

[0021] Figure 7 is a flow chart illustrating example operations of a method for continuous product authentication according to aspects of the present disclosure.

[0022] It should be understood that embodiments may be utilized and structural changes may be made without departing from the scope of the present invention. The figures are not necessarily drawn to scale. Like numbers used in the figures refer to like components. However, it should be understood that the use of numbers to refer to components in a given figure is not intended to limit the components labeled with the same number in another figure. DETAILED DESCRIPTION

[0023] Figure 1 1 is a block diagram illustrating an example medical procedure environment 100 including a product identification system 104 in accordance with various techniques of the present disclosure. The medical procedure environment can be any of many different types of environments, including a hospital, a clinic, a same-day surgery facility, etc. Aspects of the present disclosure are not limited to any particular medical procedure environment and are not limited to medical procedure environments. For example, as described above, aspects of the present disclosure can be applied in any environment where continuous product identification is desired, such as an environment where safety concerns are elevated.

[0024] Figure 1 The example shown in depicts a patient 120 undergoing a medical procedure performed by a medical professional 114 using a product 116. Figure 1 In the example shown, product 116 includes an applicator containing an antiseptic solution that can be applied to patient 120 in preparation for surgery. The medical procedure in this example is the process of applying product 116 to patient 120. Figure 1 In the example shown, product 116 has been applied to patient 120 as indicated by portion 122 .

[0025] Products 116 may be any type of product used in a medical procedure. Examples of such products include products to be applied to a patient, products to be given to a patient (e.g., medications), products used by clinicians (hand sanitizers, gloves, personal protective equipment, etc., such as protective shields, surgical gowns, etc.), medical instruments used during medical procedures, etc., products used to clean environments (such as operating rooms between cases, patient rooms, etc.). Products may also be surgical instruments that have been cleaned, packaged, and sterilized.

[0026] During a medical procedure, image capture devices 102A and 102B (collectively, “image capture devices 102”) capture video data or other image data during the procedure and provide the video data or other image data to product identification system 104. The image capture devices may be cameras, infrared cameras, or other types of sensors that can provide image data streams to product identification system 104. In some aspects, streams from multiple image capture devices may be submitted and processed by product identification system 104 in parallel with one another.

[0027] The product identification system 104 receives the video data and processes the video data to determine whether a specified product (e.g., product 116) is identified in the video data. The specified product can be a product that the product identification system is designed or trained to identify. The product identification system 104 can be designed and trained to identify more than one specified product. As an example, the product identification system 104 can be designed to identify one or more specified products. The products can be from the same manufacturer or from different manufacturers. In some aspects, the product identification system 104 processes the video data using a machine learning model 134 to determine whether a specified product is identified in the video data, as will be further described below.

[0028] Therefore, in Figure 1 In the example shown, the product identification system 104 can receive one or more video data streams and process the video data using a trained machine learning model 134 to identify the product 116. The machine learning model 134 can be trained to use one or more combinations of the following to generate a decision as to whether the product 116 is authentic: the appearance of the product 116 container, the actions taken in applying the product 116, the appearance of the product in the section 122 applied to the patient 120, and the change in the appearance of the product 116 applied in the section 122. From the above, it can be understood that the use of video data and machine learning models 134 as described herein can provide continuous identification of the product 116 and can increase the accuracy of product identification. Therefore, the use of spatiotemporal data provided by the video stream is superior to traditional systems that utilize static 2D images because such traditional systems generally lack the ability to perform the spatiotemporal analysis described herein.

[0029] If the product identification system 104 identifies the specified product in the video data, the product identification system 104 may provide an indicator that the specified product is identified in the video data. The product identification system 104 continuously processes the video data to continuously determine whether the specified product is still identified in the video data. In some aspects, the product identification system 104 may provide an indicator when the specified product is initially identified in the video data, and provide a different indicator when the specified product is no longer identified in the video data. In other aspects, the product identification system 104 may continuously provide an indicator as long as the specified product is identified in the video data, and cease providing an indicator when the specified product is not identified in the video data.

[0030] In some aspects, in addition to identifying an indicator of a specified product in the video data, other attributes of the specified product may be identified and provided in the video data. For example, the product identification system 104 may analyze the video data and determine the quantity of the specified product, the dosage of the specified product, the expiration date of the specified product, the serial number of the specified product, or other information associated with the product that can be determined or inferred from the video data. Figure 1 In the example shown, performing these additional tasks can provide additional output streams linked to specific objective functions. For example, to locate the serial number attribute, the output can be an 8-dimensional vector representing the bounding box containing the serial number. In this case, four pairs of coordinate values ​​can be represented by these 1×8 dimensional vectors. The contents of the bounding box can then be provided to a neural network trained to output the serial number. In some aspects, the neural network trained to recognize these other attributes can be separate from the machine learning model 134 and can be connected in series with the machine learning model 134.

[0031] The indication identification (and associated data (if any)) can be provided to the downstream system 106 for further operations with respect to the specified product. For example, the downstream system 106 can be a recording system that can be logged when the specified product is used in a medical procedure facility. As another embodiment, the downstream system can be an indication system that can provide guidance content about the use of the specified product in response to the presence of the specified product in the video data. As a further embodiment, the downstream system can be a product use identification system that performs analysis of the video data to determine whether the specified product is used correctly. Instruction content, suggestion content, or other types of content can be displayed on the monitor 108 and played via the speaker 110 by the downstream system 106, and the specified product is identified as being present in the video data. The downstream system 106 can perform various functions. For example, the downstream system 106 can present a score representing the task completeness involving the specified product based on the regression task performed using the machine learning model 134. As another example, the downstream system 106 can highlight the area that requires further effort / attention from the medical expert. In some aspects, the downstream system 106 can receive a feature vector or feature map generated using the machine learning model 134 from the product identification system 104.

[0032] The product identification system 104 and the downstream system 106 may be located in the same room as the room where the medical procedure is being performed, in the same facility (e.g., hospital, clinic, etc.) where the medical procedure is being performed, or they may be located remotely from the facility and coupled to equipment in the facility via a wired or wireless communication system. Furthermore, the product identification system 104 and the downstream system 106 may be in different locations from each other.

[0033] Figure 2A is a diagram depicting an example product identification training system 202 operating in accordance with techniques of this disclosure. Product identification training system 202 can execute training application 218 that can train machine learning model 134 to continuously identify products as they appear in a video data stream.

[0034] In this embodiment, the product identification training system includes one or more processors 204 and memory 206. In some embodiments, the processor provides a computer platform for executing an operating system 216, which can be, for example, a real-time multitasking operating system or other types of operating systems. In turn, the operating system 216 provides a multitasking operating environment for executing one or more software components 217. The processor 204 is connected to one or more I / O interfaces 214, which provide I / O interfaces for communicating with devices such as keyboards, controllers, display devices, image capture devices, other computing systems, etc. In addition, the one or more I / O interfaces 214 may include one or more wired or wireless network interface controllers (NICs) for communicating with a network. In addition, the processor 204 can be connected to an electronic display 208 and an image capture device 138. In some embodiments, the processor 204 and the memory 206 can be independent, discrete components. In other embodiments, the memory 206 can be an on-chip memory co-located with the processor 204 within a single integrated circuit.

[0035] The software component 217 of the product identification training system 202 is used to train the machine learning model 134 to recognize the continuous presence of a specified product in the video data. In this embodiment, the software component 217 includes a training application 218, a video data processor 220, and a machine learning engine 222.

[0036] In some aspects, the training database 230 includes training data 232, which may include many video records of various aspects of the product. For example, the video record may include a record of the container for the product and the relevant markings on the container (logo, quantity, dosage information, effective data, serial number, etc.). The video record may also include a record of the use of the product by a medical professional. In addition, the video record may include a record obtained after the product has been applied to a patient or otherwise used for a medical procedure. For example, the appearance of the product initially applied to the patient and the change in the appearance of the product over time can be used to confirm or identify the product in the video data. The video record may also include a video record in which the product does not appear. The training video record may include metadata that marks various aspects of the video record. For example, a label may indicate whether a product appears in a video record. The label may be used by the product identification training system to determine whether the machine learning model 134 correctly determines whether a specified product is continuously present in a video record. The video record may also include various views of the same product obtained from different camera or sensor angles, including simultaneous views from different angles. In addition, the video record may include a record of the same product using different lighting conditions.

[0037] Video recording can also visualize and record information through infrared (IR) spectrum cameras. For example, various labels on products can incorporate IR readable information. Therefore, the product identification system can include one or more visible spectrum cameras and one or more IR spectrum cameras.

[0038] In general, the product identification training application 218 includes functionality for controlling various aspects of training the machine learning model 134 using the training data 232. For example, the training application 218 can provide various user interfaces for selecting various sets of video records from the training data 232 and providing various parameters for training (e.g., number of scan passes, hyperparameters, etc.). Additionally, the training application 218 can provide a user interface to specify whether the machine learning engine correctly determined whether a specified product was continuously present for at least a portion of the video records presented to the machine learning engine 222.

[0039] The video data processor 220 receives the selected video record from the training data 232 and processes the video data in the video record as further described below to present it to the machine learning engine 222. The machine learning engine 222 receives the video data from the video data processor and executes the machine learning algorithm and heuristics to train the machine learning model 134. In some aspects, the machine learning engine 222 and the machine learning model 134 implement a neural network. As an embodiment, the neural network can be a convolutional neural network, a recurrent neural network, a long short-term memory (LSTM) neural network, or other types of artificial intelligence. The machine learning engine 222 can apply the machine learning model 134 to the training video data and make a prediction for at least a portion of the training video data as to whether the specified product is continuously present. If the machine learning model 134 incorrectly determines that there is (such as using labels in the training data or manual input), a "back propagation algorithm" or other optimization techniques can be used to provide adjustments to the machine learning model 134, and the training process can continue.

[0040] When the machine learning model 134 is fully trained (i.e., has achieved a desired level of accuracy to identify the continuous presence of a product in video data), the machine learning model 134 can be deployed to the product identification system 104.

[0041] The following is about Figure 3-7 Further details regarding the operation of the video data processor 220 and the machine learning engine 222 are provided.

[0042] Figure 2Bis a diagram depicting an example product identification system 104 operating in accordance with techniques of this disclosure. In some aspects, product identification system 104 receives real-time video data 236 via video capture device interface 212 and uses inference engine 242 and trained machine learning model 134 to determine in real-time or near real-time whether a specified product is continuously present within a portion of video data 236.

[0043] In this embodiment, similar to Figure 2A The product identification training system 104 includes one or more processors 244 and a memory 246. In some embodiments, the processor provides a computer platform for executing an operating system 256, which can be, for example, a real-time multitasking operating system or other types of operating systems. Furthermore, the operating system 256 provides a multitasking operating environment for executing one or more software applications 219. The processor 244 can be connected to one or more I / O interfaces 254, which provide I / O interfaces for communicating with devices such as keyboards, controllers, display devices, image capture devices, other computing systems, etc. In addition, one or more I / O interfaces 214 may include one or more wired or wireless network interface controllers (NICs) for communicating with a network. In some instances, the processor 244 and the memory 246 can be separate discrete components. In other instances, the memory 246 can be an on-chip memory co-located with the processor 244 within a single integrated circuit.

[0044] The software application 219 of the product identification system 104 is used to receive real-time video data and identify whether a specified product is continuously present in the real-time video data. In this embodiment, the software application 219 includes a product identifier 240, a video data processor 248 and an inference engine 242.

[0045] Generally, product identifier 240 includes functionality for controlling various aspects of determining whether a specified product is continuously present in real-time video data 236. For example, product identifier 240 may provide various user interfaces for selecting parameters regarding product identification (e.g., products to identify, machine learning models to use, downstream systems to notify, etc.).

[0046] Similar to the product identification training system 202 discussed above, the video data processor 248 receives video data and processes the video data as further described below for presentation to the inference engine 242. However, in this case, the video data processor 248 receives real-time video data 236 provided via the video capture device 102 during the medical procedure rather than training data.

[0047] Inference engine 242 is similar to Figure 2AThe machine learning engine 222 of the real-time video data processor receives the video data from the video data processor and executes the inference algorithm and heuristics to determine whether the specified product is continuously present during a portion of the real-time video data 236. In some aspects, the neural network implemented by the inference engine 242 and the machine learning model 134 is the same as the neural network used by the product identification training system 202 to train the machine learning model 134. As an example, the neural network can be a convolutional neural network, a recurrent neural network, a long short-term memory (LSTM) neural network, or other types of artificial intelligence. The inference engine 242 can apply the machine learning model 134 to the video data and make a prediction for at least a portion of the real-time video data 236 as to whether the specified product is continuously present. Unlike the machine learning engine 222, in some aspects, the inference engine 242 does not use back propagation or other techniques to change the machine learning model 134 to improve the accuracy of the model. In other words, the inference engine 242 operates similarly to the machine learning engine 222, with respect to predicting the continuous presence of the specified product in the video stream, but in some aspects, if the prediction is incorrect, the accuracy of the prediction is not checked or the machine learning model 134 is attempted to be corrected. However, in other aspects, the product identifier can provide an interface to indicate that the prediction is incorrect, and the product identification system 104 can update the machine learning model 134. In such aspects, the inference engine 242 is substantially the same as the machine learning engine 222. In some aspects, the online update described above can update the machine learning model 134 using a machine learning algorithm based on a stochastic gradient descent algorithm.

[0048] The following is about Figure 3-7 Further details regarding the operation of video data processor 248 and inference engine 242 are provided.

[0049] Figure 3 is a block diagram showing an example of how video data is trained by a product identification training system 202 and according to the present invention. Figure 1 , Figure 2A and Figure 2B The exemplary embodiment of the product identification system 104 is processed. Figure 3In the example shown, the video data 302 can be training video data from the training data 232, or it can be real-time video data 236. As described above, in some aspects, the video data 302 includes a 3D volume having a time axis (labeled "TIME") and two spatial axes (labeled "SPACE_1" and "SPACE_2"). In some aspects, the video data processor 220, 248 divides the video data into data windows, where each data window 304 has a dimension a along the time axis and dimensions b and c along the spatial axis. Therefore, the a dimension represents the duration of the data window 304, and the b and c dimensions represent the image size. For example, a video data window 304 with a ten frame duration and an image size of 300×300 pixels can have a×b×c dimensions of 10×300×300. In addition, each image can also have channels (e.g., red, green, and blue channels for a color image), in which case the notation can be represented as a×b×c×d, where d represents the number of channels. The following is about Figure 6 Examples using multi-channel color images are presented. However, to simplify the explanation, grayscale images with a single channel will be used below to illustrate various aspects of the technique. Therefore, the notation a×b×c can be used because d will have a constant value.

[0050] The video data processor 220, 248 may also create multiple data windows 304A, 304B, and 304C from each data window 304 by rotating or otherwise transforming one or more axes before submitting the data window to the machine learning engine 222 or the inference engine 242. Figure 3 In the example shown, the video data processors 220, 248 have generated three data streams, each having a different orientation with respect to the time and space axes. The three data streams of data windows 304A, 304B, and 304C can be submitted to the inference engine 242, which uses the machine learning model 134 to determine whether the specified product appears continuously in the data window 304. For example, an indicator of "real" can be used to indicate that the specified product is real and exists continuously in the video data window 304. An indicator of "not real" can be used to indicate that the specified product does not appear in the video data window 304. Although Figure 3 , but the techniques discussed herein can be applied to fewer or more data streams. In some aspects, at least one of the data streams has a time dimension. That is, in some aspects, the video data processor 220 receives at least one spatiotemporal image data stream.

[0051] Figure 4 is a block diagram showing an example of an exemplary embodiment of a video data processor 220, 248 performing spatiotemporal slicing of video data into 2D slices according to the present techniques. Figure 4In the example shown, the video data processor 220, 248 creates multiple data streams for submission to the machine learning engine 222 and the inference engine 242 by acquiring 2D slices 402 of the video data window 304. The angle formed between the edge of the 2D slice and the time axis and the space axis defines the orientation of the 2D slice. Therefore, two 2D slices with the same angle between the edge and the time axis and the space axis have the same orientation. Figure 4 The data window 304 shown in FIG. 3 has a time dimension a and spatial dimensions b and c. The magnitude of a is determined by the sliding window size r. Thus, at time t k , a=t k –t r Slices 402 may be taken at various orientations relative to the time axis and the two spatial axes. Figure 4 In the illustrated example, three slices 402A, 402B, and 402C (collectively referred to as slices 402) are shown. Slice 1 402A is a slice taken relative to the first spatial axis and the second spatial axis of the data window. Slice 2 402B is a slice taken relative to the time axis and the first spatial axis. Slice 3 402C is a diagonal slice taken along the second spatial axis and the hypotenuse of the time axis and the first spatial axis. As described above, in some aspects, at least one image data stream is used to provide slices that exploit temporal changes in the data stream. Slices 402B and 402C are Figure 4 402, because they include data that varies along the time axis. These slices capture the spatiotemporal shape of product behavior, which may have high predictive value for product recognition and authentication. The video data processor 220, 248 may continue to provide additional slices along the selected orientation. For example, for a slice with an XY orientation, the next slice may include the next image frame along the time axis. In a slice with a time-Y orientation, the next slice may include data at the next X position. In some aspects, the video data processor 220, 248 provides multiple data streams for the slice 402, where each data stream contains slices having the same orientation. Thus, in Figure 4 In the example shown, the video data processor 220, 248 can provide three data streams to the machine learning engine 222 or the inference engine 242. The first data stream has slices corresponding to the orientation of the slice 402A, the second data stream has slices corresponding to the orientation of the slice 402B, and the third data stream has slices corresponding to the orientation of the slice 402C. The output of the machine learning engine 222 or the inference engine 242 includes a prediction 406 as to whether the specified product appears continuously in the video data window 304 at a time t.

[0052] Other orientations can be used instead Figure 4 Slice orientation shown, or except Figure 4In addition to the slice orientation shown, other orientations can also be used. Slices for these additional orientations can be provided to the machine learning engine 222 or the inference engine 242 as separate data streams. Figure 4 The example shown in includes slices taken according to three orientations. However, the technology of the present disclosure is not limited to any particular number of orientations. In some aspects, the data window 304 can be sliced ​​in space and time according to five or six different orientations. However, for machine learning algorithms, one space and time orientation may be sufficient.

[0053] The product identification training system 202 may include transfer learning 404. As described above, conventional systems may train the machine learning model 134 to recognize products in 2D images. In some aspects, the product identification training system 202 may utilize knowledge encoded in a previously trained 2D machine learning model as transfer learning 404. This may be advantageous in reducing the amount of training time required to train the machine learning model 134 to continuously recognize a specified product in a 3D video stream. Typically, in order to successfully utilize transfer learning, the source neural network and the target neural network need to have at least some compatibility in terms of layer configuration. When dealing with 3D video classification, neural networks trained on video datasets appear to be a natural source for transfer learning. However, the number of different videos in these 3D datasets is far less than the number of different images available in 2D datasets, and if transfer learning is limited to networks trained on 3D video datasets, then the advantages of transfer become less advantageous. However, as described herein, 3D spatiotemporal is automatically captured in the spatiotemporal shape of 2D image slices (e.g., in spatiotemporal slices) without the need for additional dimensions on the neural network filters. Because 2D slices are used, transfer learning from a neural network previously trained for recognizing products on 2D images can be easily used as transfer learning 404.

[0054] Figure 5 is a block diagram depicting an example multi-stream neural network 520, wherein Figure 1 , Figure 2A and Figure 2B The exemplary instances of the product identification training system 202 and the product identification system 104 provide data slices having a particular orientation to the corresponding set of convolutional neural network layers 506. Figure 5In the example shown, slices 502 with various orientations relative to the time axis and the spatial axis are provided as data streams to corresponding sets of convolutional neural network layers 506, where each set of convolutional neural network layers 506 is trained to process slices each having the same orientation. Therefore, each set of layers 506A-506N is an orientation-specific layer group. For example, the set of convolutional neural network layers 506A receives a data stream including slices of the data window 304, each slice having the same orientation as slice 1 502A. The set of convolutional neural network layers 506B receives a data stream including slices of the data window 304, each of which has the same orientation as slice 2 502B. This continues, and each set of convolutional neural network layers 506 receives a data stream corresponding to the orientation for which the network layer is trained. Therefore, for a system that takes n slices from the data window 304, the convolutional neural network layer 506N receives a data stream including slices of the data window 304, each slice having the same orientation as slice N 502N. Each set of convolutional neural network layers has an associated weight 504 determined according to the training of the corresponding neural network layer 506.

[0055] The output of each set of convolutional neural network layers 506A-506N includes a vector (i.e., feature map) 508A-508N that summarizes the contents of the respective slice 502A-502N. In some aspects, each vector 508A-508N is provided to a fully connected high-level neural network layer 510 (also referred to as a late fusion layer). Figure 5 In the example shown, the high-level neural network layer 510 has two layers, which respectively produce vector 512 and vector 514. The output of the last layer is used to determine the time t k Prediction 406.

[0056] During the training of the multi-stream neural network 520, the training system can determine whether there is an error in the prediction 406 by comparing the prediction 406 to a known value. If the prediction 406 is erroneous, the high-level layer 510 and the CNN layer 506 can be jointly optimized to improve the predictive ability of the multi-stream neural network 520. For example, the error information can be used in a back-propagation operation applied to the high-level layer 510. Once this first back-propagation operation has been completed, the error in each vector 508 can be determined. The error of each vector 508 can then be used in a back-propagation operation to adjust the weights 504 of each CNN layer in the CNN layer 506 corresponding to a particular output vector 508. Therefore, even if the error is initially determined by the output of the high-level layer 510, each set of orientation-specific layers 506 can be updated.

[0057] The above discussion is presented in the context of video data received from a single camera or other sensor. In some aspects, video data may be received from multiple image capture devices or other image sensors simultaneously, thereby providing multiple concurrent views of a product. In such aspects, slices from the simultaneously acquired video data may be simultaneously provided to an appropriately oriented convolutional neural network, with the high-level layer 510 using the output vector 508 to correlate features from the multiple views in order to recognize the continuous presence of a given product. Applying this to Figure 5 In the example provided in , it is assumed that there are two image capture devices providing image data simultaneously, and the convolutional neural network layer 506 has been trained using five different slice orientations for the first image capture device and four different slice orientations for the second image capture device. This configuration of two image capture devices is applied to Figure 5 In the example multi-stream neural network shown, N=9, convolutional neural network layers 1-5 can process video data from a first image capture device, and convolutional neural network layers 6-9 can process synchronized video data from a second image capture device.

[0058] As described above, in some aspects a recurrent neural network or LSTM network can be used. Using a recurrent neural network or LSTM network can be advantageous because such a network can associate features from previously processed slices with currently processed slices, thereby adding an additional temporal aspect to the machine learning model. This can result in a more accurate machine learning model, but may require more training time.

[0059] Figure 6 is a diagram depicting an example data slice provided by a convolutional neural network using the present invention's techniques Figure 1 , Figure 2A and Figure 2B A block diagram of processes performed by an illustrative example of a product identification system and a product identification training system is shown. Figure 6 The example shown in provides Figure 5 506A, and a representative sample set of convolutional neural network layers 506A operating on a data stream of slice 502A is shown, where each slice has the same orientation, and the orientation corresponds to the orientation used to train the set of convolutional neural network layers 506A. Figure 6 In the example shown, the set of convolutional neural network layers 506A has a series of convolutional layers 602 and rectified linear units 604. In some aspects, the convolutional layers 602 provide two-dimensional neurons, and the rectified linear units 604 are activation functions of neurons designed to produce non-linear outputs.

[0060] Out of Figure 6For the purposes of the example shown in , it is assumed that the video data includes 300x300 pixels (i.e., b=c=300) and that the window stride used by all layers is 2. Therefore, in this example, the input slice 502A has a size of 300×300×3, which is the image size in pixels, and the three red, green, and blue (RGB) values ​​of each pixel represent the three channels. The first convolutional layer 602A has a filter size of 10 pixels×10 pixels×3 RGB values. As indicated by the "@10" symbol, there are 10 filters in the convolutional layer 602A, so the layer produces 10 output feature maps. The next convolutional layer 602B has a filter size of 6×6×10 and includes 15 filters, as indicated by the "@15" symbol. The subsequent convolutional layer 502C has a filter size of 5×5×15 and includes 18 filters. The second to last convolutional layer 602D has a filter size of 8×8×18 and also includes 18 filters. The final convolutional layer 502E has a filter size of 14×14×18 and includes 40 filters. Thus, the final output of this set of convolutional neural network layers 506A includes a vector 508A that effectively summarizes the contents of the slice 1502A. Figure 5 As shown, vector 508A is combined with output vectors 508B-508N and processed by high-level neural network layer 510. Thus, set of convolutional neural network layers 506A can receive slice 502A and produce vector output 508A of size 1×40. This vector output 508A can be fed to high-level neural network layer 510, which can jointly learn the correlations seen in the spatiotemporal features represented in vectors 508 from different slices 502.

[0061] Figure 7 700 is a flow chart illustrating example operations of a method for continuous product authentication according to aspects of the present disclosure. The example operations shown in flow chart diagram 700 can be performed continuously during the performance of a medical procedure or during the use of a product in a medical environment. A product authentication system or training system receives a video representing a medical procedure environment, such as a hospital operating room, a clinic, a same-day surgery facility, etc. (702).

[0062] In the case of a product identification training system, the video may be a pre-recorded video clip of a previously performed medical procedure or use of a specified product. The pre-recorded video clip may include metadata labeling the products and product attributes that appear in the video to aid in the automatic training of the machine learning model.

[0063] In the case of a product authentication system, the video could be a live video of a medical procedure actually being performed or a product being used in a medical setting.

[0064] The video data processor treats the input video as a three-dimensional volume with a time axis and two spatial axes (e.g., an X-axis and a Y-axis). The video data processor divides the input video into three-dimensional data windows (704). In some aspects, the duration of the three-dimensional window is τ. The value of τ may vary depending on the current response time requirements, available computing resources, and the frame rate of the captured video frames. Processing larger volumes can provide greater context and more information, but can also consume more resources (e.g., memory, GPU, etc.), and a larger neural network model can be used to provide a response. Depending on these factors, the value of τ may range from a fraction of a second to several seconds. For general medical applications with a frame rate of 25fps, a resolution of approximately 300x300x3, 400x400x3, or 500x500x3, the value of τ ranges from 1-12 seconds.

[0065] The video data processor generates multiple data streams (706) from the same three-dimensional window by spatiotemporally slicing the three-dimensional window at different orientations. For example, the video data processor may create a first data stream having slices corresponding to the X-axis and Y-axis of the three-dimensional window, a second data stream having slices corresponding to the time axis and Y axis of the three-dimensional window, and a third data stream having slices corresponding to the X-axis and a hypotenuse determined based on the Y-axis and the time axis. Other orientations may be used in addition to or in place of any of the three orientations described above.

[0066] The video data processor then provides the individual data streams to an inference engine (708). As described above, the inference engine may include individual CNN layers 506 and joint high-level layers 510, wherein the individual CNN layers 506 are specific to a particular slice orientation, and the joint high-level layers 510 relate to the output of the individual CNN layers 506. For example, the inference engine may include multiple sets of convolutional neural network layers 506, wherein each set of layers 506 is trained on slices having a different orientation. The inference engine may provide a corresponding input data stream to a corresponding convolutional neural network layer, the orientation of the training data slice used for training the convolutional neural network layer being the same as the corresponding input data stream.

[0067] The output of the inference engine may be used to determine whether a specified product is present continuously in a video data window (710). In some aspects, the output vector of each orientation-specific convolutional neural network is provided to a high-level convolutional neural network (a late fusion layer). The high-level convolutional neural network may exploit the correlation in the spatiotemporal features represented by the output vectors corresponding to different slices to predict whether a specified product is present continuously in a data window.

[0068] An indicator indicating the continued presence of a specified product may be provided to a downstream system. The downstream system may use the indicator to record the use of the specified product, determine whether the product is used correctly, or determine the content of guidance to be provided about the specified product.

[0069] The above discussion is presented in the context of a medical environment where medical procedures may be performed or medical-related products may be used. The above techniques may also be applied to other environments. For example, the techniques may be applied to a video of a security device in use to determine the presence of a specified product component of the security device.

[0070] In the specific description of the preferred embodiment, reference is made to the accompanying drawings, which illustrate specific embodiments that can be put into practice of the present invention. The illustrated embodiments are not intended to be exhaustive of all embodiments according to the present invention. It should be understood that other embodiments may be utilized without departing from the scope of the present invention, and structural or logical changes may be made. Therefore, the following detailed description should not be considered to have a limiting meaning, and the scope of the present invention is limited by the appended claims.

[0071] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood in all instances as modified by the term “about.” Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and the appended claims are approximations that may vary depending upon the desired properties sought to be obtained by one skilled in the art utilizing the teachings disclosed herein.

[0072] Unless the content clearly dictates otherwise, as used in this specification and the appended claims, the singular forms "a / kind", "the" and "the" encompass embodiments having plural referents. Unless the content clearly dictates otherwise, as used in this specification and the appended claims, the term "or" is generally employed in its sense including "and / or".

[0073] It should be appreciated that, according to this example, certain actions or events in any of the methods described herein may be performed in a different order, may be added, combined, or omitted together (e.g., not all described actions or events are necessary for the practice of the method). In addition, in some examples, actions or events may be performed simultaneously rather than sequentially, such as by multithreading, interrupt processing, or multiple processors.

[0074] The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the described techniques may be implemented in one or more processors, including one or more microprocessors, DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuits, and any combination of such components. The term "processor" or "processing circuit" may generally refer to any of the above-mentioned logic circuits, either alone or in combination with other logic circuits, or any other equivalent circuits. A control unit comprising hardware may also perform one or more of the techniques disclosed herein.

[0075] Such hardware, software, and firmware may be implemented in the same device or in different devices to support the various operations and functions described in this disclosure. In addition, any of the described units, modules, or components may be implemented together or separately as discrete but interoperable logic devices. Depicting different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be implemented by separate hardware or software components. On the contrary, the functions associated with one or more modules or units may be performed by separate hardware or software components or integrated in common or separate hardware or software components.

[0076] The techniques described in the present disclosure may also be embodied or encoded in a computer-readable medium (such as a computer-readable storage medium) containing instructions. For example, when the instructions are executed, the instructions embedded or encoded in the computer-readable storage medium may cause a programmable processor or other processor to perform the method. The computer-readable storage medium may include: random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk, CD-ROM, floppy disk, cassette, magnetic media, optical media, or other computer system readable media.

[0077] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A system comprising: at least one image capture device configured to capture video data representative of an enhanced security environment; a machine learning model trained to recognize a specified product; and At least one computing device, the at least one computing device comprising a memory and one or more processors coupled to the memory, wherein the memory comprises instructions that cause the one or more processors to successively perform operations comprising: determining from the video data a three-dimensional data window having a first spatial axis, a second spatial axis and a time axis, generating a plurality of data streams from the three-dimensional data window, the plurality of data streams comprising data varying along the time axis of the three-dimensional data window, each respective one of the plurality of data streams having a plurality of slices of the three-dimensional data window, each respective one of the plurality of data streams having an orientation with respect to the first spatial axis, the second spatial axis, and the time axis that is different from an orientation of a plurality of slices in each other one of the plurality of data streams, determining a continued presence of the designated product in the enhanced security environment based at least in part on the plurality of data streams and the machine learning model, the machine learning model comprising a plurality of sets of orientation-specific convolutional layers trained according to orientations associated with the plurality of data streams, providing each of the plurality of data streams to a set of orientation-specific convolutional layers associated with a corresponding orientation of a plurality of slices in the data stream, and An indicator of the continued presence of the designated product is provided to downstream systems.

2. The system of claim 1 , wherein the operation of determining the continued presence of the designated product in the enhanced security environment further comprises the operation of providing the output of each of the orientation-specific convolutional layer sets to one or more high-level neural network layers.

3. The system of any one of claims 1-2, wherein the machine learning model comprises a plurality of orientation-specific neural network layer sets, each of the orientation-specific neural network layer sets being connected to at least one high-level neural network layer, wherein at least one of the orientation-specific neural network layer sets is trained using data that varies along the time axis.

4. The system of claim 3, wherein the at least one high-level neural network layer comprises a plurality of fully connected neural network layers.

5. The system of any one of claims 1-2, wherein at least one data stream containing data along the time axis comprises a plurality of slices having an orientation along the first spatial axis and a diagonal formed by the hypotenuse of the second spatial axis and the time axis.

6. A system according to any one of claims 1-2, wherein the machine learning model includes multiple orientation-specific convolutional layers, and the multiple orientation-specific convolutional layers are trained to determine the presence of the specified product based at least in part on image data representing the appearance of the specified product applied to the patient.

7. A system according to any one of claims 1-2, wherein the machine learning model is trained to determine the presence of the specified product based at least in part on image data representing a change in the appearance of the specified product after application to a patient.

8. The system of any one of claims 1-2, wherein the at least one machine learning model is trained to determine the presence of the specified product based at least in part on image data representing the appearance of a container of the specified product.

9. The system of any one of claims 1-2, wherein the enhanced security environment comprises a medical procedure environment.

10. A method comprising: capturing, via at least one video capture device, video data representative of the enhanced security environment; Continuously performing operations during at least a portion of one or more actions taken relative to the enhanced security environment, the operations comprising: determining from the video data a three-dimensional data window having a first spatial axis, a second spatial axis and a time axis, generating a plurality of data streams from the three-dimensional data window, the plurality of data streams comprising data varying along the time axis of the three-dimensional data window, each respective one of the plurality of data streams having a plurality of slices of the three-dimensional data window, each respective one of the plurality of data streams having an orientation with respect to the first spatial axis, the second spatial axis, and the time axis that is different from an orientation of a plurality of slices in each other one of the plurality of data streams, determining a continued presence of the designated product in the enhanced security environment based at least in part on the plurality of data streams and a machine learning model trained to recognize the designated product, the machine learning model comprising a plurality of sets of orientation-specific convolutional layers trained according to orientations associated with the plurality of data streams; providing each of the plurality of data streams to a set of orientation-specific convolutional layers associated with a corresponding orientation of a plurality of slices in the data stream, and An indicator of the continued presence of the designated product is provided to downstream systems.

11. The method of claim 10, wherein determining the continued presence of the designated product in the enhanced security environment further comprises providing an output of each of the orientation-specific convolutional layer sets to one or more high-level neural network layers.

12. The method according to any one of claims 10-11, wherein generating the at least one data stream comprises generating slices having an orientation along the temporal axis and one of the first spatial axis and the second spatial axis.

13. The method according to any one of claims 10-11, wherein generating the at least one data stream comprises generating slices having an orientation along the first spatial axis and a diagonal formed by the hypotenuse of the second spatial axis and the temporal axis.

14. The method of any one of claims 10-11, further comprising training the machine learning model to determine the continuous presence of the specified product based at least in part on image data representing the appearance of the specified product applied to a patient.

15. The method of any one of claims 10-11, further comprising training the machine learning model to determine the continuous presence of the designated product based at least in part on image data representing the appearance of a container of the designated product.

16. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors of a product authentication system to continuously perform operations during at least a portion of one or more tasks in an enhanced security environment, the operations comprising: receiving video data representing the enhanced security environment; determining from the video data a three-dimensional data window having a first spatial axis, a second spatial axis, and a time axis; generating a plurality of data streams from the three-dimensional data window, the plurality of data streams comprising data varying along the time axis of the three-dimensional data window, each respective one of the plurality of data streams having a plurality of slices of the three-dimensional data window, each respective one of the plurality of data streams having an orientation with respect to the first spatial axis, the second spatial axis, and the time axis that is different from an orientation of a plurality of slices in each other one of the plurality of data streams; determining a continued presence of a specified product in an enhanced security environment based at least in part on the plurality of data streams and a machine learning model comprising a plurality of sets of orientation-specific convolutional layers trained according to orientations associated with the plurality of data streams; providing each data stream of the plurality of data streams to a set of orientation-specific convolutional layers associated with a corresponding orientation of a plurality of slices in the data stream; as well as An indicator of the continued presence of the designated product is provided to downstream systems.

Images