Virtual camera source to display region of interest of device

Abstract

A method of the present disclosure includes receiving an input video stream displaying at least a portion of at least one device being used in a medical environment. In one embodiment, the device has a region of interest (ROI), and the device has a region of interest (ROI). An input video stream region of a region of interest of a display device is extracted from an input video stream and provided as an ROI video stream. The ROI includes, for example, a display such that information provided in the ROI video stream is displayed on the display.

Description

Technical Field

[0001] The present invention relates to a computer-implemented data processing method for providing information about at least one device used in a medical environment as a virtual camera video stream from an input video stream, a corresponding computing program, a computer-readable storage medium storing such a program, a computer executing the program, and a digital operating room including the above computer. Background Art

[0002] The object of the present invention is to be able to record medical procedures more flexibly, for example for archival purposes, even for devices without a video signal output port used in the procedure.

[0003] The present invention can be used in the digital operating room environment of Brainlab AG.

[0004] Aspects, examples and exemplary steps of the present invention and their embodiments are disclosed below. Different exemplary features of the present invention can be combined according to the present invention as long as it is technically suitable and feasible. Summary of the Invention

[0005] Brief Description of the Invention

[0006] The specific features of the present invention are briefly described below and should not be construed as limiting the present invention to the features or combinations of features described in this part.

[0007] The method disclosed in the present invention includes: receiving an input video stream showing at least a part of at least one device being used in a medical environment. The device has a Region of Interest (ROI). Extracting from the input video stream the region of the input video stream showing the ROI of the device and providing it as an ROI video stream. The ROI includes, for example, a display or a part of a display, so as to provide the information displayed on the display or a part of the display in the ROI video stream.

[0008] Overview of the Invention

[0009] In the present invention, the general features of the present invention are described, for example, by referring to the feasible embodiments of the present invention.

[0010] Generally speaking, in order to achieve the above object, a first aspect of the present invention provides a data processing method for providing information about at least one device used in a medical environment, especially when the device is being used in a medical environment. The method includes: performing the following exemplary steps executed by at least one processor on at least one processor of at least one computer (for example, at least one computer as part of a navigation system).

[0011] In an exemplary (e.g., first) step, an input video stream captured using a camera is obtained, including a series of images showing one or more regions of interest (ROI), where the region of interest is part of a device. In this specification, the term "Device" refers to a device used or being used in a medical environment.

[0012] The camera is mounted, for example, on a tripod, a trolley, or any other mobile or stationary carrier, or on the wall or ceiling of a room (such as an operating room). The camera can be mounted, for example, behind a screen.

[0013] The video streams described below (such as the input video stream, the ROI video stream, or the output video stream) represent moving images, such as two-dimensional moving images. The moving images show at least one region of interest, where the region of interest is an area or volume in space (such as in an operating room). The area or volume is part of a device.

[0014] The video stream can be in any suitable format, such as a series of independent images in compressed or uncompressed form. Any suitable video codec can be used to compress the video stream.

[0015] In an exemplary (e.g., second) step, one or more regions in the input video stream images showing one or more regions of interest are identified. In one example, for each region of interest, there is one region. The region is part of the image. The region is, for example, a collection of adjacent pixels and, for example, forms a two-dimensional pixel set. For example, the regions showing a particular region of interest are identified in multiple or all of the images of the input video stream.

[0016] In an exemplary (e.g., third) step, an ROI video stream is generated for each of the one or more regions. The ROI video stream is a series of images showing the region of interest. In the simplest example, the images of the ROI video stream accurately represent the regions identified in the previous step. This means that the region is simply copied from the input video stream to the corresponding ROI video stream. However, the image data of the corresponding region can be edited or processed before forming the ROI video stream. This can include color manipulation and / or contrast manipulation. This can further include geometric processing, such as scaling, rotation, deformation, or a combination of the above.

[0017] In one embodiment, the region of interest includes a display, a control element, or a status indicator. The display can be any information display mechanism, such as a pixel array. In the context of the present invention, the term "Display" also includes any other physical quantity indicating mechanism, such as a rotary pointer, a segmented display, or a rotameter. The control element can be an input mechanism manipulated by a user, such as a switch, a button, a knob, an attenuator, etc. The status indicator can be, for example, a status light, and the status can be indicated by the color and / or intensity of the light emitted by the status light.

[0018] The status indicator can be a passive indicator or a mechanical or electromechanical indicator, such as using a movable cover to cover or expose a surface indicating the status.

[0019] As described above, the image data of the input video stream can be processed to form an ROI video stream. In one embodiment, the method includes the following steps: eliminating distortion from the ROI video stream to provide a planar view of the corresponding region of interest. This is particularly beneficial when the region of interest is not perpendicular to the line of sight from the camera to the region of interest.

[0020] In one embodiment, the method further includes the following steps: providing the ROI video stream as a virtual video source. The virtual video source provides an ROI video stream. The video source is called a virtual video source because it is not a physical device like a camera. The virtual video source can be used in conjunction with the input of a video editing or mixing device or software.

[0021] In one embodiment, the images in the input video stream are 360° images. The camera for capturing 360° images can be, for example, a panoramic camera that images 360° horizontally or a downward-pointing fish-eye camera or a similar camera. The camera is installed, for example, on the ceiling.

[0022] There may be various ways to identify the region in the input video stream image that shows the region of interest, some of which will be described below.

[0023] In one embodiment, identifying the region involves identifying a device in the input video stream, obtaining information defining the corresponding region of interest from a database, and identifying the region based on the obtained information. Identifying a device in the input video stream means identifying the device in the input video stream image.

[0024] Identifying the device, for example, involves image recognition, such as using a sample image of the device. The database associates a specific device with the region of interest related to that device. When the device is identified in the input video stream, the spatial position of the device in the input video stream image is known, and thus the region position of the region of interest associated with the device is also known via the database.

[0025] Identifying the device can mean identifying the model of the device. The model refers, for example, to the model name and optionally the manufacturer. The database can store templates of the models. The templates, for example, associate the model with the geometry of the device, particularly the position of the region of interest relative to the device housing.

[0026] A specific device or specific model can have more than one region of interest. In this case, the method identifies one region corresponding to one ROI, multiple regions corresponding to multiple ROIs, or one region for each ROI. Then, the method can generate an ROI video stream for each identified region. However, the ROI video stream can also include two or more regions of the same device. Which ROI to identify the region for can depend on which medical procedure or which workflow step of the medical procedure is being performed. This information can also be stored in a database.

[0027] As an alternative, the method involves presenting the ROIs associated with the specific model identified in the input video stream to the user, obtaining ROI user input identifying one or more of the presented ROIs, and identifying the regions corresponding to the selected ROIs. Then, an ROI video stream is generated for the identified regions corresponding to the selected ROIs.

[0028] In one embodiment, the identification device involves identifying the identity tag of the device in the input video stream. The identity tag can be the model number indicated on the device. In this case, text recognition is performed on the input video stream. Alternatively, the identity tag can be a barcode or QR code that identifies the device model or specific device.

[0029] In one embodiment, the region is identified using the position of the corresponding device relative to the camera. This is beneficial in cases where the settings of the camera and the device are known, such as when equipping an operating room, the camera and the device can be registered. If the relative position is known, the regional position of the region of interest displayed can be obtained through geometric calculations.

[0030] In one embodiment, the method further includes the step of obtaining user input data indicating the spatial position in the input video stream, wherein the identified region in the input video stream image is the region including the position indicated by the user input data. The identified region for displaying the region of interest is the region including the position indicated by the user and having a predetermined size and / or shape or also including the size and / or shape input by the user. The user input can also directly specify the region to be used as the identified region, for example, by drawing a boundary on the input video stream.

[0031] In one embodiment, the user input is combined with identifying the device in the input video stream. The image recognition for identifying the device is, for example, limited to the region around the spatial position indicated by the user.

[0032] In one embodiment, the method further includes the following steps: obtaining user input data indicating a device type, wherein, based on the indicated device type, regions in the input video stream image are identified. In this case, the image recognition for identifying the device in the input video stream can be limited to sample images belonging to the indicated device type, rather than using all available sample images. In the context of the present invention, the term "type" refers to the device's purpose. The type is, for example, a "heart rate monitor". The term "model" refers to a specific variant. Thus, multiple models can be of the same type.

[0033] In another example, the indicated device type is associated with a set of device model names, and the text recognition in the input video stream is limited to the device model names in the set.

[0034] The user input can, for example, involve selecting from a given list of device types or free text input of a device type.

[0035] In one embodiment, artificial intelligence (AI) is used to identify regions in the input video stream image. The artificial intelligence uses standard AI algorithms with appropriate training data added. In the current case, each data set of the training data includes an image as input data and a region showing the region of interest as the required output data. The region showing the region of interest in the training data can be manually input by the user, for example, by specifying a region in the image of the training data.

[0036] In one embodiment, identifying the region involves finding a spatial part of the input video stream that is generally static but includes variable sub-parts. It should be noted that the position of the part can change with the image sequence of the input video stream. The change in the variable sub-part indicates that the sub-part is the region of interest. In one example, the part represents a medical device with a static structure, and the sub-part represents the display of the device, which displays changing information and is thus the variable part.

[0037] In one embodiment, identifying regions in the input video stream image is based on the currently executed medical procedure. The type of medical procedure is, for example, associated with a specific set of devices to be used, for example, via a medical procedure database. The devices in the set of devices can be used to identify regions in the input video stream image, for example, through image recognition or text recognition. In this case, image recognition can be performed only using the sample images of the devices included in the set of devices.

[0038] In one embodiment, identifying a region in an input video stream image is based on the workflow steps of the currently-executed medical procedure. A medical procedure typically involves multiple workflow steps. Each workflow step can be associated with a different set of medical devices used in the workflow step, e.g., via a medical procedure database. This means that the set of medical devices searched for in the input video stream can change over time, depending on the currently-executed workflow step.

[0039] In one embodiment, the region of interest is tracked over time in the input video stream. This means that one or more of the position, shape, and size of the region associated with a particular region of interest can change from one image to another, e.g., in the case where the corresponding device moves relative to the camera, and vice versa. Thus, the ROI video stream always shows the region of interest, regardless of the region of the input video stream image that shows the region of interest.

[0040] In one embodiment, the region of interest includes a control element, and the method further includes the steps of transforming the control element state into a graphical representation and embedding the graphical representation into the corresponding ROI video stream. This embodiment does not simply use a portion of the input video stream as the ROI video stream (optionally, after removing distortion), but analyzes the identified region representing the control element to determine the control element state.

[0041] The control element state refers, for example, to a value set using the control element or to the position of the control element. If the state refers to a value, the graphical representation can be a numerical value or a graph representing the value, such as a bar graph. If the control element state refers to a position, the graphical representation can be text such as "ON" or "OFF" or any other text describing the control element state. The graphical representation can also represent the control element state history, which means that it shows the state of the control element over a past period of time.

[0042] Embedding the graphical representation can involve superimposing the graphical representation onto the image of the ROI video stream, e.g., in an opaque or semi-transparent manner. The graphical representation can include transparent pixels.

[0043] In one embodiment, the region of interest includes a status indicator, and the method further includes the steps of transforming the status indicated by the status indicator into a graphical representation and embedding the graphical representation into the corresponding ROI video stream. This embodiment does not simply use a portion of the input video stream as the ROI video stream, but analyzes the identified region representing the status indicator to determine the device status.

[0044] The status indicator can be one or more light-emitting elements. The status can be represented by whether the light-emitting element is on or off, the color by which the light-emitting element emits light, or whether the light-emitting element blinks.

[0045] The graphical representation can be text such as "on" or "off", or any other text that describes the state of the device indicated by the status indicator. The meaning of the status of the status indicator is usually explained in the device manual, and the description of this meaning can be used as the text displayed in the graphical representation. The graphical representation can also represent the device status history, which means that it shows the status of the device over a period of time in the past.

[0046] Embedding the graphical representation can involve superimposing the graphical representation onto the image of the ROI video stream, for example, in an opaque or semi-transparent manner. The graphical representation can include transparent pixels.

[0047] In one embodiment, the region of interest includes a status indicator, and the method further includes the steps of: identifying the status indicated by the status indicator, and testing the status confidence. If the result is not credible, the method can involve outputting a corresponding warning, for example, embedded as a graphical representation into the corresponding ROI video stream. For example, a specific status can be associated with a medical procedure or a workflow step of a medical procedure, for example, via a medical procedure database.

[0048] In one implementation, the graphical representation can further indicate a timestamp. Therefore, the ROI video stream is an archive of the status or set value.

[0049] In one embodiment, generating the ROI video stream involves generating a synthetic ROI video stream. The synthetic ROI video stream is not generated from the region of the input video stream, but is synthesized using the information displayed in the region of the input video stream that shows the region of interest. This embodiment is particularly beneficial when the ROI video stream is based on a part of the input video stream (the distortion correction part) and the quality of the ROI video stream is not good enough. For example, reflections may make the information displayed on the device display difficult to recognize.

[0050] In one example, the region of interest of the device includes a display that shows a graphic. Generating the synthetic video stream involves analyzing the graphic displayed in the input video stream and generating a synthetic version of the graphic as the content of the ROI video stream. The synthetic version of the graphic can be added to the general graphic of the device display to mimic the display of the real device in the ROI video stream.

[0051] In one example, the region of interest includes a digital display that shows a value with one or more digits. Generating the synthetic video stream involves detecting the digits displayed on the digital display and generating the ROI video stream according to the graphical representation of the value. Similar to the above example, the general graphic of the region of interest including the digital display is corrected by adding the graphical representation of the value to mimic the digital display of the real device.

[0052] In other examples, the region of interest includes a status indicator or a control element, the status of the status indicator or the setting of the control element is detected in the corresponding region of the input video stream, and the ROI video stream represents a composite version of the status indicator or the control element having the detected status or setting respectively.

[0053] In one embodiment, the camera includes two or more camera modules, each camera module generates an original video stream, the input video stream is composed of two or more original video streams, and the ROI video stream is generated from at least one original video stream.

[0054] The field of view of the camera is usually limited, so that the image of the input video stream can be stitched together from the outputs of two or more cameras. In this embodiment, the video data of the ROI video stream may not be taken from the resulting stitched input video stream, but from one or more original video streams. This avoids a decrease in image quality in the region where two original video streams are superimposed in the input video stream or in the region where the image geometry is distorted due to stitching. If the region of interest is completely imaged by a single original video stream, the video data of the ROI video stream can be completely derived from the single original video stream.

[0055] In one embodiment, the method further includes the step of generating an output video stream from at least one ROI video stream. The output video stream includes, for example, two or more ROI video streams. This example means that an output video stream representing an image sequence is generated, where the output video stream includes more than one ROI video stream.

[0056] In one example, the output video stream is based on a base video stream, which shows, for example, at least a part of a patient or any other content. The base video stream is supplemented by at least one ROI video stream, for example, by superimposing the ROI video stream on the base video stream. There are many different options for arranging the individual video streams (the base video stream and / or at least one ROI video stream) to obtain the output video stream. Some of these options are described below, and within the scope of the present invention, two or more options can be combined as appropriate.

[0057] In one example, at least one ROI video stream is arranged in the output video stream based on user preferences. The user for whom the output video stream is targeted is known, for example, by identifying the user by means of image recognition, reading an identity tag or manual input. The user preferences define the layout of the output video stream. The layout of the output video stream defines, for example, the base video stream to be used, the ROI video stream to be used, and how the ROI video stream to be used is combined with the base video stream. The user preferences of multiple users can be associated with the respective users in a user database.

[0058] User preferences can be created using an editor. For example, a specific user defines the layout of the output video stream in the editor.

[0059] In one example, based on the currently-executed medical procedure, at least one ROI video stream is arranged into the output video stream. The layout of the output video stream is associated with the medical procedure, for example via the aforementioned medical procedure database or a separate database. The definition in the above example also applies to this example.

[0060] In one embodiment, based on the workflow steps of the currently-executed medical procedure, at least one ROI video stream is arranged into the output video stream. Thus, the layout of the output video stream can vary with each workflow step.

[0061] In one embodiment, multiple layouts are associated with the same medical procedure or the same workflow step. In this case, selection information indicating which layout to use can be queried from the user.

[0062] In one example, artificial intelligence is used to arrange at least one ROI video stream into the output video stream. Artificial intelligence can be used, for example, to define the layout of the output video stream. Artificial intelligence can be trained using multiple sets of training data sets, each set of training data sets describing the desired layout of the output video stream as the output and at least one of the user identity and the type of medical procedure as the input data.

[0063] In one embodiment, the method further includes the steps of: adding a graphical representation of device information to one or more of the ROI video stream or multiple ROI video streams. The device information identifies the device representing the region of interest in the ROI video stream. The representation of the device information can be, for example, text identifying the model name, optionally including the manufacturer name, device name, or device image. The graphical representation of the device information is, for example, superimposed on the ROI video stream.

[0064] A second aspect of the present invention relates to a computer program comprising instructions which, when executed by at least one computer, cause the at least one computer to perform the method according to the first aspect. Alternatively or additionally, the present invention may relate to a signal wave (e.g., a physical signal wave generated by technical means, such as an electromagnetic wave) carrying information representing the program (e.g., the above-mentioned program), such as a digital signal wave, such as an electromagnetic carrier wave, the program including, for example, code means adapted to perform any or all of the steps of the method according to the first aspect. In one example, the signal wave is a data carrier signal carrying the above-mentioned computer program. A computer program stored on a disk is a data file which, when read and transmitted, becomes a data stream in the form of, for example, a signal (e.g., a physical signal generated by technical means, such as an electrical signal). The signal may be implemented as a signal wave, such as the electromagnetic carrier wave described herein. For example, the signal (e.g., the signal wave) is configured to be transmitted via a computer network (e.g., LAN, WLAN, WAN, mobile network, such as the Internet). For example, the signal (e.g., the signal wave) is configured to be transmitted by optical or acoustic data transmission. Thus, alternatively or additionally, a second aspect of the present invention may relate to a data stream representing the above-mentioned program (i.e., including the program).

[0065] A third aspect of the present invention relates to a computer-readable storage medium storing the program according to the second aspect. The above-mentioned program storage medium is, for example, a non-transitory program storage medium.

[0066] A fourth aspect of the present invention relates to at least one computer (e.g., a computer) comprising at least one processor (e.g., a processor) and at least one memory (e.g., a memory), wherein the program according to the second aspect is executed by the processor, or the at least one computer comprises the computer-readable storage medium according to the third aspect.

[0067] A fifth aspect of the present invention relates to a digital operating room system comprising the computer according to the fourth aspect. The digital operating room system optionally further comprises a camera for generating an input video stream and at least one of a display for displaying one or more ROI video streams and / or an output video stream.

[0068] The digital operating room system may further comprise a video processing mechanism for generating an output video stream and / or a recording mechanism for recording one or more ROI video streams and / or an output video stream.

[0069] The digital operating room system may optionally comprise an additional ROI camera specifically for the region of interest. The output of the ROI camera can be used to generate a corresponding ROI video stream, which can be generated and / or used as any other ROI video stream.

[0070] According to the present invention, the artificial intelligence in each embodiment can be trained using joint or collaborative learning. Multiple digital operating room systems can be installed at different locations. Local training data from each location is used to train the local models of the AI, and only the parameters generated by local training are exchanged. The parameters can be transmitted to some or all of the other locations, and the local AI models can be adjusted locally. However, the parameters from some or all of the locations can be transmitted to a central site, where the AI model is updated and distributed to different digital operating room systems.

[0071] For example, users of different digital operating room systems generate user - personal ROI video stream layouts for specific medical procedures and workflow steps, which are then used as training data for the local AI models in each digital operating room system. An overall AI model is synthesized from the parameters of multiple local AI models.

[0072] For example, the present invention does not relate to or particularly does not include or does not contain invasive steps, which represent substantial physical interference with the body, require professional medical measures to be taken on the body, and even with the necessary professional care or measures, the body may still be at significant health risk. The present invention only relates to video processing.

[0073] Definitions

[0074] This section provides definitions of specific terms used herein, which also form part of this disclosure.

[0075] Computer - implemented method

[0076] The method of the present invention is, for example, a computer - implemented method. For example, all steps or only some steps (i.e., less than the total number of steps) of the method of the present invention can be performed by a computer (e.g., at least one computer). An embodiment of a computer - implemented method is the use of a computer to perform a data - processing method. An embodiment of a computer - implemented method is a method involving computer operations such that the computer operations perform one, multiple, or all steps of the method.

[0077] The computer includes, for example, at least one processor and at least one memory, in order to (technically) process data, for example, in an electronic and / or optical manner. The processor is made of, for example, a semiconductor material or composition, such as at least partially n-type and / or p-type doped semiconductors, such as at least one of type II, III, IV, V, VI semiconductor materials, such as (doped) silicon arsenide and / or gallium arsenide. The computing step or determination step is performed, for example, by the computer. The determination step or computing step is, for example, a step of determining data within the framework of a technical method (such as within a program framework). The computer is, for example, any type of data processing device, such as an electronic data processing device. The computer can be a device that is commonly regarded as a computer, such as a desktop personal computer, a laptop computer, a netbook, etc., but can also be any programmable device, such as a mobile phone or an embedded processor. The computer can, for example, include a "sub-computer" system (network), where each sub-computer represents a computer in its own right. The term "computer" includes cloud computers, such as cloud servers. The term "computer" includes server resources. The term "cloud computer" includes cloud computer systems, such as a system including at least one cloud computer, such as a system including a plurality of operatively interconnected cloud computers, such as a server farm. Such a cloud computer is preferably connected to a wide area network such as the World Wide Web (WWW), and is located in the so-called computer cloud that is all connected to the World Wide Web. Such an infrastructure is used for "cloud computing", which describes those computing, software, data access, and storage services that do not require the end user to know the physical location and / or configuration of the computer providing a particular service. For example, the term "cloud" is used metaphorically to refer to the Internet (World Wide Web) in this context. For example, the cloud provides computing infrastructure as a service (IaaS). The cloud computer can act as a virtual host for the operating system and / or data processing applications for performing the method of the present invention. The cloud computer is, for example, provided by Amazon Web Services TM)Elastic Compute Cloud (EC2) is provided. The computer includes, for example, an interface for receiving or outputting data and / or performing analog-to-digital conversion. The data is, for example, data representing physical properties and / or data generated from technical signals. Technical signals are generated, for example, by (technical) detection devices (such as devices for detecting markers) and / or (technical) analysis devices (such as devices for performing (medical) imaging methods), where the technical signals are, for example, electrical signals or optical signals. The technical signals represent, for example, the data received or output by the computer. The computer is preferably operatively coupled to a display device that allows the information output by the computer to be displayed to, for example, a user. An example of the display device is a virtual reality device or an augmented reality device (also known as virtual reality glasses or augmented reality glasses), which can act as "goggles" for navigation. A specific example of such augmented reality glasses is Google Glass (a trademark brand under Google, Inc.). The augmented reality device or virtual reality device can be used both for inputting information into the computer through user interaction and for displaying the information output by the computer. Another example of the display device is, for example, a standard computer monitor including a liquid crystal display, which is operatively connected to a computer for receiving display control data from the computer for generating a signal for displaying image information content on the display device. A specific embodiment of such a computer monitor is a digital light box. An example of such a digital light box is a product of Brainlab AG The monitor can also be, for example, a handheld portable device such as a smart phone or a personal digital assistant or a digital media player.

[0078] The present invention also relates to a computer program including instructions that, when executed by a computer, cause the computer to perform one or more of the methods described herein, such as the steps of one or more of the methods; and / or a computer-readable storage medium (such as a non-transitory computer-readable storage medium) storing the above program; and / or a computer including the above program storage medium; and / or a signal wave (such as a physical signal wave generated by a technical mechanism, such as an electrical wave), for example, a digital signal wave, such as an electromagnetic carrier wave, carrying information representing the program (such as the above program), where the program includes, for example, code means adapted to execute any or all of the method steps described herein. In one example, the signal wave is a data carrier signal carrying the above computer program. The present invention also relates to a computer including at least one processor and / or the above computer-readable storage medium and, for example, a memory, where the program is executed by the processor.

[0079] Within the framework of the present invention, a computer program unit can be embodied as hardware and / or software (this includes firmware, resident software, microcode, etc.). Within the framework of the present invention, a computer program unit can take the form of a computer program product, which can be embodied as a computer-usable, e.g., computer-readable data storage medium, which includes computer-usable, e.g., computer-readable program instructions, and the "code" or "computer program" embodied in the data storage medium is used on or in conjunction with an instruction execution system. Such a system can be a computer; a computer can be a data processing device including means for executing the computer program unit and / or program of the present invention, e.g., a data processing device including a digital processor (central processing unit or CPU) for executing the computer program unit, and optionally including a volatile memory (e.g., random access memory or RAM) for storing data used to execute the computer program unit and / or generated by executing the computer program unit. Within the framework of the present invention, a computer-usable, e.g., computer-readable data storage medium can be any data storage medium that can contain, store, communicate, propagate, or transmit programs used on or in conjunction with those instruction execution systems, devices, or apparatuses. A computer-usable, e.g., computer-readable data storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or a propagation medium such as the Internet. A computer-usable or computer-readable data storage medium can even be, for example, paper or other suitable media on which the program can be printed, since the program can be captured electronically, e.g., by optically scanning the paper or other suitable media and then compiled, decoded, or otherwise processed appropriately. The data storage medium is preferably a non-volatile data storage medium. The computer program product and any software and / or hardware described herein form various means for performing the functions of the present invention in the exemplary embodiments. The computer and / or data processing device can, for example, include a guidance information device, which includes means for outputting guidance information. The guidance information can be output to the user, for example, visually through a visual indication means (e.g., a monitor and / or a lamp) and / or auditorily through an auditory indication means (e.g., a speaker and / or a digital voice output device) and / or tactilely through a tactile indication means (e.g., a vibration element or a vibration element incorporated in the instrument). For the purposes of this document, a computer is a technical computer, e.g., including technical components such as tangible components, e.g., mechanical components and / or electronic components. Any device mentioned herein is a technical device and is, for example, a tangible device.

[0080] Obtain data

[0081] The expression "acquiring data" includes, for example (within the framework of the computer-implemented method), scenarios where data is determined by a computer-implemented method or program. Determining data includes, for example, measuring a physical quantity and transforming the measured value into data, such as digital data, and / or calculating (e.g., outputting) the data by means of a computer and, for example, within the framework of the method of the present invention. The "determining" step as described herein includes, for example, issuing or consisting of a command to perform the determination described herein. For example, this step includes issuing or consisting of a command that causes a computer (e.g., a remote computer, e.g., a remote server, e.g., in the cloud) to perform the determination. Alternatively or additionally, the "determining" step as described herein includes, for example, the following steps or consists of: receiving the result data of the determination described herein, for example, receiving the result data from a remote computer (e.g., from the remote computer that caused it to perform the determination). The meaning of "acquiring data" also includes, for example, the following scenarios: receiving or retrieving data (e.g., by input) by a computer-implemented method or program, for example, from another program, a previous method step, or a data storage medium, for example, for further processing by a computer-implemented method or program. Generating the data to be acquired may or may not be part of the method of the present invention. Thus, the expression "acquiring data" may also, for example, mean waiting to receive data and / or receiving data. The received data may be input, for example, via an interface. The expression "acquiring data" may also mean that a computer-implemented method or program performs some steps to (actively) receive or retrieve data from a data source such as a data storage medium (e.g., ROM, RAM, database, hard disk drive, etc.) or via an interface (e.g., from other computers or a network). The data acquired respectively by the method or apparatus of the present disclosure can be obtained from a database located in a data storage device, which is operably connected to a computer for data transfer between the database and the computer, for example, data transfer from the database to the computer. The computer acquires data for use as an input to the "determining data" step. The determined data can be output again to the same or other databases for storage for subsequent use. The database or the database for implementing the method of the present disclosure may be located in a network data storage device or a network server (e.g., a cloud data storage device or a cloud server) or a local data storage device (e.g., a mass storage device operably connected to at least one computer executing the method of the present disclosure). The data can be brought to a "ready" state by performing additional steps before the acquiring step. According to this additional step, data is generated for acquisition. For example, detecting or capturing data (e.g., by an analysis device). Alternatively or additionally, according to the additional step, inputting data, for example, via an interface. For example, the generated data can be input (e.g., into a computer). According to the additional step (which is performed before the acquiring step), data can also be provided by performing an additional step of storing the data in a data storage medium (e.g., ROM, RAM, CD, and / or hard disk drive), so that, within the framework of the method or program of the present invention, the data is made ready.Accordingly, the step of "acquiring data" may also involve the command device acquiring and / or providing the data to be acquired. In particular, the acquisition step does not involve invasive steps, which represent substantial physical interference with the body and require professional medical measures. Even when the required professional care and measures are taken during execution, the body may still be at significant health risk. In particular, the step of acquiring data, such as determining data, does not involve surgical steps, especially steps of treating the human or animal body using surgery or therapy. To distinguish different data used in the methods of the present disclosure, the data is represented as (i.e., referred to as) "XY data", etc., and is defined according to the information they describe, and then preferably referred to as "XY information", etc.

[0082] Medical workflow

[0083] The medical process follows the medical workflow. The medical workflow includes multiple workflow steps performed during medical treatment and / or medical diagnosis. The workflow steps are usually but not necessarily executed in a predetermined order. For example, each workflow step refers to a specific task, which can be a single action or a set of actions. Examples of workflow steps are capturing medical images, positioning the patient, applying markers, performing excisions, moving joints, placing implants, etc. Brief description of the drawings

[0084] The present invention is described below in conjunction with the drawings, which clarify the background of the present invention and represent specific embodiments of the present invention. However, the scope of the present invention is not limited to the specific features disclosed in the context of the drawings. In the figures:

[0085] Figure 1 illustrates a digital operating room system according to the present invention;

[0086] Figure 2 shows an image of an input video stream;

[0087] Figure 3 shows an image of an ROI video stream extracted from the input video stream;

[0088] Figure 4 shows different layouts of the output video stream;

[0089] Figure 5 shows a flowchart. Detailed description of the invention

[0090] Figure 1Schematic diagram of the digital operating room system 1 according to the fifth aspect. The system 1 includes a computer 2, an input mechanism 3, an output mechanism 4, and a camera 5. The input mechanism 3 can be, for example, a keyboard, a touch-sensitive surface, a mouse, or a combination thereof. The output mechanism 4 can be a monitor or any other display. The camera 5 includes a central processing unit 6 and two camera modules 7 and 8. The fields of view of the camera modules 7 and 8 are indicated by dashed lines.

[0091] Each camera module 7, 8 has an optical system and means for generating an electrical output signal representing an image. The central processing unit 6 combines the outputs of the camera modules 7 and 8 into an input video stream, which is a series of images. The input video stream images are images stitched together from the outputs of the camera modules 7 and 8 and display a 360-degree horizontal viewing angle around the camera 5.

[0092] The computer 2 includes a central processing unit 9, a memory 10, and an interface 11. The interface 11 connects the computer 2 to the camera 5, the input mechanism 3, and the output mechanism 4. The memory 10 stores data of one or more images of the ROI video stream generated by the central processing unit 9 (such as one or more images of the input video stream obtained from the camera 5) and optionally stores data of one or more images of the output video stream generated by the central processing unit 9. The memory 10 further stores instructions to be executed by the central processing unit 9 to implement the present invention.

[0093] Figure 2 An example of an image from the input video stream 12 generated by the camera 5 and obtained by the central processing unit 9 is shown. The image shows three devices D1, D2, and D3. Device D1 has a region of interest ROI1, which includes a two-dimensional graphic display. Device D2 has a region of interest ROI2, which includes a knob as an example of a control element. And device D3 has a region of interest ROI3, which includes a light as an example of a status indicator. All other contents of the image are omitted in the figure.

[0094] The central processing unit analyzes the images of the input video stream 12 and identifies the three devices therein. The type of device D1 can be identified from the device name "KI2000" written on it. The device database stored in the memory 10 saves a template of the identified device type, which defines the region of interest related to the name written on device D1 or the boundary of device D1. Based on this information, the central processing unit identifies the region of the input video stream 12's image that shows the region of interest ROI1 of device D1.

[0095] Device D2 is identified from the QR code written on it. The device database stored in the memory 10 saves information that defines the region of interest related to the QR code written on device D2 or the boundary of device D2. Based on this information, the central processing unit identifies the region of the input video stream 12's image that shows the region of interest ROI2 of device D2.

[0096] The type of device D3 is identified by matching the device sample images included in the device database stored in the memory 10 with the input video stream 12. The device database stores information about the regions of interest within the sample images. When matching the sample images with the input video stream 12, the regions of interest in the sample images are also adjusted, and then the regions in the images of the input video stream 12 that display the region of interest ROI3 of device D3 are defined.

[0097] The central processing unit 9 extracts three regions that display the regions of interest ROI1, ROI2, and ROI3 respectively, and converts them into ROI video streams 13, 14, and 15 respectively. This conversion involves image processing such as geometric distortion correction, rotation, contrast processing, or color correction if necessary.

[0098] Figure 3 Images of each of the ROI video streams 13, 14, and 15 are shown. Each ROI video stream actually includes a series of images. The ROI video stream 13 shows the display of device D1.

[0099] The ROI video stream 14 shows the knob of device D2, but further enhanced. A graphical representation 16 indicating the set value of the knob 14 is superimposed in the lower right corner of the ROI video stream 14. The value is determined by the central processing unit 9 by analyzing the images of the input video stream 12 or the ROI video stream 14. The device database stored in the memory 10 includes, for example, the physical range of the adjustable knob and the corresponding range of settable values. The central processing unit 9 identifies the rotational position of the knob relative to device D2 and determines the corresponding value.

[0100] The lower left corner of the ROI video stream 14 includes a graphical representation 17 that is a graph showing the history of using the knob to set values within a predetermined time. The central processing unit 9 saves the set values identified from the previous images of the input video stream 12 or the ROI video stream 14 and generates the graphical representation 17.

[0101] The ROI video stream 15 shows the lamp of device 3. The central processing unit 9 further analyzes the region in the input video stream 12 or the ROI video stream 15 that displays the region of interest ROI3 to determine whether the lamp is on or off. The result of this analysis is converted into a graphical representation 18 that shows the words "on" or "off" according to the identified state of the lamp. The graphical representation 18 is added to the ROI video stream 15.

[0102] Figure 4The output video stream 19 at two time points is shown. Generally speaking, the output video stream 19 includes a basic video stream 20 enhanced by one or more ROI video streams. The basic video stream 20 can be blank, such as a continuous image of any color, or any other video stream, such as a video stream generated by a camera and showing, for example, a patient. The operating room system 1 can also include an additional camera that provides the basic video stream. The additional camera is, for example, directed at the operating room tabletop.

[0103] At Figure 4 At the first time point shown on the left, the ROI video stream 13 occupies most of the output video stream 19, while the ROI video stream 14 to the right of the ROI video stream 13 is much smaller. At the first time point, a first workflow step of a medical workflow can be performed, in which the device D2 is used for the first workflow step.

[0104] At Figure 4 At the second time point shown on the right, the ROI video stream 13 again occupies most of the output video stream 19, while the ROI video stream 15 to the right of the ROI video stream 13 is much smaller. At the second time point, a second workflow step of a medical workflow can be performed, in which the device D3 is used for the second workflow step.

[0105] The memory 10 stores the layout of the output video stream 19, which in particular defines the basic video stream to be used, the ROI video streams to be used, and the positions of the ROI video streams in the output video stream 19. The layout is optionally associated with a workflow or workflow step via a medical procedure database stored in the memory 10.

[0106] Figure 5 A flowchart showing a general method according to the present invention is shown.

[0107] The method starts with step S01: obtaining an input video stream 12 from, for example, the camera 5. Step S02 involves identifying regions in the input video stream 12 that show regions of interest (such as regions of interest ROI1, ROI2, and ROI3). Step S03 involves generating ROI video streams 13, 14, and 15 from the identified regions. Step S04 involves generating an output video stream 19 from at least one of the ROI video streams 13, 14, and 15.

[0108] It should be noted that any processing is generally based on a single image. Therefore, identifying regions in the input video stream 12, for example, means identifying regions in the images of the input video stream 12. Therefore, the term "Video Stream" can be interpreted as "Image of a Video Stream" in appropriate cases. The ROI video streams are generally generated image by image from subsequent images of the input video stream.

[0109] However, the knowledge obtained from the previous image can be used when processing the image. For example, it is expected that the region of interest will not move too far between two subsequent images, so that the position of the recognized region in the previous image can be used as a starting point for recognizing the region in the current image or for verifying the region recognized in the current image.

[0110] However, the knowledge obtained from multiple images of the input video stream can be used, for example, to generate a graphical representation added to the ROI video stream, such as graphical representation 17.

[0111] Instead of generating the output video stream 20, the computer 2 can output the ROI video streams 13, 14, and 15 via, for example, the interface 11 or any other suitable interface. For each of the ROI video streams 13, 14, and 15, the computer can act as a virtual camera source or a virtual video source. However, the central processing unit 9 forms a virtual camera or a virtual video source that provides the ROI video streams 13, 14, and 15 within the computer 2 to, for example, video mixing software running on the computer 2.

Claims

1. A data processing method for providing information about at least one device (D1, D2, D3) used in a medical environment, comprising the following steps: Obtain a video stream (12) captured using a camera (5), the video stream (12) including a series of images showing one or more regions of interest ROI, wherein, The regions of interest (ROI1, ROI2, ROI3) are part of the device (D1, D2, D3); Identifying one or more regions in the image of the input video stream (12) that display the one or more regions of interest (ROI1, ROI2, ROI3); And Generating an ROI video stream (13, 14, 15) for each of the one or more regions.

2. The method according to claim 1, wherein The regions of interest (ROI1, ROI2, ROI3) are displays, control elements or status indicators.

3. The method according to claim 1 or 2, further comprising the step of removing distortion from the ROI video stream to provide a planar view of the corresponding region of interest.

4. The method according to any one of claims 1 to 3, further comprising the step of providing the ROI video streams (13, 14, 15) as virtual video sources.

5. The method according to any one of claims 1 to 4, wherein, The images in the input video stream (12) are 360° images.

6. The method according to any one of claims 1 to 5, wherein Identifying the region involves identifying the devices (D1, D2, D3) in the input video stream (12), obtaining information defining the corresponding regions of interest (ROI1, ROI2, ROI3) from a database, and identifying the region based on the obtained information.

7. The method according to claim 6, wherein, Identifying the devices (D1, D2) involves identifying the identity tags of the devices (D1, D2) in the input video stream.

8. The method according to any one of claims 1 to 7, wherein Identifying the region uses the position of the corresponding device relative to the camera (5).

9. The method according to any one of claims 1 to 8 further comprises the following steps: obtaining user input data indicating a spatial position in the input video stream (12), wherein, The region identified in the image of the input video stream is a region including the position indicated by the user input data.

10. The method according to any one of claims 1 to 8 further comprises the following steps: obtaining user input data indicating the types of devices (D1, D2, D3), wherein, Identifying the region in the image of the input video stream based on the type of the indicated devices (D1, D2, D3).

11. The method according to any one of claims 1 to 10, wherein, Identifying the region in the image of the input video stream (12) uses artificial intelligence.

12. The method according to any one of claims 1 to 11, wherein Identifying the region in the image of the input video stream (12) is based on the currently performed medical procedure.

13. The method according to claim 12, wherein, Identifying the region in the image of the input video stream (12) is based on the workflow steps of the currently performed medical procedure.

14. The method according to any one of claims 1 to 13, wherein, Tracking the regions of interest (ROI1, ROI2, ROI3) in the video stream over time.

15. The method according to any one of claims 1 to 14, wherein, The region of interest (14) includes a control element, and the method further comprises the steps of transforming the state of the control element into a graphical representation (16, 17), and embedding the graphical representation (16, 17) into the corresponding ROI video stream (14).

16. The method according to any one of claims 1 to 15, wherein The region of interest (ROI3) includes a status indicator, and the method further comprises the steps of transforming the status indicated by the status indicator into a graphical representation (18), and embedding the graphical representation (18) into the corresponding ROI video stream (15).

17. The method according to any one of claims 1 to 16, wherein The camera (5) includes two or more camera modules (7, 8), each camera module (7, 8) generates a raw video stream, the input video stream (12) is synthesized from two or more raw video streams, and the ROI video stream is generated from at least one of the raw video streams.

18. The method according to any one of claims 1 to 17 further comprises the step of adding a graphical representation of device information to the ROI video stream.

19. The method according to any one of claims 1 to 18 further comprises the step of generating an output video stream (19) from at least one ROI video stream (13, 14, 15).

20. The method according to claim 19, wherein Arrange the at least one ROI video stream (13, 14, 15) into the output video stream (19) based on user preferences.

21. The method according to claim 19 or 20, wherein, Arrange the at least one ROI video stream (13, 14, 15) into the output video stream (19) based on the currently performed medical procedure.

22. The method according to claim 21, wherein, Arrange the at least one ROI video stream (13, 14, 15) into the output video stream (19) based on the workflow steps of the currently performed medical procedure.

23. The method according to any one of claims 19 to 22, wherein Arrange the at least one ROI video stream (13, 14, 15) into the output video stream (19) by using artificial intelligence.

24. A computer program which, when run on a computer (2), causes the computer (2) to perform the steps of the method according to any one of claims 1 to 23.

25. A computer (2) having stored thereon and / or running the computer program according to claim 24.

26. A digital operating room system (1) comprising the computer (2) according to claim 25.

27. A non-transitory computer-readable storage medium having stored thereon the program according to claim 24.

Images