Method for providing monitoring and / or control digital information to a medical device, by an external vision system

Abstract

There is described a method for providing monitoring and / or control digital information to a medical device comprising a robotic system 100 (e.g., a robotic system for medical or surgical teleoperation), by a vision system 120 adapted to acquire images and / or videos of a teleoperation area and send them to the robotic system 100 by vision system-to- medical device transmission means. The aforesaid vision system 120 is outside the robotic system 100 of the medical device. The method firstly includes encoding digital data representative of the aforesaid monitoring and / or control digital information within pixels of a raw video data stream, representative of the images and / or videos acquired by the vision system, by means of steganographic techniques, to generate a multiplexed stream containing videos and data. The method then comprises the steps of transmitting the aforesaid multiplexed stream containing videos and data, by the aforesaid vision system-to-robotic system transmission means; then, demultiplexing and decoding the aforesaid encoded and multiplexed digital data; and finally making the demultiplexed digital data available to the robotic system and / or to a control unit associated with the robotic system and / or to display means associated with the vision system. There is also described a medical system or device in which the aforesaid method is carried out, and comprising a robotic system for medical or surgical teleoperation and a vision system which is outside said robotic system, but connected thereto by data transmission means.

Description

[0001] METHOD FOR PROVIDING MONITORING AND / OR CONTROL DIGITAL INFORMATION TO A MEDICAL DEVICE, BY AN EXTERNAL VISION SYSTEM

[0002] DESCRIPTION

[0003] TECHNOLOGICAL BACKGROUND OF THE INVENTION

[0004] Field of application.

[0005] The present invention relates to a method for providing monitoring and / or control digital information to a medical device (e.g., comprising a robotic system for medical or surgical teleoperation), by a vision system and / or image acquisition system which is external to the aforesaid robotic system of the medical device.

[0006] DESCRIPTION OF THE PRIOR ART.

[0007] Endoscopic robotic systems are provided with fully incorporated vision systems, which allow a complete bidirectional exchange of information between the endoscope and the robot, in particular control and monitoring information in addition to optical / visual information, and thus allow an effective control of the vision system and also an optimal control of the robot, based on the control / monitoring information provided by the vision system.

[0008] Conversely, robotic systems for surgical or microsurgical teleoperation are typically decoupled from the vision system, and thus comprise vision systems, i.e. , digital exoscopes, which are outside the robotic system and transmit only video images of the teleoperation area thereto.

[0009] This results in several disadvantages (as compared to the previously mentioned case of endoscopic systems), including:

[0010] - optical monitoring / control technical information, such as focal length, zoom factor and lens features, are not available to the robotic system;

[0011] - movement information of the exoscope camera head (both in the event of manual exoscope and in the event of robotic exoscope) is not available to the robotic system.

[0012] Even more generally, it is not possible to carry out an effective integration of information between the vision system and the robotic system, which results in suboptimal monitoring and control performance.

[0013] In theory, a solution could be to provide an additional communication channel (in addition to the video transmission means) between the vision system and the robotic system, comprising the related hardware and software capable of managing all the communication protocols, at various levels. A solution based for example on Ethernet, whether wired or wireless, meets additional needs related to cybersecurity.

[0014] Such a solution is complex and expensive, so much so that it is impractical indeed, also because it would require, in addition to the support for the additional data communication channel, additional interface and transceiver means both on the side of the vision system and on the side of the robotic system.

[0015] In brief, the need to provide the robotic systems for surgery or microsurgery with monitoring and / or control digital data (in addition to video images) in a simple and effective manner, with external exoscopes, which need is strongly felt in the technical field considered, is not fully met by the technical solutions known to date.

[0016] SUMMARY OF THE INVENTION

[0017] It is an object of the present invention to provide a method for providing monitoring and / or control digital information to a robotic system for medical or surgical teleoperation, by a vision system outside the robotic system, which allows at least partially overcoming the drawbacks complained of above with reference to the prior art, and responding to the aforementioned needs particularly felt in the technical field considered. Such an object is achieved by a method according to claim 1 .

[0018] Further embodiments of such a method are defined by claims 2-25.

[0019] It is a further object of the present invention to provide a medical system in which the aforesaid method is carried out. Such an object is achieved by a system according to claim 26.

[0020] Further embodiments of such a system are defined by claims 27-34.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Further features and advantages of the method according to the invention will become apparent from the following description of preferred embodiments, given by way of non-limiting indication, with reference to the accompanying drawings, in which:

[0023] - figures 1A and 1 B show a medical system, in accordance with two possible embodiments of the invention, comprising a robotic system for medical or surgical teleoperation and a vision system external to the robotic system;

[0024] - figure 2 shows in more detail another embodiment of a medical system, in accordance with the invention;

[0025] - figures 3 and 4 show a block diagram of some steps of a method, according to respective embodiments of the present invention;

[0026] - figure 5 depicts a video image in which control and / or monitoring digital data is encoded, according to an embodiment of the method of the present invention, in which an enlarged rectangle of the encoding is shown in the upper left;

[0027] - figure 6 shows some aspects of conversion between video encodings, useful for understanding a step of the method according to the invention. DETAILED DESCRIPTION

[0028] With reference to Figures 1-6, a method is described for providing monitoring and / or control digital information to a medical device comprising at least one robotic system, by a vision system 120 adapted to acquire images and / or videos of a teleoperation area and send them to the medical device by vision system-to-medical device transmission means (i.e., transmission means from vision system to medical device). The aforesaid vision system 120 is outside (i.e., external to) the robotic system of the medical device.

[0029] The method firstly includes encoding digital data representative of the aforesaid monitoring and / or control digital information within pixels of a raw video data stream, representative of the images and / or videos acquired by the vision system, by means of steganographic techniques, to generate a multiplexed stream containing videos and data.

[0030] The method then comprises the steps of transmitting the aforesaid multiplexed stream containing videos and data, by the aforesaid vision system-to-medical device transmission means; then, demultiplexing and decoding the aforesaid encoded and multiplexed digital data; and finally making the demultiplexed digital data available to the robotic system and / or to a control unit associated with the robotic system and / or to display means associated with the vision system.

[0031] According to an embodiment of the method, the medical device comprises a robotic system 100 for medical or surgical teleoperation, in which the vision system 120 is adapted to acquire teleoperation images and / or videos and to send them to the robotic system 100 by vision system-to-robotic system transmission means (i.e., transmission means from vision system to robotic system).

[0032] In accordance with an embodiment of the method, the vision system 120 comprises a robotic vision head.

[0033] According to an embodiment of the method, the step of encoding the digital data comprises encoding and / or multiplying the digital data in the raw video data stream so that the multiplexed stream, containing videos and data, is substantially undisturbed from the viewpoint of displaying the video images with respect to the original raw video data stream.

[0034] According to an embodiment, the color distortions of the video contained in the multiplexed data stream are lower than a perceptibility threshold established in accordance with international standards, such as the CIE 1976 standard.

[0035] In this respect, it is noted that inserting data within a video image naturally introduces a certain level of distortion into the image, for example color distortion, which could be perceived by a human observer or software.

[0036] For example, the measurement of distortion perception is defined in the CIE 1976 standard, which uses the CIE L*a*b* encoding of each color. Such a measurement is reflected in a "distance", or "CIE distortion level" the value of which is representative of the distortion, when tested on a uniform surface of a given color with respect to another. T ogether with this "CIE distortion level", the aforesaid standard also defines a threshold value of such a level, equal to 2, the JND (Just Noticeable Level) perceptibility level.

[0037] Based on the above, according to an implementation option of the method, the encoding of the digital data in the video image is carried out in accordance with the criterion of reducing the color distortions of the modified video following the insertion of the multiplexed data stream, and more specifically with the criterion of obtaining a "CIE distortion level" lower than 2.

[0038] This can be done, for example, by taking advantage of the available degrees of freedom, as disclosed in some examples described below.

[0039] In accordance with an embodiment, the method is applied to a raw video data stream corresponding to a video signal encoded and transmitted with RGB (Red Green Blue) encoding, for example 8-bit or more per channel, or with YCbCr (Luminance and Chrominance) encoding, for example YCbCr (4:4:4, 4:2:2, 4:2:0), where 4:2:2 and 4:2:0 are spatial subsampling modes of the chrominance space.

[0040] In particular, the method applies, for example, to raw images with signals generated and transmitted in RGB (8-bit or more) or grayscale or YCbCr (4:4:4, 4:2:2, 4:2:0) with each type of ColorSpace (for example: sRGB, BT.709, BT.2020).

[0041] In accordance with a preferred embodiment of the method, the aforesaid encoding step comprises encoding each bit of digital data to be encoded into one and only one respective pixel of the raw video data stream.

[0042] According to an embodiment of the method, the aforesaid step of encoding digital data comprises generating a payload of digital data, protecting the generated payload, encoding the generated and protected payload.

[0043] In accordance with an embodiment of the method, as shown in Figure 4, for example, the aforesaid step of demultiplexing the encoded and multiplexed digital data comprises the following sub-steps:

[0044] - detecting whether the video stream contains monitoring and / or control digital data;

[0045] - if yes, separating the digital data from the data representative of the video image;

[0046] - restoring the image as in the original video stream, and providing the restored image to display means 130;

[0047] - decoding the aforesaid digital data;

[0048] - validating the decoded digital data and, if the data is validated, using the decoded digital data, i.e., proceeding with the further step of the method of making the demultiplexed and decoded digital data available.

[0049] According to an embodiment of the method, the aforesaid step of encoding the digital data or the payload of digital data comprises three levels of encoding:

[0050] - content-level encoding, concerning the general structure of the data;

[0051] - serialized-data-level encoding, concerning the sequence of the data stream bytes;

[0052] - pixel-level encoding, concerning the transformations of the pixels of the modified video stream intended to contain the digital data.

[0053] According to an implementation option of such an embodiment, the aforesaid content-level encoding comprises encoding in standard XML / JSON / ASN.1 formats, and said serialized-data-level encoding comprises serialization, encryption and protection.

[0054] The serialization is performed for example by means of JSON byte encoding or XDR encoding or ASN.1 encoding.

[0055] The encryption is performed for example by means of standard symmetric algorithms such as DES or Blowfish.

[0056] The protection is performed for example by means of CRC-8 I CRC-16 or cryptographic signature such as the SHA algorithm family.

[0057] According to an implementation option, the aforesaid pixel-level encoding is performed by means of algorithms combining the bytes sequence of the data to be encoded on the pixel sequence of the video signal, taking into account the type of video encoding.

[0058] In accordance with an embodiment, the method applies to a raw video stream that is RGB-encoded, and requires that each bit of the data to be encoded is encoded in a respective pixel.

[0059] In such a case, the encoding provides "full black" to encode 0 and "full white" to encode 1 , or vice versa.

[0060] According to another implementation example, the encoding provides using the least significant bit (LSB) of the color pixel to encode the content.

[0061] In accordance with another embodiment, the method is applied to a YCbCr-encoded raw video stream, and the encoding of the digital data is performed on the luminance channel Y so as to be robust with respect to the various chrominance down-sampling techniques (for example, 4:2:2 and 4:2:0).

[0062] In such a case, each data bit to be encoded is encoded in K bits of a respective pixel of the luminance channel Y, where, for each of said K bits, the value 0 corresponds to 0 while the value 1 corresponds to 2k-1.

[0063] According to an implementation option of the aforesaid embodiment, K = 3 such that one bit 0 of the encoding is 0 on 3 bits (e.g., 000), and a 1 bit is 7 on 3 bits (e.g., 111).

[0064] As noted above, in this case the encoding is entirely performed in YCbCr so as to ensure that the transformations carried out, such as conversion from RGB and then back to YCbCr, or sub-sampling, do not damage it.

[0065] The principle adopted includes storing each data bit in a respective pixel using the K least significant bits of the Y channel, so that bit=O is encoded with 0 while bit= 1 is encoded with 2k-1. This provides robustness to the method, which can be mathematically tested for every possible value of 8-bit YCbCr converted to 8-bit RGB and then back to YCbCr.

[0066] The decoding work includes extracting the value of the encoded bit by reading the aforesaid K bits, with the following decision criterion which takes into account the possible distortion of the values:

[0067] - Bit 0 if the value is in the range 0 ... 2k-1-1 ;

[0068] - Bit 1 if the value is in the range 2k-1... 2k-1.

[0069] If the Y component is 8-bit when K=8, this corresponds to the black / white mode disclosed above.

[0070] The criterion can be exemplified on an 8-bit component and K=3. Let X be the value of the 3 least significant bits, then:

[0071] - Decode 0 if X is from 0 to 3;

[0072] - Decode 1 if X is from 4 to 7.

[0073] By applying such an implementation option, with different values of K > 1 (therefore, by encoding each bit of digital data on at least two bits of a video pixel) the following values of the aforementioned "CIE parameter" were obtained: k=2 -> CIE = 3.15 tested on 100k color random YCbCr; k=3 -> CIE = 7.7 tested on 100k color random YCbCr; k=4 -> CIE = 16 tested on 100k color random YCbCr.

[0074] In accordance with another embodiment, the method applies to a YCbCr-encoded raw video stream, and the encoding of the digital data is performed on the Y luminance channel.

[0075] In such a case, each data bit to be encoded is encoded, as 0 or 1 , into a respective single bit of the luminance channel Y of a respective pixel.

[0076] In this case, the method also includes checking if the modified pixel in which the data bit is encoded is a valid pixel, from the viewpoint of RGB video encoding, and thus remains usable as a video pixel.

[0077] If the aforesaid modified pixel in which the data bit is encoded is an invalid pixel, the method includes modifying the Cb and / or Cr channels of the YCbCr pixel so as to obtain an adjusted modified pixel which is a valid pixel, from the viewpoint of RGB video encoding.

[0078] For example, the YCbCr pixel recognized as invalid according to the RGB encoding (i.e. , not having a valid meaning in the RGB encoding) is modified, keeping the value of the channel Y unchanged and modifying the values of the channels Cb and / or Cr (according to several possible options known per se) based on the criterion that the modified YCbCr pixel as indicated above, once reconverted into RGB format, gives rise to a triplet of valid values, which encode a significant RGB pixel.

[0079] Such an embodiment takes into account the fact that, if a generic set of values (Y,Cb,Cr) is considered and the signal Y is altered, the resulting overall value may be an invalid RGB value, which would lead to making the pixel insignificant, and ultimately to a loss of content.

[0080] For this reason, once Y has been calculated, it is verified that the resulting mapped blue or red pixels are inside the space thereof (i.e., the respective space allowed as provided for in the RGB encoding).

[0081] If not, the values of Cb and Cr are adjusted accordingly, for example by simple equations / algorithms known per se. The RGB space is contained as a polyhedron inside the parallelepiped of the YCbCr space. The validity is ensured by reprojecting the YCbCr value inside the subset of space which is mappable on valid RGBs.

[0082] Each RGB point has a corresponding YCbCr point, while the opposite is not true. For example, let YCbCr be 8-bit with the limited range convention, for which Y is from 16 to 235 and Cb,Cr from 16 to 240. In such a case the YCbCr value (16,240,16), considered in the sRGB gamut, is linearly mapped to (-179,47,226) in the RGB space, which mapping is invalid. The projection is then completed by returning it to RGB (0,47,226), the projected YCbCr of which is (62,214,95).

[0083] Applying such an implementation option, with different values of K, the following values of the aforementioned "CIE parameter" were obtained: k=1 -> CIE = 1.6 on 100k (< 2, i.e., below the JND) k=2 -> CIE = 3.8 on 100k k=3 -> CIE = 10 on 100k k=4 -> CIE = 20 on 100k.

[0084] As can be seen, the best results are obtained in such a case with k = 1 , i.e., by encoding each bit of digital data on a respective single bit of a respective pixel.

[0085] This option, in addition to being simpler and ensuring better robustness, also ensures less distortion, since it provides a "CIE parameter" value of less than 2, which, as disclosed above, is below the perceptibility threshold, i.e., the JND (Just Noticeable Level) perceptibility level. In other words, the distortion is in fact not perceptible, and the modified video stream (containing the encoded digital data) is substantially undisturbed with respect to the original video stream, with respect to the perception of the video image.

[0086] According to an implementation option of the embodiment described above, if the video signal provided by the vision system is RGB-encoded, the method comprises, before the step of encoding the digital data, the step of converting the RGB signal into a YCbCr- encoded raw video stream.

[0087] According to a particular implementation example, the aforesaid step of converting the RGB signal into a YCbCr- encoded raw video stream comprises:

[0088] - carrying out a conversion from “floating RGBf’ format to “floating YPbPr” format, by means of the following transformation matrix: where KR, KG, KB are constants defined by color space;

[0089] - carrying out a conversion from floating YPbPr to YCbCr.

[0090] To further depict the aforementioned implementation example, consider the following four possible color formats:

[0091] - " floating RGBf " in normalized units [0,1];

[0092] - " unsigned integer RBGBu " in the number of bits, typically 8, such as [0, 255];

[0093] - " floating YPbPr " in normalized units [0,1] for Y, and [-0.5, 0.5] for the others;

[0094] - " integer YCbCr " on 8 bits as [16, 235] for luminance Y, and [16,240] for chrominances Cb and Cr.

[0095] Note that the conversions between floating representations, advantageously used in this implementation example, are "lossless" by virtue of the high precision of the floating point (while the integer versions would suffer due to quantization).

[0096] The conversion between RGBf and YPbPr is governed by the above matrix, where KR, KG, KB are constants defined by color space (e.g., sRGB uses 0.2990 0.5870 0.1140).

[0097] The conversion from YPbPr to YCbCr is trivial and known per se, i.e. , for example, Y is multiplied by 219 and 16 is added, while for Cb, Cr it is multiplied by 224 and 128 is added.

[0098] The YPbPr / YCbCr frame can be interpreted as a new frame, in which the RGBf / RGBu space cube is rotated and tilted Figure 6 is a representation of YCbCr with respect to RGBu.

[0099] From the above, it also emerges that not all YPbPr / YCbCr coordinates are valid with respect to RGB color spaces.

[0100] According to another embodiment, in which the video signal provided by the vision system is YCbCr-encoded with at least 8 bits, on which the step of encoding the digital data is performed, the method comprises the further steps of:

[0101] - converting the YCbCr signal, after encoding, into a YCbCr (4:2:2) or YCbCr (4:2:0) signal, where the luminance channel Y on which the encoding was performed is not further altered;

[0102] - transmitting said YCbCr (4:2:2) or YCbCr (4:2:0) signal through said transmission means from vision system to robotic system.

[0103] With reference to such an embodiment, further details of related implementation options are provided below.

[0104] A typical robustness scenario is as follows:

[0105] 1) the input signal (e.g., provided by the camera of the vision system) is at least 8-bit;

[0106] 2) such a signal is converted to 8-bit RGB;

[0107] 3) such a signal is sent on the transmission means (e.g., HDMI cable) as YCbCr 4:2:2, 4:2:0, i.e., where Cb and Cr are subsampled horizontally (4:2:2) or are subsampled both horizontally and vertically (4:2:0);

[0108] 4) the transmitted signal is finally decoded.

[0109] In this scenario, the encoding and insertion of the digital data can be performed at step 1 or step 2. The conversion to YCbCr 4:2:0 is the reason why the encoding is done on the channel Y.

[0110] In an implementation option, step 1 is not performed.

[0111] FIRST EXAMPLE

[0112] - the signal from camera is YCbCr 4:2:2;

[0113] - directly encode the payload;

[0114] - display on RGB;

[0115] - send over HDMI cable as YCbCr 4:2:0.

[0116] SECOND EXAMPLE

[0117] - the signal from camera is YCbCr 4:2:2;

[0118] - encode as RGB;

[0119] - encode the payload;

[0120] - send over HDMI cable as YCbCr 4:2:0.

[0121] In accordance with an embodiment, before the encoding step, the method provides a step of selecting the pixels of the video image in which to encode the digital data based on the geometric position of the pixels in the video image, corresponding to the display on a screen, according to the criterion of minimizing the distortion and / or local impact of the pixels being encoded on the video image quality.

[0122] According to an implementation option of such an embodiment, the aforesaid selection step comprises selecting a plurality of pixels belonging to the same horizontal or vertical line of the video image as the pixels of the video image in which to encode the digital data.

[0123] According to a particular implementation option, the selection step comprises selecting a plurality of pixels belonging to the first half of the first or the last horizontal or vertical line of the video image as the pixels of the video image in which to encode the digital data.

[0124] In such a case, the aforesaid plurality of selected pixels comprises a sequence of consecutive pixels at the beginning of the line, or pixels evenly distributed in the first half of the line.

[0125] As for the aspects just disclosed above, it is noted that the choice of the geometric position of the pixels encoded on the screen must also appropriately consider the fact that the image can be stereoscopic, with one of several possible schemes (Vertical Interpolation, Above-Below, Top-Bottom). This determines the fact that there can be constraints in the positioning and / or distribution of the pixels to be encoded.

[0126] Considering an image with width / height (W / H), in a convenient implementation option, the pixels of any single vertical or horizontal line (preferably horizontal) up to the half thereof are encoded.

[0127] Such an implementation option is shown in Figure 5, for example, which depicts a video image in which control and / or monitoring digital data is encoded, barely visible in the upper left part (at the beginning of the first horizontal line). The example of the figure refers to an encoding of 1 pixel per bit, black / white coding, with CRC 16.

[0128] According to preferred implementation examples, the pixels of the first or last line are selected, with a width position from or up to W / 2, or N pixels of the first or last line evenly distributed distant from each other (W / 2) / N.

[0129] According to another implementation example, the distribution of the pixels to be encoded, along the chosen line, follows other temporal patterns adapted to avoid detection.

[0130] In accordance with an embodiment of the method, the aforesaid monitoring and / or control digital information comprises one or more of the following information:

[0131] - optical information related to the operation of the vision system, and / or - dynamic optical information related to the operation of the vision system, and / or

[0132] - dynamic information related to the movement of the vision system, and / or

[0133] - pose information of a surgical instrument (170) embedded in the robotic system, and / or

[0134] - signaling and / or monitoring information of the vision system, and / or

[0135] - information related to the sensors used for image and / or video acquisition.

[0136] According to an embodiment of the method, the aforesaid monitoring and / or control digital information comprises information adapted to guide the operator and / or information related to safety measures.

[0137] According to several possible implementation options, the aforesaid optical information related to the operation of the vision system comprises focal length and / or zoom information, usable by Computer Vision algorithms present in the control unit of the robotic system, for self-adjustment purposes and / or to provide metric information.

[0138] According to several possible implementation options, the aforesaid dynamic optical information related to the operation of the vision system comprises real-time variations of zoom and / or brightness and / or exposure and / or focus parameters communicated to the robotic system by the vision system.

[0139] The aforesaid dynamic information related to the movement of the vision system comprises information related to the movement of a head of the vision system, and / or changes in focal length and / or zoom, usable by an operator to decide whether to stop the movement of the surgical instrument.

[0140] The aforesaid pose information of the surgical instrument 170 comprises surgical instrument location information viewable by the operator.

[0141] The aforesaid signaling and / or monitoring information of the vision system comprises an indication of any faults and / or prioritization information of the information to be displayed.

[0142] The aforesaid sensor-related information comprises indications that the sensors are Visible Light or ICG (Indocyanine Green in Fluorescence) or other non-visible but clinically relevant spectrum.

[0143] Still referring to Figures 1-6, a medical system 10 (i.e., medical device 10), comprised in the present invention, is described below. Such a system comprises a robotic system 100 for medical or surgical teleoperation, a vision system 120 outside (i.e., external to) the robotic system, encoding means, vision system-to-robotic system transmission means, demultiplexing and decoding means.

[0144] The robotic system 100 for medical or surgical teleoperation comprises at least one surgical instrument 170, adapted to operate in teleoperation, and further comprises a control unit.

[0145] The vision system 120, which is outside the robotic system 100 (and / or, in particular, outside the surgical instrument), is configured to acquire images and / or videos of a teleoperation area and to generate a related raw video data stream, representative of the acquired images and / or videos.

[0146] The encoding means are configured to encode digital data representative of the aforesaid monitoring and / or control digital information within pixels of the raw video data stream, by means of steganographic techniques, and thus generate a multiplexed stream containing videos and data.

[0147] The vision system-to robotic system transmission means (i.e. , transmission means from vision system to robotic system) are configured to transmit data streams, comprising the aforesaid raw video stream and / or the aforesaid multiplexed stream containing videos and data, from the vision system to the robotic system.

[0148] The demultiplexing and decoding means are configured to demultiplex and decode the aforesaid encoded and multiplexed digital data and make the demultiplexed digital data available to the robotic system and / or to the control unit of the robotic system and / or to display means 130 associated with the vision system.

[0149] According to an embodiment, the medical system further comprises display means 130 associated with the vision system 120, configured to display video images of the teleoperation area from the vision system 120.

[0150] In accordance with an embodiment, the vision system 120 comprises one or more image sensors or cameras, having the same viewpoint or two different respective viewpoints, configured to provide video signals representative of two-dimensional or three-dimensional images.

[0151] According to an implementation option, the vision system comprises an exoscope.

[0152] According to an implementation option, the vision system 120 and / or the display means 130 comprise at least one electronic screen or monitor.

[0153] In accordance with an implementation option, the transmission means from vision system to robotic system comprise at least one HDMI cable.

[0154] According to an embodiment, the robotic system is a robotic system for surgery further comprising at least one master device 110, adapted to be moved by an operator 150, and at least one slave device comprising the aforesaid surgical instrument 170 adapted to be controlled by the master device 110.

[0155] In accordance with an embodiment of the system, the encoding means are configured to carry out the steps of encoding and generating a multiplexed stream containing videos and data, in accordance with the method according to any one of the embodiments disclose above.

[0156] In accordance with an embodiment of the medical system, the demultiplexing and decoding means are configured to carry out the steps of demultiplexing and decoding the digital, in accordance with the method according to any one of the embodiments disclosed above.

[0157] As can be seen, the objects of the present invention as previously indicated are fully achieved by the method and system disclosed above by virtue of the features described above in detail.

[0158] In fact, the method and system disclosed above allow the transmission channel already existing between the external vision system and the robotic system to be optimally exploited to provide the latter with all the monitoring and / or control information, coming from the vision system, which the robotic system may need.

[0159] Such information, as disclosed above, can comprise inter alia optical monitoring / control technical information (such as focal length, zoom factor and lens features) and / or camera head movement information of the vision system.

[0160] The availability of such information is particularly important and advantageous for optimizing the monitoring and / or control of the entire system, or even for optimizing the display provided to the operator.

[0161] The architecture of the system suggested with the present invention allows an effective integration of information between the vision system and the robotic system.

[0162] Furthermore, the technical solution of the present invention does not require providing a further dedicated hardware communication channel (with related software capable of managing all the communication protocols, at various levels) between vision system and robotic system, which would be complex and expensive, so as to be impractical.

[0163] Those skilled in the art may make changes and adaptations to the embodiments of the method and system described above or can replace elements with others which are functionally equivalent in order to meet contingent needs without departing from the scope of the following claims. All the features described above as belonging to one possible embodiment can be implemented irrespective of the other embodiments described.

Claims

CLAIMS1. A method for providing monitoring and / or control digital information to a medical device comprising at least one robotic system, by a vision system (120) adapted to acquire images and / or videos of a work area and send them to the medical device by vision system- to-medical device transmission means, said vision system (120) being outside the robotic system of the medical device, wherein the method comprises:- encoding digital data representative of said monitoring and / or control digital information within pixels of a raw video data stream, representative of the images and / or videos acquired by the vision system, by means of steganographic techniques, to generate a multiplexed stream containing videos and data;- transmitting said multiplexed stream containing videos and data, by said vision system-to-medical device transmission means;- demultiplexing and decoding said encoded and multiplexed digital data;- making the demultiplexed digital data available to the medical system and / or to a control unit associated with the medical system and / or to display means associated with the vision system.

2. A method according to claim 1 , wherein the medical device comprises a robotic system (100) for medical or surgical teleoperation, wherein the vision system (120) is adapted to acquire teleoperation images and / or videos and send them to the robotic system (100) by vision system-to-robotic system transmission means.

3. A method according to claim 1 or claim 2, wherein the vision system (120) comprises a robotic vision head.

4. A method according to any one of the preceding claims, wherein said step of encoding the digital data comprises encoding and / or multiplying the digital data in the raw video data stream so that the multiplexed stream, containing videos and data, is substantially undisturbed from the viewpoint of displaying the video images with respect to the original raw video data stream.

5. A method according to any one of the preceding claims, wherein the color distortions of the video contained in the multiplexed data stream are lower than a perceptibility threshold established by CIE standard.

6. A method according to any one of the preceding claims, wherein the raw video data stream corresponds to a video signal encoded and transmitted by means of RGB encoding, e.g., 8-bit or more, or by means of YCbCr encoding, e.g., 8-bit or more YCbCr (4:4:4, 4:2:2, 4:2:0).

7. A method according to any one of the preceding claims, wherein said encoding step comprises encoding each bit of digital data to be encoded into one and only one respective pixel of the raw video data stream.

8. A method according to any one of the preceding claims, wherein said step of encoding digital data comprises generating a payload of digital data, protecting the generated payload, encoding the generated and protected payload.

9. A method according to any one of the preceding claims, wherein said step of demultiplexing the encoded and multiplexed digital data comprises:- detecting whether the video stream contains said monitoring and / or control digital data;- if yes, separating the digital data from the data representative of the video image;- restoring the image as in the original video stream, and providing the restored image to display means (130);- decoding said digital data;- validating the decoded digital data and, if the data is validated, using the decoded digital data, i.e. , proceeding with the step of making the demultiplexed and decoded digital data available.

10. A method according to any one of the preceding claims, wherein said step of encoding the digital data or the digital data payload comprises three levels of encoding:- content-level encoding, concerning the general structure of the data;- serialized-data-level encoding, concerning the sequence of the data stream bytes;- pixel-level encoding, concerning the transformations of the pixels of the modified video stream intended to contain the digital data.

11. A method according to claim 9, wherein said content-level encoding comprises encoding in standard XDR / JSON formats, and wherein said serialized-data-level encodingcomprises serialization, encryption and protection, wherein the serialization is performed for example by means of JSON byte encoding or XDR or ASN.1 encoding or other binary serialization, wherein the encryption is performed for example by means of standard symmetric algorithms such as DES or Blowfish, wherein the protection is performed for example by means of CRC-81 CRC-16 or cryptographic signature.

12. A method according to claim 10 or claim 11 , wherein said pixel-level encoding is performed by means of algorithms combining the bytes sequence of the data to be encoded on the pixel sequence of the video signal, taking into account the type of video encoding.

13. A method according to any one of claims 1-12, wherein the raw video stream is RGB encoded, and wherein:- each bit of the data to be encoded is encoded in a respective pixel;- the encoding provides "full black" to encode 0 and "full white" to encode 1 , or vice versa, or the encoding provides using the least significant bit, LSB, of the color pixel to encode the content.

14. A method according to any one of claims 1-12, wherein the raw video stream is YCbCr-encoded, and wherein the encoding of the digital data is carried out on the luminance channel Y, wherein each data bit to be encoded is encoded in K bits of a respective pixel of the luminance channel Y, wherein, in the encoding step, 1 bit of the payload is encoded as K bits so that the zero value is K zero-value encoding bits, or the value one corresponds to 2k-1.

15. A method according to claim 14, wherein K = 2.

16. A method according to any one of claims 1-12, wherein the raw video stream is YCbCr-encoded, and wherein the encoding of the digital data is carried out on the luminance channel Y, wherein each data bit to be encoded is encoded, as 0 or 1 , in a single respective bit of the luminance channel Y of a respective pixel,and wherein the method further comprises:- checking if the modified pixel in which the data bit is encoded is a valid pixel, from the viewpoint of RGB video encoding, and thus remains usable as a video pixel;- if said modified pixel in which the data bit is encoded is an invalid pixel, modifying the Cb and / or Cr channels of the YCbCr pixel so as to obtain an adjusted modified pixel which is a valid pixel, from the viewpoint of RGB video encoding.

17. A method according to any one of claims 14-16, wherein, if the video signal provided by the vision system is RGB-encoded, the method comprises, before said step of encoding the digital data, the step of converting the RGB signal into a YCbCr-encoded raw video stream.

18. A method according to claim 17, wherein said step of converting the RGB signal into a YCbCr-encoded raw video stream comprises:- carrying out a conversion from floating RGBf format to floating YPbPr format, by means of the following transformation matrix:where KR, KG, KB are constants defined by color space;- carrying out a conversion from floating YPbPr to YCbCr.

19. A method according to any one of claims 14-16, wherein the video signal provided by the vision system is YCbCr-encoded with at least 8 bits, on which the step of encoding the digital data is performed, and the method comprises the further steps of:- converting the YCbCr signal, after encoding, into a YCbCr (4:2:2) or YCbCr (4:2:0) signal, wherein the luminance channel Y on which the encoding was performed is not further altered;- transmitting said YCbCr (4:2:2) or YCbCr (4:2:0) signal through said transmission means from vision system to robotic system.

20. A method according to any one of the preceding claims, wherein, before the encoding step, the method provides a step the pixels of the video image in whichto encode the digital data based on the geometric position of the pixels in the video image, corresponding to the visualization on a screen, according to the criterion of minimizing the distortion and / or local impact of the pixels being encoded on the video image quality.

21. A method according to claim 20, wherein the selection step comprises selecting a plurality of pixels belonging to the same horizontal or vertical line of the video image as the pixels of the video image in which to encode the digital data.

22. A method according to claim 21 , wherein the selection step comprises selecting a plurality of pixels belonging to the first half of the first or the last horizontal or vertical line of the video image as the pixels of the video image in which to encode the digital data, wherein said plurality of selected pixels comprises a sequence of consecutive pixels at the beginning of the line, or pixels evenly distributed in the first half of the line.

23. A method according to any one of the preceding claims, wherein said monitoring and / or control digital information comprises one or more of the following information:- optical information related to the operation of the vision system, and / or- dynamic optical information related to the operation of the vision system, and / or- dynamic information related to the movement of the vision system, and / or- pose information of a surgical instrument (170) embedded in the robotic system, and / or- signaling and / or monitoring information of the vision system, and / or- information related to the sensors used for image and / or video acquisition.

24. A method according to claim 1 or claim 23, wherein said monitoring and / or control digital information comprises information adapted to guide the operator and / or information related to safety measures.

25. A method according to claim 23, wherein:- said optical information related to the operation of the vision system comprises focal length and / or zoom information, usable by Computer Vision algorithms, which are present in the control unit of the robotic system, for self-adjustment purposes and / or to provide metric information; and / or- said dynamic optical information related to the operation of the vision system comprises real-time variations of zoom and / or brightness and / or exposure and / or focusparameters communicated to the robotic system by the vision system; and / or- said dynamic information related to the movement of the vision system comprises information related to the movement of a head of the vision system, and / or changes in focal length and / or zoom, usable by an operator to decide whether to stop the movement of the surgical instrument; and / or- said pose information of the surgical instrument (170) comprises surgical instrument location information viewable by the operator; and / or- said signaling and / or monitoring information of the vision system comprises an indication of any faults and / or prioritization information of the information to be displayed; and / or- said sensor-related information comprises indications that the sensors are Visible Light or ICG or another non-visible spectrum.

26. A medical system (10) comprising:- a robotic system (100) for medical or surgical teleoperation, comprising at least one surgical instrument (170) adapted to operate in teleoperation, and further comprising a control unit;- a vision system (120), outside the robotic system (100), configured to acquire images and / or videos of a teleoperation zone and generate a related raw video data stream, representative of the images and / or videos acquired;- encoding means, configured to encode digital data representative of said monitoring and / or control digital information within pixels of said raw video data stream, by means of steganographic techniques, and thus generate a multiplexed stream containing videos and data;- vision system-to-robotic system transmission means, configured to transmit data streams, comprising said raw video stream and / or said multiplexed stream containing videos and data, from the vision system to the robotic system;- demultiplexing and decoding means, configured to demultiplex and decode said encoded and multiplexed digital data and make the demultiplexed digital data available to the robotic system and / or to the control unit of the robotic system and / or to display means (130) associated with the vision system.

27. A medical system (10) according to claim 26, further comprising display means (130) associated with the vision system (120), configured to display video images of the teleoperation zone from the vision system (120).

28. A medical system (10) according to claim 26 or claim 27, wherein the vision system (120) comprises one or more image sensors or cameras, having the same viewpoint or two different respective viewpoints, configured to provide video signals representative of two- dimensional or three-dimensional images.

29. A medical system (10) according to claim 26 or claim 27, wherein the vision system comprises an exoscope.

30. A medical system (10) according to claim 27, wherein the vision system (120) and wherein the display means (130) comprise at least one electronic screen or monitor.

31. A medical system (10) according to any one of claims 26-30, wherein the vision system-to-robotic system transmission means comprise at least one HDMI cable.

32. A medical system (10) according to any one of claims 26-30, wherein the robotic system is a robotic system for surgery further comprising:- at least one master device (110) adapted to be moved by an operator (150);- at least one slave device comprising said surgical instrument (170) adapted to be controlled by the master device (110).

33. A medical system (10) according to any one of claims 26-32, wherein the encoding means are configured to carry out the steps of encoding and generating a multiplexed stream containing videos and data, in accordance with the method according to any one of claims 1- 25.

34. A medical system (10) according to any one of claims 26-33, wherein the demultiplexing and decoding means are configured to carry out the steps of demultiplexing and decoding the digital data in accordance with the method according to any one of claims 1-25.SUBSTITUTE SHEET (RULE 26)

Images