Distance measuring system and endoscope

Abstract

The invention relates to a distance measuring system and an endoscope. The system comprises an image acquisition device and a distance measuring device, the image acquisition device is used for capturing gesture information of a user and sending the captured gesture information to the distance measuring device; the distance measuring device is connected with the endoscope and is used for displaying a virtual cursor corresponding to a key point in gesture information in a tissue image in real time in response to a first operation gesture, captured by the image acquisition device, of the user in the process of displaying the tissue image, captured by a binocular lens of the endoscope, of the target tissue; the virtual cursor moves along with the position change of the key point; and the distance measuring device is also used for responding to a second operation gesture of the user captured by the image acquisition device, locking the mark points selected by the virtual cursor, and obtaining the distance of the target tissue between the mark points. The problems that an existing distance measurement system is complex in distance measurement process, time-consuming, high in execution difficulty and low in measurement precision are solved.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and particularly to a ranging system and an endoscope. Background Art

[0002] With the progress of technology, the endoscope technology has also developed to a certain extent. During the use of an endoscope, the endoscope is inserted into the body to observe the lesion site and perform operations. Specifically, the endoscope camera system extends the camera into the patient's body, and medical staff perform surgeries or other operations based on the internal human body images captured by the camera displayed on the display screen.

[0003] During the process of performing surgery using an endoscope, in some cases, in order to ensure the accuracy and operation effect of the surgery, precise distance measurement needs to be carried out within the surgical operation space. For example, in gastrointestinal reconstruction surgery, it is necessary to measure the length of the digestive tract, and in tumor resection, it is necessary to ensure a certain distance between the resection margin and the tumor.

[0004] The existing ranging systems in endoscopic surgery mainly adopt three methods: The first is to introduce a suture or gauze into the patient's body and place it beside the tissue to be measured, use the suture or gauze to mark the length of the tissue to be measured, and then take the suture or gauze out of the body to measure the mark; the second is to insert an instrument with scales into the patient's body, visually observe the scale corresponding to the tissue to be measured on the instrument through the image captured by the endoscope on the screen, so as to obtain the corresponding dimension information; the third is to identify the tip of the surgical instrument through an algorithm model, regard the tip of the surgical instrument as a marker, mark the tissue or tumor area to be measured in the patient's body through the movement of the surgical instrument, and then calculate the corresponding length by the ranging algorithm, so as to obtain the dimension information of the target area.

[0005] When using the method of introducing a suture or gauze into the patient's body for ranging, it is necessary to use an instrument to clamp the suture or gauze, move the other end on the basis of controlling one end fixed, and control the suture or gauze to fit the tissue to be measured along a straight line or a curved surface without overlapping. The operation is complex and time-consuming, and it is difficult to ensure the measurement accuracy; when using the method of inserting an instrument with scales into the patient's body for ranging, the object with scales is generally a rigid instrument, which can only measure the straight-line distance, and the ranging is limited. For tissues with a curved outer surface or a large size, it is difficult to meet the measurement requirements; when using the method of regarding the tip of the surgical instrument as a marker for ranging, a large number of high-quality real datasets are required for model training, and the acquisition of datasets is difficult and costly, and the operation space of the instrument is limited, and it is impossible to easily reach the target point, and the execution difficulty is great.

[0006] Aiming at the existing ranging systems, there are problems such as complex and time-consuming ranging processes, large execution difficulty, and low measurement accuracy, and no effective solutions have been proposed yet. Summary of the Invention

[0007] Based on this, it is necessary to provide a ranging system and an endoscope for the above technical problems.

[0008] In a first aspect, the present application provides a ranging system for ranging a target tissue in an endoscope scenario. The system includes an image acquisition device and a ranging device;

[0009] The image acquisition device is configured to capture gesture information of a user and send the captured gesture information to the ranging device;

[0010] The ranging device is connected to the endoscope and is configured to, during the process of displaying a tissue image of the target tissue captured by the binocular lens of the endoscope, in response to a first operation gesture of the user captured by the image acquisition device, display a virtual cursor corresponding to a key point in the gesture information in real time in the tissue image; the virtual cursor moves as the position of the key point changes; the ranging device is further configured to, in response to a second operation gesture of the user captured by the image acquisition device, lock a marked point selected by the virtual cursor and obtain the distance of the target tissue between the marked points.

[0011] In one embodiment, the first operation gesture and the second operation gesture are generated by the same hand; or, the first operation gesture and the second operation gesture are generated by different hands.

[0012] In one embodiment, the ranging device is configured to, during the process of displaying a tissue image of the target tissue captured by the binocular lens of the endoscope, in response to a first operation gesture of the user generated by one hand captured by the image acquisition device, display a virtual cursor corresponding to a key point in the gesture information in real time in the tissue image; the virtual cursor moves as the position of the key point of the hand generating the first operation gesture changes; the ranging device is further configured to, in response to a second operation gesture generated by the hand generating the first operation gesture captured by the image acquisition device, lock a marked point selected by the virtual cursor controlled by this hand and obtain the distance of the target tissue between the marked points;

[0013] Alternatively, during the process of displaying the tissue image of the target tissue captured by the binocular lens of the endoscope, the distance measuring device is configured to, in response to a first operation gesture jointly generated by the user's first hand and second hand captured by the image acquisition device, display a virtual cursor corresponding to the key points in the gesture information in real time in the tissue image; the virtual cursor moves as the position of the key points of the second hand changes; the distance measuring device is further configured to, in response to a second operation gesture generated by the user's first hand captured by the image acquisition device, lock the marked points selected by the second hand to control the virtual cursor, and obtain the distance of the target tissue between the marked points.

[0014] In one embodiment, the distance measuring device is further configured to superimpose and display the position of the virtual cursor, the movement trajectory of the virtual cursor, the coordinates of the marked points selected by each virtual cursor, the hand projection, and the distance of the target tissue between the marked points on the tissue image.

[0015] In one embodiment, the distance measuring device is further configured to, in response to a third operation gesture generated by the user captured by the image acquisition device, delete the marked points selected by the virtual cursor in the tissue image; in response to a fourth operation gesture generated by the user captured by the image acquisition device, save the marked points selected by the virtual cursor in the tissue image; in response to a fifth operation gesture generated by the user captured by the image acquisition device, continuously mark the marked points selected by the virtual cursor in the tissue image.

[0016] In one embodiment, the distance measuring device is further configured to, in response to a fifth operation gesture generated by the user captured by the image acquisition device, continuously mark the marked points selected by the virtual cursor in the tissue image, and based on the result of the continuous marking, obtain the curved distance of the target tissue between the marked points.

[0017] In one embodiment, the distance measuring device includes: an image acquisition module and a distance measurement module;

[0018] The image acquisition module is configured to, in response to a captured command received, capture a target tissue using the binocular lens of the endoscope, obtain a pair of tissue images of the target tissue, and send them to the distance measurement module; the pair of tissue images are two tissue images obtained by simultaneously photographing the target tissue using the binocular lens of the endoscope;

[0019] The distance measurement module is configured to, in response to a distance measurement start command received, generate the captured command and send it to the image acquisition module;

[0020] The ranging module is further configured to control the virtual cursor to move and mark target positions in the target tissue image based on the gesture information captured by the image acquisition device, so as to obtain each marked point of the target tissue image; the target tissue image is any one of the pair of tissue images.

[0021] The ranging module also locks the marked points selected by the virtual cursor and calculates the three-dimensional distance of the target tissue between the marked points.

[0022] In one embodiment, the ranging module further includes a parallax calculation unit, a depth value calculation unit, a coordinate calculation unit, and a distance calculation unit.

[0023] The parallax calculation unit is configured to determine the parallax of each marked point between the target tissue image and the first tissue image based on the image coordinates of the marked points in the target tissue image and the image coordinates of each marked point in the first tissue image; the first tissue image is the other image in the pair of tissue images except the target tissue image.

[0024] The depth value calculation unit is configured to determine the depth value of each marked point in the target tissue image based on the parallax of each marked point between the target tissue image and the first tissue image.

[0025] The coordinate calculation unit is configured to determine the world coordinates of each marked point based on the image coordinates of each marked point in the target tissue image, the parameters of the binocular lens of the endoscope, and the depth value of each marked point in the target tissue image.

[0026] The distance calculation unit is configured to calculate the three-dimensional distance of the target tissue between the marked points based on the world coordinates of each marked point.

[0027] In one embodiment, the ranging module is further configured to calculate the three-dimensional shape and size of the target tissue based on the world coordinates of multiple marked points.

[0028] In one embodiment, the triggering method of the ranging start command includes at least one of a handle button, a host button, a host touch screen button, a foot switch, a voice command, and a gesture action.

[0029] In one embodiment, a user configuration module is further included.

[0030] The user configuration module is configured to bind the mapping relationship between the operation gesture and the cursor operation command; the cursor operation command includes one of a ranging function on, a ranging function off, a mark, a delete, a save, and a continuous mark command.

[0031] In a second aspect, the present application also provides an endoscope. The endoscope is integrated with the ranging system described in the first aspect above.

[0032] The above ranging system and endoscope include an image acquisition device and a ranging device; the above image acquisition device is used to capture the gesture information of the user and send the captured gesture information to the ranging device. The ranging device is connected to the endoscope and is used to, during the process of displaying the tissue image of the target tissue captured by the binocular lens of the endoscope, in response to the first operation gesture of the user captured by the image acquisition device, display a virtual cursor corresponding to the key points in the gesture information in the tissue image in real time, where the virtual cursor moves as the position of the key points changes. The ranging device is further used to, in response to the second operation gesture of the user captured by the image acquisition device, lock the marked points selected by the virtual cursor and obtain the distance of the target tissue between the marked points. It uses the image acquisition device to obtain the gesture information of the user, and according to the gesture information of the user, controls the virtual cursor to select the marked points of the target tissue in the tissue image, and based on the selection result, obtains the distance of the target tissue between the marked points. In this process, it is not necessary to insert the object to be measured into the patient's body. Only through the tissue image captured by the endoscope, the image position is marked outside the body without contact, and the ranging of the target tissue is carried out using the marked image. The process is simple, saves ranging time, and the calculation result is accurate, solving the problems of the existing ranging system, such as complex and time-consuming ranging process, difficult execution, and low measurement accuracy.

[0033] Details of one or more embodiments of the present application are set forth in the following drawings and description, so that other features, objects, and advantages of the present application become more concise and understandable. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0035] Figure 1 is the first structural block diagram of the ranging system provided in this embodiment;

[0036] Figure 2 is a schematic diagram of the ranging device of the ranging system provided in this embodiment showing the virtual hand projection;

[0037] Figure 3 is a schematic diagram of the starting marked point locking process of the ranging system provided in this embodiment;

[0038] Figure 4 is a schematic diagram of the ending marked point locking process of the ranging system provided in this embodiment;

[0039] Figure 5 It is the second structural block diagram of the ranging system provided in this embodiment;

[0040] Figure 6 It is the structural block diagram of the ranging module of the ranging system provided in this embodiment. Specific embodiments

[0041] To understand the purpose, technical solution and advantages of the present application more clearly, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments.

[0042] Unless otherwise defined, the technical terms or scientific terms involved in the present application shall have the general meaning understood by those with ordinary skills in the technical field to which the present application belongs. In the present application, words such as "a", "an", "one kind", "the", "these" and the like do not indicate a limitation in quantity, and they can be singular or plural. The terms "including", "comprising", "having" and any variants thereof involved in the present application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device including a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent in these processes, methods, products or devices. The terms "connected", "coupled" and the like involved in the present application do not limit to physical or mechanical connections, but may include electrical connections, whether directly or indirectly connected. The "plurality" involved in the present application means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may mean: A exists alone, A and B exist simultaneously, and B exists alone. Usually, the character " / " indicates that the objects associated before and after are an "or" relationship. The terms "first", "second", "third" and the like involved in the present application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0043] In this embodiment, a ranging system is provided, which is applied to the ranging of a target tissue in an endoscope scenario. Figure 1 It is the first structural block diagram of the ranging system provided in this embodiment, as Figure 1As shown, the system includes an image acquisition device 110 and a distance measurement device 120. Among them, the image acquisition device 110 is used to capture the gesture information of the user and send the captured gesture information to the distance measurement device 120. Among them, the distance measurement device 120 is connected to the endoscope and is used to, during the process of displaying the tissue image of the target tissue captured by the binocular lens of the endoscope, in response to the first operation gesture of the user captured by the image acquisition device 110, display a virtual cursor corresponding to the key point in the gesture information in real time in the tissue image; the virtual cursor moves as the position of the key point changes. The distance measurement device 120 is further used to, in response to the second operation gesture of the user captured by the image acquisition device 110, lock the marked points selected by the virtual cursor and obtain the distance of the target tissue between the marked points.

[0044] The above-mentioned target tissue can be the tissue observed by using the endoscope lens in the endoscope scenario. The specific tissue can be vascular tissue, esophageal tissue, gastric tissue, rectal tissue, colonic tissue, duodenal tissue, tracheal tissue, abdominal tissue, etc. The above-mentioned locking of the marked points selected by the virtual cursor can be at least locking the starting marked point and the ending marked point selected by the virtual cursor. The above-mentioned starting marked point is the marked point corresponding to the starting position where the distance needs to be measured in the target tissue. The above-mentioned ending marked point is the marked point corresponding to the ending position where the distance needs to be measured in the target tissue. The distance measurement of the above-mentioned target tissue can be the straight-line distance or the curved distance between a certain target point and other target points in the target tissue. In this embodiment, the distance measurement of the above-mentioned target tissue refers to the three-dimensional distance, that is, the distance including three dimensions: the abscissa, the ordinate, and the depth information.

[0045] In this embodiment, the above image acquisition device 110 may be a distance measurement device 120 embedded in a certain display screen of an endoscope or an external camera installed outside the endoscope display screen. It should be noted that in general, the user measures the distance of the target tissue while observing the endoscope distance measurement device 120. In order to facilitate the user to perform distance measurement control in front of the endoscope distance measurement device 120 during the distance measurement process, the camera can be embedded in a certain display screen of the endoscope distance measurement device 120, or the image acquisition device 110 can be installed at other target positions according to the actual situation or actual needs. The specific installation position is not specifically limited in this embodiment as long as the gesture information of the user in the target area can be captured by the image acquisition device 110. The above target area may be an area within a certain distance directly in front of the endoscope display screen, or other position areas preset according to specific circumstances for capturing the gesture information of the user. The specific area range and area position are not specifically limited in this embodiment. The above distance measurement device 120 may include one or more display screens. For convenience of distinction, one of the display screens can be named the main display screen, and the other display screens can be named sub-display screens. The setting of multiple display screens is to facilitate multiple medical staff to observe the target tissue images in the target area captured by the endoscope and to view the distance measurement results of the target tissue. Generally, the image acquisition device 110 can be embedded above the main display screen.

[0046] The gesture information of the user in the target area captured by the above image acquisition device 110 may be the gesture information of the user in the target area captured by using a preset gesture capture method. The above preset gesture capture method may be gesture capture by one or more trained artificial intelligence models. The above artificial intelligence models may include, but are not limited to, machine learning models, supervised learning models, semi-supervised learning models, unsupervised learning models, deep learning models, neural network models, reinforcement learning models, natural language processing models, etc.

[0047] Among them, the gesture information of the above user can include single - hand information and two - hand information, and can include hand key - point information and gesture information. The above hand key - point information can be the position information of hand key - points and the position - change information of hand key - points. The above gesture information can include various operation gestures. The above operation gestures can be gestures composed of one or more consecutive hand movements. For example, palm opening can be used as an operation gesture, and the combination of palm opening - fist clenching - palm opening can also be used as an operation gesture. Among them, the above operation gestures at least include a first operation gesture and a second operation gesture. The above first operation gesture can be the gesture corresponding to the function of enabling the real - time display of a virtual cursor corresponding to the key - points in the gesture information in the tissue image, and the virtual cursor moves with the position change of the key - points. The above second operation gesture can be the gesture corresponding to locking the marked points selected by the virtual cursor in the tissue image and displaying the distance of the target tissue between the locked marked points. The specific first operation gesture and second operation gesture are specifically set according to specific requirements.

[0048] For example, the first hand can be opened, the second hand can extend the index finger, and the other fingers can be curled as the first operation gesture. The virtual cursor moves with the position change of the index finger of the second hand. At this time, when the first hand disappears or the open gesture is cancelled, the display of the virtual cursor stops. The first hand can be continuously reversed in a short time, the second hand can extend the index finger, and the other fingers can be curled as the second operation gesture. The virtual cursor moves with the position change of the index finger of the second hand. When the first hand is continuously reversed once in a short time, the position where the virtual cursor is located is locked as the starting marked point. When the first hand is continuously reversed twice, the position where the virtual cursor is located is locked as the ending marked point, and the distance of the target tissue between the starting marked point and the ending marked point is obtained. This example only provides one of the situations of the first operation gesture and the second operation gesture. The specific operation gestures can be specifically set according to the needs and habits of the user, and this embodiment does not make specific limitations here.

[0049] In another example, the first hand can extend two fingers to form a V shape, and the second hand extends the index finger while curling the other fingers as the first operation gesture. The virtual cursor moves with the position change of the index finger of the second hand. At this time, when the first hand disappears or cancels the V shape, the display of the virtual cursor stops. The first hand can make a fist continuously for a short time, and the second hand extends the index finger while curling the other fingers as the second operation gesture. The virtual cursor moves with the position change of the index finger of the second hand. When the first hand makes a fist continuously once for a short time, the position where the virtual cursor is located is locked as the starting mark point. When the first hand makes a fist continuously twice, the position where the virtual cursor is located is locked as the ending mark point, and the distance of the target tissue between the starting mark point and the ending mark point is obtained. It should be noted that this example only provides one of the first operation gesture and the second operation gesture, and the specific operation gesture can be specifically set according to the user's needs and habits, and this embodiment does not make specific limitations here.

[0050] In addition, in order to obtain the depth information of the tissue, there are two cameras in the endoscope, and each camera includes a lens, that is, the lens of the endoscope is a binocular lens, and the endoscope is a binocular endoscope. In order to calculate the depth information of the target tissue, when the endoscope captures the tissue image of the target tissue using the binocular lens, it is necessary to ensure that the binocular lens captures the tissue image of the target tissue simultaneously to obtain a pair of tissue images, and use the pair of tissue images and the parameters of the binocular endoscope to calculate the parallax of the target tissue, that is, the depth information of the target tissue. The above parameters of the binocular endoscope can include the internal parameters and external parameters of the endoscope camera. The above internal parameters include focal length, principal point coordinates, distortion coefficients, etc. The above external parameters can include baseline, rotation matrix, translation vector, etc. The above parameters of the binocular endoscope can be obtained through a calibration process, or can be obtained through software algorithms or models. It should be noted that when using the endoscope to capture the tissue image of the target tissue, the parameters of the binocular endoscope have been determined and belong to known parameters.

[0051] In addition, in one embodiment, the ranging device 120 is further configured to superimpose and display the position of the virtual cursor, the moving trajectory of the virtual cursor, the coordinates of the mark points selected by each virtual cursor, the hand projection, and the distance of the target tissue between the mark points on the tissue image.

[0052] The coordinates of the mark points selected by the above virtual cursor can be the three-dimensional coordinates of the mark points, and the above three-dimensional coordinates are the coordinates in the world coordinate system.

[0053] Preferably, in order to facilitate human-computer interaction more, the gesture information obtained by the image acquisition device 110 can also be projected onto the ranging device 120 for display, that is, a virtual hand projection is displayed on the ranging device 120. Figure 2Schematic diagram showing the virtual hand projection displayed by the distance measurement device 120 of the distance measurement system provided in this embodiment. As Figure 2 shown, it is the hand projection with both hands outstretched. The above-mentioned hand projection is a hand gesture projection or a hand gesture change projection. The above-mentioned hand gesture change projection is a projection of the change process of the hand gesture. Specifically, the above-mentioned hand projection can be set to projections of different colors and different transparencies according to requirements. Preferably, in order to prevent the hand projection from blocking the tissue structure, the hand projection can also be set to a projection with only the outer contour of the hand (the inside of the contour is set to a transparent state). The specific setting rules of the projection can be specifically set according to specific requirements, as long as the current hand gesture or the change process of the hand gesture can be determined through the hand projection. This embodiment does not make specific limitations here.

[0054] In this embodiment, the embedded camera of the endoscope display screen or the external camera connected to the endoscope display screen is used to capture the hands of the surgeon or other medical staff, capture the key points of the hands, and based on the captured key points of the hands as marking points, a virtual cursor corresponding to the key points of the hands is displayed in real time on the endoscope display screen. Furthermore, through the operation gestures in the gesture information recognized by the camera, the virtual cursor is controlled to lock the respective marking points of the target tissue. Then, based on the locked respective marking points, the distance of the target tissue is calculated and displayed on the endoscope display screen. This enables the surgeon or other medical staff to perform human-computer interaction and achieve real-time three-dimensional measurement of the target tissue in the endoscope scenario by simply making gestures in front of the endoscope display screen without any contact between any person and the device and without any instrument contacting the patient's internal environment.

[0055] Furthermore, in this embodiment, the key points in the above-mentioned gesture information can be a certain key point or multiple key points of the human hand recognized in the target shooting area, and specifically, one or more points such as the fingertip of the index finger, the fingertip of the middle finger, and the center point of the palm can be specified. The above-mentioned virtual cursor is displayed in the tissue image displayed on the endoscope display screen when the distance measurement function is turned on and a human hand is recognized in the target shooting area. It moves with the change of the position of the key points of the hand and can be used to point to the target position of the tissue image, and can simulate the functions of a real cursor such as marking, deleting, and distance measurement at the target position, and is used to interact with the user's gesture information. The above-mentioned virtual cursor can be a custom mouse pointer, and specifically can be set in forms such as a hand shape, a cross shape, a text selection cursor, etc. It should be noted that the style and function of the virtual cursor can be customized according to its application scenario and user requirements to provide a better user experience and interaction effect.

[0056] Among them, in one embodiment, the first operation gesture and the second operation gesture are generated by the same hand; or, the first operation gesture and the second operation gesture are generated by different hands.

[0057] When the first operation gesture and the second operation gesture are generated by the same hand, the same hand of the user first generates the first operation gesture, and a certain point in this hand is used as a key point. A virtual cursor corresponding to the key point in the gesture information is displayed in real time in the tissue image. As the position of the key point of this hand changes, the position of the virtual cursor changes. When the second operation gesture generated by this hand is recognized, the marking points are locked, and the distance of the target tissue between the locked marking points is displayed.

[0058] In addition, when the first operation gesture and the second operation gesture are generated by different hands, the user uses different hands to generate the first operation gesture and uses different hands to generate the second operation gesture. Or, the user generates the first operation gesture and the second operation gesture respectively with different hands. For example, the user generates the first operation gesture with the first hand and generates the second operation gesture with the second hand. Specifically, during the distance measurement process, one hand serves as the operation hand to perform operations such as starting the distance measurement and locking the marking points, and the other hand is used as the control hand for the position of the virtual cursor. Through the change of the position of the other hand, the position change of the virtual cursor is realized. The two hands work together to lock the marking points of the target position, and based on the locking result of the marking points, the distance of the target tissue between the starting marking point and the ending marking point is calculated.

[0059] For example, to measure the straight-line distance between two points on a tissue, the medical staff can expose both hands. In one example, the medical staff can use the right hand as the calibration hand for measuring the distance and the left hand as the command-issuing hand for measuring the distance. Specifically: the left hand is opened, and the right hand extends the index finger and curls the other fingers so that the virtual cursor corresponding to the right index finger reaches the starting point position of the tissue length to be measured, and the position of the starting marking point is determined. After that, the right hand remains stationary, and the left hand makes a fist and then releases it, so that the position where the right index finger is located when the left hand makes a fist and then releases it, that is, the position of the virtual cursor corresponding to the right index finger, is marked as the starting marking point, that is, the starting marking point is locked. Figure 3 It is a schematic diagram of the starting marking point locking process of the distance measurement system provided in this embodiment. As Figure 3 shown, the arrow indicates the sequential action order of the left hand. Then, the right index finger can be moved to the ending point position of the tissue length to be measured and remain stationary. The left hand repeats the series of actions of "opening - making a fist - opening" again to issue a calibration command, so that the position where the virtual cursor corresponding to the right index finger is located is marked as the end point, that is, the ending marking point is locked. Figure 4 It is a schematic diagram of the ending marking point locking process of the distance measurement system provided in this embodiment. As Figure 4As shown, the arrows indicate the sequential action order of the left hand, which, together with the hand pose projections corresponding to each sequential action, generate the projection of the hand pose change process. When the starting point and the ending point are successfully marked, the medical staff can relax their hands. At this time, the coordinates and depth information of these two points in the image coordinate system can be obtained, and then the coordinates of these two points in the world coordinate system can be calculated, and the straight-line distance between the two points can be obtained.

[0060] Specifically, in one embodiment, the distance measuring device 120 is configured to, during the process of displaying the tissue image of the target tissue captured by the binocular lens of the endoscope, in response to the first operation gesture generated by a hand of the user captured by the image acquisition device 110, display a virtual cursor corresponding to the key points in the gesture information in the tissue image in real time; the virtual cursor moves with the change of the position of the key points of the hand generating the first operation gesture; the distance measuring device 120 is further configured to, in response to the second operation gesture generated by the hand generating the first operation gesture captured by the image acquisition device 110, lock the marked points selected by the virtual cursor controlled by this hand, and obtain the distance of the target tissue between the marked points; or, the distance measuring device 120 is configured to, during the process of displaying the tissue image of the target tissue captured by the binocular lens of the endoscope, in response to the first operation gesture jointly generated by the first hand and the second hand of the user captured by the image acquisition device 110, display a virtual cursor corresponding to the key points in the gesture information in the tissue image in real time; the virtual cursor moves with the change of the position of the key points of the second hand; the distance measuring device 120 is further configured to, in response to the second operation gesture generated by the first hand of the user captured by the image acquisition device 110, lock the marked points selected by the virtual cursor controlled by the second hand, and obtain the distance of the target tissue between the marked points.

[0061] In one embodiment, the distance measuring device 120 is further configured to, in response to the third operation gesture of the user captured by the image acquisition device 110, delete the marked points selected by the virtual cursor in the tissue image; in response to the fourth operation gesture of the user captured by the image acquisition device 110, save the marked points selected by the virtual cursor in the tissue image.

[0062] The above-mentioned third operation gesture can be the gesture corresponding to deleting the marked points selected by the virtual cursor in the tissue image. By using the third operation gesture, the marked points can be deleted when the marked points are mislabeled. The above-mentioned fourth operation gesture can be the gesture corresponding to saving the marked points selected by the virtual cursor in the tissue image. The above-mentioned fifth operation gesture can be the gesture corresponding to continuously marking the marked points selected by the virtual cursor in the tissue image.

[0063] For example, the first hand can be opened and waved left and right, and the index finger of the second hand can be extended while the other fingers are curled as the third operation gesture. The first hand can be clenched into a fist, and the index finger of the second hand can be extended while the other fingers are curled as the fourth operation gesture. This example only provides one of the situations of the third operation gesture and the fourth operation gesture. The specific operation gesture can be specifically set according to the user's needs and habits, and this embodiment does not make specific limitations here.

[0064] It should be noted that after saving multiple marked points using the fourth operation gesture, the initial marked point and the termination marked point can be selected from the multiple saved marked points using the second operation gesture. Furthermore, the initial marked point and the termination marked point can be locked, and the distance of the target tissue between the locked marked points can be displayed.

[0065] In one embodiment, the distance measuring device 120 is further configured to, in response to a fifth operation gesture of the user captured by the image acquisition device 110, continuously mark the marked points selected by the virtual cursor in the tissue image, and based on the result of the continuous marking, obtain the curved distance of the target tissue between the marked points.

[0066] Among them, in order to make the distance of the target tissue more accurate, between the starting marked point and the termination marked point, multiple marked points selected by the virtual cursor are continuously marked, that is, between the starting marked point and the termination marked point, multiple process marked points can be locked. The operation gesture of the process marked point, that is, the fifth operation gesture, can be specifically set according to specific needs. For example, three fingers of the first hand can be extended, or the little finger of the second hand can be straightened, and the position where the virtual cursor is located is locked as the process marked point. It should be noted that this example only provides one or two situations of locking the process marked point. The specific operation gesture of locking the process marked point can be specifically set according to the user's needs and habits, and this embodiment does not make specific limitations here.

[0067] The above-mentioned obtaining the curved distance of the target tissue between the marked points based on the result of the continuous marking may be that when calculating the distance between the initial marked point and the termination marked point, if there are process marked points between the starting marked point and the termination marked point, it is necessary to first determine the world coordinates of each marked point, and then use the world coordinates of all marked points to construct an irregular curve. Furthermore, through a preset calculation method, the irregular curve can be segmented into several line segments approximated to straight lines, and length calculation methods including but not limited to Euclidean distance, Manhattan distance, Chebyshev distance, Minkowski distance, cosine similarity, Pearson correlation coefficient, Jaccard correlation coefficient, etc. are used to calculate the lengths of each line segment. Finally, the lengths of each line segment are summed up to obtain the length of the curve between the starting marked point and the termination marked point, that is, the distance of the target tissue between the starting marked point and the termination marked point. The above-mentioned preset calculation algorithm can be the infinitesimal method or other calculation methods applicable to the curved distance.

[0068] Further, a data saving gesture can be set. For example, when it is necessary to save relevant data, medical staff can make a data saving gesture in the target shooting area, and the current data can be saved. For example, the data saving gesture can be that the index finger and thumb of the left hand are extended, and the other three fingers are curled. When the ranging system recognizes this gesture, the current screenshot and relevant measurement parameters can be saved. This example only shows one data saving gesture. Other gestures can also be set as data saving gestures according to needs. A fourth operation gesture can also be set as the data saving gesture, that is, other data is saved while the marked points are saved. This embodiment does not make specific limitations here.

[0069] In some cases, when it is necessary to re-measure the target tissue and it is necessary to clear all the marked points and the calculated data, a data clearing gesture needs to be set. For example, the data clearing gesture can be that the medical staff keep their hands outstretched. This example only shows one data clearing gesture. Other gestures can also be set as data clearing gestures according to needs. This embodiment does not make specific limitations here.

[0070] In addition, Figure 5 This is the second structural block diagram of the ranging system provided in this embodiment. As Figure 5 shown, the ranging device 120 includes: an image acquisition module 122 and a ranging module 124; the image acquisition module 122 is configured to capture the target tissue using the binocular lens of the endoscope in response to the received capture command, obtain a pair of tissue images of the target tissue, and send them to the ranging module 124; the pair of tissue images is two tissue images obtained by simultaneously photographing the target tissue using the binocular lens of the endoscope; the ranging module 124 is configured to generate a capture command and send it to the image acquisition module 122 in response to the received ranging start command; the ranging module 124 is further configured to control the virtual cursor to move and mark the target position in the target tissue image based on the gesture information captured by the image acquisition device 110, so as to obtain each marked point of the target tissue image; the target tissue image is any one of the pair of tissue images; the ranging module 124 is further configured to lock the marked points selected by the virtual cursor and calculate the three-dimensional distance of the target tissue between the marked points.

[0071] Among them, in one embodiment, the triggering method of the ranging start command includes at least one of a handle button, a host button, a host touch screen button, a foot switch, a voice command, and a gesture action.

[0072] The above-mentioned handle buttons can be the buttons on the handle for controlling the endoscope or the handle for starting the distance measurement of the endoscope. The above-mentioned host buttons can be the buttons on the endoscope host for starting the distance measurement. The above-mentioned host touch screen buttons can be the touch screen buttons on the endoscope host screen that indicate starting the distance measurement function. The above-mentioned gesture actions can be the gesture actions corresponding to the distance measurement start command obtained by the image acquisition device 110.

[0073] Further, in one embodiment, Figure 6 is the structural block diagram of the distance measurement module of the distance measurement system provided in this embodiment, as Figure 6 shown, the distance measurement module 124 further includes a parallax calculation unit 1242, a depth value calculation unit 1244, a coordinate calculation unit 1246, and a distance calculation unit 1248; the parallax calculation unit 1242 is configured to determine the parallax between each marker point in the target tissue image and the first tissue image based on the image coordinates of the marker points in the target tissue image and the image coordinates of each marker point in the first tissue image; the first tissue image is the other image in the tissue image pair except the target tissue image; the depth value calculation unit 1244 is configured to determine the depth value of each marker point in the target tissue image based on the parallax between each marker point in the target tissue image and the first tissue image; the coordinate calculation unit 1246 is configured to determine the world coordinates of each marker point based on the image coordinates of each marker point in the target tissue image, the parameters of the binocular lens of the endoscope, and the depth value of each marker point in the target tissue image; the distance calculation unit 1248 is configured to calculate the three-dimensional distance of the target tissue between the marker points based on the world coordinates of each marker point.

[0074] Among them, the above-mentioned parallax calculation unit 1242 determines the parallax of each fiducial point between the target tissue image and the first tissue image based on the image coordinates of the fiducial points in the target tissue image and the image coordinates of each fiducial point in the first tissue image. The process can be to determine the position of the fiducial point of the target tissue image in the corresponding first tissue image based on the fiducial points selected by the user using the virtual cursor, and then read the image coordinates of each fiducial point, that is, the coordinates of the image coordinate system, in the target tissue image and the corresponding first tissue image. Furthermore, the parallax of each fiducial point between the target tissue image and the corresponding first tissue image is determined using the image coordinates of each fiducial point in the target tissue image and the corresponding first tissue image. The above-mentioned parallax can be the difference in the abscissa of the same fiducial point on the target tissue image and the corresponding first tissue image in the image coordinate system. The above-mentioned method for determining the position of the fiducial point of the target tissue image in the corresponding first tissue image can be one or more feature matching methods such as SIFT (Scale-Invariant Feature Transform) and ORB (Oriented FAST and Rotated BRIEF).

[0075] In addition, the process of reading the image coordinates of each fiducial point using the pixel coordinates can be expressed as follows:

[0076] The transformation between the point (u, v) in the pixel coordinate system and the point (x, y) in the image coordinate system can be obtained through the following relationship:

[0077] ;

[0078] Or it can be obtained through the following relationship:

[0079] ;

[0080] where dx and dy respectively represent the length corresponding to a single pixel in the x direction and the y direction, and u0 and v0 respectively represent the pixel coordinates of the intersection of the camera optical axis and the imaging plane.

[0081] The above-mentioned depth value calculation unit 1244 determines the depth value of each fiducial point in the target tissue image based on the parallax of each fiducial point between the target tissue image and the first tissue image, and can calculate the depth value of each fiducial point in the target tissue image using a preset depth calculation formula.

[0082] The above-mentioned preset depth calculation formula can be:

[0083] ;

[0084] Among them, the above-mentioned z represents the depth value of the marked point, indicating the distance from the target tissue to the lens of the endoscope. The above-mentioned f is the focal length of the lens of the endoscope, B is the baseline of the binocular lens of the endoscope, indicating the distance between the binocular lenses of the endoscope, and d is the parallax of the marked point between the target tissue image and the first tissue image.

[0085] Furthermore, the coordinate calculation unit 1246 determines the world coordinates of each marked point based on the image coordinates of each marked point in the target tissue image, the parameters of the binocular lens of the endoscope, and the depth value of each marked point in the target tissue image, which can be achieved by using a preset first conversion formula and a preset second conversion formula. The above-mentioned first conversion formula is the conversion formula between the image coordinate system and the camera coordinate system. The above-mentioned second conversion formula is the conversion formula between the camera coordinate system and the world coordinate system.

[0086] Among them, the first conversion formula is:

[0087] ;

[0088] ;

[0089] Among them, f x and f y are the focal lengths of the binocular lenses of the endoscope, (c x , c y ) are the optical center coordinates of the binocular lenses of the endoscope, and (X, Y) is the two-dimensional representation result of the coordinate point in the camera coordinate system.

[0090] The second conversion formula can include two cases:

[0091] The first is rotation first and then translation. That is, assuming that the world coordinate system first rotates around the X-axis, Y-axis, and Z-axis by α, β, and γ respectively according to the right-hand rule, and then translates along the X-axis, Y-axis, and Z-axis by T x , T y , T z to obtain the camera coordinate system. At this time, the second conversion formula is:

[0092] ;

[0093] Among them, (X c , Y c , Z c ) is the three-dimensional representation result of the coordinate point in the camera coordinate system, (X w , Y w , Z w ) is the coordinate point representation result in the world coordinate system, R x , R y , R z，Represent the rotation matrices about the X, Y, and Z axes respectively, i.e., the rotation matrix of the external camera parameters, [T x , T y , T z -1 is the translation vector in the external camera parameter matrix.

[0094] Second, translation first and then rotation. That is, assume that the world coordinate system is first translated along the X-axis, Y-axis, and Z-axis by T x , T y , T z respectively, and then rotated by α, β, and γ about the X-axis, Y-axis, and Z-axis according to the right-hand rule to obtain the camera coordinate system.

[0095] At this time, the second conversion formula is:

[0096] ;

[0097] The above distance calculation unit 1248 is used to calculate the three-dimensional distance of the target tissue between the marker points based on the world coordinates of each marker point. It can first use the world coordinates of all marker points to construct an irregular curve. Furthermore, through a preset calculation method, the irregular curve can be segmented into several line segments approximately similar to straight lines, and length calculation methods including but not limited to Euclidean distance, Manhattan distance, Chebyshev distance, Minkowski distance, cosine similarity, Pearson correlation coefficient, Jaccard correlation coefficient, etc. are used to calculate the lengths of each line segment. Finally, the lengths of each line segment are summed up to obtain the three-dimensional distance of the target tissue. The above preset calculation algorithm can be the infinitesimal element method or other calculation methods applicable to curve distances.

[0098] In one embodiment, the ranging module is further used to calculate the three-dimensional shape and size of the target tissue based on the world coordinates of multiple marker points.

[0099] The above calculation of the three-dimensional shape and size of the target tissue based on the world coordinates of multiple marker points can be to first use the obtained world coordinates of the marker points to construct the point cloud data of the target tissue. Furthermore, through the point cloud data, a preset three-dimensional reconstruction algorithm is used to reconstruct the three-dimensional model of the target tissue. Then, based on the reconstructed three-dimensional model, the size of the target tissue, such as length, width, height, etc., can be calculated. The above preset three-dimensional reconstruction algorithm can be one or more of Delaunay triangulation, Poisson reconstruction, etc., and this embodiment does not make specific limitations in this regard.

[0100] In addition, in one embodiment, a user configuration module is further included; the user configuration module is used to bind the mapping relationship between the operation gesture and the cursor operation command; the cursor operation command includes one of the commands of ranging function on, ranging function off, marking, deleting, saving, and continuous marking. ​

[0101] The above user configuration module facilitates various operations on the marking points and distance measurement by binding the mapping relationship between the operation gestures and the cursor operation commands and using the recognized operation gestures according to the binding result.

[0102] It should be noted that the above-mentioned various modules can be functional modules or program modules, and can be implemented either by software or by hardware. For the modules implemented by hardware, the above-mentioned various modules can be located in the same processor; or the above-mentioned various modules can also be located in different processors respectively in any combination form.

[0103] In one embodiment, an endoscope is further provided. The endoscope integrates the distance measurement system described in any of the above embodiments and is used to measure the distance of the target tissue captured by the binocular lens of the endoscope.

[0104] It should be understood that the specific embodiments described here are only used to explain this application, rather than to limit it. According to the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of this application.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0106] Obviously, the drawings are only some examples or embodiments of this application. For those of ordinary skill in the art, this application can also be applied to other similar situations based on these drawings without creative work. In addition, it can be understood that although the work done during this development process may be complex and time-consuming, for those of ordinary skill in the art, certain design, manufacturing, or production changes based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient disclosure of this application.

[0107] The term "embodiment" in this application means that the specific features, structures, or characteristics described in combination with the embodiment may be included in at least one embodiment of this application. The phrase appears in various positions in the specification does not necessarily mean the same embodiment, nor does it mean being independent or alternative to other embodiments and mutually exclusive. Those of ordinary skill in the art can clearly or implicitly understand that the embodiments described in this application can be combined with other embodiments without conflict.

[0108] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of patent protection. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A ranging system is applied to the ranging of a target tissue in an endoscope scenario, and is characterized in that, The system includes an image acquisition device and a ranging device; The image acquisition device is configured to capture the gesture information of the user and send the captured gesture information to the ranging device; The ranging device, which is connected to the endoscope, is configured to, during the process of displaying the tissue image of the target tissue captured by the binocular lens of the endoscope, in response to the first operation gesture of the user captured by the image acquisition device, display a virtual cursor corresponding to the key point in the gesture information in real time in the tissue image; the virtual cursor moves as the position of the key point changes; the ranging device is further configured to, in response to the second operation gesture of the user captured by the image acquisition device, lock the marked points selected by the virtual cursor and obtain the distance of the target tissue between the marked points.

2. The ranging system according to claim 1, wherein The first operation gesture and the second operation gesture are generated by the same hand; or, the first operation gesture and the second operation gesture are generated by different hands.

3. The ranging system according to claim 2, wherein The ranging device is configured to, during the process of displaying the tissue image of the target tissue captured by the binocular lens of the endoscope, in response to the first operation gesture of the user generated by one hand captured by the image acquisition device, display a virtual cursor corresponding to the key point in the gesture information in real time in the tissue image; the virtual cursor moves as the position of the key point of the hand generating the first operation gesture changes; the ranging device is further configured to, in response to the second operation gesture generated by the hand that generates the first operation gesture captured by the image acquisition device, lock the marked points selected by the virtual cursor controlled by that hand and obtain the distance of the target tissue between the marked points; Or, the ranging device is configured to, during the process of displaying the tissue image of the target tissue captured by the binocular lens of the endoscope, in response to the first operation gesture jointly generated by the first hand and the second hand of the user captured by the image acquisition device, display a virtual cursor corresponding to the key point in the gesture information in real time in the tissue image; the virtual cursor moves as the position of the key point of the second hand changes; the ranging device is further configured to, in response to the second operation gesture generated by the first hand of the user captured by the image acquisition device, lock the marked points selected by the virtual cursor controlled by the second hand and obtain the distance of the target tissue between the marked points.

4. The ranging system according to any one of claims 1 to 3, characterized in that, The ranging device is further configured to superimpose and display the position of the virtual cursor, the movement trajectory of the virtual cursor, the coordinates of the marked points selected by each virtual cursor, the hand projection, and the distance of the target tissue between the marked points on the tissue image.

5. The ranging system according to any one of claims 1 to 3, characterized in that The ranging device is further configured to, in response to the third operation gesture of the user captured by the image acquisition device, delete the marked points selected by the virtual cursor in the tissue image; and in response to the fourth operation gesture of the user captured by the image acquisition device, save the marked points selected by the virtual cursor in the tissue image.

6. The ranging system according to claim 5, characterized in that, The ranging device is further configured to, in response to a fifth operation gesture of the user captured by the image acquisition device, continuously mark the marked points selected by the virtual cursor in the tissue image, and based on the result of the continuous marking, obtain the curve distance of the target tissue between the marked points.

7. The ranging system according to claim 1, characterized in that, The ranging device includes: an image acquisition module and a ranging module; The image acquisition module is configured to, in response to a captured command received, capture a target tissue by using the binocular lens of the endoscope, obtain a pair of tissue images of the target tissue, and send the pair of tissue images to the ranging module; the pair of tissue images are two tissue images obtained by simultaneously photographing the target tissue by using the binocular lens of the endoscope. The ranging module is configured to, in response to a ranging start command received, generate the captured command and send the captured command to the image acquisition module. The ranging module is further configured to, based on the gesture information captured by the image acquisition device, control the virtual cursor to move and mark target positions in the target tissue image, and obtain each marked point of the target tissue image; the target tissue image is any one of the pair of tissue images. The ranging module is further configured to lock the marked points selected by the virtual cursor, and calculate the three-dimensional distance of the target tissue between the marked points.

8. The ranging system according to claim 7, characterized in that, The ranging module further includes a parallax calculation unit, a depth value calculation unit, a coordinate calculation unit, and a distance calculation unit. The parallax calculation unit is configured to, based on the image coordinates of the marked points in the target tissue image and the image coordinates of each of the marked points in the first tissue image, determine the parallax of each of the marked points between the target tissue image and the first tissue image. The first tissue image is the other image in the pair of tissue images excluding the target tissue image. The depth value calculation unit is configured to, based on the parallax of each of the marked points between the target tissue image and the first tissue image, determine the depth value of each of the marked points in the target tissue image. The coordinate calculation unit is configured to, based on the image coordinates of each of the marked points in the target tissue image, the parameters of the binocular lens of the endoscope, and the depth value of each of the marked points in the target tissue image, determine the world coordinates of each of the marked points. The distance calculation unit is configured to, based on the world coordinates of each of the marked points, calculate the three-dimensional distance of the target tissue between the marked points.

9. The ranging system according to claim 8, characterized in that, The ranging module is further configured to calculate the three-dimensional shape and size of the target tissue based on the world coordinates of multiple marked points.

10. The ranging system according to claim 7, wherein The triggering manner of the ranging start command includes at least one of a handle button, a host button, a host touch screen button, a foot switch, a voice instruction, and a gesture action.

11. The ranging system according to claim 1, wherein It further includes a user configuration module. The user configuration module is configured to bind the mapping relationship between the operation gesture and the cursor operation command; the cursor operation command includes one of a ranging function on, a ranging function off, a mark, a delete, a save, and a continuous mark command.

12. An endoscope, characterized in that, Integrated with the ranging system according to any one of claims 1 to 11.

Images