Instrument adjustment method and device, electronic equipment and storage medium

Abstract

The invention discloses an instrument adjusting method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a target instrument image of a to-be-adjusted target instrument; inputting the target instrument image into an image recognition model for recognizing the position of an instrument forceps head to obtain forceps head bounding box position information and recognition confidence of the target instrument; under the condition that the recognition confidence is higher than a preset confidence threshold value, on the basis of the forceps head bounding box position information, determining forceps head center point position information of the target instrument; under the condition that the position information of the center point of the tong head is not in the view range, orientation adjustment information is determined based on the position information of the center point of the tong head, the position information of the center point of the view range and the position information of the view frame; the direction adjustment information is sent to the target terminal, the endoscope is adjusted according to the direction adjustment information, when the tong head of the target instrument is located in the view field, the tong head of the target instrument is adjusted, and the accuracy, stability and safety of instrument adjustment are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an instrument adjustment method, device, electronic equipment and storage medium. Background Art

[0002] Surgical robots are widely used in the field of surgery. As the only medium for obtaining surgical area information during laparoscopic robotic surgery, the endoscope has a very limited field of view. Since the surgical instruments may exceed the field of view of the endoscope during the operation, the position of the surgical instruments is usually adjusted to make them within the field of view of the endoscope.

[0003] In the prior art, the position of surgical instruments is mostly adjusted based on the surgical experience of relevant personnel. However, it is easy to make mistakes in judging the position of surgical instruments based on human experience, which may cause damage to human tissues during the adjustment of surgical instruments. Summary of the invention

[0004] The present invention provides an instrument adjustment method, device, electronic equipment and storage medium, which realize real-time monitoring of the position information of the surgical instrument and improve the accuracy, stability and safety of the surgical instrument adjustment.

[0005] According to one aspect of the present invention, there is provided a method for adjusting an instrument, the method comprising:

[0006] Acquire a target device image acquired for the target device to be adjusted;

[0007] Input the target instrument image into a pre-trained image recognition model to identify the position of the instrument clamp head, and obtain the clamp head bounding box position information of the target instrument and the recognition confidence corresponding to the clamp head bounding box position information;

[0008] When the recognition confidence is higher than a preset confidence threshold, the position information of the center point of the clamp head of the target instrument in the two-dimensional coordinate system is determined based on the position information of the clamp head bounding box, wherein the two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin;

[0009] When the center point position information of the clamp head is not within the field of view, the position adjustment information of the target device is determined based on the center point position information of the clamp head, the center point position information of the field of view, and the field of view frame position information corresponding to the field of view;

[0010] The orientation adjustment information is sent to the target terminal to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp head of the target instrument is within the field of view, the clamp head of the target instrument is adjusted according to the orientation adjustment information.

[0011] According to another aspect of the present invention, there is provided an instrument adjustment device, the device comprising:

[0012] An image acquisition module, used for acquiring a target device image collected for the target device to be adjusted;

[0013] An image recognition module is used to input the target instrument image into a pre-trained image recognition model to perform instrument clamp head position recognition, and obtain the clamp head bounding box position information of the target instrument and the recognition confidence corresponding to the clamp head bounding box position information;

[0014] A clamp head position determination module, used to determine the clamp head center point position information of the target instrument in a two-dimensional coordinate system based on the clamp head bounding box position information when the recognition confidence is higher than a preset confidence threshold, wherein the two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin;

[0015] An adjustment information determination module, for determining the orientation adjustment information of the target device based on the position information of the center point of the clamp head, the position information of the center point of the visual field, and the position information of the visual field frame corresponding to the visual field when the position information of the center point of the clamp head is not within the visual field;

[0016] The instrument adjustment module is used to send the orientation adjustment information to the target terminal to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp head of the target instrument is within the field of view, adjust the clamp head of the target instrument according to the orientation adjustment information.

[0017] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0018] at least one processor; and

[0019] a memory communicatively connected to at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the instrument adjustment method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the instrument adjustment method of any embodiment of the present invention when executed by a processor.

[0022] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements an instrument adjustment method according to any embodiment of the present invention.

[0023] The technical solution of the embodiment of the present invention obtains the target instrument image collected for the target instrument to be adjusted. The target instrument image is input into the pre-trained image recognition model to identify the position of the instrument clamp head, and the clamp head bounding box position information of the target instrument and the recognition confidence corresponding to the clamp head bounding box position information are obtained, so as to realize the real-time detection of the position of the target instrument, and provide data support for the subsequent determination of whether the target instrument needs to be adjusted. When the recognition confidence is higher than the preset confidence threshold, the clamp head center point position information of the target instrument in the two-dimensional coordinate system is determined based on the clamp head bounding box position information, so as to judge whether the target instrument is out of the field of view based on the clamp head center point position information. When the clamp head center point position information is not within the field of view, the orientation adjustment information of the target instrument is accurately determined based on the clamp head center point position information, the center point position information of the field of view range, and the field of view frame position information corresponding to the field of view range, so as to provide a basis for the adjustment of the target instrument. The orientation adjustment information is sent to the target terminal, so as to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp head of the target instrument is within the field of view, the clamp head of the target instrument is adjusted according to the orientation adjustment information. The present invention solves the problem of misjudgment and tissue loss caused by relying on manual experience to adjust the position of surgical instruments in the prior art. By determining the position information of the instrument in real time to determine whether the target instrument is out of the field of view, the accuracy, stability and safety of surgical instrument adjustment are improved.

[0024] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a flow chart of an instrument adjustment method provided by an embodiment of the present invention;

[0027] Figure 2 is a flow chart of an instrument adjustment method provided by an embodiment of the present invention;

[0028] Figure 3 is a structural schematic diagram of an instrument adjustment device provided by an embodiment of the present invention;

[0029] Figure 4It is a schematic diagram of the structure of an electronic device for implementing the device adjustment method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Embodiment 1

[0033] Figure 1 This is a flow chart of an instrument adjustment method provided by the first embodiment of the present invention. This embodiment can be applied to the situation of determining whether a surgical instrument is out of the field of view and prompting and adjusting the surgical instrument that is out of the field of view. The method can be executed by an instrument adjustment device, which can be implemented in the form of hardware and / or software. The instrument adjustment device can be configured in electronic devices such as mobile phones, computers or servers. Figure 1 As shown, the method includes:

[0034] S110, acquiring a target device image collected for the target device to be adjusted.

[0035] The target instrument to be adjusted may be a surgical instrument that currently needs to be adjusted. Since there may be one or more surgical instruments that currently need to be adjusted, the currently selected surgical instrument is used as the target instrument to be adjusted. Accordingly, the image obtained by capturing the image of the target instrument is the target instrument image.

[0036] In actual application, surgical instruments may be blocked by tissue or exceed the corresponding field of view of the endoscope. Based on this, the currently selected surgical instrument is used as the target instrument to be adjusted. The target instrument to be adjusted is imaged and processed to obtain the target instrument image, so as to determine the position information of the target instrument to be adjusted based on the target instrument image.

[0037] In an embodiment of the present invention, before performing image recognition based on a pre-trained image recognition model, the image recognition model needs to be trained. The specific training process is: obtaining training sample data, wherein the training sample data includes a plurality of training sample images and the clamp head label information of the sample instrument in each training sample image, wherein the clamp head label information of the sample instrument includes the sample instrument category information, the clamp head bounding box position information of the sample instrument and the recognition confidence; inputting the training sample data into the image recognition model to be trained to obtain the actual output result; determining the loss value based on the clamp head label information and the actual output result, so as to adjust the model parameters of the image recognition model to be trained based on the loss value, and obtain a trained image recognition model.

[0038] The training sample data includes a plurality of training sample images and the label information of the clamp head of the sample instrument in each training sample image. The training sample image may be an image or video frame of the clamp head of the sample instrument collected in advance. Optionally, in order to improve the recognition accuracy of the surgical instrument, the images of the clamp head of the sample instrument under different instrument application scenarios, different viewing angles and different lighting conditions, i.e., the training sample images, may be collected. One or more sample instrument clamp heads may be included in the training sample image. The sample instrument may be understood as a surgical instrument as a sample. The clamp head label information may be label information obtained by labeling the clamp head of the sample instrument in the training sample image using a labeling tool. The clamp head label information includes sample instrument category information, sample instrument clamp head bounding box position information and recognition confidence. The sample instrument category information is the type corresponding to the sample instrument. The clamp head bounding box position information may be understood as the position information of the bounding box corresponding to the clamp head of the sample instrument. The recognition confidence of the clamp head label information may be understood as the degree of confidence in the accurate recognition and interpretation of the sample instrument clamp head. Optionally, the recognition confidence of the clamp head label information may be 100%.

[0039] The model parameters in the image recognition model to be trained are default values. The model parameters in the image recognition model to be trained are corrected by the training sample data to obtain the image recognition model. Optionally, the image recognition model to be trained can be a model determined based on the YOLO (You Only Look Once) algorithm. It should be noted that the output layer of the image recognition model to be trained can be configured according to the number of categories of sample instruments in the training sample data. The actual output result includes the actual instrument category, the actual clamp head bounding box position information and the actual recognition confidence. The actual instrument category is the category output after the training sample image is input into the image recognition model to be trained. The actual clamp head bounding box position information is the position information of the clamp head bounding box corresponding to the sample instrument output after the training sample image is input into the image recognition model to be trained. The actual recognition confidence is the credibility of the instrument clamp head output after the training sample image is input into the image recognition model to be trained. The loss value is the difference between the actual output result and the clamp head label information.

[0040] Specifically, in different instrument clamp application scenarios, different types of sample instrument clamps are imaged under different viewing angles and different lighting conditions to obtain training sample images to ensure that the image recognition model can accurately detect in different environments. The sample instrument clamps in each training sample image are annotated using an annotation tool to obtain a training sample image containing the clamp label information of the sample instrument. According to the preset image size, the size of all training sample images containing the clamp label information of the sample instrument is adjusted to facilitate the recognition processing of the image recognition model. At the same time, the pixel values ​​of the training sample images containing the clamp label information of the sample instrument are normalized to obtain training sample data.

[0041] Input the training sample data into the image recognition model to be trained to obtain the actual output result. According to the loss function, the sample instrument category information in the clamp head label information of the actual instrument category in the actual output result is subjected to loss processing to obtain the category loss value. According to the loss function, the actual clamp head bounding box position information and the clamp head bounding box position information of the sample instrument are subjected to loss processing to obtain the position loss value. According to the loss function, the actual recognition confidence and the recognition confidence of the sample instrument are subjected to loss processing to determine the confidence loss value. The model parameters of the image recognition model to be trained are adjusted according to the category loss value, the position loss value and the confidence loss value. When the model parameters in the image recognition model to be trained are corrected using the category loss value, the position loss value and the confidence loss value, the convergence of the loss function can be used as a training target, such as whether the training error is less than the preset error, or whether the error change tends to be stable, or whether the current number of iterations is equal to the preset number. If the detection reaches the convergence condition, such as the training error of the loss function is less than the preset error, or the error change trend tends to be stable, it indicates that the training of the image recognition model to be trained is completed, and the iterative training can be stopped at this time. If it is detected that the convergence condition is not reached at present, other training samples can be further obtained to continue training the image recognition model to be trained until the training error of the loss function is within the preset range. When the training error of the loss function reaches convergence, the trained image recognition model is obtained.

[0042] It should be noted that in order to eliminate duplicate actual output results, the non-maximum suppression technology is applied to select the highest recognition confidence corresponding to the clamp head bounding box of the sample instrument as the actual recognition confidence. The position information of the clamp head bounding box corresponding to the highest recognition confidence is used as the actual position information of the clamp head bounding box. At the same time, other clamp head bounding boxes with high overlap are suppressed to improve the accuracy and precision of image processing of the image recognition model.

[0043] S120, inputting the target instrument image into a pre-trained image recognition model to perform instrument clamp head position recognition, and obtaining clamp head bounding box position information of the target instrument and recognition confidence corresponding to the clamp head bounding box position information.

[0044] The pre-trained image recognition model is used to identify the clamp head bounding box position information and recognition confidence of the target instrument image. The clamp head bounding box position information may be the position information of the bounding box of the instrument clamp head of the target instrument. The recognition confidence corresponding to the clamp head bounding box position information may be understood as the degree of confidence in the accurate recognition and interpretation of the target instrument clamp head.

[0045] Specifically, the target instrument image is input into the pre-trained image recognition model to identify the position of the instrument clamp head, and obtain the clamp head bounding box position information corresponding to the target instrument and the recognition confidence corresponding to the clamp head bounding box information. Based on this, the real-time detection of the position of the target instrument clamp head is achieved, providing data support for the subsequent determination of whether the clamp head of the target instrument is out of the field of view.

[0046] S130. When the recognition confidence is higher than a preset confidence threshold, determine the position information of the center point of the clamp head of the target instrument in a two-dimensional coordinate system based on the position information of the clamp head bounding box, wherein the two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin.

[0047] The preset confidence threshold can be understood as a pre-set standard value for identification confidence. The clamp head center point position information can be understood as the coordinate information of the center point of the clamp head boundary box of the target instrument in the two-dimensional coordinate system. The endoscope is a device used to obtain surgical area information. The two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin.

[0048] Specifically, when the recognition confidence is higher than a preset confidence threshold, that is, when the instrument clamp of the target instrument is accurately identified, the clamp center point position information of the target instrument in the two-dimensional coordinate system is determined according to the clamp bounding box position information, so as to determine whether the instrument clamp of the target instrument is within the corresponding field of view of the endoscope based on the clamp center point position information.

[0049] S140. When the center point position information of the clamp head is not within the field of view, determine the orientation adjustment information of the target device based on the center point position information of the clamp head, the center point position information of the field of view, and the field of view frame position information corresponding to the field of view.

[0050] The field of view is the effective field of view corresponding to the endoscope. The center point position information of the field of view is the origin coordinate information of the two-dimensional coordinate system. The field of view frame position information can be the coordinate information of the field of view frame corresponding to the endoscope. The orientation adjustment information can be information for prompting the target device to adjust its orientation. The orientation adjustment information includes the orientation information of the clamp head of the target device relative to the center point of the field of view.

[0051] Specifically, the field of view corresponding to the endoscope is determined according to the field of view resolution corresponding to the endoscope. According to the position information of the center point of the clamp head of the target instrument, it is determined whether the position information of the center point of the clamp head is within the field of view corresponding to the endoscope. If the position information of the center point of the clamp head is not within the field of view, the center point of the clamp head and the center point of the field of view are connected according to the position information of the center point of the clamp head and the position information of the center point of the field of view, so as to determine the orientation information of the instrument clamp head of the target instrument relative to the center point of the field of view according to the relationship between the line segment and the field of view frame position information corresponding to the field of view. The orientation adjustment information of the target instrument is generated according to the orientation information, so as to adjust the clamp head of the target instrument based on the orientation adjustment information.

[0052] For example, a two-dimensional coordinate system is established with the center of the field of view of the endoscope as the origin. The position information of the center point of the clamp head of the target instrument in the two-dimensional coordinate system is determined as (x obj ,y obj ). According to the field of view resolution of the endoscope, the length value corresponding to the field of view of the endoscope can be determined as W view , the width value is H view Then, the field of view corresponding to the endoscope can be:

[0053]

[0054] In (x obj ,y obj ) does not meet the conditions corresponding to the above-mentioned field of view, it means that the clamp head center point position information is not within the field of view. obj ,y obj ), the center point position information (0,0) of the field of view, and the field of view border position information corresponding to the field of view as well as Determine the orientation adjustment information corresponding to the target device.

[0055] In an embodiment of the present invention, the method for determining the orientation adjustment information of the target device can be: determining the line segment between the center point of the clamp head and the center point of the field of view based on the position information of the center point of the clamp head and the center point of the field of view, and determining the intersection position information between the line segment and the field of view border based on the field of view border position information corresponding to the line segment and the field of view; based on the intersection position information, determining the orientation adjustment information of the target device.

[0056] The intersection position information may be understood as the intersection coordinate information between the line segment between the center point of the clamp head and the center point of the field of view, and the field of view border.

[0057] Specifically, according to the position information of the center point of the clamp head and the position information of the center point of the field of view, the center point of the clamp head and the center point of the field of view are connected to obtain a line segment between the center point of the clamp head and the center point of the field of view. According to the position information of the field of view frame corresponding to the field of view, the intersection position information between the line segment and the field of view frame is determined. According to the intersection position information, the orientation information of the instrument clamp head of the target instrument relative to the center point of the field of view can be determined. According to the orientation information, the orientation adjustment information of the target instrument is generated.

[0058] In an embodiment of the present invention, a method for determining the intersection position information between a line segment and a field of view border may be: based on the position information of the center point of the clamp head and the position information of the center point of the field of view range, determining the instrument orientation function corresponding to the target instrument; based on the position information of the field of view border corresponding to the field of view range, determining the field of view border function; based on the instrument orientation function and the field of view border function, determining the intersection coordinates between the line segment and the field of view border, and using the intersection coordinates as the intersection position information between the line segment and the field of view border.

[0059] The instrument orientation function may be a function corresponding to the center point of the clamp head and the center point of the field of view. The field of view frame function may be a function corresponding to the field of view frame of the endoscope. The intersection coordinates are the coordinate data of the intersection between the line segment and the field of view frame in a two-dimensional coordinate system.

[0060] Specifically, according to the position information of the center point of the clamp head and the position information of the center point of the field of view, the instrument orientation function corresponding to the line segment between the center point of the clamp head and the center point of the field of view is determined. According to the field of view frame position information corresponding to the field of view, the field of view frame function is determined. The instrument orientation function and the field of view frame function are calculated jointly to obtain the intersection coordinates between the line segment and the field of view frame. The intersection coordinates are used as the intersection position information between the line segment and the field of view frame, so as to determine the orientation adjustment information of the target instrument based on the intersection position information.

[0061] Exemplarily, in combination with the above example, according to the clamp head center point position information (x obj ,y obj ) and the center point position information of the field of view (0,0), determine the device orientation function corresponding to the target device Determine the field of view border function based on the field of view border position information as well as The instrument orientation function and the field of view border function are processed together to determine the obj ),max(0,x obj )) and y∈(min(0,y obj ),max(0,y objThe intersection coordinates are used as the intersection position information between the line segment and the field of view frame, so as to determine that the target instrument is not within the field of view of the endoscope according to the intersection coordinates, and determine the orientation information of the target instrument clamp head relative to the center of the field of view of the endoscope, so as to adjust the target instrument according to the orientation information.

[0062] S150, sending the orientation adjustment information to the target terminal to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp head of the target instrument is within the field of view, adjusting the clamp head of the target instrument according to the orientation adjustment information.

[0063] The target terminal may be a terminal device corresponding to the operator of the target device. Optionally, the target terminal may be a device for displaying the orientation adjustment information.

[0064] Specifically, the orientation adjustment information is sent to the target terminal of the target instrument operator, so that the target instrument operator first adjusts the endoscope according to the orientation adjustment information displayed on the target terminal, so that the clamp head of the target instrument is within the corresponding field of view of the endoscope. When the clamp head of the target instrument is within the corresponding field of view of the endoscope, the endoscope and the clamp head of the target instrument are adjusted together to the original position of the clamp head of the target instrument according to actual needs and under the condition that the clamp head of the target instrument meets the safety adjustment requirements, the orientation adjustment information is used to adjust the clamp head of the endoscope and the target instrument to the original position of the clamp head of the target instrument. By adjusting the endoscope first, the safety of the clamp head of the subsequent target instrument during movement can be guaranteed.

[0065] The technical solution of this embodiment is to obtain the target instrument image collected for the target instrument to be adjusted. The target instrument image is input into the pre-trained image recognition model to identify the position of the instrument clamp head, and the clamp head bounding box position information of the target instrument and the recognition confidence corresponding to the clamp head bounding box position information are obtained, so as to realize the real-time detection of the position of the target instrument, and provide data support for the subsequent determination of whether the target instrument needs to be adjusted. When the recognition confidence is higher than the preset confidence threshold, the clamp head center point position information of the target instrument in the two-dimensional coordinate system is determined based on the clamp head bounding box position information, so as to judge whether the target instrument is out of the field of view based on the clamp head center point position information. When the clamp head center point position information is not within the field of view, the orientation adjustment information of the target instrument is accurately determined based on the clamp head center point position information, the center point position information of the field of view range, and the field of view frame position information corresponding to the field of view range, so as to provide a basis for the adjustment of the target instrument. The orientation adjustment information is sent to the target terminal, so as to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp head of the target instrument is within the field of view, the clamp head of the target instrument is adjusted according to the orientation adjustment information. The present invention solves the problem of misjudgment and tissue loss caused by relying on manual experience to adjust the position of surgical instruments in the prior art. By determining the position information of the instrument in real time to determine whether the target instrument is out of the field of view, the accuracy, stability and safety of surgical instrument adjustment are improved.

[0066] Embodiment 2

[0067] Figure 2 This is a flowchart of a device adjustment method provided by the second embodiment of the present invention. This embodiment is based on the above embodiment and adjusts the target device when the recognition confidence is lower than the preset confidence threshold. The specific implementation method can refer to the technical solution of this embodiment. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here. Figure 2 As shown, the method includes:

[0068] S210. When the recognition confidence is lower than a preset confidence threshold, based on the field of view reference information corresponding to the endoscope and the position information of the center point of the clamp head of the target instrument in the three-dimensional coordinate system, detect whether the target instrument is within the field of view of the endoscope; wherein the three-dimensional coordinate system is determined based on the two-dimensional coordinate system and the preset depth.

[0069] The field of view reference information can be understood as the field of view position information corresponding to the endoscope. The position information of the center point of the clamp head of the target instrument in the three-dimensional coordinate system can be understood as the three-dimensional coordinate data corresponding to the center point of the clamp head of the target instrument in the three-dimensional coordinate system. The preset depth can be a depth pre-set according to actual needs.

[0070] Specifically, during the image recognition process of the target device image, there may be problems such as the target device being blocked or affected by reflections, resulting in the recognition confidence being lower than the preset confidence threshold. At this time, in order to accurately determine whether the target device is within the field of view corresponding to the endoscope, the center point position information of the target device's clamp head in the three-dimensional coordinate system can be determined based on the center point position information of the target device's clamp head in the two-dimensional coordinate system and the vertical distance between the center point of the target device's clamp head and the endoscope. According to the field of view reference information corresponding to the endoscope and the center point position information of the target device's clamp head in the three-dimensional coordinate system, it is detected whether the target device is within the field of view of the endoscope, so as to adjust the clamp head of the target device according to the detection result.

[0071] Optionally, the field of view reference information may be a field of view normal vector. The field of view reference information may be determined by: for multiple field of view corner points corresponding to the endoscope, when the instrument reference point of the reference instrument coincides with the current field of view corner point, obtaining multiple reference position information corresponding to the current field of view corner point in the three-dimensional coordinate system; based on the multiple reference position information, determining the field of view function corresponding to the current field of view corner point and the field of view direction vector corresponding to the field of view function; based on the field of view direction vector corresponding to each field of view corner point, determining the field of view normal vector of the field of view plane corresponding to each two adjacent field of view direction vectors, and using the field of view normal vector as the field of view reference information corresponding to the endoscope.

[0072] Among them, the reference instrument can be a surgical instrument that is pre-set and used to determine the field of view reference information. The instrument reference point can be a point on the instrument yaw axis of the reference instrument that is set according to actual needs. The field of view corner points can be understood as the four corner points of the field of view corresponding to the endoscope. For example, the field of view corresponding to the endoscope is a rectangle, and the field of view corner points are the four vertices of the matrix. The reference position information can be the three-dimensional coordinate data corresponding to the instrument yaw axis. The field of view function can be a linear function established by multiple field of view corner points. For each corner point, there is a corresponding field of view function. The field of view direction vector can be a direction vector corresponding to the field of view function. The field of view plane can be a plane formed by two adjacent field of view direction vectors. The field of view normal vector is a direction vector perpendicular to the field of view plane. Among them, the direction corresponding to the field of view normal vector points to the inside of the field of view corresponding to the endoscope.

[0073] Specifically, there are multiple field of view corner points in the field of view corresponding to the endoscope. For each field of view corner point, when the instrument reference point of the reference instrument coincides with the current field of view corner point, obtain multiple reference position information corresponding to the current field of view corner point in the three-dimensional coordinate system. According to the multiple reference position information corresponding to the current field of view corner point, determine the field of view function corresponding to the current field of view corner point. And determine the field of view direction vector of the field of view function. Based on this, the field of view direction vector corresponding to each field of view corner point can be obtained. For multiple field of view direction vectors, determine the field of view plane corresponding to every two adjacent field of view direction vectors, and determine the field of view normal vector of the field of view plane. Use the field of view normal vector as the field of view reference information corresponding to the endoscope to detect whether the target instrument is within the field of view of the endoscope based on the field of view normal vector.

[0074] Exemplarily, taking the rectangular field of view corresponding to the endoscope as an example, the rectangular field of view corresponding to the reference instrument and the endoscope can constitute a virtual field of view in the shape of a tetrahedron. For the four field of view corner points corresponding to the endoscope, the instrument reference point on the instrument yaw axis of the reference instrument is overlapped with each field of view corner point, and the 15 three-dimensional coordinate data corresponding to each field of view corner point are collected by adjusting the depth of field, for a total of 60 three-dimensional coordinate data. According to the 15 three-dimensional coordinate data corresponding to the current field of view corner point, the field of view function corresponding to an edge line is fitted using the binary linear regression method. Based on this, the field of view function corresponding to the four edges of the virtual field of view corresponding to the endoscope can be obtained. According to the field of view function corresponding to the four edges, the field of view direction vectors corresponding to the four edges are determined. Since any two edges can constitute a field of view plane. Based on this, the field of view normal vector corresponding to each field of view plane can be obtained, that is, four field of view normal vectors are obtained. The direction of the field of view normal vector points to the inside of the virtual field of view in the shape of a tetrahedron.

[0075] In an embodiment of the present invention, a method for detecting whether a target device is within the field of view of an endoscope may be: performing a dot multiplication process on the clamp center point position information of the target device and each field of view normal vector in the field of view reference information to obtain a dot multiplication result corresponding to each field of view normal vector; performing a multiplication process on multiple dot multiplication results to obtain a multiplication result of the target device; if the multiplication result is greater than a preset value, it is determined that the target device is within the field of view corresponding to the endoscope; if the multiplication result is less than a preset value, it is determined that the target device is not within the field of view corresponding to the endoscope.

[0076] The dot product result may be a calculation result obtained by multiplying the position information of the center point of the clamp head with the normal vector of the field of view. The multiplication result is a calculation result obtained by multiplying multiple dot product results. The preset value may be zero.

[0077] Specifically, the center point position information of the clamp head in the three-dimensional coordinate system is multiplied with each field of view normal vector in the field of view reference information to obtain the point product result corresponding to each field of view normal vector, that is, to obtain multiple point product results. The multiple point product results are multiplied to obtain the multiplication result. If the multiplication result is greater than zero, it means that the clamp head of the target instrument is within the field of view corresponding to the endoscope. If the multiplication result is less than zero, it means that the clamp head of the target instrument is not within the field of view corresponding to the endoscope.

[0078] Exemplarily, in combination with the above example, taking the position information of the center point of the clamp head of the target instrument as (x, y, z) as an example, perform dot multiplication of (x, y, z) and the four field of view normal vectors respectively to obtain four dot product results. Multiply the four dot product results to obtain the multiplication result. If the multiplication result is a positive number, it means that the center point of the clamp head of the target instrument is in the virtual field of view of the four-sided pyramid shape, that is, the clamp head of the target instrument is in the field of view corresponding to the endoscope. Correspondingly, if the multiplication result is a negative number, the clamp head of the target instrument is not in the field of view corresponding to the endoscope.

[0079] S220, displaying the visual field detection result on the target terminal, so as to adjust the endoscope according to the visual field detection result displayed on the target terminal, and when the clamp head of the target instrument is within the visual field, adjusting the clamp head of the target instrument according to the visual field detection result.

[0080] The visual field detection result may be a result used to characterize whether the clamp head of the target instrument is within the visual field corresponding to the endoscope. That is, the visual field detection result may be that the clamp head of the target instrument is within the visual field corresponding to the endoscope, or that the clamp head of the target instrument is not within the visual field corresponding to the endoscope.

[0081] Specifically, it is determined whether the clamp head of the target instrument is within the field of view corresponding to the endoscope based on the field of view detection result. If the clamp head of the target instrument is not within the field of view corresponding to the endoscope, the field of view detection result is sent to the target terminal, so that the surgical instrument operator adjusts the endoscope based on the field of view detection result displayed by the target terminal, so that the clamp head of the target instrument is displayed within the field of view of the endoscope. And when the clamp head of the target instrument is displayed within the field of view of the endoscope, the clamp head of the target instrument and the endoscope are adjusted together according to actual needs and when the clamp head of the target instrument meets the safety adjustment conditions, so as to adjust the endoscope and the clamp head of the target instrument to the original position of the clamp head of the target instrument.

[0082] The technical solution of this embodiment detects whether the target instrument is within the field of view of the endoscope based on the field of view reference information corresponding to the endoscope and the position information of the center point of the clamp head of the target instrument in the three-dimensional coordinate system when the recognition confidence is lower than the preset confidence threshold, thereby improving the accuracy and comprehensiveness of the detection of the target instrument out of the field of view. The field of view detection result is displayed on the target terminal, and the endoscope is adjusted according to the field of view detection result displayed on the target terminal, so that when the clamp head of the target instrument is within the field of view, the clamp head of the target instrument is adjusted according to the field of view detection result, thereby realizing the adjustment of the instrument clamp head and improving the comprehensiveness, accuracy and safety of the adjustment of the surgical instrument.

[0083] Embodiment 3

[0084] Figure 3 Schematic diagram of the structure of an instrument adjustment device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an image acquisition module 310 , an image recognition module 320 , a clamp position determination module 330 , an adjustment information determination module 340 and an instrument adjustment module 350 .

[0085] The image acquisition module 310 is used to acquire the target instrument image collected for the target instrument to be adjusted; the image recognition module 320 is used to input the target instrument image into a pre-trained image recognition model to perform instrument clamp head position recognition, and obtain the clamp head bounding box position information of the target instrument and the recognition confidence corresponding to the clamp head bounding box position information; the clamp head position determination module 330 is used to determine the clamp head center point position information of the target instrument in a two-dimensional coordinate system based on the clamp head bounding box position information when the recognition confidence is higher than a preset confidence threshold, wherein the two-dimensional coordinate system is based on the endoscope. A coordinate system with the center point of the corresponding field of view as the origin; an adjustment information determination module 340, for determining the orientation adjustment information of the target instrument based on the center point position information of the clamp head, the center point position information of the field of view and the field of view frame position information corresponding to the field of view when the center point position information of the clamp head is not within the field of view; an instrument adjustment module 350, for sending the orientation adjustment information to the target terminal, so as to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp head of the target instrument is within the field of view, adjust the clamp head of the target instrument according to the orientation adjustment information.

[0086] The technical solution of this embodiment is to obtain the target instrument image collected for the target instrument to be adjusted. The target instrument image is input into the pre-trained image recognition model to identify the position of the instrument clamp head, and the clamp head bounding box position information of the target instrument and the recognition confidence corresponding to the clamp head bounding box position information are obtained, so as to realize the real-time detection of the position of the target instrument, and provide data support for the subsequent determination of whether the target instrument needs to be adjusted. When the recognition confidence is higher than the preset confidence threshold, the clamp head center point position information of the target instrument in the two-dimensional coordinate system is determined based on the clamp head bounding box position information, so as to judge whether the target instrument is out of the field of view based on the clamp head center point position information. When the clamp head center point position information is not within the field of view, the orientation adjustment information of the target instrument is accurately determined based on the clamp head center point position information, the center point position information of the field of view range, and the field of view frame position information corresponding to the field of view range, so as to provide a basis for the adjustment of the target instrument. The orientation adjustment information is sent to the target terminal, so as to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp head of the target instrument is within the field of view, the clamp head of the target instrument is adjusted according to the orientation adjustment information. The present invention solves the problem of misjudgment and tissue loss caused by relying on manual experience to adjust the position of surgical instruments in the prior art. By determining the position information of the instrument in real time to determine whether the target instrument is out of the field of view, the accuracy, stability and safety of surgical instrument adjustment are improved.

[0087] On the basis of the above embodiment, optionally, the adjustment information determination module includes: an intersection position information determination unit, which is used to determine the line segment between the center point of the clamp head and the center point of the field of view according to the center point position information of the clamp head and the center point position information of the field of view, and determine the intersection position information between the line segment and the field of view border based on the field of view border position information corresponding to the line segment and the field of view; an orientation adjustment information determination unit, which is used to determine the orientation adjustment information of the target instrument based on the intersection position information.

[0088] Optionally, the intersection position information determination unit includes: an instrument orientation function determination subunit, which is used to determine the instrument orientation function corresponding to the target instrument based on the forceps center point position information and the field of view center point position information; a field of view border function determination subunit, which is used to determine the field of view border function based on the field of view border position information corresponding to the field of view range; an intersection position information determination subunit, which is used to determine the intersection coordinates between the line segment and the field of view border based on the instrument orientation function and the field of view border function, and use the intersection coordinates as the intersection position information between the line segment and the field of view border.

[0089] Optionally, the device also includes: an image recognition model training module, which is used to obtain training sample data, wherein the training sample data includes multiple training sample images and clamp head label information of a sample instrument in each training sample image, wherein the clamp head label information of the sample instrument includes sample instrument category information, clamp head bounding box position information of the sample instrument and recognition confidence; inputting the training sample data into the image recognition model to be trained to obtain an actual output result; determining a loss value based on the clamp head label information and the actual output result, and adjusting the model parameters of the image recognition model to be trained based on the loss value to obtain a trained image recognition model.

[0090] Optionally, the device also includes: an instrument field of view range detection module, which includes: an instrument position judgment unit, which is used to detect whether the target instrument is within the field of view of the endoscope based on the field of view reference information corresponding to the endoscope and the center point position information of the clamp head of the target instrument in the three-dimensional coordinate system when the recognition confidence is lower than a preset confidence threshold; wherein the three-dimensional coordinate system is determined based on the two-dimensional coordinate system and a preset depth; an instrument adjustment unit, which is used to display the field of view detection result on the target terminal, so as to adjust the endoscope according to the field of view detection result displayed by the target terminal, so as to adjust the clamp head of the target instrument according to the field of view detection result when the clamp head of the target instrument is within the field of view.

[0091] Optionally, the device also includes: a field of view reference information determination module, which is used to obtain multiple reference position information corresponding to the current field of view corner point in a three-dimensional coordinate system for multiple field of view corner points corresponding to the endoscope, when the instrument reference point of the reference instrument coincides with the current field of view corner point; based on the multiple reference position information, determine the field of view function corresponding to the current field of view corner point and the field of view direction vector corresponding to the field of view function; based on the field of view direction vector corresponding to each field of view corner point, determine the field of view normal vector of the field of view plane corresponding to every two adjacent field of view direction vectors, and use the field of view normal vector as the field of view reference information corresponding to the endoscope.

[0092] Optionally, an instrument position judgment unit is used to perform point multiplication processing on the clamp center point position information of the target instrument and each field of view normal vector in the field of view reference information to obtain the point product result corresponding to each field of view normal vector; multiply multiple point product results to obtain the multiplication result of the target instrument; if the multiplication result is greater than a preset value, it is determined that the target instrument is within the field of view corresponding to the endoscope; if the multiplication result is less than the preset value, it is determined that the target instrument is not within the field of view corresponding to the endoscope.

[0093] The instrument adjustment device provided in the embodiment of the present invention can execute the instrument adjustment method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0094] Embodiment 4

[0095] Figure 4 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0096] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0097] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0098] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the instrument adjustment method.

[0099] In some embodiments, the instrument adjustment method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the instrument adjustment method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the instrument adjustment method in any other suitable manner (e.g., by means of firmware).

[0100] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The computer programs for implementing the instrument adjustment methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0102] Embodiment 5

[0103] Embodiment 5 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a processor to execute an instrument adjustment method, the method comprising:

[0104] Acquire a target instrument image captured for the target instrument to be adjusted; input the target instrument image into a pre-trained image recognition model to perform instrument clamp position recognition, and obtain the clamp bounding box position information of the target instrument and the recognition confidence corresponding to the clamp bounding box position information; when the recognition confidence is higher than a preset confidence threshold, determine the clamp center point position information of the target instrument in a two-dimensional coordinate system based on the clamp bounding box position information, wherein the two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin; when the clamp center point position information is not within the field of view, determine the orientation adjustment information of the target instrument based on the clamp center point position information, the center point position information of the field of view, and the field of view frame position information corresponding to the field of view; send the orientation adjustment information to the target terminal, so as to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp of the target instrument is within the field of view, adjust the clamp of the target instrument according to the orientation adjustment information.

[0105] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0107] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0108] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0109] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0110] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for adjusting an instrument, characterized in that: include: Acquire a target device image acquired for the target device to be adjusted; Inputting the target instrument image into a pre-trained image recognition model to perform instrument clamp head position recognition, and obtaining clamp head bounding box position information of the target instrument and recognition confidence corresponding to the clamp head bounding box position information; In the case where the recognition confidence is higher than a preset confidence threshold, based on the position information of the clamp head bounding box, determining the position information of the center point of the clamp head of the target instrument in a two-dimensional coordinate system, wherein the two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin; In the case where the center point position information of the clamp head is not within the field of view, based on the center point position information of the clamp head, the center point position information of the field of view, and the field of view frame position information corresponding to the field of view, determining the orientation adjustment information of the target device; The orientation adjustment information is sent to the target terminal to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp head of the target instrument is within the field of view, the clamp head of the target instrument is adjusted according to the orientation adjustment information.

2. The method according to claim 1, characterized in that The determining the orientation adjustment information of the target device based on the center point position information of the clamp head, the center point position information of the field of view, and the field of view frame position information corresponding to the field of view includes: Determine a line segment between the center point of the clamp head and the center point of the field of view according to the position information of the center point of the clamp head and the center point of the field of view, and determine the position information of the intersection point between the line segment and the field of view frame based on the position information of the field of view frame corresponding to the line segment and the field of view; Based on the intersection position information, the orientation adjustment information of the target device is determined.

3. The method according to claim 2, characterized in that The method of determining a line segment between the center point of the clamp head and the center point of the field of view according to the center point position information of the clamp head and the center point position information of the field of view, and determining the intersection position information between the line segment and the field of view frame based on the field of view frame position information corresponding to the line segment and the field of view, comprises: Determining an instrument orientation function corresponding to the target instrument based on the center point position information of the clamp head and the center point position information of the field of view; Determining a field of view border function based on the field of view border position information corresponding to the field of view range; Based on the instrument orientation function and the visual field frame function, the coordinates of the intersection between the line segment and the visual field frame are determined, and the coordinates of the intersection are used as the position information of the intersection between the line segment and the visual field frame.

4. The method according to claim 1, characterized in that: The training process of the image recognition model includes: Acquire training sample data, wherein the training sample data includes a plurality of training sample images and clamp head label information of a sample instrument in each of the training sample images, wherein the clamp head label information of the sample instrument includes sample instrument category information, clamp head bounding box position information of the sample instrument, and recognition confidence; Inputting the training sample data into the image recognition model to be trained to obtain an actual output result; Based on the clamp head label information and the actual output result, a loss value is determined, so as to adjust the model parameters of the image recognition model to be trained based on the loss value to obtain a trained image recognition model.

5. The method according to claim 1, characterized in that Also includes: In the case where the recognition confidence is lower than a preset confidence threshold, based on the field of view reference information corresponding to the endoscope and the position information of the center point of the clamp head of the target instrument in the three-dimensional coordinate system, detecting whether the target instrument is within the field of view of the endoscope; wherein the three-dimensional coordinate system is determined based on the two-dimensional coordinate system and the preset depth; The visual field detection result is displayed on the target terminal, so that the endoscope is adjusted according to the visual field detection result displayed on the target terminal, and when the clamp head of the target instrument is within the visual field, the clamp head of the target instrument is adjusted according to the visual field detection result.

6. The method according to claim 5, characterized in that The method further comprises: For multiple visual field corner points corresponding to the endoscope, when the instrument reference point of the reference instrument coincides with the current visual field corner point, obtaining multiple reference position information corresponding to the current visual field corner point in the three-dimensional coordinate system; Based on the plurality of reference position information, determining a field of view function corresponding to the current field of view corner point and a field of view direction vector corresponding to the field of view function; Based on the field of view direction vector corresponding to each field of view corner point, the field of view normal vector of the field of view plane corresponding to every two adjacent field of view direction vectors is determined, and the field of view normal vector is used as the field of view reference information corresponding to the endoscope.

7. The method according to claim 5, characterized in that The detecting whether the target instrument is within the field of view of the endoscope based on the field of view reference information corresponding to the endoscope and the center point position information of the clamp head of the target instrument in the three-dimensional coordinate system includes: Performing a dot multiplication process on the clamp head center point position information of the target instrument and each field of view normal vector in the field of view reference information to obtain a dot multiplication result corresponding to each field of view normal vector; Multiplying a plurality of the point product results to obtain a multiplication result of the target device; If the multiplication result is greater than a preset value, it is determined that the target device is within the field of view corresponding to the endoscope; If the multiplication result is less than a preset value, it is determined that the target instrument is not within the field of view corresponding to the endoscope.

8. An instrument adjustment device, characterized in that: include: An image acquisition module, used for acquiring a target device image collected for the target device to be adjusted; An image recognition module is used to input the target instrument image into a pre-trained image recognition model to perform instrument clamp head position recognition, and obtain the clamp head bounding box position information of the target instrument and the recognition confidence corresponding to the clamp head bounding box position information; A clamp head position determination module, used to determine the clamp head center point position information of the target instrument in a two-dimensional coordinate system based on the clamp head bounding box position information when the recognition confidence is higher than a preset confidence threshold, wherein the two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin; an adjustment information determination module, configured to determine the orientation adjustment information of the target instrument based on the position information of the center point of the clamp head, the position information of the center point of the visual field, and the position information of the visual field frame corresponding to the visual field when the position information of the center point of the clamp head is not within the visual field; An instrument adjustment module is used to send the orientation adjustment information to the target terminal to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and when the clamp head of the target instrument is within the field of view, adjust the clamp head of the target instrument according to the orientation adjustment information.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, wherein the computer program is executed by the at least one processor so as to enable the at least one processor to perform the instrument adjustment method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the instrument adjustment method according to any one of claims 1 to 7 when executed.

Images