Instrument adjustment devices, electronic devices and storage media

Abstract

The application discloses an instrument adjusting device, an electronic device and a storage medium. The target instrument image of a target instrument to be adjusted is acquired; the target instrument image is input into an image recognition model for instrument jaw position recognition, so that jaw bounding box position information and recognition confidence of the target instrument are obtained; in the case that the recognition confidence is higher than a preset confidence threshold, the jaw center point position information of the target instrument is determined based on the jaw bounding box position information; in the case that the jaw center point position information is not in a field of view range, the orientation adjustment information is determined based on the jaw center point position information, the center point position information of the field of view range and the field of view bounding box position information; and the orientation adjustment information is sent to a target terminal, so that the endoscope is adjusted according to the orientation adjustment information, and the jaw of the target instrument is adjusted when the jaw of the target instrument is in the field of view range, thereby improving the accuracy, stability and safety of instrument adjustment.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an instrument adjustment device, electronic device, and storage medium. Background Technology

[0002] Surgical robots are widely used in the field of surgery. Endoscopes, as the only medium for obtaining information about the surgical area during laparoscopic robotic surgery, have a very limited field of view. Because surgical instruments may sometimes extend beyond the endoscope's field of view during surgery, their positions are usually adjusted to ensure they remain within the endoscope's view.

[0003] Current techniques often rely on the surgical experience of personnel to adjust the position of surgical instruments. However, relying solely on human experience to determine the position of surgical instruments is prone to errors, which can easily cause damage to human tissues during the adjustment process. Summary of the Invention

[0004] This invention provides an instrument adjustment device, electronic device, and storage medium, which enables real-time monitoring of the position information of surgical instruments, thereby improving the accuracy, stability, and safety of surgical instrument adjustment.

[0005] According to one aspect of the present invention, a device adjustment method is provided, the method comprising:

[0006] Acquire images of the target instrument to be adjusted;

[0007] The target instrument image is input into a pre-trained image recognition model to identify the position of the instrument's clamp head, thereby obtaining the position information of the clamp head bounding box and the corresponding recognition confidence level.

[0008] When the confidence level is higher than the preset confidence threshold, the position information of the center point of the target instrument's head is determined in a two-dimensional coordinate system based on the position information of the head bounding box. The two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin.

[0009] When the center point of the clamp head is not within the field of view, the orientation adjustment information of the target instrument is determined based on the center point of the clamp head, the center point of the field of view, and the position of the field of view boundary corresponding to the field of view.

[0010] The orientation adjustment information is sent to the target terminal, and the endoscope is adjusted based on the orientation adjustment information displayed on the target terminal. When the forceps of the target instrument are within the field of view, the forceps of the target instrument are adjusted according to the orientation adjustment information.

[0011] According to another aspect of the present invention, a medical device adjustment apparatus is provided, the apparatus comprising:

[0012] The image acquisition module is used to acquire images of the target instrument to be adjusted.

[0013] The image recognition module is used to input the target instrument image into a pre-trained image recognition model to identify the position of the instrument's clamp head, and obtain the position information of the clamp head bounding box of the target instrument and the recognition confidence level corresponding to the position information of the clamp head bounding box.

[0014] The clamp head position determination module is used to determine the position information of the center point of the clamp head of the target instrument in a two-dimensional coordinate system based on the clamp head bounding box position information when the identification confidence level is higher than a preset confidence threshold. The two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin.

[0015] The adjustment information determination module is used to determine the orientation adjustment information of the target instrument based on the position information of the center point of the clamp head, the position information of the center point of the field of view, and the position information of the field of view boundary corresponding to the field of view when the position information of the center point of the clamp head is not within the field of view.

[0016] The instrument adjustment module is used to send orientation adjustment information to the target terminal, so as to adjust the endoscope based on the orientation adjustment information displayed on the target terminal, and adjust the forceps of the target instrument according to the orientation adjustment information when the forceps of the target instrument are in the field of view.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory that is communicatively connected to at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the device adjustment method of any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided that stores computer instructions for causing a processor to execute and implement the instrument adjustment method of any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, characterized in that the computer program, when executed by a processor, implements the instrument adjustment method as described in any embodiment of the present invention.

[0023] The technical solution of this invention involves acquiring an image of the target instrument to be adjusted. This image is then input into a pre-trained image recognition model to identify the position of the instrument's clamp head. This yields the clamp head bounding box position information and the corresponding recognition confidence level, enabling real-time detection of the target instrument's position and providing data support for determining whether adjustment is necessary. If the recognition confidence level exceeds a preset confidence threshold, the clamp head bounding box position information is used to determine the clamp head center point position in a two-dimensional coordinate system, allowing for the determination of whether the target instrument is outside the field of view. If the clamp head center point position is not within the field of view, the orientation adjustment information of the target instrument is accurately determined based on the clamp head center point position information, the center point position information of the field of view, and the corresponding field of view border position information, providing a basis for adjustment. This orientation adjustment information is sent to the target terminal, allowing for endoscope adjustment based on the displayed orientation adjustment information. When the clamp head of the target instrument is within the field of view, the clamp head is adjusted according to the orientation adjustment information. This invention solves the problems of judgment errors and tissue damage caused by relying on human experience to adjust the position of surgical instruments in the prior art. By determining the position information of the instruments in real time, it can determine whether the target instrument is out of the field of view, thereby improving the accuracy, stability and safety of surgical instrument adjustment.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a device adjustment method provided in an embodiment of the present invention;

[0027] Figure 2 This is a flowchart of a device adjustment method provided in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of an instrument adjustment device provided in an embodiment of the present invention;

[0029] Figure 4This is a schematic diagram of the structure of an electronic device that implements the instrument adjustment method of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Example 1

[0033] Figure 1 This is a flowchart of an instrument adjustment method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where it is necessary to determine whether a surgical instrument is out of the field of view and to provide prompts and adjustments for surgical instruments that are out of the field of view. This method can be executed by an instrument adjustment device, which can be implemented in hardware and / or software. This instrument adjustment device can be configured in electronic devices such as mobile phones, computers, or servers. Figure 1 As shown, the method includes:

[0034] S110. Acquire the target instrument image for the target instrument to be adjusted.

[0035] The target instrument to be adjusted can be any surgical instrument that currently needs adjustment. Since there may be one or more surgical instruments that need adjustment, the currently selected surgical instrument is taken as the target instrument to be adjusted. Accordingly, the image obtained by acquiring images of the target instrument is the target instrument image.

[0036] In practical applications, surgical instruments may be obscured by tissue or extend beyond the endoscopic field of view. Therefore, the currently selected surgical instrument is designated as the target instrument to be adjusted. Image acquisition and processing are performed on the target instrument to obtain its image, which is then used to determine its position.

[0037] In this embodiment of the invention, before performing image recognition based on a pre-trained image recognition model, the image recognition model needs to be trained. The specific training process is as follows: Training sample data is acquired, including multiple training sample images and the pincer head label information of the sample instrument in each training sample image. The pincer head label information includes sample instrument category information, pincer head bounding box position information, and recognition confidence. The training sample data is input into the image recognition model to be trained to obtain the actual output result. Based on the pincer head label information and the actual output result, a loss value is determined, and the model parameters of the image recognition model to be trained are adjusted based on the loss value to obtain the trained image recognition model.

[0038] The training sample data includes multiple training sample images and the forceps head label information of the sample instruments in each training sample image. The training sample images can be pre-acquired images or video frames of sample instrument forceps heads. Optionally, to improve the recognition accuracy of surgical instruments, sample instrument forceps head images can be acquired under different instrument application scenarios, different viewpoints, and different lighting conditions; these are the training sample images. The training sample images may include one or more sample instrument forceps heads. A sample instrument can be understood as a surgical instrument used as a sample. The forceps head label information can be the label information obtained by annotating the forceps heads of the sample instruments in the training sample images using an annotation tool. The forceps head label information includes sample instrument category information, the bounding box position information of the sample instrument forceps head, and the recognition confidence level. The sample instrument category information refers to the type of sample instrument. The bounding box position information can be understood as the position information of the bounding box corresponding to the sample instrument forceps head. The recognition confidence level of the forceps head label information can be understood as the reliability of the accurate recognition and interpretation of the sample instrument forceps head. Optionally, the recognition confidence level of the forceps head label information can be 100%.

[0039] The model parameters in the image recognition model to be trained are default values. These parameters are corrected using training sample data to obtain the final image recognition model. Optionally, the image recognition model to be trained can be a model determined based on the YOLO (You Only Look Once) algorithm. It should be noted that the output layer of the image recognition model to be trained can be configured according to the number of sample instrument categories in the training sample data. The actual output results include the actual instrument category, the actual clamp head bounding box location information, and the actual recognition confidence score. The actual instrument category is the category output after inputting the training sample image into the image recognition model to be trained. The actual clamp head bounding box location information is the location information of the clamp head bounding box corresponding to the sample instrument, output after inputting the training sample image into the image recognition model to be trained. The actual recognition confidence score is the degree of confidence of the clamp head output after inputting the training sample image into the image recognition model to be trained. The loss value is the difference between the actual output result and the clamp head label information.

[0040] Specifically, images of different types of instrument pliers were acquired under various application scenarios, viewing angles, and lighting conditions to obtain training sample images, ensuring the image recognition model can accurately detect pliers in different environments. Annotation tools were used to label the instrument pliers in each training sample image, resulting in training sample images containing the pliers label information. The dimensions of all training sample images containing the instrument label information were adjusted according to a preset image size to facilitate recognition by the image recognition model. Simultaneously, the pixel values ​​of the training sample images containing the instrument label information were normalized to obtain training sample data.

[0041] Training sample data is input into the image recognition model to be trained to obtain the actual output results. Based on the loss function, loss processing is applied to the sample instrument category information from the actual instrument category pliers label information in the actual output results to obtain the category loss value. Loss processing is then applied to the actual pliers bounding box position information and the sample instrument's pliers bounding box position information to obtain the position loss value. Loss processing is then applied to the actual recognition confidence and the sample instrument recognition confidence to determine the confidence loss value. The model parameters of the image recognition model to be trained are adjusted based on the category loss value, position loss value, and confidence loss value. When correcting the model parameters in the image recognition model to be trained using the category loss value, position loss value, and confidence loss value, the convergence of the loss function can be used as a training objective. For example, whether the training error is less than a preset error, whether the error change tends to stabilize, or whether the current iteration number is equal to the preset number. If the convergence condition is met, such as the training error of the loss function being less than the preset error, or the error change trend tending to stabilize, it indicates that the image recognition model to be trained has completed training, and iterative training can be stopped at this point. If the convergence condition is not met, additional training samples can be obtained to continue training the image recognition model until the training error of the loss function is within a preset range. When the training error of the loss function converges, the trained image recognition model is obtained.

[0042] It should be noted that, in order to eliminate duplicate actual output results, non-maximum suppression is applied, and the highest recognition confidence score corresponding to the bounding box of the sample instrument's jaws is selected as the actual recognition confidence score. The jaw bounding box position information corresponding to the highest recognition confidence score is then used as the actual jaw bounding box position information. Simultaneously, other jaw bounding boxes with high overlap are suppressed to improve the accuracy and precision of the image recognition model's image processing.

[0043] S120. Input the target instrument image into a pre-trained image recognition model to identify the position of the instrument's clamp head, and obtain the position information of the clamp head bounding box of the target instrument and the recognition confidence level corresponding to the position information of the clamp head bounding box.

[0044] The pre-trained image recognition model is used to identify the bounding box location information of the pliers head in the target instrument image and to determine the recognition confidence level. The bounding box location information refers to the position of the bounding box of the pliers head of the target instrument. The recognition confidence level corresponding to the bounding box location information can be understood as the degree of reliability in accurately identifying and interpreting the pliers head of the target instrument.

[0045] Specifically, the image of the target instrument is input into a pre-trained image recognition model to identify the position of the instrument's jaws, obtaining the bounding box position information of the jaws corresponding to the target instrument and the recognition confidence level of the bounding box information. Based on this, real-time detection of the jaw position of the target instrument is achieved, providing data support for subsequent determination of whether the jaws of the target instrument have left the field of view.

[0046] S130. When the confidence level is higher than the preset confidence threshold, the position information of the center point of the target instrument's clamp head is determined in a two-dimensional coordinate system based on the clamp head bounding box position information. The two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin.

[0047] The pre-set confidence threshold can be understood as a pre-set standard value for identifying confidence. The forceps head center point location information can be understood as the coordinate information of the center point of the target instrument's forceps head bounding box in a two-dimensional coordinate system. An endoscope is a device used to acquire information about the surgical area. The two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as its origin.

[0048] Specifically, when the confidence level is higher than the preset confidence threshold, that is, when the instrument forceps of the target instrument are accurately identified, the position information of the center point of the instrument forceps in the two-dimensional coordinate system is determined based on the position information of the forceps bounding box. Based on the position information of the center point of the forceps, it is determined whether the instrument forceps of the target instrument is within the field of view of the endoscope.

[0049] S140. When the position information of the center point of the clamp head is not within the field of view, determine the orientation adjustment information of the target instrument based on the position information of the center point of the clamp head, the position information of the center point of the field of view, and the position information of the field of view border corresponding to the field of view.

[0050] The field of view (FAV) refers to the effective field of view corresponding to the endoscope. The center point of the FAV is located at the origin of the two-dimensional coordinate system. The border of the FAV can be the coordinates of the border of the endoscope's field of view. Orientation adjustment information is used to prompt the target instrument to adjust its orientation. This information includes the orientation of the instrument's forceps relative to the center point of the FAV.

[0051] Specifically, the field of view (FAV) of the endoscope is determined based on its resolution. The position of the forceps head center point is then used to determine if it falls within the FAV. If not, a line is drawn connecting the center point to the FAV's center point. The relationship between this line and the FAV's bounding box position is then used to determine the orientation of the forceps head relative to the FAV's center point. Orientation adjustment information is then generated based on this information, and the forceps head is adjusted accordingly.

[0052] For example, a two-dimensional coordinate system is established with the center of the endoscope's field of view as the origin. The position information of the center point of the instrument's forceps tip in the two-dimensional coordinate system is then determined. The length of the endoscopic field of view can be determined based on the resolution of the endoscope. The width value is Therefore, the field of view corresponding to the endoscope can be:

[0053]

[0054] exist If the conditions corresponding to the above field of view are not met, it means that the position information of the pliers head center point is not within the field of view. In this case, based on the pliers head center point position information... Information on the center point of the field of view and the position information of the field of view bounding box corresponding to the field of view range. , , as well as Determine the orientation adjustment information corresponding to the target instrument.

[0055] In this embodiment of the invention, the orientation adjustment information of the target instrument can be determined as follows: based on the position information of the center point of the clamp head and the position information of the center point of the field of view, the line segment between the center point of the clamp head and the center point of the field of view is determined, and based on the position information of the line segment and the field of view border corresponding to the field of view, the intersection point position information between the line segment and the field of view border is determined; based on the intersection point position information, the orientation adjustment information of the target instrument is determined.

[0056] The intersection point information can be understood as the coordinates of the intersection point between the line segment between the center point of the pliers head and the center point of the field of view, and the boundary of the field of view.

[0057] Specifically, based on the position information of the center point of the clamp head and the center point of the field of view, a line segment is obtained by connecting the center points of the clamp head and the field of view. The intersection point between the line segment and the field of view boundary is determined based on the position information of the corresponding field of view boundary. The orientation of the clamp head of the target instrument relative to the center point of the field of view can be determined based on the intersection point. Orientation adjustment information for the target instrument is then generated based on this orientation information.

[0058] In this embodiment of the invention, the method for determining the intersection point information between the line segment and the field of view border can be as follows: based on the position information of the center point of the clamp head and the position information of the center point of the field of view, determine the instrument orientation function corresponding to the target instrument; based on the position information of the field of view border corresponding to the field of view, determine the field of view border function; based on the instrument orientation function and the field of view border function, determine the intersection point coordinates between the line segment and the field of view border, and use the intersection point coordinates as the intersection point position information between the line segment and the field of view border.

[0059] The instrument orientation function can be a function corresponding to the center point of the forceps tip and the center point of the field of view. The field of view bounding function can be a function corresponding to the endoscopic field of view bounding. The intersection coordinates are the coordinate data of the intersection point between the line segment and the field of view bounding in a two-dimensional coordinate system.

[0060] Specifically, based on the position information of the clamp head center point and the center point of the field of view, the instrument orientation function corresponding to the line segment between the center point of the clamp head and the center point of the field of view is determined. Based on the position information of the field of view boundary corresponding to the field of view, the field of view boundary function is determined. The instrument orientation function and the field of view boundary function are simultaneously calculated to obtain the coordinates of the intersection point between the line segment and the field of view boundary. These intersection point coordinates are used as the intersection point position information between the line segment and the field of view boundary, and the orientation adjustment information of the target instrument is determined based on this intersection point position information.

[0061] For example, in conjunction with the above example, based on the center point position information of the pliers head and the location information of the center point of the field of view. Determine the instrument orientation function corresponding to the target instrument. Based on the position information of the view bounding box, determine the view bounding box function. , , as well as The instrument orientation function and the field of view bounding function are combined and processed to determine the position of the instrument. and The coordinates of the intersection points within the range are used as the location information of the intersection point between the line segment and the field of view boundary. Based on these coordinates, it is determined whether the target instrument is within the field of view of the endoscope, and the orientation information of the target instrument forceps head relative to the center of the endoscope's field of view is determined, so as to adjust the target instrument according to the orientation information.

[0062] S150. Send the orientation adjustment information to the target terminal, adjust the endoscope based on the orientation adjustment information displayed on the target terminal, and adjust the head of the target instrument according to the orientation adjustment information when the head of the target instrument is within the field of view.

[0063] The target terminal can be the terminal device corresponding to the operator of the target instrument. Optionally, the target terminal can be a device used to display orientation adjustment information.

[0064] Specifically, orientation adjustment information is sent to the target terminal of the instrument operator, allowing the operator to adjust the endoscope according to the orientation adjustment information displayed on the target terminal, ensuring the forceps head of the target instrument is within the endoscope's corresponding field of view. Once the forceps head is within the endoscope's field of view, based on actual needs and ensuring safe adjustment of the forceps head, the orientation adjustment information is used to adjust both the endoscope and the forceps head back to their original position. By adjusting the endoscope first, the safety of the forceps head during subsequent movement is ensured.

[0065] The technical solution of this embodiment acquires an image of the target instrument to be adjusted. The image is input into a pre-trained image recognition model for instrument head position recognition, obtaining the bounding box position information of the target instrument head and the corresponding recognition confidence level. This enables real-time detection of the target instrument's position, providing data support for subsequent determination of whether adjustment is needed. If the recognition confidence level is higher than a preset confidence threshold, the center point position of the target instrument head in a two-dimensional coordinate system is determined based on the bounding box position information, allowing for judgment of whether the target instrument is out of the field of view. If the center point position is not within the field of view, the orientation adjustment information of the target instrument is accurately determined based on the center point position information, the center point position information of the field of view, and the corresponding border position information, providing a basis for adjustment. The orientation adjustment information is sent to the target terminal, allowing for endoscope adjustment based on the orientation adjustment information displayed on the target terminal. When the target instrument head is within the field of view, the head is adjusted according to the orientation adjustment information. This invention solves the problems of judgment errors and tissue damage caused by relying on human experience to adjust the position of surgical instruments in the prior art. By determining the position information of the instruments in real time, it can determine whether the target instrument is out of the field of view, thereby improving the accuracy, stability and safety of surgical instrument adjustment.

[0066] Example 2

[0067] Figure 2 This is a flowchart of a device adjustment method provided in Embodiment 2 of the present invention. This embodiment, based on the above embodiments, focuses on adjusting the target device when the identification confidence level is lower than a preset confidence threshold. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 2 As shown, the method includes:

[0068] S210. When the confidence level is lower than the preset confidence threshold, the system detects whether the target instrument is within the field of view of the endoscope based on the field of view reference information corresponding to the endoscope and the position information of the center point of the clamp of the target instrument in the three-dimensional coordinate system; wherein, the three-dimensional coordinate system is determined based on the two-dimensional coordinate system and the preset depth.

[0069] The field of view reference information can be understood as the position information of the endoscope within its field of view. The position information of the center point of the target instrument's jaws in the three-dimensional coordinate system can be understood as the three-dimensional coordinate data corresponding to the center point of the target instrument's jaws in the three-dimensional coordinate system. The preset depth can be a depth pre-set according to actual needs.

[0070] Specifically, during image recognition of a target instrument, issues such as occlusion or glare may arise, causing the recognition confidence level to fall below a preset confidence threshold. In such cases, to accurately determine whether the target instrument is within the endoscope's field of view, the position of the instrument's clamp head center point in a three-dimensional coordinate system can be determined based on the two-dimensional coordinate system's information about the instrument's clamp head center point and the vertical distance between the clamp head center point and the endoscope. Using the endoscope's field of view reference information and the three-dimensional coordinate system's clamp head center point position information, the instrument's position is checked to determine if it is within the endoscope's field of view, allowing for adjustments to the clamp head based on the detection results.

[0071] Optionally, the field of view reference information can be a field of view normal vector. The field of view reference information can be determined as follows: for multiple corner points of the field of view corresponding to the endoscope, when the instrument reference point of the reference instrument coincides with the current corner point of the field of view, obtain multiple reference position information corresponding to the current corner point of the field of view in the three-dimensional coordinate system; based on the multiple reference position information, determine the field of view function corresponding to the current corner point of the field of view and the field of view direction vector corresponding to the field of view function; based on the field of view direction vector corresponding to each corner point of the field of view, determine the field of view normal vector of the field plane corresponding to every two adjacent field of view direction vectors, and use the field of view normal vector as the field of view reference information corresponding to the endoscope.

[0072] The reference instrument can be a pre-set surgical instrument used to determine the field of view reference information. The instrument reference point can be a point on the instrument's tilt axis, set according to actual needs. The field of view corner points can be understood as the four corner points of the field of view corresponding to the endoscope. For example, if the field of view corresponding to the endoscope is rectangular, the corner points are the four vertices of the matrix. The reference position information can be the three-dimensional coordinate data corresponding to the instrument tilt axis. The field of view function can be a linear function established by multiple field of view corner points. Each corner point has its corresponding field of view function. The field of view direction vector can be the direction vector corresponding to the field of view function. The field of view plane can be a plane formed by two adjacent field of view direction vectors. The field of view normal vector is a direction vector perpendicular to the field of view plane. The direction corresponding to the field of view normal vector points into the field of view corresponding to the endoscope.

[0073] Specifically, the endoscope's field of view contains multiple corner points. For each corner point, assuming the instrument reference point of the reference instrument coincides with the current corner point, multiple reference position information corresponding to the current corner point in the three-dimensional coordinate system is obtained. Based on the multiple reference position information corresponding to the current corner point, the field function corresponding to the current corner point is determined, and the field direction vector of the field function is determined. Based on this, the field direction vector corresponding to each corner point can be obtained. For multiple field direction vectors, the field plane corresponding to every two adjacent field direction vectors is determined, and the field normal vector of the field plane is determined. The field normal vector is used as the field reference information corresponding to the endoscope to detect whether the target instrument is within the endoscope's field of view.

[0074] For example, taking the field of view corresponding to the endoscope as a rectangle, the reference instrument and the rectangular field of view corresponding to the endoscope can form a virtual field of view in the shape of a square pyramid. For the four corner points of the field of view corresponding to the endoscope, the instrument reference point on the instrument yaw axis of the reference instrument is aligned with each corner point of the field of view. By adjusting the depth of field, 15 three-dimensional coordinate data points are collected for each corner point, for a total of 60 three-dimensional coordinate data points. Based on the 15 three-dimensional coordinate data points corresponding to the current corner points of the field of view, a field of view function corresponding to one edge is fitted using a binary linear regression method. Based on this, the field of view functions corresponding to the four edges of the virtual field of view corresponding to the endoscope can be obtained. Based on the field of view functions corresponding to the four edges, the field of view direction vectors corresponding to the four edges are determined. Since any two edges can form a field of view plane, the field of view normal vector corresponding to each field of view plane can be obtained, that is, four field of view normal vectors are obtained. The direction of the field of view normal vectors points into the interior of the virtual field of view in the shape of a square pyramid.

[0075] In this embodiment of the invention, the method for detecting whether the target instrument is within the field of view of the endoscope may be as follows: perform a dot product on each field of view normal vector in the position information of the center point of the forceps head of the target instrument and the field of view reference information to obtain the dot product result corresponding to each field of view normal vector; multiply the multiple dot product results to obtain the multiplied result of the target instrument; if the multiplied result is greater than a preset value, it is determined that the target instrument is within the field of view corresponding to the endoscope; if the multiplied result is less than a preset value, it is determined that the target instrument is not within the field of view corresponding to the endoscope.

[0076] The dot product result can be obtained by multiplying the pincer center point position information with the field-of-view normal vector. The multiplication result is the result of multiplying multiple dot product results. The default value can be zero.

[0077] Specifically, the position information of the clamp tip center point in the three-dimensional coordinate system is multiplied by each field-of-view normal vector in the field-of-view reference information to obtain a dot product result for each field-of-view normal vector, resulting in multiple dot product results. These multiple dot product results are then multiplied together to obtain a final product. If the final product is greater than zero, it indicates that the clamp tip of the target instrument is within the field of view corresponding to the endoscope. If the final product is less than zero, it indicates that the clamp tip of the target instrument is not within the field of view corresponding to the endoscope.

[0078] For example, in conjunction with the above example, the center point position information of the target instrument's clamp head is used as... For example, The four dot products are multiplied individually with the four field-of-view normal vectors, resulting in four dot product results. These four dot product results are then multiplied together to obtain the final product. If the final product is positive, it indicates that the center point of the target instrument's clamp head is within the virtual field of view shaped like a square pyramid, meaning the clamp head is within the field of view corresponding to the endoscope. Conversely, if the final product is negative, the clamp head is not within the field of view corresponding to the endoscope.

[0079] S220. Display the field of view detection results on the target terminal, and adjust the endoscope according to the field of view detection results displayed on the target terminal, so that when the forceps of the target instrument are within the field of view, the forceps of the target instrument are adjusted according to the field of view detection results.

[0080] The visual field detection result can be a result indicating whether the forceps head of the target instrument is within the visual field corresponding to the endoscope. That is, the visual field detection result can be that the forceps head of the target instrument is within the visual field corresponding to the endoscope, or it can be that the forceps head of the target instrument is not within the visual field corresponding to the endoscope.

[0081] Specifically, the visual field detection results determine whether the forceps head of the target instrument is within the corresponding field of view of the endoscope. If the forceps head is not within the corresponding field of view of the endoscope, the visual field detection result is sent to the target terminal, allowing the surgical instrument operator to adjust the endoscope according to the visual field detection result displayed on the target terminal, so that the forceps head of the target instrument is displayed within the field of view of the endoscope. Once the forceps head is displayed within the field of view of the endoscope, based on actual needs and under safe adjustment conditions, the forceps head of the target instrument and the endoscope are adjusted together to bring them back to their original positions.

[0082] The technical solution of this embodiment, when the confidence level is lower than a preset confidence threshold, detects whether the target instrument is within the endoscope's field of view based on the endoscope's corresponding field of view reference information and the center point position information of the target instrument's forceps head in a three-dimensional coordinate system. This improves the accuracy and comprehensiveness of the target instrument's field of view detection. The field of view detection results are displayed on the target terminal, allowing for endoscope adjustments based on these results. Furthermore, when the target instrument's forceps head is within the field of view, adjustments are made based on the field of view detection results, thus improving the comprehensiveness, accuracy, and safety of surgical instrument adjustments.

[0083] Example 3

[0084] Figure 3 This is a schematic diagram of the structure of an instrument adjustment device provided in Embodiment 3 of the present invention. Figure 3 As shown, the device includes: an image acquisition module 310, an image recognition module 320, a clamp head position determination module 330, an adjustment information determination module 340, and an instrument adjustment module 350.

[0085] Image acquisition module 310 is used to acquire images of the target instrument to be adjusted; image recognition module 320 is used to input the target instrument image into a pre-trained image recognition model to identify the position of the instrument's clamp head, obtaining the clamp head bounding box position information and the corresponding recognition confidence level; clamp head position determination module 330 is used to determine the position information of the center point of the target instrument's clamp head in a two-dimensional coordinate system based on the clamp head bounding box position information, provided that the recognition confidence level is higher than a preset confidence threshold. The two-dimensional coordinate system is based on the endoscope's position. The coordinate system is based on the center point of the corresponding field of view; the adjustment information determination module 340 is used to determine the orientation adjustment information of the target instrument based on the center point position information of the forceps, the center point position information of the field of view, and the position information of the field of view border corresponding to the field of view when the center point position information of the forceps is not within the field of view; the instrument adjustment module 350 is used to send the orientation adjustment information to the target terminal to adjust the endoscope based on the orientation adjustment information displayed on the target terminal, and to adjust the forceps of the target instrument according to the orientation adjustment information when the forceps of the target instrument is within the field of view.

[0086] The technical solution of this embodiment acquires an image of the target instrument to be adjusted. The image is input into a pre-trained image recognition model for instrument head position recognition, obtaining the bounding box position information of the target instrument head and the corresponding recognition confidence level. This enables real-time detection of the target instrument's position, providing data support for subsequent determination of whether adjustment is needed. If the recognition confidence level is higher than a preset confidence threshold, the center point position of the target instrument head in a two-dimensional coordinate system is determined based on the bounding box position information, allowing for judgment of whether the target instrument is out of the field of view. If the center point position is not within the field of view, the orientation adjustment information of the target instrument is accurately determined based on the center point position information, the center point position information of the field of view, and the corresponding border position information, providing a basis for adjustment. The orientation adjustment information is sent to the target terminal, allowing for endoscope adjustment based on the orientation adjustment information displayed on the target terminal. When the target instrument head is within the field of view, the head is adjusted according to the orientation adjustment information. This invention solves the problems of judgment errors and tissue damage caused by relying on human experience to adjust the position of surgical instruments in the prior art. By determining the position information of the instruments in real time, it can determine whether the target instrument is out of the field of view, thereby improving the accuracy, stability and safety of surgical instrument adjustment.

[0087] Based on the above embodiments, optionally, the adjustment information determination module includes: an intersection position information determination unit, used to determine the line segment between the center point of the clamp head and the center point of the field of view based on the position information of the center point of the clamp head and the position information of the center point of the field of view, and to determine the intersection position information between the line segment and the field of view border based on the position information of the line segment and the field of view border corresponding to the field of view; and an orientation adjustment information determination unit, used to determine the orientation adjustment information of the target instrument based on the intersection position information.

[0088] Optionally, the intersection location information determination unit includes: an instrument orientation function determination subunit, used to determine the instrument orientation function corresponding to the target instrument based on the center point location information of the clamp head and the center point location information of the field of view; a field of view boundary function determination subunit, used to determine the field of view boundary function based on the field of view boundary location information corresponding to the field of view; and an intersection location information determination subunit, used to determine the intersection coordinates between the line segment and the field of view boundary based on the instrument orientation function and the field of view boundary function, and use the intersection coordinates as the intersection location information between the line segment and the field of view boundary.

[0089] Optionally, the device further includes: an image recognition model training module, which is used to acquire training sample data, wherein the training sample data includes multiple training sample images and the clamp head label information of the sample instrument in each training sample image, wherein the clamp head label information of the sample instrument includes sample instrument category information, clamp head bounding box position information and recognition confidence; inputting the training sample data into the image recognition model to be trained to obtain the actual output result; determining the loss value based on the clamp head label information and the actual output result, and adjusting the model parameters of the image recognition model to be trained based on the loss value to obtain the trained image recognition model.

[0090] Optionally, the device further includes: an instrument field of view detection module, which includes: an instrument position determination unit, used to detect whether the target instrument is within the field of view of the endoscope based on the field of view reference information corresponding to the endoscope and the position information of the center point of the clamp head of the target instrument in the three-dimensional coordinate system when the recognition confidence is lower than a preset confidence threshold; wherein the three-dimensional coordinate system is determined based on the two-dimensional coordinate system and a preset depth; and an instrument adjustment unit, used to display the field of view detection results on the target terminal, so as to adjust the endoscope according to the field of view detection results displayed on the target terminal, so as to adjust the clamp head of the target instrument according to the field of view detection results when the clamp head of the target instrument is within the field of view.

[0091] Optionally, the device further includes: a field of view reference information determination module, which is used to obtain multiple reference position information corresponding to the current field of view corner points in a three-dimensional coordinate system when the instrument reference point of the reference instrument coincides with the current field of view corner point; based on the multiple reference position information, determine the field of view function corresponding to the current field of view corner point and the field of view direction vector corresponding to the field of view function; based on the field of view direction vector corresponding to each field of view corner point, determine the field of view normal vector of the field of view plane corresponding to every two adjacent field of view direction vectors, and use the field of view normal vector as the field of view reference information corresponding to the endoscope.

[0092] Optionally, the instrument position determination unit is used to perform dot product processing on each field normal vector in the center point position information of the clamp head of the target instrument and the field reference information to obtain the dot product result corresponding to each field normal vector; to perform multiplication processing on multiple dot product results to obtain the multiplication result of the target instrument; if the multiplication result is greater than a preset value, it is determined that the target instrument is within the field of view corresponding to the endoscope; if the multiplication result is less than a preset value, it is determined that the target instrument is not within the field of view corresponding to the endoscope.

[0093] The instrument adjustment device provided in the embodiments of the present invention can execute the instrument adjustment method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0094] Example 4

[0095] Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0096] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0097] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0098] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the instrument adjustment method.

[0099] In some embodiments, the device adjustment method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the device adjustment method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the device adjustment method by any other suitable means (e.g., by means of firmware).

[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] Computer programs used to implement the instrument adjustment method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] Example 5

[0103] Embodiment 5 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a device adjustment method, the method comprising:

[0104] Acquire images of the target instrument to be adjusted; input the target instrument image into a pre-trained image recognition model to identify the position of the instrument's clamp head, obtaining the clamp head bounding box position information and the corresponding recognition confidence level; if the recognition confidence level is higher than a preset confidence threshold, determine the position information of the center point of the target instrument's clamp head in a two-dimensional coordinate system based on the clamp head bounding box position information, where the two-dimensional coordinate system is a coordinate system with the center point of the field of view corresponding to the endoscope as the origin; if the clamp head center point position information is not within the field of view, determine the orientation adjustment information of the target instrument based on the clamp head center point position information, the center point position information of the field of view, and the position information of the field of view bounding box corresponding to the field of view; send the orientation adjustment information to the target terminal, adjust the endoscope based on the orientation adjustment information displayed on the target terminal, and adjust the clamp head of the target instrument according to the orientation adjustment information when the clamp head of the target instrument is within the field of view.

[0105] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0108] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0109] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An apparatus adjustment device, characterized by, The method comprises the following steps: An image acquisition module is configured to acquire a target instrument image collected for a target instrument to be adjusted. An image recognition module is configured to input the target instrument image into a pre-trained image recognition model to recognize a position of a jaw of the target instrument, so as to obtain jaw bounding box position information of the target instrument and recognition confidence corresponding to the jaw bounding box position information. A jaw position determination module is configured to, when the recognition confidence is higher than a preset confidence threshold, determine, based on the jaw bounding box position information, a jaw center point position information of the target instrument in a two-dimensional coordinate system, wherein the two-dimensional coordinate system is a coordinate system with a center point of a field of view range corresponding to an endoscope as an origin. An adjustment information determination module is configured to, when the jaw center point position information is not in the field of view range, determine, based on the jaw center point position information, center point position information of the field of view range, and field of view frame position information corresponding to the field of view range, orientation adjustment information of the target instrument. An instrument adjustment module is configured to send the orientation adjustment information to a target terminal, so as to adjust the endoscope based on the orientation adjustment information displayed by the target terminal, and adjust the jaw of the target instrument according to the orientation adjustment information when the jaw of the target instrument is in the field of view range.

2. The apparatus adjustment device of claim 1, wherein, The adjustment information determination module comprises: An intersection position information determination unit is configured to determine a line segment between the jaw center point and the center point of the field of view range according to the jaw center point position information and the center point position information of the field of view range, and determine intersection position information between the line segment and a field of view frame corresponding to the field of view range based on the line segment and the field of view frame position information. An orientation adjustment information determination unit is configured to determine the orientation adjustment information of the target instrument based on the intersection position information.

3. The apparatus adjustment device of claim 2, wherein, The intersection position information determination unit comprises: An instrument orientation function determination subunit is configured to determine an instrument orientation function corresponding to the target instrument based on the jaw center point position information and the center point position information of the field of view range. A field of view frame function determination subunit is configured to determine a field of view frame function based on the field of view frame position information corresponding to the field of view range. An intersection position information determination subunit is configured to determine intersection coordinates between the line segment and the field of view frame based on the instrument orientation function and the field of view frame function, and take the intersection coordinates as the intersection position information between the line segment and the field of view frame.

4. The apparatus adjustment device of claim 1, wherein Further comprising: An image recognition model training module is configured to: acquire training sample data, wherein the training sample data comprises a plurality of training sample images and jaw label information of a sample instrument in each training sample image, wherein the jaw label information of the sample instrument comprises sample instrument category information, jaw bounding box position information of the sample instrument, and recognition confidence; input the training sample data into a to-be-trained image recognition model to obtain an actual output result; and Based on the forceps head tag information and the actual output result, a loss value is determined, and model parameters of the image recognition model to be trained are adjusted based on the loss value to obtain a trained image recognition model.

5. The apparatus adjustment device of claim 1, wherein, Further comprising: An instrument field of view range detection module, the instrument field of view range detection module comprising: An instrument position determination unit configured to, in a case where the recognition confidence is lower than a preset confidence threshold, detect whether the target instrument is within a field of view range of the endoscope based on field of view reference information corresponding to the endoscope and forceps head center point position information of the target instrument in a three-dimensional coordinate system, wherein the three-dimensional coordinate system is determined based on the two-dimensional coordinate system and a preset depth; An instrument adjustment unit configured to display the field of view detection result on the target terminal, and adjust the endoscope according to the field of view detection result displayed on the target terminal, and adjust the forceps head of the target instrument according to the field of view detection result when the forceps head of the target instrument is within the field of view range.

6. The apparatus adjustment device of claim 5, wherein, Further comprising: A field of view reference information determination module, the field of view reference information determination module configured to: For a plurality of field of view corner points corresponding to the endoscope, in a case where an instrument reference point of a reference instrument coincides with a current field of view corner point, obtain a plurality of reference position information corresponding to the current field of view corner point in a three-dimensional coordinate system; Determine a field of view function corresponding to the current field of view corner point and a field of view direction vector corresponding to the field of view function based on a plurality of the reference position information; Determine a field of view normal vector of a field of view plane corresponding to each two adjacent field of view direction vectors based on the field of view direction vector corresponding to each field of view corner point, and take the field of view normal vector as the field of view reference information corresponding to the endoscope.

7. The apparatus adjustment device of claim 5, wherein, The instrument position determination unit is configured to: Perform dot multiplication processing on the forceps head center point position information of the target instrument and each field of view normal vector in the field of view reference information to obtain a dot multiplication result corresponding to each field of view normal vector; Perform multiplication processing on a plurality of the dot multiplication results to obtain a multiplication result of the target instrument; If the multiplication result is greater than a preset value, it is determined that the target instrument is within the field of view range corresponding to the endoscope; If the multiplication result is less than a preset value, it is determined that the target instrument is not within the field of view range corresponding to the endoscope.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method corresponding to the instrument adjustment device in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to execute the method corresponding to the instrument adjustment device in any one of claims 1-7 when executed.

Images