Image tag generation method, apparatus and system, medium and computing device

Abstract

A label generation method, device and system of an image, a medium and a computing device, the method comprising: acquiring a first image comprising a target object and an initial pose of the target object when the first image is captured; acquiring a three-dimensional model of the target object; projecting the three-dimensional model onto the first image based on the initial pose to obtain a projected pixel region; calibrating the initial pose based on an overlap degree between the projected pixel region and a target pixel region in the first image where the target object is located to obtain a calibrated pose; generating a second image comprising the target object based on the first image; and generating label information of the target object in the second image based on the calibrated pose.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to image tag generation methods, apparatus and systems, media and computing devices. Background Technology

[0002] Typically, neural networks are used to track target objects. However, training these networks relies on a large number of labeled images. If the image labels are inaccurate, the trained neural network will also be inaccurate, leading to inaccurate tracking results. Furthermore, related technologies generally rely on sensor measurements of the target object to obtain its label; however, due to errors in sensor measurements, the acquired labels are also inaccurate. Summary of the Invention

[0003] Based on this, embodiments of the present disclosure provide a method, apparatus, system, medium, and computing device for generating image labels, so as to generate more accurate labels for images.

[0004] In a first aspect, embodiments of this disclosure provide a method for generating image labels, the method comprising: acquiring a first image including a target object and an initial pose of the target object when the first image was acquired; acquiring a three-dimensional model of the target object; projecting the three-dimensional model onto the first image based on the initial pose to obtain a projected pixel region; calibrating the initial pose based on the overlap between the projected pixel region and a target pixel region in the first image where the target object is located to obtain a calibrated pose; generating a second image including the target object based on the first image; and generating label information of the target object in the second image based on the calibrated pose.

[0005] In some embodiments, the first image is obtained by acquiring an image of the target object under a first preset background.

[0006] In some embodiments, generating a second image including the target object based on the first image includes: replacing the first preset background in the first image with a second preset background to obtain the second image.

[0007] In some embodiments, the method further includes: post-processing the second image; the post-processing includes at least one of the following: blurring, sharpening, noise reduction, and enhancement.

[0008] In some embodiments, the calibration pose is the pose of the target object when the overlap is maximized.

[0009] In some embodiments, calibrating the initial pose based on the overlap between the projected pixel region and the target pixel region where the target object is located in the first image to obtain a calibrated pose includes: after optimizing the initial pose using a preset pose optimization algorithm, recalculating the overlap between the projected pixel region and the target pixel region; and determining the pose corresponding to the projected pixel region with the largest overlap with the target pixel region as the calibrated pose.

[0010] In some embodiments, the overlap between the projected pixel region and the target pixel region is determined based on the IoU, GIoU, or dice loss between the projected pixel region and the target pixel region.

[0011] In some embodiments, the method further includes: obtaining a mask of the target object in the first image; and determining the overlap based on the mask of the target object and the projected pixel region.

[0012] In some embodiments, before determining the overlap based on the mask of the target object and the projected pixel region, the method further includes: smoothing the mask.

[0013] In some embodiments, the target object includes at least one surgical instrument, each surgical instrument being held on a robotic arm of a surgical robot, and the robotic arm is equipped with a sensor for acquiring the initial pose of the surgical instrument held on the robotic arm; the first image is acquired by an image acquisition device.

[0014] In some embodiments, the three-dimensional model of the surgical instrument corresponds to the type and model of the surgical instrument; obtaining the three-dimensional model of the target object includes: obtaining the three-dimensional model of the surgical instrument held on the robotic arm according to the type and model of the surgical instrument held on the robotic arm.

[0015] In some embodiments, the method further includes: determining the type and model of surgical instruments held on each robotic arm based on the operation log of the surgical robot; or determining the type and model of surgical instruments held on each robotic arm based on user input.

[0016] In some embodiments, the label information includes the calibration pose, the type and model of the surgical instrument.

[0017] In some embodiments, the label information of the second image and the target object in the second image is used to train a neural network for tracking the target object.

[0018] Secondly, embodiments of this disclosure provide an image label generation apparatus, the apparatus comprising: an acquisition module, configured to acquire a first image including a target object and an initial pose of the target object when the first image is acquired, and a three-dimensional model of the target object; a projection module, configured to project the three-dimensional model of the target object onto the first image based on the initial pose, to obtain a projected pixel region; a calibration module, configured to calibrate the initial pose based on the overlap between the projected pixel region and the target pixel region in the first image where the target object is located, to obtain a calibrated pose; a generation module, configured to generate a second image including the target object based on the first image; and a determination module, configured to determine label information of the target object in the second image based on the calibrated pose.

[0019] Thirdly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in any of the embodiments of the first aspect.

[0020] Fourthly, embodiments of this disclosure provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any embodiment of the first aspect.

[0021] Fifthly, embodiments of this disclosure provide an image tag generation system, the system comprising: an image acquisition device for acquiring a first image of a target object; a pose sensor for acquiring an initial pose of the target object when acquiring the first image; and a computing device according to the fourth aspect.

[0022] In some embodiments, the target object is a surgical instrument; the system further includes: a surgical robot, including at least one robotic arm, each robotic arm being used to hold a surgical instrument, and each robotic arm being provided with the pose sensor.

[0023] In some embodiments, the target object is a surgical instrument; the system further includes: a surgical robot, comprising at least two robotic arms, wherein the image acquisition device and the surgical instrument are held by different robotic arms, and at least the robotic arm holding the surgical instrument is provided with the pose sensor.

[0024] In this embodiment, a first image including the target object is acquired. Based on the initial pose of the target object when the first image is acquired, a 3D model of the target object is projected onto the first image to obtain a projected pixel region. Then, the initial pose is calibrated based on the overlap between the projected pixel region and the target pixel region, thereby obtaining a more accurate calibrated pose of the target object in the first image. Label information of the target object in a second image including the target object is generated based on this calibrated pose, enabling the acquisition of more accurate label information and improving the accuracy of the label information.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to illustrate the technical solutions of this disclosure.

[0027] Figure 1 This is a schematic diagram of a robotic surgical system according to an embodiment of the present disclosure.

[0028] Figure 2 This is a schematic diagram of the patient-side robot according to an embodiment of the present disclosure.

[0029] Figure 3 This is a flowchart of an image tag generation method according to an embodiment of the present disclosure.

[0030] Figure 4 yes Figure 3 The overall flowchart of the method shown is as follows.

[0031] Figure 5 yes Figure 4 A schematic diagram of the images in the processing flow shown.

[0032] Figure 6 This is a flowchart of an image tag generation method according to another embodiment of the present disclosure.

[0033] Figure 7 yes Figure 6 The overall flowchart of the method shown is as follows.

[0034] Figure 8 This is a flowchart of a method for tracking a target object in an image according to an embodiment of this disclosure.

[0035] Figure 9A This is a schematic diagram of the structure of a neural network according to an embodiment of the present disclosure.

[0036] Figure 9B This is a schematic diagram of a more specific neural network structure according to an embodiment of this disclosure.

[0037] Figure 10A yes Figure 8 The overall flowchart of the method shown is as follows.

[0038] Figure 10B This is a schematic diagram of the multi-target tracking process.

[0039] Figure 11 This is a block diagram of an image tag generation apparatus according to an embodiment of the present disclosure.

[0040] Figure 12 This is a block diagram of an image tag generation apparatus according to another embodiment of the present disclosure.

[0041] Figure 13 This is a block diagram of a target object tracking device according to an embodiment of the present disclosure.

[0042] Figure 14 This is a schematic diagram of a computing device according to an embodiment of the present disclosure.

[0043] Figure 15 This is a schematic diagram of an image tag generation system according to an embodiment of the present disclosure. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0045] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0046] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0047] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0048] Before using the neural network Net to track the target object Obj, a large number of labeled images are needed to train the neural network Net. Examples of specific application scenarios are given below. It should be understood that the application scenarios described below are merely illustrative and are not intended to limit this disclosure.

[0049] In a surgical setting, the target object Obj includes surgical instruments X. Surgical instruments X are widely used in various surgeries, and surgeons can manipulate them using a robotic surgical system 10 to perform surgical procedures. For example... Figure 1 The diagram shows a schematic of a robotic surgical system 10. During surgery, the patient is positioned in front of a patient-side robot (PSR) 101, which includes one or more robotic arms 101a. The end of each robotic arm 101a is used to hold one or more surgical instruments X. The surgeon can control the robotic arms 101a via a surgeon console (SGC) 102, thereby controlling the surgical instruments X to perform surgical procedures on the patient. The robotic arms 101a can also hold image acquisition devices (e.g., an endoscope camera, not shown). The surgeon can control the robotic arm 101a holding the endoscope camera via the surgeon console 102, thereby moving and holding the endoscope camera near the patient's lesion area to acquire surgical images including the lesion, surrounding tissue, and the surgical instruments X. During the surgery, the surgical instrument X and / or endoscopic camera on the robotic arm 101a are inserted into the patient's body through a pre-set incision and can rotate around the center point of that incision (generally referred to as the Remote Center of Motion point, RCM). The images captured by the endoscopic camera are transmitted to the Vision Cart (VCT) 103 for image processing and recording, and the processed images are displayed on the respective display devices of the Vision Cart 103 and the doctor's console 102 for observation by the doctor and other surgical staff.

[0050] During surgery, it is often desirable to obtain accurate positional information of surgical instruments for real-time tracking. This is particularly advantageous when instruments are outside the endoscopic camera's field of view or obstructed from it. In some cases, neural networks are used to track surgical instruments. Before deploying a neural network model in a real-world application, it needs to be trained.

[0051] Dataset preparation

[0052] Training a neural network model requires a large dataset containing images of the target object and labels within those images that describe the target object. In the application scenario of this embodiment, the target object Obj includes a surgical instrument X, and the label information may include the accurate pose information of the surgical instrument X.

[0053] Figure 2 A schematic diagram of the patient-side robot 101 is shown. (As shown) Figure 2 As shown, the patient-side robot 101 includes a chassis 101b, a push handle 101c, and at least one robotic arm 101a (only one robotic arm 101a is shown in the figure for ease of illustration). Each robotic arm 101a includes an adjustment arm 101a-1 and an operating arm 101a-2. The robotic arm 101a-2 includes one or more sensors, such as displacement sensors, orientation sensors, and / or position sensors. Kinematic data, such as the pose information of the surgical instrument X, can be obtained from the detection values ​​of these sensors. However, due to errors in the sensor measurements and the cumulative error in the transmission of the robotic arm 101a, the kinematic data acquired by the sensors is noisy. The pose information of the surgical instrument X generated solely from this noisy kinematic data has low accuracy and cannot be used for neural network training.

[0054] Example 1

[0055] To at least address the aforementioned problems, embodiments of this disclosure provide a method for generating image labels, see below. Figure 3 The method includes:

[0056] Step S11: Obtain the first image Img1 of the target object Obj and the initial pose Pose0 of the target object Obj when the first image Img1 is acquired;

[0057] Step S12: Obtain the 3D model Mod of the target object Obj;

[0058] Step S13: Based on the initial pose Pose0, project the 3D model Mod onto the first image Img1 to obtain the projected pixel region Rm;

[0059] Step S14: Based on the overlap between the projected pixel region Rm and the target pixel region Ro of the target object Obj in the first image Img1, the initial pose Pose0 is calibrated to obtain the calibrated pose Pose1.

[0060] Step S15: Generate a second image Img2, which includes the target object Obj, based on the first image Img1;

[0061] Step S16: Generate the label information of the target object Obj in the second image Img2 based on the calibration pose Pose1.

[0062] The following provides an example illustrating the implementation details of image label generation in this disclosure.

[0063] In step S11, the target object Obj can be a surgical instrument X. However, it is understood that in other application scenarios, the target object Obj can also be other objects. For example, in an image surveillance scenario, the target object Obj can be a monitored object such as a person or animal; in a traffic scenario, the target object Obj can be a vehicle. For ease of explanation, the following will use... Figure 1 and Figure 2 The following describes the solution of the present disclosure embodiment using a surgical scenario as an example.

[0064] An image acquisition device can be used to acquire images of surgical instruments X, resulting in a first image Img1 including the surgical instruments X. The surgical instruments X include, but are not limited to, one or more of a scalpel, tissue scissors, surgical forceps, needle holder, and hemostat. Each surgical instrument X can be held on a robotic arm 101a of the surgical robot. The image acquisition device can also be held on the robotic arm 101a of the surgical robot, mounted on a support, or fixed in another location (e.g., a wall or tabletop). A pose sensor can be installed on the robotic arm 101a holding the surgical instruments X to acquire the initial pose Pose0 of the surgical instruments X held on the robotic arm 101a when acquiring the first image Img1. In one example, the robotic arm 101a includes multiple sequentially connected connecting arms, with adjacent connecting arms connected by rotary joints. The surgical instruments X are mounted on the end connecting arm. The pose sensor may include encoders located at each rotary joint, displacement gauges located on the linear drive module of the end connecting arm, and encoders located on the tool drive module of the end connecting arm. Due to errors in the pose sensor, the initial pose Pose0 is a noisy pose and cannot accurately reflect the true pose of the surgical instruments. Furthermore, a pose sensor can also be installed on the robotic arm 101a holding the image acquisition device to detect the pose of the image acquisition device.

[0065] In some embodiments, the first image Img1 can be obtained by acquiring an image of the surgical instrument X against a first preset background. The difference between the pixel values ​​of the first preset background and the pixel values ​​of the surgical instrument X can be greater than a preset value. For example, if the surgical instrument X is white, the first preset background can be black. Furthermore, the first preset background can also be a solid color background (i.e., containing only one color) with minimal texture. This reduces the interference of the color and texture of the first preset background on subsequent processing of the first image Img1, thereby improving the accuracy of the acquired label information. Alternatively, the first image Img1 can also be obtained by acquiring an image of the surgical instrument X in a practical application scenario (e.g., a surgical procedure).

[0066] In step S12, a three-dimensional model Mod of the surgical instrument X can be obtained. Each surgical instrument X has a specific type and model number, as well as a corresponding three-dimensional model Mod. For example, surgical instruments can be classified into the following types according to their functions: scalpel, tissue scissors, surgical forceps, hemostats, etc. For each type of surgical instrument, it can be divided into different models according to its structure, size, and other characteristics. The three-dimensional model Mod of the surgical instrument X is established when its design is completed or before production. This disclosure does not specifically limit the method of establishing the three-dimensional model Mod. The three-dimensional model Mod of the surgical instrument X can be pre-stored in a corresponding memory, thereby obtaining the three-dimensional model Mod of the surgical instrument X held on the robotic arm 101a from the memory according to the type and model number of the surgical instrument X held on the robotic arm 101a. For example, if the surgical instrument X held on robotic arm 101a is a scalpel of size 10, then the obtained 3D model Mod is the 3D model Mod corresponding to scalpel of size 10; if the surgical instrument X held on robotic arm 201a is a straight hemostat, then the obtained 3D model Mod is the 3D model Mod corresponding to straight hemostat.

[0067] In some embodiments, the surgical robot can automatically identify the type and model of the surgical instruments X held on each robotic arm 101a of the patient-side robot 101 and record it in the corresponding operation log. The type and model of the surgical instruments X held on each robotic arm 101a can be determined based on the surgical robot's operation log. In other embodiments, the type and model of the surgical instruments X held on each robotic arm 101a can also be determined based on user input. For example, the surgeon can manually input the type and model of the surgical instruments X held on each robotic arm 101a on the input interface of the surgeon's console 102.

[0068] Furthermore, a correspondence between various types and models of surgical instruments X and their corresponding 3D model Mods can be established in advance. After obtaining the type and model of the surgical instruments X held on each robotic arm 101a, the corresponding 3D model Mod of the surgical instrument X can be obtained based on the aforementioned correspondence. In this way, the 3D model Mod corresponding to the surgical instrument X can be automatically obtained, reducing manual operation and lowering labor costs.

[0069] In step S13, the 3D model Mod can be a 3D model in a physical coordinate system, for example, a standard model located at the origin of the physical coordinate system and having a specified pose. When the target object is a surgical instrument X, the initial pose Pose0 of the surgical instrument X can be the pose measured by the encoder on the robotic arm on which the surgical instrument X is located. This pose can be the pose of the surgical instrument X in a physical coordinate system such as the PSR base coordinate system or the world coordinate system. Based on the aforementioned initial pose Pose0, the 3D model Mod can be projected onto the first image Img1. Specifically, a transformation matrix of the image acquisition device can be obtained. This transformation matrix can be obtained by calibrating the image acquisition device and is used to realize the transformation between the physical coordinate system (such as the aforementioned PSR base coordinate system) and the coordinate system of the image acquisition device. Based on the aforementioned transformation matrix of the image acquisition device, the 3D model Mod can be projected onto the first image Img1.

[0070] Ideally, the projected pixel region Rm of the 3D model Mod on the first image Img1 completely overlaps with the target pixel region Ro where the target object Obj is located in the first image Img1. However, due to a certain error in the initial pose Pose0, they do not completely overlap in reality, and the error of the initial pose Pose0 is inversely correlated to some extent with the degree of overlap between the projected pixel region Rm and the target pixel region Ro. Therefore, in step S14, the initial pose Pose0 can be calibrated based on the degree of overlap between the projected pixel region Rm and the target pixel region Ro.

[0071] The overlap between the projected pixel region Rm and the target pixel region Ro can be determined based on the intersection over union (IoU), generalized intersection over union (GIoU), or dice loss between the projected pixel region Rm and the target pixel region Ro, or other parameters that can characterize the overlap.

[0072] In some embodiments, a mask of the target object Obj in the first image Img1 can be obtained, and the overlap between the projected pixel region Rm and the target pixel region Ro can be determined based on the mask of the target object Obj and the projected pixel region Rm. Specifically, image processing can be performed on the first image Img1 to remove background regions, thereby obtaining the mask of the target object Obj in the first image Img1. Alternatively, the mask of the target object Obj in the first image Img1 can be obtained through manual annotation. Optionally, when the first image Img1 is obtained by image acquisition of the surgical instrument X under a first preset background, since the first preset background generally differs significantly from the target object Obj, the mask can be automatically obtained through image processing. However, in practical applications where the first image Img1 is obtained by image acquisition of the surgical instrument X, the mask can be obtained through manual annotation to reduce the influence of complex backgrounds and improve the accuracy of mask acquisition. By obtaining a mask, the influence of background regions on the overlap calculation process can be reduced, thereby improving the accuracy of the calculated overlap and consequently the accuracy of the obtained label information. Furthermore, before determining the overlap between the projected pixel region Rm and the target pixel region Ro based on the mask of the target object Obj and the projected pixel region Rm, the mask can be smoothed. Smoothing reduces the influence of random noise and eliminates abnormal pixels, thus improving the accuracy and reliability of the obtained mask.

[0073] In some embodiments, the calibration pose Pose1 is the pose of the target object Obj when the overlap is maximized. Specifically, the calibration pose Pose1 can be obtained as follows: After optimizing the initial pose Pose0 using a preset pose optimization algorithm, the overlap between the projected pixel region Rm and the target pixel region Ro is recalculated; the pose corresponding to the projected pixel region Rm with the largest overlap with the target pixel region Ro is determined as the calibration pose Pose1. The pose optimization algorithm can be a gradient-based optimization algorithm or other global optimization algorithms. Several iterations of optimization can be used to determine the calibration pose Pose1. In the first iteration, the overlap between the projected pixel region Rm and the target pixel region Ro corresponding to the initial pose Pose0 is determined, and the initial pose Pose0 is optimized to obtain the pose optimized in the first iteration. In the second iteration, the overlap between the projected pixel region Rm and the target pixel region Ro corresponding to the pose optimized in the first iteration is determined, and the pose optimized in the first iteration is used to obtain the pose optimized in the second iteration. This process continues until a preset iteration termination condition is met. For example, the number of iterations reaches a preset threshold, the algorithm execution time reaches a preset threshold, the maximum overlap obtained during iteration reaches a preset overlap threshold, or the overlap obtained during iteration reaches a local maximum. Therefore, the calibration pose Pose1 can be considered the true pose of the target object Obj.

[0074] In step S15, a second image Img2, including the target object Obj, can be generated based on the first image Img1. For example, the first preset background in the first image Img1 can be replaced with a second preset background to obtain the second image Img2. The second preset background is typically obtained by an image acquisition device, such as an endoscope, from photographing human tissue. Furthermore, before replacing the background with the second preset background, operations such as brightness adjustment, orientation transformation, and / or scale transformation can be performed on the target object Obj in the first image Img1. The second preset background can be related to the application scenario. For example, in the surgical scenario of the aforementioned embodiment, the second preset background can be the background of the surgical scenario.

[0075] In some embodiments, the second image Img2 can also be post-processed to make it more closely resemble the actual surgical scene. Post-processing may include, but is not limited to, at least one of the following: blurring, sharpening, noise reduction, and enhancement.

[0076] In step S16, label information for the target object Obj in the second image Img2 can be generated based on the calibration pose Pose1. The label information may include the calibration pose Pose1. If the target object Obj is a surgical instrument X, the label information may also include the type and model of the surgical instrument X. Since the second image Img2 is obtained by processing the first image Img1 through grayscale processing, background transformation, etc., the pose of the target object Obj in the second image Img2 is the same as that in the first image Img1. Therefore, the calibration pose Pose1 can be directly determined as one of the label information for the target object Obj in the second image Img2. When the target object Obj in the first image Img1 is subjected to orientation transformation or scale transformation, the pose of the target object Obj in the second image Img2 is different from that in the first image Img1. Therefore, the calibration pose Pose1 can be mapped based on the pixel mapping relationship between the second image Img2 and the first image Img1 to obtain the mapped pose, and the mapped pose is determined as one of the label information of the target object Obj in the second image Img2.

[0077] When the first image includes multiple surgical instruments X, the above process can be performed on each surgical instrument X separately to obtain the label information of each surgical instrument X in the second image corresponding to the first image.

[0078] The following example uses a surgical scenario, combined with... Figure 4 The overall flow of the image tag generation method according to embodiments of this disclosure will be described. The image tag generation method includes the following steps:

[0079] Step S21: Acquire a first image Img1 including surgical instrument X against a first preset background.

[0080] Step S22: Obtain the initial pose Pose0 of the surgical instrument X.

[0081] Step S23: Obtain the 3D model Mod of surgical instrument X.

[0082] Step S24: Extract the mask of surgical instrument X from the first image Img1.

[0083] Step S25: Project the 3D model Mod onto the first image Img1.

[0084] Step S26: Calculate the overlap between the mask of the surgical instrument X and the projected image region of the 3D model Mod on the first image Img1.

[0085] Step S27: Determine if the overlap is maximized. If yes, proceed to step S29; otherwise, proceed to step S28.

[0086] Step S28: Adjust the initial pose Pose0 and return to step S26.

[0087] Step S29: Determine the pose at which the overlap is maximized as the calibration pose Pose1.

[0088] Step S30: Replace the first preset background in the first image Img1 with the second preset background.

[0089] Step S31: Post-process the second image Img2. The calibration pose Pose1 can be used as the label information corresponding to the post-processed second image Img2.

[0090] It is understood that the execution order of the steps in the above method does not need to follow the step numbers. For example, steps S21, S22 and S23, and steps S24 and S25 can be executed in parallel or in any order.

[0091] Figure 5 It shows Figure 4 The diagram illustrates the images generated during the processing flow. First, an image of the surgical instrument X is acquired against a single-color background, resulting in a first image Img1. The 3D model Mod of the target object Obj is then projected onto the first image Img1, yielding a projected pixel region Rm. The first image Img1 is then segmented to obtain a mask for the surgical instrument X within it. Based on this mask and the projected pixel region Rm, pose optimization is performed to obtain the calibrated pose Pose1. Finally, the background in the first image Img1 is replaced with the surgical scene, resulting in a second image Img2.

[0092] The above embodiment acquires a first image Img1 containing the target object Obj. Based on the initial pose Pose0 of the target object Obj when acquiring the first image Img1, the 3D model Mod of the target object Obj is projected onto the first image Img1 to obtain the projected pixel region Rm. Then, the initial pose Pose0 is calibrated based on the overlap between the projected pixel region Rm and the target pixel region Ro, thereby obtaining a more accurate calibrated pose Pose1 for the target object Obj in the first image Img1. Based on this calibrated pose Pose1, the label information of the target object Obj in the second image Img2 containing the target object Obj is generated, which can obtain more accurate label information.

[0093] In some embodiments, the label information of the target object Obj in the second image Img2 and the second image Img2 can be used to train the neural network Net. The trained neural network Net can then be used to track the target object Obj. For example, in a surgical scenario, the second image Img2, which includes surgical instruments as the target object and actual human tissue as the background, obtained by the method in the foregoing embodiments, and the label information obtained by the method in the foregoing embodiments, can be used as a dataset to train the neural network Net. During the surgical procedure, the trained neural network Net can be used to track the surgical instrument X. To improve the accuracy of the tracking results, a large dataset is needed to train the neural network Net. The above method can automatically and quickly generate a large dataset without relying on actual surgical operation footage. The datasets generated by the above method are diverse, including images of different surgical instruments in different surgical scenarios and the calibration poses of the surgical instruments in those images.

[0094] In addition, corresponding to the above method, this disclosure also provides an image tag generation apparatus, see [link to relevant documentation]. Figure 11 The device includes:

[0095] The acquisition module 110 is used to acquire a first image Img1 of the target object Obj and the initial pose Pose0 of the target object Obj when the first image Img1 is acquired, and to acquire a three-dimensional model Mod of the target object Obj.

[0096] The projection module 120 is used to project the three-dimensional model Mod of the target object Obj onto the first image Img1 based on the initial pose Pose0, so as to obtain the projection pixel area Rm.

[0097] The calibration module 130 is used to calibrate the initial pose Pose0 based on the overlap between the projected pixel region Rm and the target pixel region Ro where the target object Obj is located in the first image Img1, so as to obtain the calibrated pose Pose1.

[0098] The generation module 140 is used to generate a second image Img2, which includes the target object Obj, based on the first image Img1;

[0099] The determination module 150 is used to determine the label information of the target object Obj in the second image Img2 based on the calibration pose Pose1.

[0100] In some embodiments, the first image is obtained by acquiring an image of the target object under a first preset background.

[0101] In some embodiments, the generation module is specifically used to: replace the first preset background in the first image Img1 with a second preset background to obtain the second image Img2.

[0102] In some embodiments, the apparatus further includes: a post-processing module for post-processing the second image Img2; the post-processing includes at least one of the following: blurring, sharpening, noise reduction, and enhancement.

[0103] In some embodiments, the calibration pose Pose1 is the pose of the target object Obj when the overlap is maximized.

[0104] In some embodiments, the calibration module is specifically used to: after optimizing the initial pose Pose0 using a preset pose optimization algorithm, recalculate the overlap between the projected pixel region Rm and the target pixel region Ro; and determine the pose corresponding to the projected pixel region Rm with the largest overlap with the target pixel region Ro as the calibration pose Pose1.

[0105] In some embodiments, the overlap between the projected pixel region Rm and the target pixel region Ro is determined based on the IoU, GIoU, or dice loss between the projected pixel region Rm and the target pixel region Ro.

[0106] In some embodiments, the apparatus further includes: a mask acquisition module, configured to acquire a mask of a target object Obj in a first image Img1; and an overlap determination module, configured to determine the overlap based on the mask of the target object Obj and the projected pixel region Rm.

[0107] In some embodiments, prior to the calibration module, the apparatus further includes a smoothing module for smoothing the mask.

[0108] In some embodiments, the target object Obj includes at least one surgical instrument X, each surgical instrument X is held on a robotic arm 101a of the surgical robot, and the robotic arm 101a is equipped with a sensor for acquiring the initial pose of the surgical instrument X held on the robotic arm 101a; the first image Img1 is acquired by an image acquisition device.

[0109] In some embodiments, the three-dimensional model Mod of surgical instrument X corresponds to the type and model of surgical instrument X; the acquisition module is specifically used to: acquire the three-dimensional model Mod of surgical instrument X held on robotic arm 101a according to the type and model of surgical instrument X held on robotic arm 101a.

[0110] In some embodiments, the apparatus further includes: a type and model determination module, configured to determine the type and model of the surgical instruments X held on each robotic arm 101a based on the operation log of the surgical robot; or to determine the type and model of the surgical instruments X held on each robotic arm 101a based on user input.

[0111] In some embodiments, the label information includes the calibration pose Pose1 and the type and model of the surgical instrument X.

[0112] In some embodiments, the label information of the target object Obj in the second image Img2 and the second image Img2 is used to train a neural network, which is used to track the target object Obj.

[0113] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0114] Example 2

[0115] See Figure 6 This disclosure also provides a method for generating image tags, the method comprising:

[0116] Step S41: Obtain the original video frame f0 of the target object Obj and the initial pose Pose0 of the target object Obj when the original video frame f0 is acquired;

[0117] Step S42: Obtain the 3D model Mod of the target object Obj;

[0118] Step S43: Render the 3D model Mod based on the initial pose Pose0 to obtain the rendered video frame fr;

[0119] Step S44: Based on the optical flow field of the target object Obj in the original video frame f0 and the optical flow field of the target object Obj in the rendered video frame fr, determine the pose transformation relationship T between the target object Obj in the original video frame f0 and the target object Obj in the rendered video frame fr.

[0120] Step S45: Based on the pose transformation relationship T, calibrate the initial pose Pose0 to obtain the calibrated pose Pose1;

[0121] Step S46: Generate the label information of the target object Obj in the original video frame f0 based on the calibration pose Pose1.

[0122] In step S41, the original video frame f0 may include one or more video frames from a video. The original video frame f0 may be a video frame captured in a specified scene. Generally, the specified scene refers to the actual application scenario of the target object Obj, such as a surgical scene. In addition to including the target object Obj, the original video frame f0 may also include the background of the specified scene. For example, in the case of a surgical scene, the background of the specified scene may include tissues within the body of the object undergoing surgery, such as a human or animal. In some embodiments, each original video frame f0 may be the first image Img1 in the aforementioned embodiment one.

[0123] The target object Obj can be a surgical instrument X or other objects, and the specific category of the target object Obj may vary depending on the actual application scenario. In an embodiment where the target object Obj is a surgical instrument X, the surgical instrument X can be held on the robotic arm 101a of the surgical robot. An image acquisition device can also be held on the robotic arm 101a of the surgical robot for acquiring raw video frames f0. Alternatively, the image acquisition device for acquiring raw video frames f0 can be mounted on a bracket or fixed in other locations (e.g., a wall, a desktop, or beside a patient's bed). A pose sensor can be provided on the robotic arm 101a holding the surgical instrument X to acquire the initial pose Pose0 of the surgical instrument X held on the robotic arm 101a when acquiring the first image Img1. Furthermore, a pose sensor can also be provided on the robotic arm 101a holding the image acquisition device to detect the pose of the image acquisition device. In one example, the robotic arm 101a includes a plurality of sequentially connected connecting arms, with adjacent connecting arms connected by a rotary joint. The pose sensor may include an encoder disposed at each rotary joint for measuring the relative rotation angle between two adjacent connecting arms.

[0124] In step S42, a three-dimensional model Mod of the surgical instrument X can be obtained. A specific embodiment of step S42 can be found in the preceding description of step S12, and will not be repeated here.

[0125] In step S43, the 3D model Mod can be a 3D model in a physical coordinate system, for example, a standard model located at the origin of the physical coordinate system and having a specified pose. When the target object is a surgical instrument X, the initial pose Pose0 of the surgical instrument X can be the pose measured by the encoder on the robotic arm where the surgical instrument X is located. This pose can be the pose of the surgical instrument X in a physical coordinate system such as the PSR base coordinate system or the world coordinate system. Based on the initial pose Pose0, the 3D model Mod can be rendered to obtain a rendered video frame fr including the 3D model Mod. During rendering, the 3D model Mod can be projected onto the coordinate system of the image acquisition device based on the initial pose Pose0 to obtain a projected video frame, which is then rendered to obtain the rendered video frame fr. For example, the initial pose Pose0 can be converted into a pose in the coordinate system of the image acquisition device based on the transformation matrix of the image acquisition device, and then the 3D model Mod can be projected onto the coordinate system of the image acquisition device based on the converted pose to obtain a projected video frame. The transformation matrix is ​​used to characterize the transformation relationship between the coordinate system of the image acquisition device and the physical coordinate system. After obtaining the projected video frame, the target object Obj in the projected video frame can be rendered based on the pre-generated color map and texture map.

[0126] Assuming the number of original video frames f0 is N (N is a positive integer), the 3D model Mod can be rendered based on the initial pose Pose0 of the target object Obj when each original video frame f0 is acquired, resulting in N rendered video frames fr. Each rendered video frame fr corresponds to one original video frame f0.

[0127] In step S44, the optical flow field is used to describe the motion information in the image sequence. The optical flow field can be understood as the temporal displacement of each pixel in the image. Based on the optical flow fields of the target object Obj in the original video frame f0 and the rendered video frame fr, the motion and changes between the target object Obj in the original video frame f0 and the rendered video frame fr can be analyzed to determine the pose difference of the target object Obj in the two video frames. This pose difference can be characterized by a pose transformation relation T (which can be a transformation matrix). For example, the RANSAC (Random Sample Consensus) algorithm or a two-dimensional template matching algorithm can be used to estimate the pose transformation relation T.

[0128] In some embodiments, image segmentation can be performed on the original video frame f0 to obtain the target pixel region Ro where the target object Obj is located in the original video frame f0. The image segmentation operation on the original video frame f0 can be implemented by a pre-trained image segmentation network or based on user-annotated information. By performing image segmentation, the background region in the original video frame can be removed, retaining only the target pixel region Ro where the target object Obj is located in the original video frame f0, thereby reducing the impact of the background region on subsequent operations. Similarly, the target pixel region Ro where the target object Obj is located in subsequent video frames can also be obtained from the original video frame f0. Subsequent video frames can be the h-th (h is a positive integer) video frame after the original video frame f0. For example, assuming the original video frame f0 is the first video frame in the video, then subsequent video frames can be the second frame or a subsequent video frame in the video. Similar to the processing method for the original video frame f0, image segmentation can also be performed on subsequent video frames of the original video frame f0 to obtain the target pixel region Ro where the target object Obj is located in the subsequent video frames of the original video frame f0. Then, based on the target pixel region Ro where the target object Obj is located in the original video frame f0 and the target pixel region Ro where the target object Obj is located in the subsequent video frames of the original video frame f0, the optical flow field of the target object Obj in the original video frame f0 can be determined.

[0129] In other embodiments, image segmentation may be omitted, and instead the optical flow field of the target object Obj in the original video frame f0 and the optical flow field of the target object Obj in the rendered video frame fr may be directly obtained.

[0130] In step S45, the initial pose Pose0 can be transformed based on the pose transformation relationship T obtained in step S44 to obtain the calibrated pose Pose1.

[0131] Step S46 can be found in step S16 of Embodiment 1, and will not be repeated here.

[0132] This embodiment of the disclosure acquires the optical flow field of the target object Obj in the original video frame f0, and renders the 3D model Mod of the target object Obj based on the initial pose Pose0 of the target object Obj when acquiring the original video frame f0, obtaining a rendered video frame fr. Then, it acquires the optical flow field of the target object Obj in the rendered video frame fr, and determines the pose transformation relationship T between the target object Obj in the original video frame f0 and the target object Obj in the rendered video frame fr based on the two optical flow fields. This pose transformation relationship T reflects the difference between the motion information of the target object Obj in the original video frame f0 and the motion information of the target object Obj in the rendered video frame fr. Therefore, calibrating the initial pose Pose0 based on the pose transformation relationship T can obtain a more accurate calibrated pose Pose1, thereby making the tag information generated based on the calibrated pose Pose1 more accurate.

[0133] In some embodiments, the label information may include a calibration pose (Pose1). If the target object (Obj) is a surgical instrument (X), the label information may also include the type and model of the surgical instrument (X).

[0134] The following example uses a surgical scenario, combined with... Figure 7 The overall flow of the image tag generation method according to embodiments of this disclosure will be described. The image tag generation method includes the following steps:

[0135] Step S51: Obtain the original video frame f0.

[0136] Step S52: Obtain the rendered video frame fr.

[0137] Step S53: Input the original video frame f0 into the image segmentation network for image segmentation.

[0138] Step S54: Obtain the foreground region (i.e., the target pixel region Ro including the target object Obj) in the original video frame f0 through an image segmentation network.

[0139] Step S55: Calculate the optical flow field of the target object Obj in the original video frame f0.

[0140] Step S56: Calculate the optical flow field of the target object Obj in the rendered video frame fr.

[0141] Step S57: Based on the optical flow field of the target object Obj in the original video frame f0 and the optical flow field of the target object Obj in the rendered video frame fr, the initial pose Pose0 of the target object Obj in the original video frame f0 is calibrated to obtain the calibrated pose Pose1, and the calibrated pose Pose1 is used as the label information corresponding to the original video frame f0.

[0142] It is understood that the execution order of the steps in the above method does not need to follow the step numbers. For example, steps S51 and S52, and steps S55 and S56 can be executed in parallel or in any order.

[0143] In some embodiments, the original video frame f0 and the label information of the target object Obj in the original video frame f0 can be used to train a neural network Net, which can be used to track the target object Obj. For example, in a surgical scenario, video frames from the surgical operation video captured during the actual surgery, along with the label information obtained by the method in the foregoing embodiments, can be used as a dataset to train the neural network Net. During the surgical procedure, the trained neural network Net can be used to track the surgical instrument X. To improve the accuracy of the tracking results, a large dataset is needed to train the neural network Net. The above method can automatically and quickly generate a large dataset based on recorded real surgical operation videos. The generated dataset better reflects the real surgical operation scenario and can improve the quality of neural network Net training.

[0144] In addition, corresponding to the above method, this disclosure also provides an image tag generation apparatus, see [link to relevant documentation]. Figure 12 The device includes:

[0145] The acquisition module 210 is used to acquire the original video frame f0 of the target object Obj and the initial pose Pose0 of the target object Obj when the original video frame f0 is acquired, and to acquire the three-dimensional model Mod of the target object Obj.

[0146] Rendering module 220 is used to render the 3D model Mod based on the initial pose Pose0 to obtain the rendered video frame fr;

[0147] The determination module 230 is used to determine the pose transformation relationship T between the target object Obj in the original video frame f0 and the target object Obj in the rendered video frame fr based on the optical flow field of the target object Obj in the original video frame f0 and the optical flow field of the target object Obj in the rendered video frame fr.

[0148] The calibration module 240 is used to calibrate the initial pose Pose0 based on the pose transformation relationship T to obtain the calibrated pose Pose1.

[0149] The generation module 250 is used to generate the label information of the target object Obj in the original video frame f0 based on the calibrated pose Pose1.

[0150] In some embodiments, the apparatus further includes: an image segmentation module, configured to perform image segmentation on the original video frame f0 to obtain the target pixel region Ro where the target object Obj is located in the original video frame f0; and an optical flow field determination module, configured to determine the optical flow field of the target object Obj in the original video frame f0 based on the target pixel region Ro where the target object Obj is located in the original video frame f0 and the target pixel region Ro where the target object Obj is located in a subsequent video frame of the original video frame f0.

[0151] In some embodiments, the rendering module is specifically used to: project the 3D model Mod onto the coordinate system of the image acquisition device according to the initial pose Pose0 to obtain a projected video frame; and render the projected video frame to obtain a rendered video frame.

[0152] In some embodiments, the target object Obj includes at least one surgical instrument X, each surgical instrument X is held on a robotic arm 101a of the surgical robot, and the robotic arm 101a is equipped with a sensor for acquiring the initial pose Pose0 of the surgical instrument X held on the robotic arm 101a; the original video frame f0 is acquired by an image acquisition device.

[0153] In some embodiments, the three-dimensional model Mod of surgical instrument X corresponds to the type and model of surgical instrument X; the acquisition module is specifically used to: acquire the three-dimensional model Mod of surgical instrument X held on robotic arm 101a according to the type and model of surgical instrument X held on robotic arm 101a.

[0154] In some embodiments, the apparatus further includes: a type and model determination module, configured to determine the type and model of the surgical instruments X held on each robotic arm 101a based on the operation log of the surgical robot; or to determine the type and model of the surgical instruments X held on each robotic arm 101a based on user input.

[0155] In some embodiments, the label information includes the calibration pose Pose1, the type and model of the surgical instrument X.

[0156] In some embodiments, the original video frame f0 and the label information of the target object Obj in the original video frame f0 are used to train a neural network, which is used to track the target object Obj.

[0157] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0158] The embodiments of this disclosure can generate a large number of accurate calibration poses based on noisy initial poses. In related technologies, obtaining accurate pose information requires precise control of the robotic arm or manual annotation, which is costly. By using Embodiment 1 or Embodiment 2 of this disclosure, the accuracy of pose acquisition can be effectively improved while reducing costs.

[0159] Neural network models and target object tracking

[0160] See Figure 8 This disclosure also provides a method for tracking a target object Obj, the method comprising:

[0161] Step S61: Obtain the video frame f including the target object Obj and the initial pose Pose0 of the target object Obj when the video frame f is acquired;

[0162] Step S62: Obtain the 3D model Mod of the target object Obj;

[0163] Step S63: Based on video frame f, obtain the predicted pose information (Pose) of the target object Obj using a pre-trained neural network Net. pre ;

[0164] Step S64: Based on the initial pose Pose0 and the 3D model Mod, obtain the detection pose information Pose of the target object Obj. det ;

[0165] Step S65: Pose the predicted pose information pre And detect pose information Pose det Perform a match and obtain the matching results;

[0166] Step S66: Track the target object Obj based on the matching results.

[0167] In step S61, the target object Obj can be a surgical instrument X or other objects. An image acquisition device can be used to acquire images of the surgical instrument X, resulting in a video frame f including the surgical instrument X. Both the surgical instrument X and the image acquisition device can be held on the robotic arm 101a of the surgical robot. The image acquisition device can also be mounted on a support or fixed in other locations (e.g., a wall, a tabletop, or beside a patient's bed). A pose sensor can be installed on the robotic arm 101a holding the surgical instrument X to acquire the initial pose Pose0 of the surgical instrument X held on the robotic arm 101a when the video frame f is acquired. In one example, the robotic arm 101a includes multiple sequentially connected connecting arms, with adjacent connecting arms connected by rotary joints. The pose sensor can include encoders located at each rotary joint to measure the relative rotation angle between two adjacent connecting arms.

[0168] In step S62, the 3D model Mod of the target object Obj can be obtained. A specific embodiment of step S62 can be found in the preceding description of step S12, and will not be repeated here.

[0169] In step S63, the predicted pose information (Pose) of the target object Obj in video frame f can be obtained through a pre-trained neural network Net. pre The neural network Net can be trained based on sample images and their label information. The sample image can be the second image Img2 from Example 1, and the label information can be obtained using the method described in Example 1. Alternatively, the sample image can be the original video frame f0 from Example 2, and the label information can be obtained using the method described in Example 2.

[0170] The predicted pose information obtained by the neural network Net pre This can include the predicted pixel region, predicted keypoint information, and predicted orientation information of the target object Obj. The predicted pixel region is the pixel region where the target object Obj is located in the video frame. The predicted keypoint information can include the position information of one or more keypoints of the target object Obj in the video frame. The predicted orientation information is used to indicate the attitude of the target object Obj in the video frame and can include the yaw angle, roll angle, and / or pitch angle of the target object Obj.

[0171] In some embodiments, the bounding box of the target object Obj in video frame f can be obtained, and the predicted pose information (Pose) of the target object Obj in video frame f can be obtained based on the bounding box of the target object Obj in video frame f. pre .

[0172] Furthermore, it can be based on video frame f and its preceding video frame f. prior The tracking result of the target object Obj is used to obtain the bounding box of the target object Obj in video frame f. In the earlier video frame f... prior This can include at least one video frame in the video to which video frame f belongs, that precedes video frame f. Assuming video frame f is the m-th video frame (m is a positive integer), then the preceding video frame f... prior It can include at least one of the following: the (m-1)th video frame, the (m-2)th video frame, the (m-3)th video frame, etc., in the video. The preceding video frame f prior The tracking results of the target object Obj can include the previous video frame f prior The detection results of the bounding box (box) of the target object Obj. Since the position of the target object Obj satisfies the constraints of the physical world and will not undergo abrupt changes, the results are based on the previous video frame f. prior By tracking the target object Obj in the video frame f, the bounding box of the target object Obj can be obtained, which can effectively improve the accuracy of the bounding box acquisition.

[0173] After obtaining the bounding box of the target object Obj, the features within the bounding box of the target object Obj in video frame f can be pooled to obtain the predicted pose information Pose of the target object Obj. pre Pooling can be performed using methods such as max pooling or average pooling. Pooling helps in locating the target object during pose estimation. After locating the target object, its bounding box can be restored to its original size before predicting the pose information. pre The calculation.

[0174] In some embodiments, the neural network Net described above may include multiple feature extraction layers l f This is used to extract features from video frame f. Multiple feature extraction layers l f The features output by at least one first feature extraction layer are used to obtain the bounding box (box) of the target object Obj in the video frame. Multiple feature extraction layers l f The features output by at least one of the second feature extraction layers are used to obtain the predicted pose information (Pose) of the target object Obj in the video frame. pre Each first feature extraction layer is located after each second feature extraction layer.

[0175] Figure 9A and Figure 9B The structure of the neural network Net according to an embodiment of this disclosure is shown. It is assumed that the neural network Net includes n-1 (n is a positive integer and n is greater than 1) feature extraction layers l. fThe first feature extraction layer l f Up to the k-th feature extraction layer (k is a positive integer and k is less than n-1) f The output features are used to obtain the predicted pose information (Pose) of the target object Obj in the video frame. pre The (k+1)th feature extraction layer l f up to the (n-1)th feature extraction layer l f The output features are used to obtain the bounding box (box) of the target object Obj in the video frame. This embodiment employs a two-stage model as a neural network Net for bounding box detection and pose estimation. In the first stage, high-level features are used to detect bounding boxes from the video frame, obtaining more feature information and thus improving the accuracy of bounding box detection. In the second stage, low-level features are used to estimate the pose of features within the bounding box. These low-level features are typically associated with the geometric structure in the video frame. By using low-level features, the neural network Net can obtain geometric structure information such as edges and corners from the video frame, improving the accuracy of pose estimation. Furthermore, low-level features generally have better stability and are less affected by factors such as lighting changes and noise, improving the performance stability of pose estimation. Additionally, based on the (k+1)th feature extraction layer l... f up to the (n-1)th feature extraction layer l f Each feature output in the algorithm can detect bounding box information, including the bounding box's geometric information (width and height) and its corresponding confidence score. By applying Non-Maximum Suppression (NMS) to each detected bounding box, the bounding box with the highest confidence score is identified as the bounding box of the target object Obj, while other bounding boxes are filtered out. This effectively improves the accuracy of bounding box detection.

[0176] In some embodiments, such as Figure 9A and Figure 9B As shown, each feature extraction layer l f It includes an encoder and a decoder, and the i-th feature extraction layer l f encoder C i The output is connected to the (i+1)th feature extraction layer l f encoder C i+1 The input terminal and the i-th feature extraction layer l f decoder P i The input terminal; the i-th feature extraction layer l f decoder P i The input terminal is connected to the (i+1)th feature extraction layer l f decoder P i+1The output of ; i is a positive integer. Each encoder is used to downsample the features input to it, and each decoder is used to upsample the features input to it. Each feature encoder and feature decoder can include a convolutional neural network (Net), multiple transformer layers, or multiple pooling layers. Each encoder is used to upsample the features from the previous feature extraction layer l. f The features output by the encoder are downsampled to reduce the feature dimensionality. Each decoder is used to downsample the features from the previous feature extraction layer. f The decoder and the corresponding feature extraction layer l f The features output by the encoder are upsampled to recover the feature dimension, thereby obtaining high-resolution pose prediction results.

[0177] See Figure 9B Each encoder includes an encoder formed by connecting several bottleneck structures of the ResNet network, and an encoder formed by connecting several BasicBlock structures of the ResNet network. Each decoder can be formed by connecting several convolutional layers. For example, the encoder of the first feature extraction layer in the figure is formed by connecting 3 BasicBlock structures, the encoder of the second feature extraction layer is formed by connecting 4 BasicBlock structures, and the encoders of the third and fourth feature extraction layers are each formed by connecting 2 bottleneck structures. The BasicBlock structure and bottleneck structure are shown as ResNet BasicBlock and ResNetBottleneck in the figure, respectively. The "×x" in each rectangle indicates the number of the corresponding structure. For example, in the encoder of the first feature extraction layer, ResNet BasicBlock×3 means that the encoder is formed by connecting 3 BasicBlock structures.

[0178] If the encoder of a feature extraction layer is obtained by connecting bottleneck structures, then the decoder of the same feature extraction layer is obtained by connecting several depthwise separable convolution layers. Figure 9B In the illustrated embodiment, the decoders for both the third and fourth feature extraction layers are formed by connecting two depthwise separable convolutional layers, as shown in the figure (DWSeparableConv). Conv2d in the figure represents a two-dimensional convolution. By adding an encoder connected by a bottleneck structure and a decoder connected by depthwise separable convolutional layers, the number of feature extraction layers can be increased, thereby extracting more features.

[0179] The encoder and decoder of the last feature extraction layer can also include an Atrous Spatial Pyramid Pooling (ASPP) module. The ASPP module first uses multiple parallel convolutional modules to perform dilated convolutions on the features output by the last encoder. Different convolutional modules use different dilation parameters to obtain different receptive fields. The ASPP module shown in the figure uses four parallel convolutional modules, with dilation parameters of 1, 3, 6, and 9 for each module.

[0180] The features output from each convolutional module are then concatenated, and a depthwise separable convolution is performed on the concatenated features. The features obtained from the depthwise separable convolution can be output to the decoder of the last feature extraction layer. By using the ASPP module, the receptive field can be increased, thereby extracting more features.

[0181] The bounding box information obtained after NMS processing can be output to the pooling layer, so that the pooling layer can perform pooling based on the bounding box information on the first feature extraction layer. f Up to the k-th feature extraction layer (k is a positive integer and k is less than n-1) f The output features are pooled. The pooling layer can use ROI alignment to pool the acquired features. The pooled features are used to obtain the target object's mask and its predicted pose information. pre (Including the predicted keypoint locations and poses of the target object). In the diagram, Conv2d and DeConv2d represent 2D convolution and 2D deconvolution processes, respectively; Softmax represents normalization; and Regression represents regression. When determining keypoint locations and poses, the features output by each feature extraction layer can be processed using a swin transformer to effectively extract global features and improve the accuracy of keypoint detection.

[0182] It is understood that the structure shown in the figure is merely an exemplary structure of a neural network. In addition to the structure described above, other neural network structures may be used in the embodiments of this disclosure, which will not be elaborated here.

[0183] In step S64, corresponding to the predicted pose information, the detected pose information may include the detected pixel region, detected key point information, and detected direction information of the target object Obj.

[0184] In some embodiments, the 3D model Mod can be projected onto the coordinate system of the image acquisition device according to the initial pose Pose0, and the detection key point information and detection direction information of the target object Obj in the coordinate system of the image acquisition device can be obtained. Alternatively, the 3D model Mod can be projected onto the 2D image plane corresponding to the video frame according to the initial pose Pose0, and the detection pixel area of ​​the target object Obj in the 2D image plane can be obtained.

[0185] The initial pose Pose0 represents the relative pose between the 3D model Mod and the image acquisition device. Based on Pose0, the 3D key points on the 3D model Mod can be projected into the coordinate system of the image acquisition device to obtain key point detection information and detection direction information. Furthermore, based on Pose0, the 3D model Mod can also be projected onto the 2D image plane corresponding to the video frame, and the image projected onto the 2D image plane can be detected using the aforementioned neural network Net to obtain the detection pixel region.

[0186] In step S65, the predicted pixel region and the detected pixel region, the predicted key point information and the detected key point information, and the predicted direction information and the detected direction information can be matched respectively to obtain the pixel region matching result, key point matching result, and direction information matching result. For example, bipartite matching can be used to match the above information.

[0187] In step S66, a first confidence level can be determined based on the pixel region matching result to match the predicted pixel region with the detected pixel region, a second confidence level can be determined based on the key point matching result to match the predicted key point information with the detected key point information, and a third confidence level can be determined based on the direction information matching result to match the predicted direction information with the detected direction information. Based on the first confidence level, the second confidence level, and the third confidence level, the target object Obj is tracked.

[0188] For example, a weighted average can be calculated from the first, second, and third confidence levels to obtain a weighted average confidence level. If the weighted average confidence level is greater than a preset confidence threshold, a successful match is determined, and the initial pose of the target object Obj is calibrated based on the predicted pose information, with the calibration result saved. If the weighted average confidence level is less than or equal to the preset confidence threshold, a failed match is determined.

[0189] The following example uses a surgical scenario, combined with... Figure 10A The overall flow of the method for tracking a target object Obj in an image according to embodiments of this disclosure will be described. The method for tracking the target object Obj includes the following steps:

[0190] Step S71: Acquire video frame f including surgical instrument X.

[0191] Step S72: Perform bounding box detection on video frame f to obtain the bounding box of surgical instrument X in video frame f.

[0192] Step S73: Based on the bounding box obtained in step S72, predict the pose of the surgical instrument X in video frame f to obtain the predicted pose information Pose. pre .

[0193] Step S74: Obtain detection pose information Pose based on the 3D model Mod of surgical instrument X and the initial pose Pose0 of surgical instrument X. det And predict pose information Pose pre And detect pose information Pose det Perform bipartite graph matching.

[0194] Step S75: Based on the matching results and confidence level, calibrate the initial pose Pose0 to obtain the calibrated pose Pose1.

[0195] This embodiment of the present disclosure obtains the predicted pose information of the target object through a neural network, obtains the detected pose information of the target object based on the initial pose and the three-dimensional model of the target object, and matches the predicted pose information and the detected pose information. The target object is then tracked according to the matching result, which can effectively improve the tracking accuracy.

[0196] The tracking method of this disclosure can be used to track a single target object Obj or multiple target objects Obj.

[0197] The following is combined with Figure 10B The example shown illustrates the multi-object tracking process. In this example, three surgical instruments X (shown as X1, X2, and X3 in the figure) need to be tracked. Therefore, after inputting the video frame f into the neural network, three sets of predicted pose information (Pose) are obtained. pre (as shown in the pose) pre，1 Pose pre，2 and Pose pre，3 (As shown). Based on the kinematic data (including initial pose) of surgical instruments X1, X2, and X3, and the 3D model, the detection pose information of each surgical instrument X1, X2, and X3 can be obtained, denoted as Pose. det，X1 Pose det，X2 and Pose det , X3 For the three sets of predicted pose information {Pose} pre，1 Posepre，2 and Pose pre，3 Each of the three sets of detection pose information {Pose} det , X1 Pose det，X2 and Pose det，X3 Each element in the sequence is subjected to bipartite graph matching, and the configuration confidence is calculated. The matching confidence is compared with a preset confidence threshold to determine the matching result. For example, if the matching confidence is greater than the preset confidence threshold, the matching is considered successful, and the initial pose is calibrated based on the predicted position information. In one example, when the predicted pose information is Pose... pre，1 With detection pose information Pose det，X1 If a match is successful, the predicted pose information (Pose) is determined. pre，1 The predicted pose information for surgical instrument X1 is used, and the predicted pose information is used to pose. pre，1 To calibrate the initial pose of the surgical instrument X1, for example, the predicted pose information (Pose) can be saved. pre，1 The pose offset from the initial pose. The processing of the other two surgical instruments X2 and X3 is similar, and will not be elaborated here. Finally, the tracking results of surgical instruments X1, X2, and X3 can be obtained, as shown in the Pose diagram. X1 Pose X2 and Pose X3 As shown.

[0198] Corresponding to the above method, this disclosure also provides a tracking device for a target object Obj, see [link to relevant documentation]. Figure 13 The device includes:

[0199] The first acquisition module 310 is used to acquire video frame f of the target object Obj and the initial pose Pose0 of the target object Obj when acquiring video frame f, and to acquire the three-dimensional model Mod of the target object Obj.

[0200] The second acquisition module 320 is used to acquire the predicted pose information (Pose) of the target object Obj based on video frame f using a pre-trained neural network Net. pre ;

[0201] The third acquisition module 330 is used to acquire the detection pose information Pose of the target object Obj based on the initial pose Pose0 and the 3D model Mod. det ;

[0202] Matching module 340 is used to match the predicted pose information Pose pre And detect pose information Pose det Perform a match and obtain the matching results;

[0203] The tracking module 350 is used to track the target object Obj based on the matching results.

[0204] In some embodiments, the predicted pose information is Pose. pre This includes the predicted pixel region of the target object, predicted key point information, and predicted direction information, as well as the detected pose information (Pose). det The system includes the detection pixel region, detection key point information, and detection direction information of the target object. The matching module is specifically used to match the predicted pixel region and the detected pixel region, the predicted key point information and the detected key point information, and the predicted direction information and the detection direction information, respectively.

[0205] In some embodiments, the matching result includes pixel region matching result, key point matching result, and direction information matching result. The tracking module is specifically used to: determine a first confidence level that the predicted pixel region matches the detected pixel region based on the pixel region matching result; determine a second confidence level that the predicted key point information matches the detected key point information based on the key point matching result; determine a third confidence level that the predicted direction information matches the detected direction information based on the direction information matching result; and track the target object based on the first confidence level, the second confidence level, and the third confidence level.

[0206] In some embodiments, the third acquisition module is specifically used to: project the three-dimensional model Mod onto the coordinate system of the image acquisition device according to the initial pose Pose0, and acquire the detection key point information and detection direction information of the target object Obj in the coordinate system of the image acquisition device, wherein the image acquisition device is used to acquire the video frame f; project the three-dimensional model Mod onto the two-dimensional image plane corresponding to the video frame f according to the initial pose Pose0, and acquire the detection pixel area of ​​the target object Obj in the two-dimensional image plane.

[0207] In some embodiments, the second acquisition module is specifically used to: acquire the bounding box of the target object Obj in the video frame f; and acquire the predicted pose information Pose of the target object Obj in the video frame f based on the bounding box of the target object Obj in the video frame f. pre .

[0208] In some embodiments, the neural network Net includes multiple feature extraction layers l f The multiple feature extraction layers l are used to extract features from the video frame f. fThe features output by at least one first feature extraction layer in the plurality of feature extraction layers are used to obtain the bounding box of the target object in the video frame f; f The features output by at least one second feature extraction layer in the video frame f are used to obtain the predicted pose information of the target object Obj in the video frame f; wherein each first feature extraction layer is located after each second feature extraction layer.

[0209] In some embodiments, each feature extraction layer includes an encoder and a decoder. The output of the encoder of the i-th feature extraction layer is connected to the input of the encoder of the (i+1)-th feature extraction layer and the input of the decoder of the i-th feature extraction layer. The input of the decoder of the i-th feature extraction layer is connected to the output of the decoder of the (i+1)-th feature extraction layer. i is a positive integer. Each encoder is used to downsample the features input to the encoder, and each decoder is used to upsample the features input to the decoder.

[0210] In some embodiments, the neural network Net obtains the bounding box of the target object Obj in the video frame f based on the following method: obtaining the bounding box of the target object Obj in the video frame f based on the tracking results of the target object Obj in the video frame f and the previous video frames of the video frame f.

[0211] In some embodiments, the neural network Net obtains the predicted pose information (Pose) of the target object Obj in the video frame f based on the following method. pre Pooling is performed on the features within the bounding box of the target object Obj in the video frame f to obtain the predicted pose information Pose of the target object Obj. pre .

[0212] In some embodiments, the target object Obj includes at least one surgical instrument X, each surgical instrument X is held on a robotic arm 101a of a surgical robot, and the robotic arm 101a is provided with a sensor for acquiring the initial pose Pose0 of the surgical instrument X held on the robotic arm 101a; the video frame f is acquired by an image acquisition device.

[0213] In some embodiments, the three-dimensional model Mod of the surgical instrument X corresponds to the type and model of the surgical instrument X; the first acquisition module is specifically used to: acquire the three-dimensional model Mod of the surgical instrument X held on the robotic arm 101a according to the type and model of the surgical instrument X held on the robotic arm 101a.

[0214] In some embodiments, the apparatus further includes: a type and model acquisition module, configured to determine the type and model of the surgical instruments X held on each robotic arm 101a based on the operation log of the surgical robot; or to determine the type and model of the surgical instruments X held on each robotic arm 101a based on user input.

[0215] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0216] This disclosure also provides a computing device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the foregoing embodiments.

[0217] Figure 14 A more specific hardware structure diagram of a computing device 400 provided in this disclosure embodiment is shown. The device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, memory 420, input / output interface 430, and communication interface 440 are internally connected to each other via the bus 450.

[0218] The processor 410 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The processor 410 may also include a graphics card, such as an Nvidia Titan X graphics card or a 10120Ti graphics card.

[0219] The memory 420 can be implemented in the form of read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this disclosure are implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0220] Input / output interface 430 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0221] The communication interface 440 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0222] Bus 450 includes a pathway for transmitting information between various components of the device (e.g., processor 410, memory 420, input / output interface 430, and communication interface 440).

[0223] It should be noted that although the above-described device only shows the processor 410, memory 420, input / output interface 430, communication interface 440, and bus 450, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this disclosure, and not necessarily all the components shown in the figures.

[0224] See Figure 15 This disclosure also provides an image tag generation system, the system comprising:

[0225] Image acquisition device 510 is used to acquire the first image Img1 or the original video frame f0 of the target object;

[0226] Pose sensor 520 is used to acquire the initial pose Pose0 of target object Obj when the first image Img1 or the original video frame f0 is captured; and

[0227] The computing device 400 in the foregoing embodiments.

[0228] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0229] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0230] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computing device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0231] The systems, devices, modules, or units described in the above embodiments can be implemented by computer devices or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0232] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this disclosure, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0233] The above description is merely a specific implementation of the embodiments of this disclosure. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered within the protection scope of the embodiments of this disclosure.

Claims

1. A method for generating labels for images, characterized in that, The method includes: Acquire a first image of a target object and the initial pose of the target object when the first image is acquired; the target object includes at least one surgical instrument, each surgical instrument is held on a robotic arm of a surgical robot, and the robotic arm is equipped with a sensor for detecting the initial pose of the surgical instrument held on the robotic arm; the first image is obtained by acquiring an image of the surgical instrument under a first preset background. Obtain the three-dimensional model of the target object; Based on the initial pose, the 3D model is projected onto the first image to obtain the projected pixel region; Based on the overlap between the projected pixel region and the target pixel region where the target object is located in the first image, the initial pose is calibrated to obtain the calibrated pose. The first preset background in the first image is replaced with a second preset background to obtain a second image including the target object; the difference between the pixel value of the first preset background and the pixel value of the surgical instrument is greater than a preset value, and the second preset background is the background of the surgical scene; The label information of the target object in the second image is generated based on the calibration pose.

2. The method according to claim 1, characterized in that, The method further includes: The second image is post-processed; the post-processing includes at least one of the following: blurring, sharpening, noise reduction, and enhancement.

3. The method according to claim 1, characterized in that, The calibration pose is the pose of the target object when the overlap is maximized.

4. The method according to claim 3, characterized in that, The step of calibrating the initial pose based on the overlap between the projected pixel region and the target pixel region where the target object is located in the first image to obtain the calibrated pose includes: After optimizing the initial pose using a preset pose optimization algorithm, the overlap between the projected pixel region and the target pixel region is recalculated. The pose corresponding to the projected pixel region with the greatest overlap with the target pixel region is determined as the calibration pose.

5. The method according to claim 1, characterized in that, The overlap between the projected pixel region and the target pixel region is determined based on the IoU, GIoU, or dice loss between the projected pixel region and the target pixel region.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the mask of the target object in the first image; The overlap is determined based on the mask of the target object and the projected pixel region.

7. The method according to claim 6, characterized in that, Before determining the overlap based on the mask of the target object and the projected pixel region, the method further includes: The mask is then smoothed.

8. The method according to claim 1, characterized in that, The three-dimensional model of the surgical instrument corresponds to the type and model of the surgical instrument; obtaining the three-dimensional model of the target object includes: Based on the type and model of the surgical instruments held on the robotic arm, obtain a three-dimensional model of the surgical instruments held on the robotic arm.

9. The method according to claim 8, characterized in that, The method further includes: The type and model of the surgical instruments held on each robotic arm are determined based on the operation logs of the surgical robot; or The type and model of the surgical instruments held on each robotic arm are determined based on user input.

10. The method according to claim 1, characterized in that, The label information includes the calibration pose, the type and model of the surgical instrument.

11. The method according to any one of claims 1 to 9, characterized in that, The second image and the label information of the target object in the second image are used to train a neural network, which is used to track the target object.

12. An image tag generation device, characterized in that, The device includes: An acquisition module is used to acquire a first image of a target object and the initial pose of the target object when the first image is acquired, and to acquire a three-dimensional model of the target object; the target object includes at least one surgical instrument, each surgical instrument is held on a robotic arm of a surgical robot, and the robotic arm is equipped with a sensor for detecting the initial pose of the surgical instrument held on the robotic arm; the first image is obtained by acquiring an image of the surgical instrument under a first preset background; The projection module is used to project the three-dimensional model of the target object onto the first image based on the initial pose to obtain the projected pixel area; The calibration module is used to calibrate the initial pose based on the overlap between the projected pixel region and the target pixel region where the target object is located in the first image, so as to obtain the calibrated pose. The generation module is used to replace the first preset background in the first image with a second preset background to obtain a second image including the target object; the difference between the pixel value of the first preset background and the pixel value of the surgical instrument is greater than a preset value, and the second preset background is the background of the surgical scene; The determination module is used to determine the label information of the target object in the second image based on the calibration pose.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1 to 11.

14. A computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 11.

15. An image tag generation system, characterized in that, The system includes: An image acquisition device is used to acquire a first image of a target object; A pose sensor is used to acquire the initial pose of the target object when the first image is captured; and The computing device according to claim 14.

16. The system according to claim 15, characterized in that, The target object is a surgical instrument; the system also includes: A surgical robot includes at least one robotic arm, each robotic arm being used to hold a surgical instrument, and each robotic arm being provided with the pose sensor.

17. The system according to claim 15, characterized in that, The target object is a surgical instrument; the system also includes: A surgical robot includes at least two robotic arms, wherein the image acquisition device and the surgical instruments are held by different robotic arms, and at least the robotic arm holding the surgical instruments is provided with the pose sensor.

Images