Surgical robot system and surgical robot punching guiding method and device

Abstract

The invention relates to a surgical robot system and a surgical robot punching guiding method and device, and the method comprises the steps: obtaining first image data of a focus region, and building a three-dimensional model of the focus region according to the first image data; and acquiring second image data of the working environment of the scanning equipment, and determining the hole position of the holding arm under the visual equipment coordinate system according to the three-dimensional model of the focus area and the second image data. In the whole process, a doctor can visually observe the punching position and the focus area of the instrument holding arm through the visual device, and therefore accurate punching guidance of the surgical robot is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of robot-assisted surgery, and in particular to a surgical robot system, a surgical robot drilling guidance method, a device, a storage medium, and a computer program product. Background Art

[0002] With the development of science and technology, minimally invasive surgical robots are increasingly being used in minimally invasive surgeries. The design concept of surgical robots is to use minimally invasive methods to accurately perform complex surgical operations. Surgical instruments enter the body cavity through specific holes on the body surface, approach the lesion, and the doctor controls the surgical robot to complete the operation. The holes on the patient's body surface for surgical instruments to pass through are usually planned by the doctor before the operation based on the lesion and surgical procedure, and the doctor drills the holes based on experience.

[0003] However, it is difficult to combine the doctor's experience with the configuration and workspace of the minimally invasive surgical robot, resulting in the inability of traditional surgical robot systems to achieve accurate drilling guidance. Summary of the invention

[0004] Based on this, it is necessary to provide a surgical robot system, surgical robot drilling guidance method, device, storage medium and computer program product that can achieve accurate drilling guidance in order to address the technical problem that the surgical robot system in the prior art cannot achieve accurate drilling guidance.

[0005] In a first aspect, the present application provides a surgical robot system, including a scope holding arm, and:

[0006] A scanning unit, used for acquiring first image data of a lesion area;

[0007] A modeling unit, used for establishing a three-dimensional model of the lesion area according to the first image data;

[0008] A visual unit, used for acquiring second image data of a working environment of a scanning device, wherein the scanning device is a scanning device that scans and generates the first image data of the lesion area;

[0009] A processing unit is used to determine the position of a first hole position in the visual device coordinate system according to the three-dimensional model of the lesion area and the second image data, wherein the first hole position is the hole position of the mirror holding arm.

[0010] In one embodiment, the processing unit determines the position of the first hole in the visual device coordinate system according to the three-dimensional model of the lesion area and the second image data, including:

[0011] Acquiring a first pose relationship between the scanning device and the lesion area;

[0012] Acquire a second posture relationship between the scanning device and the visual device;

[0013] According to the first posture relationship and the second posture relationship, obtaining position information of the three-dimensional model of the lesion area in the visual device coordinate system;

[0014] According to the position information of the three-dimensional model of the lesion area in the visual device coordinate system, the position of the first hole in the visual device coordinate system is determined.

[0015] In one embodiment, the processing unit is further used to obtain the pre-drilling position of the lesion area; and determining the position of the first hole in the visual device coordinate system according to the position information of the three-dimensional model of the lesion area in the visual device coordinate system includes:

[0016] According to the position of the three-dimensional model of the lesion area in the visual device coordinate system, the pre-drilling position is corrected to obtain the position of the first hole in the visual device coordinate system.

[0017] In one embodiment, the surgical robot system further includes a robotic arm, and the processing unit is further configured to obtain a position of a second hole according to a position of the first hole, wherein the position of the second hole is a hole position of the robotic arm.

[0018] In one embodiment, acquiring the position of the second hole position according to the position of the first hole position comprises:

[0019] Obtaining a preset distance between the first hole position and the second hole position;

[0020] The position of the second hole is acquired according to the preset distance and the position of the first hole.

[0021] In one embodiment, obtaining the position of the second hole position according to the preset distance and the position of the first hole position includes:

[0022] Determining the position of the end of the robotic arm according to the preset distance, the position of the first hole and the lesion area;

[0023] Performing inverse kinematics solution on the robotic arm according to the end posture of the robotic arm;

[0024] If the inverse kinematics has a solution, the contact point between the line connecting the end posture of the robotic arm and the lesion area and the body surface is determined as the position of the second hole.

[0025] In one embodiment, determining the position of the end of the robotic arm according to the preset distance, the position of the first hole and the lesion area includes:

[0026] Determining the axis of the sleeve of the robotic arm according to the preset distance, the position of the first hole, and the lesion area;

[0027] Based on the axis of the sleeve of the robotic arm, the rotational freedom of the robotic arm sleeve around itself is selected to determine the middle position of the joint travel;

[0028] According to the middle position of the joint travel, the end position of the robotic arm is determined.

[0029] In one embodiment, there are multiple preset distances; and obtaining the position of the second hole according to the preset distance and the position of the first hole includes:

[0030] Determining the positions of a plurality of initial second hole positions according to a plurality of preset distances and the positions of the first hole positions;

[0031] Calculating the arm distances between the mirror holding arm and the mechanical holding arm in a plurality of initial second hole positions;

[0032] The hole position at which the arm distance between the mirror holding arm and the mechanical holding arm is the largest is determined as the second hole position.

[0033] In one embodiment, the calculating the arm spacing between the mirror holding arm and the robot holding arm in the plurality of initial second hole positions comprises:

[0034] Performing inverse kinematics solution on the robotic arm according to the end posture of the robotic arm to obtain the target posture of the robotic arm;

[0035] The arm distances between the mirror holding arm and the robot arm in a plurality of initial second hole positions are calculated according to the target posture of the robot arm and the posture of the mirror holding arm.

[0036] In a second aspect, the present application also provides a surgical robot drilling guidance method, the method comprising:

[0037] Acquiring first image data of a lesion area;

[0038] Establishing a three-dimensional model of the lesion area according to the first image data;

[0039] Acquire second image data of a working environment of a scanning device, wherein the scanning device is a scanning device that scans and generates the first image data of the lesion area;

[0040] According to the three-dimensional model of the lesion area and the second image data, the position of the first hole position in the visual device coordinate system is determined, and the first hole position is the hole position of the mirror holding arm.

[0041] In one embodiment, determining the position of the first hole in the visual device coordinate system according to the three-dimensional model of the lesion area and the second image data includes:

[0042] Acquiring a first pose relationship between the scanning device and the lesion area;

[0043] Acquire a second posture relationship between the scanning device and the visual device;

[0044] According to the first posture relationship and the second posture relationship, obtaining position information of the three-dimensional model of the lesion area in the visual device coordinate system;

[0045] According to the position information of the three-dimensional model of the lesion area in the visual device coordinate system, the position of the first hole in the visual device coordinate system is determined.

[0046] In one embodiment, the surgical robot drilling guidance method further includes:

[0047] Obtaining a preset distance between a first hole position and a second hole position, where the second hole position is a hole position of the robotic arm;

[0048] Determining the position of the end of the robotic arm according to the preset distance and the lesion area;

[0049] Performing inverse kinematics solution on the robotic arm according to the end posture of the robotic arm;

[0050] If the inverse kinematics has a solution, the contact point between the line connecting the end posture of the robotic arm and the lesion area and the body surface is determined as the position of the second hole.

[0051] In a third aspect, the present application also provides a surgical robot drilling guidance device, including a scanning component, a visual component, and a control component;

[0052] The scanning component generates first image data of the lesion area, and sends the first image data to the control component;

[0053] The control component uses the surgical robot drilling guidance method as described above to generate the position of the first hole in the visual component.

[0054] In one embodiment, the scanning component includes an ultrasonic probe, and a visual target is provided on the ultrasonic probe; the visual target emits a cursor signal to illuminate the lesion area;

[0055] The visual component receives the cursor signal on the lesion area and sends the position information of the cursor signal to the control component.

[0056] In one embodiment, the visual component includes:

[0057] An imaging element, collecting the cursor signal on the lesion area, and sending the position information of the cursor signal to the control component;

[0058] An angle adjustment element, carrying the imaging element and connected to the control component, wherein the control component controls the angle adjustment element to adjust the image tracking angle of the imaging element;

[0059] The mixed reality component displays the hole position in the visual component coordinate system pushed by the control component.

[0060] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0061] Acquiring first image data of a lesion area;

[0062] Establishing a three-dimensional model of the lesion area according to the first image data;

[0063] Acquire second image data of a working environment of a scanning device, wherein the scanning device is a scanning device that scans and generates the first image data of the lesion area;

[0064] According to the three-dimensional model of the lesion area and the second image data, the position of the first hole position in the visual device coordinate system is determined, and the first hole position is the hole position of the mirror holding arm.

[0065] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0066] Acquiring first image data of a lesion area;

[0067] Establishing a three-dimensional model of the lesion area according to the first image data;

[0068] Acquire second image data of a working environment of a scanning device, wherein the scanning device is a scanning device that scans and generates the first image data of the lesion area;

[0069] According to the three-dimensional model of the lesion area and the second image data, the position of the first hole position in the visual device coordinate system is determined, and the first hole position is the hole position of the mirror holding arm.

[0070] The above-mentioned surgical robot system, surgical robot drilling guidance method, device, storage medium and computer program product obtain the first image data of the lesion area, and establish a three-dimensional model of the lesion area based on the first image data; obtain the second image data of the working environment of the scanning device, and determine the hole position of the mechanical arm in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data. During the whole process, the doctor can intuitively observe the drilling position of the mechanical arm and the lesion area through the visual device, thereby realizing accurate surgical robot drilling guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic diagram of an application scenario of the surgical robot system of the present application in one embodiment;

[0072] Figure 2 is a structural block diagram of a surgical robot system in one embodiment;

[0073] Figure 3 A schematic diagram of the relative relationship between a scanning device, a visual device and a lesion area in one embodiment;

[0074] Figure 4 A schematic diagram of the relative positions of the mechanical arm and the scope arm in a surgical robot system in one embodiment;

[0075] Figure 5 A schematic diagram of obtaining the relative position relationship between the lesion point and the scope holding arm in one embodiment;

[0076] Figure 6 A schematic diagram of a process for determining a pre-punching position in a specific application example;

[0077] Figure 7 A schematic diagram of a flow chart of a surgical robot drilling guidance method in one embodiment;

[0078] Figure 8 A schematic diagram of the structure of a drilling guidance device for a surgical robot in one embodiment;

[0079] Fig. 9 Schematic diagram of the structure of a scanning component and a visual component in one embodiment. DETAILED DESCRIPTION

[0080] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0081] In one embodiment, if Figure 1As shown, a surgical robot system is provided, and the surgical robot system includes a mirror holding arm. This embodiment is explained by taking the surgical robot system used to assist doctors in guiding the drilling positions as an example. In this embodiment, the surgical robot system also includes a scanning unit, a modeling unit, a visual unit and a processing unit, and these system units can be integrated in one or more main consoles. In actual applications, the surgical robot system also includes a scanning device and a visual device, and one or more main consoles integrating the above-mentioned system units are respectively connected to the scanning device and the visual device. The patient lies flat on the operating bed, and the scanning device collects the medical image of the patient and sends the collected medical image to the above-mentioned computer device. The above-mentioned main console processes the hole position of the mirror holding arm in the coordinate system of the visual device, and sends the data to the visual device. The doctor wears the visual device, and the visual device displays the hole position of the mirror arm, so that the entire surgical robot system can achieve accurate drilling guidance.

[0082] Specifically, if Figure 2 As shown, in one embodiment, the surgical robot system includes:

[0083] The scanning unit 200 is used to obtain first image data of the lesion area.

[0084] The scanning device scans and images the scanned object to obtain medical image data, and the scanning unit 200 analyzes the medical image data to obtain the first image data of the lesion area. Specifically, the scanned object refers to the object that the surgical robot punches this time. Taking the punched object as a patient as an example, the scanned object is the surgical patient, and the medical image of the patient is scanned by the scanning device. The lesion area refers to the area on the body of the object that the surgical robot punches this time where the lesion occurs. For example, it can be the location where the lesion occurs in the kidney or gallbladder in the abdominal cavity of the scanned object. Further, the scanning unit 200 can first extract the image of the preset scanning area from the medical image data sent by the scanning device. The preset scanning area refers to the area corresponding to the surgery required this time, which is specifically a scanning area pre-delineated on the scanned object, for example, it can be the abdomen, chest, back and other areas of the scanned object, for example, through the image data of the patient's abdomen.

[0085] The modeling unit 400 is used to establish a three-dimensional model of the lesion area according to the first image data.

[0086] The first image data is used as basic data, and a virtual three-dimensional model is constructed by virtual three-dimensional modeling, and the corresponding lesion area in the patient's body is identified on the virtual three-dimensional model. In this way, it is convenient to calculate and plan the drilling position for the lesion area in the virtual three-dimensional model in subsequent processing.

[0087] The visual unit 600 is used to obtain second image data of the working environment of a scanning device, where the scanning device is a scanning device that scans and generates first image data of a lesion area.

[0088] The working environment of the scanning device refers to the scene environment when the scanning device scans the patient to generate medical image data. Specifically, as mentioned above, the scanning device scans the patient to generate medical images, and at the same time, the image data of the scanning device when performing the scanning work is collected, so that the phase posture relationship with the scanning device can be accurately located later. Furthermore, in order to facilitate the accurate determination of the relative posture relationship between the visual device and the scanning device in the subsequent period, so as to realize the accurate presentation of the hole position of the robotic arm in the visual device.

[0089] The processing unit 800 is used to determine the position of the first hole position in the visual device coordinate system according to the three-dimensional model of the lesion area and the second image data, where the first hole position is the hole position of the mirror holding arm.

[0090] The visual device can assist the doctor to observe the surrounding environment intuitively, and can present the specific drilling position on it so that the doctor can view the structure. For example, the visual device can be a mixed reality helmet (MR helmet for short) worn by the doctor. Specifically, the hole position of the mirror holding arm in the surgical robot system can be determined based on the three-dimensional model of the lesion area. At this time, the hole position of the mirror holding arm is constructed in the scanning device coordinate system. In order to accurately map the hole position to the visual device coordinate system, the relative posture relationship between the scanning device and the visual device is further determined at this time, that is, the point mapping relationship between the scanning device coordinate system and the visual device coordinate system is clarified. On the basis of determining the posture relationship, the hole position of the mirror holding arm determined based on the three-dimensional model of the lesion area is mapped to the visual device coordinate system to determine the hole position of the mirror holding arm in the visual device coordinate system.

[0091] Furthermore, when the hole position of the surgical arm is determined based on the three-dimensional model of the lesion area, the drilling position is pre-calculated in the virtual three-dimensional model based on the lesion location in the patient's body and combined with the configuration of the surgical robot system. For example, the lesion is located on an organ in the abdominal area, and the drilling position corresponds to the body surface of the abdomen. After determining the hole position of the scope arm in the visual device coordinate system, the data can be pushed to the visual device so that the doctor can display the accurate hole position of the scope arm in the display interface of the identification device after wearing the visual device, so that the doctor can intuitively observe the drilling position, thereby achieving accurate drilling guidance.

[0092] The above surgical robot system obtains the first image data of the lesion area, and builds a three-dimensional model of the lesion area based on the first image data; obtains the second image data of the working environment of the scanning device, and determines the hole position of the robotic arm in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data. During the whole process, the doctor can visually observe the drilling position of the robotic arm and the lesion area through the visual device, thereby realizing accurate surgical robot drilling guidance.

[0093] In one of the embodiments, the processing unit 800 determines the position of the first hole in the visual device coordinate system based on the three-dimensional model of the lesion area and the second image data, including: obtaining a first pose relationship between the scanning device and the lesion area; obtaining a second pose relationship between the scanning device and the visual device; obtaining position information of the three-dimensional model of the lesion area in the visual device coordinate system based on the first pose relationship and the second pose relationship; determining the position of the first hole in the visual device coordinate system based on the position information of the three-dimensional model of the lesion area in the visual device coordinate system.

[0094] The scanning device refers to a device used to scan a patient and generate an image of a preset scanning area, for example, it can be any one of an ultrasonic imaging device, an X-ray device, and a magnetic resonance imaging device (MRI device for short). In this embodiment, the scanning device is an ultrasonic imaging device. The first posture relationship between the scanning device and the lesion area corresponding to the scanned object refers to the posture relationship between the scanning device and the lesion area of ​​the scanned object. For example, it can be the relationship between the relative position and posture between the ultrasonic probe of the ultrasonic imaging device and the lesion area of ​​the patient.

[0095] The relative position and posture relationship between the scanning device and the visual device refers to the relationship between the relative position and posture between the structure of the scanning device used to scan the scanning area pre-demarcated on the patient's body and the visual device. For example, it can be the relationship between the position and posture between the ultrasound probe of the ultrasound imaging device and the MR helmet. The second position and posture relationship between the scanning device and the visual device refers to the relationship between the relative position and posture between the scanning device and the visual device. For example, it can be the relationship between the position and posture between the ultrasound probe of the ultrasound imaging device and the MR helmet. By combining the first position and the second position and posture relationship, it is possible to map the position in the three-dimensional model of the lesion area to the visual device coordinate system, and then the position of the mirror holding arm hole determined in the three-dimensional model of the lesion area can be mapped to the visual device coordinate system, and the hole position of the mirror holding arm in the visual device coordinate system is obtained.

[0096] For example, refer to Figure 3It is understood that the processing unit 800 can obtain a first posture relationship based on the relative posture relationship between the ultrasound probe on the ultrasound imaging device and the corresponding lesion area of ​​the patient, and the first posture relationship is the relative posture relationship between the probe and the lesion area. The processing unit 800 can also obtain a second posture relationship based on the relative posture relationship between the probe on the ultrasound imaging device and the MR helmet, and the second posture relationship is the posture relationship between the probe and the MR helmet. The processing unit 800 can map the three-dimensional model of the lesion area to the visual device coordinate system based on the relationship between the first posture relationship and the second posture relationship, and then determine the hole position of the mirror holding arm in the visual device coordinate system based on the hole position of the mirror holding arm determined in the three-dimensional model of the lesion area.

[0097] In this embodiment, the posture relationship between the lesion area and the MR helmet is obtained by converting the posture relationship between the lesion area, the scanning device and the visual device, so as to realize the conversion of the position information of the lesion area into the position information in the visual device coordinate system. It should be noted that the posture includes both position information and posture information, so the posture relationship also includes the relationship between positions and the relationship between postures.

[0098] It should be noted that, in this embodiment, the visual device may include a depth camera and a DOF gimbal, the depth camera may detect three-dimensional information of the environment, and the DOF gimbal may adjust the posture of the depth camera. For example, the angle of the depth camera may be adjusted by rotating the DOF gimbal, thereby adjusting the position of the field of view of the depth camera. Since the depth camera can only detect three-dimensional information of the surrounding environment and cannot detect intracavitary information of the patient, the present embodiment obtains the position information of the probe in the scanning device through the depth camera, and the position information of the lesion area can be obtained through the probe, and then the posture relationship between the depth camera and the lesion area can be calculated based on the probe in the scanning device.

[0099] In one of the embodiments, the processing unit 800 is also used to obtain the pre-punching position of the lesion area; determining the position of the first hole in the visual device coordinate system based on the position information of the three-dimensional model of the lesion area in the visual device coordinate system includes: correcting the pre-punching position according to the position of the three-dimensional model of the lesion area in the visual device coordinate system to obtain the position of the first hole in the visual device coordinate system.

[0100] The pre-punching position of the lesion area refers to the pre-determined punching position of the scope arm based on the three-dimensional model of the lesion area before the operation. In actual applications, an inspection will be conducted before the operation to preliminarily determine the punching position. During the actual operation, in order to further improve the accuracy of the operation, the preset punching position is corrected here. After the correction is completed in the three-dimensional model of the lesion area, the position of the first hole in the visual device coordinate system is obtained according to the position of the three-dimensional model of the lesion area in the visual device coordinate system.

[0101] In one embodiment, the surgical robot system further includes a robotic arm, and the processing unit is further configured to obtain a position of a second hole according to a position of the first hole, the position of the second hole being the hole position of the robotic arm.

[0102] like Figure 4 As shown, the surgical robot system includes a scope holding arm 420 and a tool holding arm 410, wherein the scope holding arm 420 is fixedly connected to an endoscope 422 and drives the endoscope 422 to move, and the tool holding arm 410 is fixedly connected to a surgical instrument 412 and drives the surgical instrument 412 to move. After the hole position of the scope holding arm 420 is determined, the hole position of the tool holding arm 410 needs to be further determined based on the hole position of the scope holding arm 420. Specifically, since a relative distance relationship is generally maintained between the scope holding arm 420 and the tool holding arm 410, after the hole position of the scope holding arm 420 is obtained, the hole position of the tool holding arm 410 can be determined based on the relative distance.

[0103] In one embodiment, obtaining the position of the second hole position according to the position of the first hole position includes: obtaining a preset distance between the first hole position and the second hole position; and obtaining the position of the second hole position according to the preset distance and the position of the first hole position.

[0104] The preset distance is a predetermined distance, which can be obtained based on prior experience. Specifically, the minimum punching distance and the maximum punching distance between the robot arm and the mirror holding arm can be obtained based on prior experience, and then the appropriate preset distance is determined based on the movement trajectory, size parameters, patient size parameters, etc. of the robot arm and the mirror holding arm in the minimum punching distance and the maximum punching distance. More specifically, in a certain application scenario, the minimum punching distance between the robot arm and the mirror holding arm can be 6 cm, and the maximum punching distance between the robot arm and the mirror holding arm can be 10 cm.

[0105] In one embodiment, obtaining the position of the second hole according to the preset distance and the position of the first hole includes: determining the posture of the end of the robotic arm according to the preset distance, the position of the first hole and the lesion area; solving the inverse kinematics of the robotic arm according to the posture of the end of the robotic arm; if the inverse kinematics has a solution, determining the contact point between the line connecting the posture of the end of the robotic arm and the lesion area and the body surface as the position of the second hole.

[0106] like Figure 5 As shown, after the lesion area is determined, the lesion point is determined. Since the surgical instrument fixed by the holding arm needs to point to the lesion point, and the preset distance fixes the distance between the holding arm and the mirror holding arm, the end position of the holding arm can be determined based on the position of the first hole. On the basis of obtaining the end position of the holding arm, the inverse kinematics of the holding arm is solved according to the end position of the holding arm. Specifically, the inverse kinematics solution here refers to calculating the solution corresponding to the requirement that the arm spacing between the telecentric mechanisms of the holding arm is the maximum during the movement of the holding arm and the preset distance between the holding arm and the mirror holding arm is satisfied. If the inverse kinematics has a solution, it means that the appropriate target joint angle of the holding arm can be calculated, and then the arm spacing value of the telecentric mechanism of the holding arm can be obtained. In the case of a solution to the inverse kinematics, the line connecting the end position of the holding arm and the lesion area is further marked, and the contact point between the line and the patient's body surface is the hole position of the holding arm.

[0107] In one embodiment, determining the end position of the robotic arm according to the preset distance, the position of the first hole, and the lesion area includes: determining the axis of the robotic arm sleeve according to the preset distance, the position of the first hole, and the lesion area; based on the axis of the robotic arm sleeve, selecting the rotational freedom of the robotic arm sleeve around itself to determine the median position of the joint travel; and determining the end position of the robotic arm according to the median position of the joint travel.

[0108] Continue to refer Figure 5 , considering that if the preoperative positioning step can ensure that the axis of the instrument arm (puncture tube) points to the lesion point, then the end instrument points to the lesion point, which can simplify the process of adjusting the telecentric mechanism of the instrument arm. Under this concept, in this embodiment, after the preset distance between the instrument arm and the scope arm, the hole position of the scope arm, and the axis of the instrument arm tube are determined, the tube selects the middle position of the joint travel around its own rotational freedom, and the end position of the instrument arm can be determined.

[0109] In one embodiment, there are multiple preset distances; the position of the second hole position is obtained according to the preset distance and the position of the first hole position, including: determining the positions of multiple initial second hole positions according to the multiple preset distances and the position of the first hole position; calculating the arm distances between the mirror holding arm and the robot holding arm in the multiple initial second hole positions; and determining the hole position with the largest arm distance between the mirror holding arm and the robot holding arm as the second hole position.

[0110] The preset distance between the mechanical arm and the mirror holding arm can be set to multiple according to the actual needs. Under the premise that the number of preset distances is multiple, the positions of multiple alternative second hole positions can be determined according to the multiple preset distances and the position of the first hole position, that is, the positions of multiple initial second hole positions are obtained. In view of this situation, it is necessary to select the hole position with the largest arm spacing between the mirror holding arm and the mechanical arm from the positions of the multiple initial second hole positions determined as the final second hole position. Specifically, the maximum arm spacing between the mirror holding arm and the mechanical arm means that the possibility of collision between the mechanical arm and the mirror holding arm is the smallest, and the second hole position determined at this time is the optimal second hole position. Further, multiple preset distances can be calculated at equal step spacing within the number of cyclic iterative searches k, with ΔP as the step increment of the preset distance, and the arm spacing between the mirror holding arm and the mechanical arm corresponding to different preset distances is continuously iterated, and the hole position corresponding to the largest arm spacing between the mirror holding arm and the mechanical arm is selected as the second hole position.

[0111] In one of the embodiments, the arm spacing between the mirror holding arm and the robot holding arm in multiple initial second hole positions is calculated, including: performing inverse kinematics solution on the robot holding arm according to the end posture of the robot arm to obtain the target posture of the robot arm; calculating the arm spacing between the mirror holding arm and the robot arm in multiple initial second hole positions according to the target posture of the robot arm and the posture of the mirror holding arm.

[0112] The inverse kinematics of the robotic arm is solved based on the end position of the robotic arm. The target joint angle of the robotic arm can be calculated through the inverse kinematics of the robotic arm, that is, the target position of the robotic arm is obtained. In order to further obtain the arm spacing between the telescope holding arm and the robotic arm, it is necessary to calculate the arm spacing between the telescope holding arm and the robotic arm in multiple initial second hole positions according to the target position of the robotic arm, the position of the telescope holding arm and different initial second hole positions.

[0113] In order to further explain in detail the technical principles and working process of the surgical robot system of the present application, the process of determining the preset drilling position will be described in detail below.

[0114] In one of the embodiments, the preset punching position is obtained in the following manner: determining the surgical area based on the position information of the lesion area; determining the candidate punching area based on the position and size of the surgical area and the structural parameters of the robotic arm in the surgical robot; screening out feasible punching areas from the candidate punching areas based on the first image information; determining the preset punching position in the feasible punching area based on the structural parameters of the robotic arm in the surgical robot.

[0115] The location information of the lesion area is the specific location of the lesion point in the patient's body. For example, the lesion point can be the gallbladder, stomach and other parts in the patient's abdominal cavity. Through the specific location of the lesion point, the approximate area of ​​the operation can be determined, that is, the approximate location and size of the operation area. Through the approximate location and size of the operation area, combined with the length of the mechanical arm in the surgical robot, and the distance between the mechanical arms, the candidate punching area can be determined. Then, the area that can be punched is screened out in the candidate punching area, that is, the feasible punching area. Based on the length of the mechanical arm in the surgical robot and the distance between the mechanical arms, the preset punching position is determined in the feasible punching area. Among them, the mechanical arm includes a mirror holding arm for clamping an endoscope and a mechanical holding arm for clamping surgical instruments. Usually the number of mirror holding arms is one, and the number of mechanical holding arms is three. Through the processing flow of this embodiment, the accuracy of the preset punching position can be improved, thereby reducing the risk of surgery.

[0116] In one embodiment, screening out feasible perforation areas from candidate perforation areas according to the first image information includes:

[0117] According to the first image information, risky perforation positions are identified, wherein the risky perforation positions include perforation positions that pass through bones, blood vessels, or nerves; the risky perforation positions are eliminated from candidate perforation areas, and feasible perforation areas are screened out.

[0118] Since the human body has structures such as blood vessels, nerves, and bones, there will be surgical risks if a hole is made at a location with blood vessels, nerves, or bones, so these locations need to be screened out. In this embodiment, it is necessary to screen out locations with blood vessels, nerves, or bones based on the first image information, thereby obtaining a feasible drilling area. In this way, the drilling risk is reduced, thereby ensuring the reliability and safety of the operation.

[0119] In one embodiment, according to the structural parameters of the robot arm, determining the position of the pre-punching position of the lesion area in the feasible punching area includes:

[0120] Based on the structural parameters of the robotic arm, the length parameters of the scope holding arm are obtained; according to the length parameters of the scope holding arm and the preset depth of field limit constraints, the feasible drilling area of ​​the scope holding arm is screened in the feasible drilling area; the scope holding arm drilling points and the robotic arm drilling points are selected in the feasible drilling area of ​​the scope holding arm; the scope holding arm drilling points and the robotic arm drilling points are aggregated to obtain the pre-drilling position of the lesion area.

[0121] Since the accuracy requirement for the punching position of the endoscope holding arm is lower than that of the instrument holding arm, the punching position of the endoscope holding arm can be determined first. In the present application, the preset depth of field of the endoscope is that when the endoscope is placed in the abdominal cavity, objects in the abdominal cavity can be clearly imaged within the depth of field. In this way, the punching accuracy of the endoscope holding arm can be guaranteed, thereby ensuring that during surgery, a clear picture of the specific location of the lesion can be obtained through the endoscope, thereby ensuring the smooth progress of the surgical procedure. Further, the point closest to the lesion area within the feasible punching area of ​​the endoscope holding arm can be selected as the punching point of the endoscope holding arm (endoscope cannula). By selecting the point closest to the lesion area as the punching point of the endoscope holding arm, the situation of the lesion point can be well observed after the endoscope is placed in the abdominal cavity.

[0122] In one embodiment, if Figure 6 As shown, the present application also provides another method for determining the punching position of the robotic arm, which is as follows:

[0123] Acquire the first image data of the lesion area, and determine the surgical area according to the position information of the lesion area. Determine the candidate punching area according to the position and size of the surgical area and the structural parameters of the mechanical arm in the surgical robot. Identify the risky punching position according to the first image information, wherein the risky punching position includes the punching position through the bone, blood vessel, or nerve. Eliminate the risky punching position from the candidate punching area and filter out the feasible punching area. Based on the structural parameters of the mechanical arm in the surgical robot, obtain the length parameter of the mirror holding arm. According to the length parameter of the mirror holding arm and the preset depth of field limit constraint, filter the feasible punching area of ​​the mirror holding arm in the feasible punching area. Select the punching point of the mirror holding arm and the punching point of the mechanical arm in the feasible punching area of ​​the mirror holding arm. Collect the punching points of the mirror holding arm and the punching points of the mechanical arm in the surgical robot to obtain the preset punching position. According to the converted position information of the lesion area and the preset punching position in the lesion area, obtain the punching position of the preset punching position in the visual device coordinate system. Push the punching position in the visual device coordinate system to the visual device.

[0124] Based on the same inventive concept, Figure 7 As shown, the embodiment of the present application also provides a surgical robot drilling guidance method, the method comprising:

[0125] S200: Acquire first image data of the lesion area;

[0126] S400: Establishing a three-dimensional model of the lesion area according to the first image data;

[0127] S600: Acquire second image data of a working environment of a scanning device, where the scanning device is a scanning device that scans and generates first image data of a lesion area;

[0128] S800: Determine the position of a first hole in the visual device coordinate system according to the three-dimensional model of the lesion area and the second image data, where the first hole is the hole of the mirror holding arm.

[0129] In one embodiment, determining the position of the first hole in the visual device coordinate system according to the three-dimensional model of the lesion area and the second image data includes:

[0130] Obtain a first pose relationship between the scanning device and the lesion area; obtain a second pose relationship between the scanning device and the visual device; obtain position information of the three-dimensional model of the lesion area in the visual device coordinate system based on the first pose relationship and the second pose relationship; determine the position of the first hole in the visual device coordinate system based on the position information of the three-dimensional model of the lesion area in the visual device coordinate system.

[0131] In one of the embodiments, the above-mentioned surgical robot drilling guidance method also includes: obtaining a preset distance between the first hole position and the second hole position, the second hole position being the hole position of the robotic arm; determining the posture of the end of the robotic arm according to the preset distance and the lesion area; performing inverse kinematics solution on the robotic arm according to the posture of the end of the robotic arm; if the inverse kinematics has a solution, determining the contact point between the line connecting the posture of the end of the robotic arm and the lesion area and the body surface as the position of the second hole position.

[0132] The implementation solution to the problem provided by the surgical robot drilling guidance method in the above-mentioned embodiment is similar to the implementation solution recorded in the above-mentioned surgical robot system. Therefore, the specific limitations in one or more surgical robot drilling guidance method embodiments provided above can be referred to the limitations in the surgical robot system above and will not be repeated here.

[0133] Based on the same inventive concept, Figure 8 , the embodiment of the present application also provides a surgical robot drilling guidance device, the entire device specifically includes a scanning component 820, a visual component 840 and a control component 860;

[0134] The scanning component 820 generates first image data of the lesion area and sends the first image data to the control component 860 ; the control component 860 generates the position of the first hole in the visual component 840 using the surgical robot drilling guidance method as described above.

[0135] In one embodiment, if Figure 8 , Fig. 9 As shown, the scanning component 820 includes an ultrasonic probe 822, on which a visual target is arranged; the visual target transmits a cursor signal to illuminate the lesion area; the visual component 840 receives the cursor signal on the lesion area, and sends the position information of the cursor signal to the control component.

[0136] The scanning component 820 includes an ultrasonic probe 822, on which a visual target is arranged; the visual target transmits a cursor signal to illuminate the lesion area of ​​the scanned object; the visual component 840 receives the cursor signal on the lesion area and sends the position information of the cursor signal to the control component.

[0137] It should be noted that the visual target is a structure that can emit a cursor signal. Specifically, the optical signal can be an infrared cursor, or a light emitted by a light emitting diode, etc. In this embodiment, the scanning component 820 is described as an ultrasound device, and in this embodiment, the lesion area includes the lesion location in the abdominal cavity and the patient's body surface corresponding to the lesion location. Among them, the cursor signal emitted by the visual target is irradiated on the body surface corresponding to the lesion location of the scanned object. Specifically, when the patient is scanned by an ultrasound device, the probe of the ultrasound device is against the patient's body surface. At this time, the cursor signal emitted by the visual target on the probe just irradiates the body surface. Since the depth camera can detect the three-dimensional information of the environment, the depth camera can receive the cursor signal and send the position information of the received cursor signal to the processing component.

[0138] In this embodiment, a visual target capable of emitting a cursor signal is provided on the ultrasonic probe 822, the cursor signal is received by the visual component 840, and the position information of the cursor signal is sent to the processing component to obtain the relative posture relationship between the scanning component 820 and the visual component 840. Through the ultrasonic probe 822 and the lesion area detected by the ultrasonic probe 822, the posture relationship between the ultrasonic probe 822 and the lesion area can be obtained.

[0139] In one embodiment, the control component is further used to control the scanning component 820 to emit a cursor signal to illuminate the lesion area of ​​the scanned object.

[0140] The cursor signal refers to a visible light identification signal, such as a bright red spot or a visible light with a certain pattern. The cursor signal irradiating the lesion area of ​​the scanned object means that the cursor signal emitted by the scanning component 820 is irradiated onto the body surface corresponding to the lesion area of ​​the scanned object. For example, when scanning the abdomen of a patient, the red cursor or a cursor with a specific pattern emitted by the ultrasound target on the ultrasound probe 822 can be irradiated onto the body surface corresponding to the abdomen of the patient.

[0141] Controlling the visual component 840 to continuously track the cursor signal on the lesion area so that the cursor signal is within the field of view of the visual component 840. Controlling the visual component 840 to continuously track the cursor signal on the lesion area means controlling the field of view of the visual component 840 so that the cursor signal is always within the field of view of the visual component 840. For example, the field of view of the visual component 840 can be adjusted by controlling the movement or rotation of the visual component 840 so that the cursor signal is always within the field of view of the visual component 840.

[0142] Only when the cursor signal is always within the field of view of the visual component 840, that is, within the field of view of the depth camera, can the visual component 840 always detect the specific position of the ultrasonic probe 822 on the body surface, so that the processing component can always obtain the posture relationship between the ultrasonic probe 822 and the visual component 840. Thus, the posture relationship between the ultrasonic probe 822 and the lesion area is obtained through the posture relationship between the ultrasonic probe 822 and the lesion area. Therefore, in this embodiment, the processing component controls the visual component 840 to continuously track the cursor signal on the lesion area, so that the cursor signal is within the field of view of the visual component 840, so that the processing component can always obtain the posture relationship between the visual component 840 and the lesion area, thereby ensuring the smooth progress of the drilling guidance work and ensuring the reliability of the operation of the surgical robot system.

[0143] In one embodiment, if Fig. 9 As shown, the visual component 840 includes a camera element 842, an angle adjustment element 844 and a mixed reality element 846, wherein the camera element 842 receives a cursor signal on the lesion area and sends the position information of the cursor signal to the processing component; the angle adjustment element 844 is connected to the camera element 842 and is communicatively connected to the processing component, and the processing component controls the angle adjustment element 844 to adjust the image tracking angle of the camera element 842; the mixed reality element 846 is communicatively connected to the processing component, and the mixed reality element 846 displays the punching position in the coordinate system of the visual component 840 pushed by the processing component.

[0144] It should be noted that the camera element 842 can be the depth camera mentioned above, the angle adjustment element 844 can be the degree of freedom gimbal mentioned above, and the mixed reality element 846 can present the information of the punching position in front of the eyes of the person wearing the MR helmet in a mixed reality manner. The camera element 842, that is, the depth camera tracks the cursor signal emitted by the visual target on the ultrasonic probe 822, and sends the position information of the detected cursor signal to the processing component. By connecting the camera element 842 to the angle adjustment element 844, the camera element 842 can change its position as the angle adjustment element 844 rotates, thereby adjusting the angle and position of the field of view. By connecting the angle adjustment element 844 to the processing component in communication, the angle of the angle adjustment element 844132 can be controlled by the processing component, thereby adjusting the image tracking angle of the camera element 842.

[0145] Exemplarily, the degree of freedom gimbal has two degrees of freedom, namely the pitch degree of freedom and the yaw degree of freedom. When the doctor puts on the MR helmet, the doctor adjusts the head posture so that the cursor signal emitted by the visual target in the ultrasonic probe 822 is within the field of view of the depth camera of the MR helmet. This process is the initialization process of the visual component 840. After the initialization is completed, the degree of freedom gimbal adjustment function of the depth camera is turned on, and the depth camera can detect the cursor signal emitted by the visual target. After the processing component obtains the image of the cursor signal sent by the depth camera, it can identify the position and posture of the cursor signal, and control the depth camera to track the cursor signal in real time, thereby ensuring that the processing component can always obtain the posture relationship between the visual component 840 and the ultrasonic target to ensure the reliability of the drilling guidance work.

[0146] In one embodiment, the processing component is further used to control the visual component 840 to track a cursor signal on the lesion area; if the cursor signal is not within the field of view of the visual component 840, the image tracking angle of the visual component 840 is adjusted.

[0147] The image tracking angle refers to the angle of the field of view of the visual component 840. For example, when the visual component 840 is a MR helmet, the depth camera on the MR helmet is used to detect the cursor signal. When the cursor signal is outside the lens of the depth camera, the cursor signal can be placed within the field of view of the visual component 840 by adjusting the angle of the depth camera or adjusting the angle of the MR helmet.

[0148] If the image tracking angle exceeds the preset angle limit, a message prompting that the field of view is out of range will be pushed.

[0149] The preset angle limit value refers to the maximum or minimum value preset for adjusting the image tracking angle. When this value is exceeded, the image tracking angle cannot be adjusted any further. The out-of-field-range prompt message is a prompt signal sent to the user to facilitate the user to adjust the image tracking angle. For example, when the DOF gimbal of the depth camera moves to the limit of the travel, the DOF gimbal cannot move any further. At this time, the doctor needs to be reminded to adjust the angle of the MR helmet so that the cursor signal is within the lens of the depth camera.

[0150] It is understandable that the visual component 840 tracks the cursor signal in real time and sends the acquired image information to the processing component. The processing component identifies whether there is a cursor signal in the image information. When there is a cursor signal in the image information and the cursor signal is in the central area of ​​the visual field, the processing component only needs to control the visual component 840 to track the cursor signal in real time. When the processing component identifies that there is a cursor signal in the image information, but the cursor signal is not in the central area of ​​the visual field, the processing component will calculate the deviation between the current cursor signal and the boundary of the central area, and control the angle adjustment element 844 to move to drive the camera element 842 to move, thereby adjusting the image tracking angle of the camera element 842 so that the cursor signal is in the central area of ​​the visual field. When the processing component controls the angle adjustment element 844 to move, the angle adjustment element 844 has reached the limit value and cannot move, indicating that the image tracking angle has exceeded the limit value of the preset angle, and at this time, the image tracking angle cannot be adjusted further. The processing component will push a prompt message of exceeding the visual field range to the visual component 840 to remind the doctor to adjust the direction of the visual component 840 worn on the head, thereby adjusting the visual field of the visual component 840 so that the cursor signal is located in the central area of ​​the visual field. Specifically, the adjustment direction can be determined by the intersection of the cursor signal and the central area, and the adjustment distance of the angle adjustment element 844 can be determined according to the distance between the cursor signal and the center of the field of view, thereby determining the adjustment direction and adjustment distance of the angle adjustment element 844.

[0151] For example, when the cursor signal is at the left boundary of the central area, the angle adjustment element needs to be moved to the left to make the cursor signal in the central area of ​​the visual field. If the angle adjustment element cannot be moved to the left, it means that the image tracking angle exceeds the preset angle limit. At this time, the processing component pushes a prompt message to the visual component, that is, the MR helmet, so that the doctor adjusts the angle of the head so that the cursor signal is in the central area of ​​the visual field. In this way, it can effectively ensure that the visual component can always track the cursor signal, thereby ensuring the smooth progress of the drilling guidance work.

[0152] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0153] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0154] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A surgical robot system, comprising a scope holding arm, characterized in that: include: A scanning unit, used for acquiring first image data of a lesion area; A modeling unit, used for establishing a three-dimensional model of the lesion area according to the first image data; A visual unit, used for acquiring second image data of a working environment of a scanning device, wherein the scanning device is a scanning device that scans and generates the first image data of the lesion area; A processing unit is used to determine the position of a first hole position in the visual device coordinate system according to the three-dimensional model of the lesion area and the second image data, wherein the first hole position is the hole position of the mirror holding arm.

2. The surgical robot system according to claim 1, characterized in that: The processing unit determines the position of the first hole in the visual device coordinate system according to the three-dimensional model of the lesion area and the second image data, including: Acquiring a first pose relationship between the scanning device and the lesion area; Acquire a second posture relationship between the scanning device and the visual device; According to the first posture relationship and the second posture relationship, obtaining position information of the three-dimensional model of the lesion area in the visual device coordinate system; According to the position information of the three-dimensional model of the lesion area in the visual device coordinate system, the position of the first hole in the visual device coordinate system is determined.

3. The surgical robot system according to claim 2, characterized in that: The processing unit is also used to obtain the pre-punching position of the lesion area; Determining the position of the first hole in the visual device coordinate system according to the position information of the three-dimensional model of the lesion area in the visual device coordinate system includes: According to the position of the three-dimensional model of the lesion area in the visual device coordinate system, the pre-drilling position is corrected to obtain the position of the first hole in the visual device coordinate system.

4. The surgical robot system according to claim 1, characterized in that: The surgical robot system also includes a robotic arm, and the processing unit is further used to obtain the position of a second hole position according to the position of the first hole position, and the position of the second hole position is the hole position of the robotic arm.

5. The surgical robot system according to claim 4, characterized in that: The acquiring the position of the second hole position according to the position of the first hole position comprises: Obtaining a preset distance between the first hole position and the second hole position; The position of the second hole is acquired according to the preset distance and the position of the first hole.

6. The surgical robot system according to claim 5, characterized in that: The acquiring the position of the second hole position according to the preset distance and the position of the first hole position comprises: Determining the position of the end of the robotic arm according to the preset distance, the position of the first hole and the lesion area; Performing inverse kinematics solution on the robotic arm according to the end posture of the robotic arm; If the inverse kinematics has a solution, the contact point between the line connecting the end posture of the robotic arm and the lesion area and the body surface is determined as the position of the second hole.

7. The surgical robot system according to claim 6, characterized in that: Determining the position of the end of the robotic arm according to the preset distance, the position of the first hole and the lesion area includes: Determining the axis of the sleeve of the robotic arm according to the preset distance, the position of the first hole, and the lesion area; Based on the axis of the sleeve of the robotic arm, the rotational freedom of the robotic arm sleeve around itself is selected to determine the middle position of the joint travel; According to the middle position of the joint travel, the end position of the robotic arm is determined.

8. The surgical robot system according to claim 6, characterized in that: The number of the preset distances is multiple; The obtaining the position of the second hole position according to the preset distance and the position of the first hole position includes: Determining the positions of a plurality of initial second hole positions according to a plurality of preset distances and the positions of the first hole positions; Calculating the arm distances between the mirror holding arm and the mechanical holding arm in a plurality of initial second hole positions; The hole position at which the arm distance between the mirror holding arm and the mechanical holding arm is the largest is determined as the second hole position.

9. The surgical robot system according to claim 8, characterized in that: The calculating the arm distance between the mirror holding arm and the robot holding arm in the plurality of initial second hole positions comprises: Performing inverse kinematics solution on the robotic arm according to the end posture of the robotic arm to obtain the target posture of the robotic arm; The arm distances between the mirror holding arm and the robot arm in a plurality of initial second hole positions are calculated according to the target posture of the robot arm and the posture of the mirror holding arm.

10. A surgical robot drilling guidance method, characterized in that: The method comprises: Acquiring first image data of a lesion area; Establishing a three-dimensional model of the lesion area according to the first image data; Acquire second image data of a working environment of a scanning device, wherein the scanning device is a scanning device that scans and generates the first image data of the lesion area; According to the three-dimensional model of the lesion area and the second image data, the position of the first hole position in the visual device coordinate system is determined, and the first hole position is the hole position of the mirror holding arm.

11. The surgical robot drilling guidance method according to claim 10, characterized in that: Determining the position of the first hole in the visual device coordinate system according to the three-dimensional model of the lesion area and the second image data includes: Acquiring a first pose relationship between the scanning device and the lesion area; Acquire a second posture relationship between the scanning device and the visual device; According to the first posture relationship and the second posture relationship, obtaining position information of the three-dimensional model of the lesion area in the visual device coordinate system; According to the position information of the three-dimensional model of the lesion area in the visual device coordinate system, the position of the first hole in the visual device coordinate system is determined.

12. The surgical robot drilling guidance method according to claim 10, characterized in that: Also includes: Obtaining a preset distance between a first hole position and a second hole position, where the second hole position is a hole position of the robotic arm; Determining the position of the end of the robotic arm according to the preset distance and the lesion area; Performing inverse kinematics solution on the robotic arm according to the end posture of the robotic arm; If the inverse kinematics has a solution, the contact point between the line connecting the end posture of the robotic arm and the lesion area and the body surface is determined as the position of the second hole.

13. A surgical robot drilling guidance device, characterized in that: Including scanning components, visual components and control components; The scanning component generates first image data of the lesion area, and sends the first image data to the control component; The control component uses the surgical robot drilling guidance method as described in any one of claims 10 to 12 to generate the position of the first hole in the visual component.

14. The surgical robot drilling guidance device according to claim 13, characterized in that: The scanning component includes an ultrasonic probe, on which a visual target is arranged; the visual target emits a cursor signal to illuminate the lesion area; The visual component receives the cursor signal on the lesion area and sends the position information of the cursor signal to the control component.

15. The surgical robot drilling guidance device according to claim 14, characterized in that: The visual components include: An imaging element, collecting the cursor signal on the lesion area, and sending the position information of the cursor signal to the control component; An angle adjustment element, carrying the imaging element and connected to the control component, wherein the control component controls the angle adjustment element to adjust the image tracking angle of the imaging element; The mixed reality component displays the hole position in the visual component coordinate system pushed by the control component.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 10 to 12 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 10 to 12 are implemented.

Images