Adjustment method and device of image acquisition component and electronic equipment

Abstract

The present specification provides an image acquisition component adjustment method and device and electronic equipment, and relates to the technical field of medical apparatuses, and the method comprises the following steps: acquiring a target image formed by a signal collected when an image acquisition component is in a current pose; determining first distances of at least two target objects in the target image from a center point of the target image and second distances of the at least two target objects from predetermined points on a boundary of the target image; calculating first distribution data corresponding to the current pose of the image acquisition component according to the first distances and the second distances; when the first distribution data is less than or equal to a preset threshold, determining a target pose within a pose adjustment range of the image acquisition component, and adjusting the image acquisition component to the target pose, wherein the first distribution data corresponding to the target pose is greater than the first distribution data corresponding to the current pose. The present scheme automatically adjusts the pose of the image acquisition component based on the field of view data of the quantitative value, which can reduce the workload of the operator and shorten the execution time of the operation task.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and in particular to methods, apparatus and electronic devices for adjusting image acquisition components. Background Technology

[0002] In some existing technologies, the operation of automated equipment requires the user to operate the tools on the equipment when the real-time operation is not visible to the naked eye (for example, the operation result occurs in a tiny area that is not visible to the naked eye, or the operation result needs to be magnified to be discernible; or the user is not in an area where the operation result can be seen, such as being in a different room). Therefore, the controllable automated equipment is typically equipped with image acquisition components such as cameras to collect the real-time operation status and feed it back to the user.

[0003] To more accurately monitor the real-time operational status of the image acquisition component, users typically need to frequently adjust its pose during operation to capture data from the optimal angle. This makes the process of operating automated equipment to complete tasks cumbersome and time-consuming. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, and electronic device for adjusting an image acquisition component, so as to solve the problem that the operation process is cumbersome and the task execution time is long when users need to operate the automated equipment while adjusting the image acquisition component when the real-time operation status cannot be seen with the naked eye.

[0005] To address the aforementioned technical problems, this specification provides a method for adjusting an image acquisition component, comprising: acquiring a target image formed by signals collected by the image acquisition component when it is in a current pose; determining a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image; the predetermined point being the point on the boundary of the target image closest to the target object; calculating first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance between each target object and the predetermined point on the boundary of the target image; when the first distribution data is less than or equal to a preset threshold, determining a target pose within the pose adjustment range of the image acquisition component, and adjusting the image acquisition component to the target pose, wherein the first distribution data corresponding to the target pose is greater than the first distribution data corresponding to the current pose.

[0006] In some embodiments, calculating first distribution data corresponding to the current pose of the image acquisition component based on a first distance between each of the at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image, includes: acquiring a reference pose, wherein the reference pose is determined based on a target object in a stationary state among the at least two target objects; determining the pose deviation between the current pose of the image acquisition component and the reference pose; and calculating the first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, and the pose deviation.

[0007] In some embodiments, the first distribution data corresponding to the current pose of the image acquisition component is calculated based on a first distance between each of the at least two target objects in the target image and the center point of the target image, a second distance between each target object and a predetermined point on the boundary of the target image, and the pose deviation. This includes: when a target object is occluded, acquiring the area occupied by the occluder in the target image; and calculating the first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, the pose deviation, and the area.

[0008] In some embodiments, calculating the first distribution data corresponding to the current pose of the image acquisition component based on a first distance between each of at least two target objects in the target image and the center point of the target image, a second distance between each target object and a predetermined point on the boundary of the target image, the pose deviation, and the area, includes: performing a weighted summation of the first distances between each target object in at least two target images and the center point of the target image to obtain a first value; performing a weighted summation of the second distances between each target object in at least two target images and a predetermined point on the boundary of the target image to obtain a second value; calculating the norm of the pose deviation to obtain a third value; and calculating the first distribution data corresponding to the current pose of the image acquisition component based on the first value, the second value, the third value, and the area.

[0009] In some embodiments, before adjusting the image acquisition component to the target pose, the method further includes: performing path planning based on the target pose to obtain an adjustment path for the image acquisition component during the process of adjusting from the current pose to the target pose; determining, based on the path planning result, whether the image acquisition component will collide with a movable object among the at least two target objects during the process of adjusting from the current pose to the target pose; and automatically adjusting the image acquisition component to the target pose based on the path planning result when the image acquisition component will not collide with a movable object.

[0010] In some embodiments, determining a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image, includes: determining the first distance between each of at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image, in the following manner: determining the motion trajectory of the current target object based on multiple historical images formed from historical signals acquired by the image acquisition component; determining a first expected position of the current target object relative to the center point of the target image based on the motion trajectory; identifying an image region in the target image within a predetermined range centered on the first expected position; and when the current target object is identified, determining the first distance between the identified current target object and the center point of the target image, and the second distance between the identified current target object and the predetermined point on the boundary of the target image.

[0011] In some embodiments, after acquiring a target image formed by signals collected by the image acquisition component when it is in the current pose, the method further includes: when there is no target object in the target image, determining a second expected position of the target object outside the target image; performing path planning based on the second expected position to obtain an adjustment path for the image acquisition component from the current pose toward the second expected position; and generating a prompt message on the adjustment direction of the image acquisition component based on the path planning result, so that the operator can adjust the pose of the image acquisition component according to the adjustment direction using a pose adjustment manipulator.

[0012] In some embodiments, after path planning is performed based on the second expected position to obtain the adjustment path of the image acquisition component from the current pose toward the second expected position, the method further includes: feeding back control resistance on the pose adjustment manipulator of the image acquisition component, wherein the control resistance in the first operating direction of the pose adjustment manipulator is less than the control resistance in other operating directions, wherein the first operating direction is the operating direction corresponding to the adjustment direction of the image acquisition component in the path planning result, and the other operating directions are operating directions other than the first operating direction in a preset set of operating directions.

[0013] In some embodiments, after generating the prompt information for the adjustment direction of the image acquisition component, the method further includes: during the process of adjusting the pose of the image acquisition component by the pose adjustment manipulator, calculating in real time the second distribution data corresponding to the pose of the image acquisition component based on the image formed by the signal acquired by the image acquisition component; when the second distribution data corresponding to the pose of the image acquisition component reaches a predetermined range, determining a target pose within the pose adjustment range of the image acquisition component, and adjusting the image acquisition component to the target pose.

[0014] In some embodiments, the first distribution data and the second distribution data are calculated in the same way.

[0015] In some embodiments, calculating the second distribution data corresponding to the pose of the image acquisition component in real time based on the image formed by the signal acquired by the image acquisition component includes: acquiring a first image formed by the signal acquired by the image acquisition component at a first moment; acquiring a first distance between each of at least two target objects in the first image and the center point of the first image, and a second distance between each target object and a predetermined point on the boundary of the first image; wherein, when the target object is not in the first image, determining the first distance between the second expected position of the target object and the center point of the first image, and the second distance between the second expected position of the target object and the predetermined point on the boundary of the first image; and calculating the second distribution data corresponding to the pose of the image acquisition component at the first moment based on the first distance and the second distance.

[0016] In some embodiments, calculating the second distribution data corresponding to the pose of the image acquisition component at a first moment based on the first distance and the second distance includes: forming a first distribution matrix corresponding to the pose at the first moment based on the first distance and the second distance; determining the second image formed by the signal acquired by the image acquisition component at a second moment and the estimated positions of the at least two target objects, and determining the first distance between the estimated positions of the at least two target objects and the center point of the second image, and the second distance between the estimated positions of the at least two target objects and predetermined points on the boundary of the second image; forming a second distribution matrix corresponding to the pose at the second moment based on the first distance between the estimated positions of the at least two target objects and the center point of the second image, and the second distance between the estimated positions of the at least two target objects and predetermined points on the boundary of the second image; and calculating the second distribution data corresponding to the pose of the image acquisition component at the first moment based on the first distribution matrix and the second distribution matrix.

[0017] In some embodiments, the elements in the first distribution matrix further include: the area occupied by the occluder in the first image when the target object is occluded; and / or, the pose deviation between the pose at a first moment and a reference pose, wherein the reference pose is determined in advance based on the target object in a stationary state among the at least two target objects.

[0018] In some embodiments, the image acquisition component is an endoscope lens or an ultrasound detector.

[0019] A second aspect of this specification provides an adjustment device for an image acquisition component, comprising: an acquisition unit for acquiring a target image formed by signals collected when the image acquisition component is in a current pose; a first determination unit for determining a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image; a first calculation unit for calculating first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of at least two target objects in the target image and the center point of the target image, and the second distance between each target object and the predetermined point on the boundary of the target image; a second determination unit for determining a target pose within the pose adjustment range of the image acquisition component when the first distribution data is less than or equal to a preset threshold, wherein the first distribution data corresponding to the target pose is greater than the first distribution data corresponding to the current pose; and an adjustment unit for adjusting the image acquisition component to the target pose.

[0020] In some embodiments, the first calculation unit includes: a first acquisition subunit, configured to acquire a reference pose, wherein the reference pose is determined based on a target object in a stationary state among the at least two target objects; a first determination subunit, configured to determine the pose deviation between the current pose of the image acquisition component and the reference pose; and a first calculation subunit, configured to calculate first distribution data corresponding to the current pose of the image acquisition component based on a first distance between each of the at least two target objects in the target image and the center point of the target image, a second distance between each target object and a predetermined point on the boundary of the target image, and the pose deviation.

[0021] In some embodiments, the first calculation subunit includes: a second acquisition subunit, configured to acquire the area occupied by the occupant in the target image when the target object is occluded; and a second calculation subunit, configured to calculate first distribution data corresponding to the current pose of the image acquisition component based on a first distance between each of at least two target objects in the target image and the center point of the target image, a second distance between each target object and a predetermined point on the boundary of the target image, the pose deviation, and the area.

[0022] In some embodiments, the second calculation subunit calculates the first distribution data corresponding to the current pose of the image acquisition component according to the following method: weighted summation of the first distance between each target object in at least two target images and the center point of the target image to obtain a first value; weighted summation of the second distance between each target object in at least two target images and a predetermined point on the boundary of the target image to obtain a second value; calculation of the norm of the pose deviation to obtain a third value; and calculation of the first distribution data corresponding to the current pose of the image acquisition component based on the first value, the second value, the third value, and the area.

[0023] In some embodiments, the apparatus further includes: a first planning unit, configured to perform path planning based on the target pose to obtain an adjustment path for the image acquisition component during the process of adjusting from the current pose to the target pose; and a judging unit, configured to judge, based on the path planning result, whether the image acquisition component will collide with a movable object among the at least two target objects during the process of adjusting from the current pose to the target pose; when the image acquisition component will not collide with a movable object, the adjusting unit performs the adjustment of the image acquisition component to the target pose.

[0024] In some embodiments, the first determining unit includes: determining a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image, in the following manner: a second determining subunit, configured to determine the motion trajectory of the current target object based on multiple historical images formed by historical signals acquired by the image acquisition component; a third determining subunit, configured to determine a first expected position of the current target object relative to the center point of the target image based on the motion trajectory; an identification subunit, configured to identify an image region in the target image within a predetermined range centered on the first expected position; and a fourth determining subunit, configured to determine, when the current target object is identified, the first distance between the identified current target object and the center point of the target image, and the second distance between the identified current target object and the predetermined point on the boundary of the target image.

[0025] In some embodiments, the apparatus further includes: a second determining unit, configured to determine a second expected position of the target object outside the target image when the target object is not present in the target image; a second planning unit, configured to perform path planning based on the second expected position to obtain an adjustment path for the image acquisition component from its current pose toward the second expected position; and a prompting unit, configured to generate prompt information about the adjustment direction of the image acquisition component based on the path planning result, so that an operator can adjust the pose of the image acquisition component according to the adjustment direction using a pose adjustment manipulator.

[0026] In some embodiments, the device further includes: a force feedback unit, configured to provide feedback control resistance on the pose adjustment manipulator of the image acquisition component, wherein the control resistance in a first operating direction of the pose adjustment manipulator is less than the control resistance in other operating directions, wherein the first operating direction is the operating direction corresponding to the adjustment direction of the image acquisition component in the path planning result, and the other operating directions are operating directions other than the first operating direction in a preset set of operating directions.

[0027] In some embodiments, the apparatus further includes: a second calculation unit, configured to calculate in real time second distribution data corresponding to the pose of the image acquisition component based on the image formed by the signal acquired by the image acquisition component during the process of adjusting the pose of the image acquisition component by the pose adjustment manipulator; a second determining unit and the adjusting unit, configured to determine a target pose within the pose adjustment range of the image acquisition component when the second distribution data corresponding to the pose of the image acquisition component reaches a predetermined range, and adjust the image acquisition component to the target pose.

[0028] In some embodiments, the first distribution data and the second distribution data are calculated in the same way.

[0029] In some embodiments, the second calculation unit includes: a third acquisition subunit, configured to acquire a first image formed by signals acquired by the image acquisition component at a first moment; a fourth acquisition subunit, configured to acquire a first distance between each of at least two target objects in the first image and the center point of the first image, and a second distance between each target object and a predetermined point on the boundary of the first image; wherein, when the target object is not in the first image, the first distance between the second expected position of the target object and the center point of the first image, and the second distance between the second expected position of the target object and the predetermined point on the boundary of the first image are determined; and a third calculation subunit, configured to calculate second distribution data corresponding to the pose of the image acquisition component at the first moment based on the first distance and the second distance.

[0030] In some embodiments, the third calculation subunit includes: a first forming subunit, configured to form a first distribution matrix corresponding to the pose at a first time moment based on the first distance and the second distance; a fifth determining subunit, configured to determine the second image formed by the signal acquired by the image acquisition component at a second time moment and the estimated positions of the at least two target objects, and to determine the first distance between the estimated positions of the at least two target objects and the center point of the second image, and the second distance between the estimated positions of the at least two target objects and predetermined points on the boundary of the second image, respectively; a second forming subunit, configured to form a second distribution matrix corresponding to the pose at a second time moment based on the first distance between the estimated positions of the at least two target objects and the center point of the second image, and the second distance between the estimated positions of the at least two target objects and predetermined points on the boundary of the second image, respectively; and a fourth calculation subunit, configured to calculate the second distribution data corresponding to the pose of the image acquisition component at a first time moment based on the first distribution matrix and the second distribution matrix.

[0031] In some embodiments, the elements in the first distribution matrix further include: the area occupied by the occluder in the first image when the target object is occluded; and / or, the pose deviation between the pose at a first moment and a reference pose, wherein the reference pose is determined in advance based on the target object in a stationary state among the at least two target objects.

[0032] In some embodiments, the image acquisition component is an endoscope lens or an ultrasound detector.

[0033] A third aspect of this specification provides an electronic device, comprising: a memory and a processor, wherein the processor and the memory are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to implement the steps of any of the methods described in the first aspect.

[0034] A fourth aspect of this specification provides a computer storage medium storing computer program instructions that, when executed, implement the steps of any of the methods described in the first aspect.

[0035] The image acquisition component adjustment method, apparatus, and electronic device provided in this specification, based on the target image formed by the signal acquired by the image acquisition component when it is in its current pose, determines a first distance between each of at least two target objects and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image. Based on the first and second distances, it calculates first distribution data corresponding to the current pose of the image acquisition component, thereby quantifying the distribution of the target image corresponding to the current pose. This facilitates the quantitative evaluation of the field of view corresponding to the current pose by a computer or electronic device. When the first distribution data is less than or equal to a preset threshold, this solution can automatically search for a target pose with a first distribution data greater than that corresponding to the current pose within the pose adjustment range, i.e., find a target pose with a better field of view. Therefore, this solution can achieve automatic adjustment of the image acquisition component, and this automatic adjustment method can improve the field of view of the image acquisition component. This allows the operator to only operate the operating tool without switching back and forth between operating the operating tool and the image acquisition component, reducing the operator's workload and shortening the execution time of the operation task. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 A three-dimensional structural diagram of the surgical robot system is shown;

[0038] Figure 2 A three-dimensional structural diagram of the doctor's control console is shown;

[0039] Figure 3 A flowchart illustrating an embodiment of the adjustment method for the image acquisition component provided in this specification is shown;

[0040] Figure 4 A three-dimensional structural diagram showing the field of view axis and field of view range of the object to be operated on, the operation tool, and the image acquisition component is shown;

[0041] Figure 5 It shows Figure 4 A schematic diagram of the image formed by the signals acquired by the corresponding image acquisition component;

[0042] Figure 6 A flowchart illustrating another embodiment of the adjustment method for the image acquisition component provided in this specification is shown;

[0043] Figure 7 It shows in Figure 4 A diagram showing the coordinate system established based on the three-dimensional structural schematic shown;

[0044] Figure 8 A side view showing the adjustable range of the image acquisition component;

[0045] Figure 9 A top view showing the adjustable range of the image acquisition component and the corresponding field of view;

[0046] Figure 10 A flowchart of yet another embodiment of the adjustment method for the image acquisition component provided in this specification is shown;

[0047] Figure 11 A flowchart of yet another embodiment of the adjustment method for the image acquisition component provided in this specification is shown;

[0048] Figure 12 A schematic diagram of the panoramic map is shown;

[0049] Figure 13 Another schematic diagram of the panoramic map is shown;

[0050] Figure 14 A flowchart illustrating one method for calculating the second distribution data is shown.

[0051] Figure 15 A flowchart illustrating another method for calculating the second distribution data is shown.

[0052] Figure 16 A schematic block diagram of the adjustment device for the image acquisition component provided in this specification is shown;

[0053] Figure 17 A schematic block diagram of the electronic device provided in this specification is shown. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0055] To address the issue of cumbersome operation and long execution time caused by users needing to simultaneously operate automated equipment and adjust the image acquisition component when the real-time operation status cannot be seen with the naked eye, this specification provides a method for automatically adjusting the image acquisition component based on an image. This method uses the target image formed by the signal acquired by the image acquisition component when it is in its current pose. It calculates the current field of view evaluation value corresponding to the current pose of the image acquisition component based on the position of the target object in the image. Within the pose adjustment range, it searches for a target pose with a field of view evaluation value greater than the current field of view evaluation value and adjusts the image acquisition component to that target pose.

[0056] The following section will primarily use a surgical robot system for performing minimally invasive surgery as an example to illustrate the adjustment method of the image acquisition component provided in this manual.

[0057] like Figure 1 As shown, a surgical robot system typically consists of a control unit 100, an execution unit 200, and an imaging unit 300. The control unit 100, usually called the doctor's console, is located outside the sterile area of ​​the operating room and is used to send control commands to the execution unit 200. The execution unit 200, i.e., the surgical robot device (referred to as the surgical robot in this specification), is used to hold surgical instruments and perform specific surgical operations on the patient according to the control commands. The surgical robot can also be equipped with an endoscope or ultrasound detector (the following example uses an endoscope). The imaging unit 300, usually called an imaging cart, is used to process the information acquired by the endoscope to form a three-dimensional high-definition image and feed it back to the control unit 100, etc.

[0058] like Figure 2 As shown, the control terminal device 100, also known as the doctor's console, is equipped with a main operator hand and an imaging device. The main operator hand detects the surgeon's hand movements, serving as the control signal for the entire surgical robot system. The imaging device provides the surgeon with a three-dimensional image of the patient's body detected by the endoscope, providing reliable image information for the surgeon's surgical operations. During surgery, the surgeon observes the transmitted three-dimensional image of the cavity through the imaging device and uses the main operator hand to control the movement of the robotic arm mechanism and surgical instruments on the surgical robot to complete various operations, thereby achieving the goal of performing surgery on the patient. In existing technology, if the field of vision is poor when controlling surgical instruments to perform surgical operations, the surgeon needs to first switch to the endoscope control mode, adjust the endoscope's position and posture to the optimal surgical field of vision through the main operator hand, and then switch back to the surgical instrument control mode to continue controlling the surgical instruments through the main operator hand.

[0059] like Figure 3 As shown, the adjustment method for the image acquisition component provided in this specification includes the following steps:

[0060] S10: Acquire the target image formed by the signals collected by the image acquisition component when it is in the current pose.

[0061] In this specification, "target object" refers to the object of interest when adjusting the pose of the image acquisition component. For example, when the image acquisition component is an endoscope used in surgery, the target object can be the tissue to be processed or surgical instruments, wherein there can be one or two surgical instruments. The method for acquiring the target object can be: identifying multiple objects in the target image, and selecting the object of interest from the identified objects as the target object. Here, the object of interest refers to the object of interest when adjusting the position of the image acquisition component, such as the object to be operated on or at least one operating tool.

[0062] An image acquisition component is used to acquire images of a target object. Some image acquisition components can acquire images on their own, such as those with a built-in CMOS image sensor; others cannot acquire images themselves, but only acquire intermediate data that can be used to form an image. The intermediate data needs to be processed by a processor to obtain the image, such as an ultrasound detector, which can only acquire ultrasound echo signals and needs to process the ultrasound echo signals to obtain an ultrasound image.

[0063] Image acquisition components can be endoscope lenses, ultrasonic probes of ultrasonic detectors, etc.

[0064] An endoscope is a commonly used medical device, consisting of a beam guide structure and a set of lenses. For example, the laparoscope commonly used in clinical practice uses a series of optical cylindrical lenses to transmit images to an eyepiece, which then displays the image via a separate camera. Based on their imaging principles, endoscopes can be classified into optical endoscopes (cylindrical lenses), fiber optic endoscopes, and electronic endoscopes, among others.

[0065] Changes in the pose of the image acquisition component will cause changes in the field of view of the corresponding image. The image corresponding to the pose refers to the image formed by the signals acquired by the image acquisition component when it is in that pose.

[0066] For example, in Figure 1 and Figure 2 In the surgical robot system shown, the image acquisition component can be an endoscope, which is inserted into the patient's cavity through a small incision. The endoscope's position can be adjusted by a robotic arm of the surgical robot, or a tiny motor can be built into the endoscope's head for further adjustment. The image acquisition device 300 generates a high-magnification stereoscopic image based on the signals acquired by the endoscope lens and feeds it back to the doctor's control panel.

[0067] S20: Determine the first distance between each of at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image, wherein the predetermined point is the point on the boundary of the target image that is closest to the target object.

[0068] The target object refers to the object associated with the operation result. For example, in the case of a surgical robot system performing minimally invasive surgery, it is usually necessary to process the lesion tissue with surgical instruments. In this case, the target object is the lesion tissue and the end of the surgical instruments.

[0069] In operating automated equipment, at least the positions of the following two objects must be considered: the end effector of the tool held by the robotic arm, and the object to be operated. In some cases, the total number of target objects may be three or more, such as when the total number of end effectors held by the robotic arm is four, and the number of operations of interest is two or more.

[0070] For example, when a surgical robot performs minimally invasive surgery, it focuses on three surgical instruments and two biological tissues, one of which is the lesion tissue and the other is the healthy tissue (such as an artery). During the surgery, it is necessary to ensure that the surgical instrument tip only treats the lesion tissue and does not touch the healthy tissue.

[0071] Therefore, the adjustment method for the image acquisition component provided in this specification requires obtaining a first distance between each target object and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image. The predetermined point is the point on the boundary of the target image that is closest to the target object.

[0072] The field of view of an image acquisition component (i.e., the range of signals that the image acquisition component can acquire) is usually a regular geometric shape, such as a circle, ellipse, square, rectangle, or other axially or centrally symmetrical shape. The image formed by the signal acquired by the image acquisition component is also a shape corresponding to the field of view shape; the center of the image can be the center of the geometric shape, and the boundaries of the image can be the boundaries of the geometric shape.

[0073] Figure 4 The diagram shows a three-dimensional structure of the object to be manipulated (a circular or elliptical area filled with diagonal lines), the manipulation tool B, the field of view axis O of the image acquisition component, and the field of view range. Figure 5 It shows Figure 4 A schematic diagram of the image formed by the signal acquired by the corresponding image acquisition component. Figure 4 and Figure 5In the diagram, the triangular area formed by the dashed and solid lines represents the conical field of view of the image acquisition component; O' is the center point of the image, which is the projection of the field of view axis O onto the image; d1 is the distance between the object to be operated on A and the center point O' of the image; d2 is the distance between the end of the operation tool B and the center point O' of the image; 2R is the diameter of the image; the distance D1 between the object to be operated on A and a predetermined point on the image boundary is R-d1; and the distance D2 between the end of the operation tool B and a predetermined point on the image boundary is R-d2.

[0074] It should be noted that the "center of the image" mentioned in this specification may not be the absolute center point of the geometrically shaped image, but a point whose distance from the absolute center point is within an acceptable error range; the "boundary of the image" mentioned in this specification may not be the absolute boundary of the image, but a boundary line whose distance from the absolute boundary is within an acceptable error range.

[0075] In some embodiments, step S20 may determine a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image, in the following manner:

[0076] S21: Determine the motion trajectory of the current target object based on multiple historical images formed by historical signals collected by the image acquisition component.

[0077] During operation, if the pose of the image acquisition component changes, the position of the target object in the image will typically change. If the target object remains stationary while the pose of the image acquisition component changes, the position of the target object in the image will also change, and it may even become out of the image. However, whether it is a change in the pose of the image acquisition component or a change in the position of the target object, the essence is a change in their relative positions. Therefore, the representation on the image is that the position of the target object relative to the center of the image changes.

[0078] Based on this, multiple historical images formed by the historical signals collected by the image acquisition component can be analyzed to determine the motion trajectory and speed of the target object in the image, that is, the trajectory and speed of the target object's position change in the image.

[0079] For example, if the current time is t, then n historical images at times t-1, t-2...tn can be obtained, and the motion trajectory of the target object can be determined based on these n historical images.

[0080] S22: Determine the first expected position of the current target object relative to the center point of the target image based on the motion trajectory.

[0081] Based on the trajectory of the motion prior to the current moment, we can determine the approximate orientation of the current position. Based on the velocity of the motion prior to the current moment, we can determine the distance between the current position and the previous position. Therefore, by combining the approximate orientation and distance, we can determine the exact position of the current moment. Since this position is calculated rather than actually measured, it may deviate from the actual position; hence, it is called the predicted position.

[0082] S23: Identify the image region within a predetermined range centered on the first expected position in the target image.

[0083] S24: When the current target object is identified, determine the first distance between the identified current target object and the center point of the target image, and the second distance between the identified current target object and a predetermined point on the boundary of the target image.

[0084] The steps S21 to S24 above first use kinematics to estimate the approximate position of the target object in the image based on historical images, and then identify the image near the approximate position. Thus, the actual position of the target object in the image is determined based on the identified target object. This can obtain the exact position without having to identify the entire image, thus reducing the amount of computation.

[0085] S30: Based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image, calculate the first distribution data corresponding to the current pose of the image acquisition component.

[0086] The first distribution data is used to characterize the quality of the field of view corresponding to the pose of the image acquisition component. The larger the first distribution data, the better the field of view corresponding to the pose of the image acquisition component, which means that it can more clearly acquire details of the relative position and pose of the target objects.

[0087] It should be noted that there are multiple ways to calculate the first distribution data. Depending on the calculation method, a larger value of the first distribution data may indicate a better field of view corresponding to the pose, or a smaller value may indicate a better field of view corresponding to the pose. Furthermore, these two methods can be converted to each other through simple transformations (e.g., replacing the increasing function in the calculation formula with a decreasing function, or replacing a positive correlation with a negative correlation). This specification and claims only use the example of a larger value of the first distribution data indicating a better field of view corresponding to the pose to illustrate the adjustment method of the image acquisition component provided in this specification. However, the technical solutions obtained through the above simple modifications should also be within the scope of protection of the claims.

[0088] In some embodiments, when calculating the first distribution data, only the first distance and the second distance may be considered. For example, the first distribution data can be calculated using the function FV = f(d, D), where FV is the first distribution data, d is the first distance between the target object in the target image and the center point of the target image, D is the second distance between the target object in the target image and a predetermined point on the boundary of the target image, and f(x) is an increasing function.

[0089] In some embodiments, when calculating the first distribution data, in addition to considering the first distance and the second distance, the pose deviation between the current pose of the image acquisition component and the reference pose, the area occupied by occluders in the target image, etc., are also considered. Figure 6 As shown, step S30 may include the following steps:

[0090] S31: Obtain a reference pose, wherein the reference pose is determined based on at least two target objects that are in a stationary state.

[0091] During the operation, the object to be operated on (e.g., lesion tissue) is usually in a stationary state, while the operating tool (e.g., the end of the tool held by the robotic arm) is in a movable state. Before performing the operation, the operator can first determine a reference pose based on the stationary object to be operated on. This reference pose is used to determine the pose of the operating tool.

[0092] like Figure 4 and Figure 7 As shown, the reference pose can be determined as follows: Based on the observed morphology of the object to be operated on (e.g., lesion tissue) and their own operating habits, the operator determines a coordinate system xyz based on the position of the object to be operated on (e.g., lesion tissue). In some embodiments, the unit vector with the smallest angle to the field of view axis of the image acquisition component in this coordinate system can be used as the reference pose; for example, the unit vector on the z-axis in the figure can be used as the reference pose.

[0093] In some embodiments, the orientation of the object to be operated on (e.g., lesion tissue) can be further determined based on the coordinate system xyz, and the orientation can be used as a reference pose.

[0094] In some embodiments, the pose of the image acquisition component can be represented using the xyz coordinate system described above. Alternatively, a new coordinate system abc can be established along the axes of the image acquisition component's field of view, such as... Figure 7 As shown, the pose of the image acquisition component is represented by coordinates in the abc coordinate system.

[0095] S32: Determine the pose deviation between the current pose of the image acquisition component and the reference pose.

[0096] In this specification, parameters related to pose and attitude, such as pose and attitude pointing vector, can be uniformly represented by one of the following: Euler angles, angular axes, or quaternions.

[0097] For example, with This represents the current pose quaternion of the image acquisition component. ε is the real part of the quaternion. d The imaginary part of the quaternion; The pose pointer quaternion of the object to be operated on is also known as the reference pose quaternion. ε is the real part of the quaternion. e Since the quaternion is the imaginary part, the posture deviation of the target biological tissue can be calculated using the following formula: Where e0 is the pose deviation between the current pose and the reference pose of the image acquisition component, S(·) is the antisymmetric operator, and * represents the product operation.

[0098] S33: When the target object is occluded, obtain the area occupied by the occluding object in the target image.

[0099] Before obtaining the area occupied by the occupant, you can first determine whether any target object is occluded. If the target object is occluded, you can obtain the area occupied by the occupant; if the target object is not occluded, you can set the area of ​​the occupant to a special preset value.

[0100] In some embodiments, the area occupied by the occupant in the target image can be the area value itself. In some embodiments, the area of ​​the occupant in the target object can also refer to the ratio of the area occupied by the occupant to the total area of ​​the target image.

[0101] S34: Based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance, pose deviation, and area between each target object and a predetermined point on the boundary of the target image, calculate the first distribution data corresponding to the current pose of the image acquisition component.

[0102] In some embodiments, step S34 may calculate the first distribution data corresponding to the current pose of the image acquisition component in the following manner: first, calculate the first distance between each target object in at least two target images and the center point of the target image by weighted summation to obtain a first value; calculate the second distance between each target object in at least two target images and a predetermined point on the boundary of the target image by weighted summation to obtain a second value; calculate the norm of the pose deviation to obtain a third value; and then calculate the first distribution data corresponding to the current pose of the image acquisition component based on the first value, the second value, the third value, and the area.

[0103] For example, the calculation formula can be:

[0104] Where FV represents the first distribution data corresponding to the current pose, n is the number of target objects, and u1, u2…u n v1, v2…v n w1, w2 are weighting coefficients, C is the paranoia coefficient, and C≥1, d1, d2…d n Let D1, D2, ..., D be the first distances between n target objects in the target image and the center point of the target image, respectively. n Let S0 be the second distance between n target objects in the target image and a predetermined point on the boundary of the target image, S0 be the area occupied by the occlusion in the target image, e0 be the pose deviation, ||e0|| be the norm of the pose deviation, and * denotes product operation.

[0105] The above formula is only one feasible implementation method for calculating the first distribution data. Based on the above formula, those skilled in the art should be able to conceive of other forms of calculation formulas, such as changing the ln function to a log function with other bases, or using a power function or exponential function, etc.

[0106] It should be noted that, in the absence of obstructions, Figure 6 The method shown can also remove S33, and accordingly, S34 is changed to "calculate the first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image, and the pose deviation", thus forming a new embodiment.

[0107] In some embodiments, when calculating the first distribution data, attitude deviation may be disregarded, i.e., in Figure 6 The method shown can remove S31 and S32. Accordingly, S34 is changed to "calculate the first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance and area between each target object and a predetermined point on the boundary of the target image", thus forming a new embodiment.

[0108] S40: When the first distribution data is less than or equal to a preset threshold, the target pose is determined within the pose adjustment range of the image acquisition component, wherein the first distribution data corresponding to the target pose is greater than the first distribution data corresponding to the current pose.

[0109] S50: Adjust the image acquisition component to the target pose.

[0110] In step S40, determining the target pose and adjusting the image acquisition component to the target pose should be performed automatically. That is, the target pose is automatically determined by electronic devices such as computers, and adjustment instructions for the image acquisition component are generated to control the movement of devices such as motors used to adjust the position of the image acquisition component.

[0111] The pose of the image acquisition component can include position and orientation. Position can refer to the distance between the image acquisition component and the target object, and orientation can be an angle at the position, such as the angle between the field of view axis of the image acquisition component and the reference axis, or the rotation angle around the reference axis.

[0112] Figure 8 A side view of the adjustable range of the image acquisition component is shown, where the small circle on the left represents the patient's head, the large ellipse on the right represents the patient's body (the same symbols in the accompanying drawings have the same meaning), and the triangle is actually a conical spatial dimension representing the pose adjustment range of the image acquisition component. The cone height h and cone diameter 2r can be input by the operator according to the actual needs during operation, or they can be input by the operator based on experience. Figure 9 A top view shows the adjustable range of the image acquisition component and the corresponding field of view. Here, J represents the image acquisition component, K represents the adjustable range of the image acquisition component, and L represents the field of view of the image acquisition component.

[0113] Existing methods such as iterative methods and heuristic search can be used to determine the target pose.

[0114] The image acquisition component adjustment method, apparatus, and electronic device provided in this specification, based on the target image formed by the signal acquired by the image acquisition component when it is in its current pose, determines a first distance between each of at least two target objects and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image. Based on the first and second distances, it calculates first distribution data corresponding to the current pose of the image acquisition component, thereby quantifying the distribution of the target image corresponding to the current pose. This facilitates the quantitative evaluation of the field of view corresponding to the current pose by a computer or electronic device. When the first distribution data is less than or equal to a preset threshold, this solution can automatically search for a target pose with a first distribution data greater than that corresponding to the current pose within the pose adjustment range, i.e., find a target pose with a better field of view. Therefore, this solution can achieve automatic adjustment of the image acquisition component, and this automatic adjustment method can improve the field of view of the image acquisition component. This allows the operator to only operate the operating tool without switching back and forth between operating the operating tool and the image acquisition component, reducing the operator's workload and shortening the execution time of the operation task.

[0115] In some embodiments, such as Figure 10 As shown, before step S50, the procedure further includes:

[0116] S61: Perform path planning based on the target pose to obtain the adjustment path of the image acquisition component from the current pose to the target pose.

[0117] The path planning method can use the object to be operated on as an obstacle and employ the RRT algorithm to plan a path. The image acquisition component will not encounter the object to be operated on during the adjustment process along this path. Of course, other existing algorithms that can be used for path planning can also be employed, such as the rectangular acceleration curve algorithm, which will not be elaborated upon in this application.

[0118] The result of path planning, namely path adjustment, refers to the motion trajectory of the image acquisition component itself, as well as the motion trajectory of each joint on the robotic arm equipped with the image acquisition component on the automated equipment.

[0119] S62: Based on the path planning results, determine whether the image acquisition component will collide with at least two movable objects in the process of adjusting from the current pose to the target pose.

[0120] S63: When the image acquisition component does not collide with the movable object, adjust the image acquisition component to the target pose according to the path planning result.

[0121] During the adjustment of the image acquisition component, movable objects within the target object, i.e., the end effector of the tool, may also move. If the image acquisition component collides with a movable object during adjustment, it may affect the operation result or cause quality issues such as jitter in the image formed from the signals acquired by the image acquisition component. After obtaining the path planning result, an additional judgment step is added to determine whether a collision with a movable object has occurred. If no collision occurs, the image acquisition component is adjusted to the target pose according to the path planning result. This prevents the impact on the operation result and avoids image quality problems. If the judgment result indicates that a collision will occur, path planning can be re-performed to obtain a different path, or the target pose can be redefined.

[0122] In some embodiments, such as Figure 11 As shown, after S10, it also includes:

[0123] S70: When the target object does not exist in the target image, determine the second expected position of the target object outside the target image.

[0124] Since the operation usually requires focusing on at least two target objects, it often happens that some target objects are in the target image while others are not. For target objects that are not in the target image, they need to be adjusted to be in the target image. To do this, it is first necessary to determine the approximate location of the target object outside the target image.

[0125] In step S70, determining the second expected position of the target object outside the target image can be achieved as follows: Based on multiple historical images formed from historical signals acquired by the image acquisition component, the motion trajectory of the target object is determined; based on the motion trajectory, the second expected position of the target object outside the target image is determined. For details of this method, please refer to the descriptions of steps S21 and S22, which will not be repeated here.

[0126] S80: Perform path planning based on the second expected position to obtain the adjustment path of the image acquisition component from the current pose toward the second expected position.

[0127] For details on the path planning method, please refer to the description of step S 141.

[0128] S90: Based on the path planning results, generate prompts for the adjustment direction of the image acquisition component, so that the operator can adjust the pose of the image acquisition component according to the adjustment direction using the pose adjustment manipulator.

[0129] The prompts can be arrow indicators, voice prompts, or a panoramic map showing the positional relationship between the target object and the target image, with the panoramic image serving as a prompt. Figure 12 and Figure 13 A schematic diagram of a panoramic map is shown, in which circles represent target images, A1 and A2 are movable target objects, and B is a target object in a stationary state.

[0130] It should be noted that when the movable target object is outside the target image, in order to bring it into the target image, in addition to adjusting the image acquisition component, the position of the movable object can also be adjusted. Since this solution focuses on the adjustment method of the image acquisition component, the specific methods for adjusting the position of the movable object will not be described.

[0131] In some embodiments, such as Figure 11 As shown, following S80, it also includes:

[0132] S100: Feedback control resistance on the pose adjustment manipulator of the image acquisition component, and the control resistance in the first operation direction of the pose adjustment manipulator is less than the control resistance in other operation directions, wherein the first operation direction is the operation direction corresponding to the adjustment direction of the image acquisition component in the path planning result, and the other operation directions are the operation directions other than the first operation direction in the preset operation direction set.

[0133] When the operator uses the pose adjustment manipulator to adjust the pose of the image acquisition component, the pose adjustment manipulator will provide a control force prompt, that is, the resistance in the adjustment direction of the image acquisition component in the planning result is small, and the operator can confirm the control direction of the manipulator based on the magnitude of the resistance.

[0134] When the operator determines that the adjustment direction derived from the path planning result is incorrect, the operator may not control the manipulator in the indicated direction. For example, according to the path planning result, the operator should control the manipulator to the left. There is no resistance in the left direction, but there is resistance in the right, up, and down directions. Instead of controlling the manipulator to the left, the operator should use a force that can overcome the resistance to control the manipulator to the right.

[0135] In some embodiments, such as Figure 11 As shown, following S90, it also includes:

[0136] S110: During the process of adjusting the pose of the image acquisition component through the pose adjustment manipulator, the second distribution data corresponding to the pose of the image acquisition component is calculated in real time based on the image formed by the signal collected by the image acquisition component.

[0137] S120: When the second distribution data corresponding to the pose of the image acquisition component reaches a predetermined range, a target pose is determined within the pose adjustment range of the image acquisition component, and the image acquisition component is adjusted to the target pose, wherein the first distribution data corresponding to the target pose is greater than the first distribution data corresponding to the current pose.

[0138] In other words, while the operator is manually adjusting the pose of the image acquisition component, the second distribution data corresponding to the pose is calculated in real time. Based on this second distribution data, it is determined whether the conditions for automatic pose adjustment have been met. When the conditions for automatic pose adjustment are met, the automatic pose adjustment method can be activated, and the manual pose adjustment permission can be revoked.

[0139] In some embodiments, the calculation of the first distribution data and the second distribution data can be the same.

[0140] In some embodiments, such as Figure 14 As shown, the second distribution data can be calculated in the following way:

[0141] S1410: Acquire the first image formed by the signal acquired by the image acquisition component at the first moment.

[0142] S1420: Obtain the first distance between each of the at least two target objects in the first image and the center point of the first image, and the second distance between each target object and a predetermined point on the boundary of the first image; wherein, when the target object is not in the first image, determine the first distance between the second expected position of the target object and the center point of the first image, and the second distance between the second expected position of the target object and the predetermined point on the boundary of the first image.

[0143] S1430: Based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image, the second distribution data corresponding to the pose of the image acquisition component at the first moment is calculated.

[0144] In some embodiments, such as Figure 15 As shown, S1430 may include the following steps:

[0145] S1431: Based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image, a first distribution matrix corresponding to the pose at the first moment is formed.

[0146] S1432: Determine the second image formed by the signal acquired by the image acquisition component at the second time and the estimated positions of at least two target objects, and determine the first distance between the estimated positions of at least two target objects and the center point of the second image, and the second distance between the estimated positions of at least two target objects and predetermined points on the boundary of the second image.

[0147] The estimated position can be determined by referring to the descriptions of the first and second estimated positions. Simultaneously with determining the estimated position, the estimated pose of the image acquisition component at the second time step can also be determined.

[0148] S1433: Based on the first distance between the estimated positions of at least two target objects and the center point of the second image, and the second distance between the estimated positions of at least two target objects and a predetermined point on the boundary of the second image, a second distribution matrix corresponding to the pose at the second time moment is formed.

[0149] S1434: Based on the first distribution matrix and the second distribution matrix, calculate the second distribution data corresponding to the pose of the image acquisition component at the first moment.

[0150] In some embodiments, the elements in the first distribution matrix may also include: the area occupied by the image of the occluder in the first image; and / or, the pose deviation between the pose at the first moment and the reference pose, wherein the reference pose is determined in advance based on at least two target objects that are in a stationary state.

[0151] For example, the first distribution matrix corresponding to the pose at the first moment can be:

[0152] V t =W*[d 1t d 2t …d nt D 1t D 2t …D nt ||e 0t ||S 0t ],

[0153] Among them, V t The first distribution matrix corresponding to the pose of the image acquisition component at the first moment, W∈R (n +2)×(n+2) Let d be the coefficient matrix. 1t ,d 2t …d nt Let D be the first distance between each of the n target objects and the center point of the first image at the first time step. 1t D 2t …D nt S represents the second distance between the n target objects at the first time step and a predetermined point on the boundary of the first image. 0t e represents the area occupied by the occluder in the first image at the first moment. 0t For the attitude deviation at the first moment, ||e 0t || represents the norm of the attitude deviation at the first moment, and * represents the product operation.

[0154] The second distribution matrix corresponding to the pose at the second time step can be:

[0155] V t+1 =W*[d 1t+1 d 2t+1 …d nt+1 D 1t+1 D 2t+1 …D nt+1 ||e 0t+1 ||S 0t+1 ],

[0156] Among them, V t The first distribution matrix corresponding to the pose of the image acquisition component at the first moment, W∈R (n +2)×(n+2) Let d be the coefficient matrix. 1t+1 ,d 2t+1 …d nt+1 The first distances d and d are respectively the first distances between the n target objects at the second time point and the center point of the first image. 1t+1 ,d 2t+1 …d nt+1S represents the second distance between the n target objects at the second time point and a predetermined point on the boundary of the first image. 0t e represents the area occupied by the occluder in the first image at the second time step. 0t The attitude deviation at the second time step can be calculated using the estimated pose and the reference pose at the second time step, ||e 0t || represents the norm of the attitude deviation at the second moment, and * represents the product operation.

[0157] The calculation methods for the first distance, second distance, area, and attitude deviation mentioned above can be found in the relevant descriptions above.

[0158] Finally, the second distribution data corresponding to the pose at the first moment can be calculated using the following formula:

[0159] FV=||V t ×V t+1 ||, where FV is the second distribution data corresponding to the pose at the first moment.

[0160] This specification provides an adjustment device for an image acquisition component, which can be used to implement the adjustment method for the image acquisition component provided in this specification. Figure 16 As shown, the device includes an acquisition unit 10, a first determination unit 20, a first calculation unit 30, a second determination unit 40, and an adjustment unit 50.

[0161] The acquisition unit 10 is used to acquire a target image formed by signals collected by the image acquisition component when it is in its current pose. The first determination unit 20 is used to determine a first distance between each of the at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image; the predetermined point is the point on the boundary of the target image that is closest to the target object. The first calculation unit 30 is used to calculate a first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance between each target object and the predetermined point on the boundary of the target image. The second determination unit 40 is used to determine a target pose within the pose adjustment range of the image acquisition component when the first distribution data is less than or equal to a preset threshold, wherein the first distribution data corresponding to the target pose is greater than the first distribution data corresponding to the current pose. The adjustment unit 50 is used to adjust the image acquisition component to the target pose.

[0162] In some embodiments, the first calculation 30 includes a first acquisition subunit 31, a first determination subunit 32, and a first calculation subunit 33.

[0163] The first acquisition subunit 31 is used to acquire a reference pose, wherein the reference pose is determined based on a target object that is stationary among at least two target objects. The first determination subunit 32 is used to determine the pose deviation between the current pose of the image acquisition component and the reference pose. The first calculation subunit 33 is used to calculate the first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, and the pose deviation.

[0164] In some embodiments, the first calculation subunit 33 includes a second acquisition subunit 331 and a second calculation subunit 332.

[0165] The second acquisition subunit 331 is used to acquire the area occupied by the occluder in the target image when the target object is occluded. The second calculation subunit 332 is used to calculate the first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance, pose deviation, and area between each target object and a predetermined point on the boundary of the target image.

[0166] In some embodiments, the second calculation subunit 332 calculates the first distribution data corresponding to the current pose of the image acquisition component according to the following method: a first value is obtained by weighted summation of the first distance between each target object in at least two target images and the center point of the target image; a second value is obtained by weighted summation of the second distance between each target object in at least two target images and a predetermined point on the boundary of the target image; a third value is obtained by calculating the norm of the pose deviation; and the first distribution data corresponding to the current pose of the image acquisition component is calculated based on the first value, the second value, the third value, and the area.

[0167] In some embodiments, the device further includes a first planning unit 60 and a judgment unit 70.

[0168] The first planning unit 60 is used to perform path planning based on the target pose to obtain the adjustment path of the image acquisition component from the current pose to the target pose. The judgment unit 70 is used to determine, based on the path planning result, whether the image acquisition component will collide with at least two movable objects during the adjustment process from the current pose to the target pose. When the image acquisition component will not collide with any movable objects, the adjustment unit 50 performs adjustment of the image acquisition component to the target pose.

[0169] In some embodiments, the first determining unit 20 includes a second determining subunit 21, a third determining subunit 22, an identification subunit 23, and a fourth determining subunit 24. The second determining subunit 21, the third determining subunit 22, the identification subunit 23, and the fourth determining subunit 24 determine a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image, in the following manner: The second determining subunit 21 is used to determine the motion trajectory of the current target object based on multiple historical images formed from historical signals acquired by the image acquisition component. The third determining subunit 22 is used to determine a first expected position of the current target object relative to the center point of the target image based on the motion trajectory. The identification subunit 23 is used to identify an image region within a predetermined range centered on the first expected position in the target image. The fourth determining subunit 24 is used to determine, when the current target object is identified, the first distance between the identified current target object and the center point of the target image, and the second distance between the identified current target object and the predetermined point on the boundary of the target image.

[0170] In some embodiments, the device further includes a second determining unit 80, a second planning unit 90, and a prompting unit 100.

[0171] The second determining unit 80 is used to determine a second expected position of the target object outside the target image when the target object does not exist in the target image. The second planning unit 90 is used to perform path planning based on the second expected position to obtain an adjustment path for the image acquisition component from its current pose toward the second expected position. The prompting unit 100 is used to generate prompt information about the adjustment direction of the image acquisition component based on the path planning result, so that the operator can adjust the pose of the image acquisition component according to the adjustment direction using the pose adjustment manipulator.

[0172] In some embodiments, the device further includes: a force feedback unit 110 for feeding back control resistance on the pose adjustment manipulator of the image acquisition component, wherein the control resistance in a first operating direction of the pose adjustment manipulator is less than the control resistance in other operating directions, wherein the first operating direction is the operating direction corresponding to the adjustment direction of the image acquisition component in the path planning result, and the other operating directions are operating directions other than the first operating direction in a preset set of operating directions.

[0173] In some embodiments, the apparatus further includes: a second calculation unit 120 configured to calculate, in real time, second distribution data corresponding to the pose of the image acquisition component based on the image formed by the signal acquired by the image acquisition component during the process of adjusting the pose of the image acquisition component by the pose adjustment manipulator; and a second determination unit 40 and an adjustment unit 50 configured to determine a target pose within the pose adjustment range of the image acquisition component when the second distribution data corresponding to the pose of the image acquisition component reaches a predetermined range, and adjust the image acquisition component to the target pose, wherein the first distribution data corresponding to the target pose is greater than the first distribution data corresponding to the current pose.

[0174] In some embodiments, the first distribution data and the second distribution data are calculated in the same way.

[0175] In some embodiments, the second calculation unit 120 includes a third acquisition subunit 121, a fourth acquisition subunit 122, and a third calculation subunit 123.

[0176] The third acquisition subunit 121 is used to acquire a first image formed by the signals acquired by the image acquisition component at the first moment. The fourth acquisition subunit 122 is used to acquire a first distance between each of at least two target objects in the first image and the center point of the first image, and a second distance between each target object and a predetermined point on the boundary of the first image; wherein, when the target object is not in the first image, the first distance between the second expected position of the target object and the center point of the first image, and the second distance between the second expected position of the target object and the predetermined point on the boundary of the first image are determined. The third calculation subunit 123 is used to calculate the second distribution data corresponding to the pose of the image acquisition component at the first moment based on the first distance and the second distance.

[0177] In some embodiments, the third calculation subunit 123 includes a first forming subunit 1231, a fifth determining subunit 1232, a second forming subunit 1233, and a fourth calculation subunit 1234.

[0178] The first forming subunit 1231 is used to form a first distribution matrix corresponding to the pose at the first moment based on the first distance and the second distance. The fifth determining subunit 1232 is used to determine the second image formed by the signal acquired by the image acquisition component at the second moment and the estimated positions of at least two target objects, and to determine the first distance between the estimated positions of at least two target objects and the center point of the second image, and the second distance between the estimated positions of at least two target objects and predetermined points on the boundary of the second image. The second forming subunit 1233 is used to form a second distribution matrix corresponding to the pose at the second moment based on the first distance between the estimated positions of at least two target objects and the center point of the second image, and the second distance between the estimated positions of at least two target objects and predetermined points on the boundary of the second image. The fourth calculation subunit 1234 is used to calculate the second distribution data corresponding to the pose of the image acquisition component at the first moment based on the first distribution matrix and the second distribution matrix.

[0179] In some embodiments, the elements in the first distribution matrix further include: the area occupied by the occluder in the first image when the target object is occluded; and / or, the pose deviation between the pose at the first moment and the reference pose, wherein the reference pose is determined in advance based on the target object in a stationary state among at least two target objects.

[0180] In some embodiments, the image acquisition component is an endoscope lens or an ultrasound detector.

[0181] For details regarding the adjustment device of the aforementioned image acquisition component, please refer to [link / reference needed]. Figures 1 to 15 The relevant descriptions and effects in the corresponding embodiments are for reference only and will not be repeated here.

[0182] This invention also provides an electronic device, such as... Figure 17 As shown, the electronic device may include a processor 1701 and a memory 1702, wherein the processor 1701 and the memory 1702 may be connected via a bus or other means. Figure 17 Taking the example of a connection between China and Israel via a bus.

[0183] Processor 1701 can be a Central Processing Unit (CPU). Processor 1701 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0184] Memory 1702, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the adjustment method of the image acquisition component in this embodiment of the invention (e.g., Figure 16 The processor 1701 comprises an acquisition unit 10, a first determination unit 20, a first calculation unit 30, a second determination unit 40, and an adjustment unit 50. The processor 1701 executes various functional applications and data classification by running non-transitory software programs, instructions, and modules stored in the memory 1702, thereby implementing the adjustment method of the image acquisition component in the above method embodiment.

[0185] Memory 1702 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by processor 1701, etc. Furthermore, memory 1702 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 1702 may optionally include memory remotely located relative to processor 1701, and these remote memories may be connected to processor 1701 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0186] The one or more modules are stored in the memory 1702, and when executed by the processor 1701, they perform the following: Figures 1 to 15 The adjustment method of the image acquisition component in the illustrated embodiment.

[0187] For specific details regarding the aforementioned electronic devices, please refer to [link / reference]. Figures 1 to 15 The relevant descriptions and effects in the corresponding embodiments are for reference only and will not be repeated here.

[0188] This specification also provides a computer storage medium storing computer program instructions, which, when executed, implement... Figures 1 to 15 The steps corresponding to the embodiments.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0190] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0191] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0192] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0193] The above description is merely an embodiment of the present specification and is not intended to limit the embodiments of the present specification. For those skilled in the art, various modifications and variations can be made to the embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of the present specification should be included within the scope of the claims of the embodiments of the present specification.

Claims

1. A method for adjusting an image acquisition component, characterized in that, include: The image acquisition component acquires the target image formed by signals collected when it is in the current pose. Determine a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image; the predetermined point is the point on the boundary of the target image that is closest to the target object. Based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image, the first distribution data corresponding to the current pose of the image acquisition component is calculated. When the first distribution data is less than or equal to a preset threshold, a target pose is determined within the pose adjustment range of the image acquisition component, and the image acquisition component is adjusted to the target pose, wherein the first distribution data corresponding to the target pose is greater than the first distribution data corresponding to the current pose. Specifically, based on a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image, the first distribution data corresponding to the current pose of the image acquisition component is calculated, including: When the target object is occluded, obtain the area occupied by the occluding object in the target image; Based on the first distance between each of the at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, and the area, the first distribution data corresponding to the current pose of the image acquisition component is calculated.

2. The method according to claim 1, characterized in that, Based on the first distance between each of at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image, the first distribution data corresponding to the current pose of the image acquisition component is calculated, including: Obtain a reference pose, wherein the reference pose is determined based on a target object that is stationary among the at least two target objects; Determine the pose deviation between the current pose of the image acquisition component and the reference pose; Based on the first distance between each of the at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, and the pose deviation, the first distribution data corresponding to the current pose of the image acquisition component is calculated.

3. The method according to claim 2, characterized in that, Based on the first distance between each of at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, and the pose deviation, the first distribution data corresponding to the current pose of the image acquisition component is calculated, including: When the target object is occluded, obtain the area occupied by the occluding object in the target image; Based on the first distance between each of the at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, the pose deviation, and the area, the first distribution data corresponding to the current pose of the image acquisition component is calculated.

4. The method according to claim 3, characterized in that, Based on the first distance between each of at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, the pose deviation, and the area, the first distribution data corresponding to the current pose of the image acquisition component is calculated, including: A weighted summation is performed on the first distance between each target object in at least two target images and the center point of the target image to obtain a first value. A second value is obtained by weighted summation of the second distances between each target object in at least two target images and a predetermined point on the boundary of the target image; Calculate the norm of the attitude deviation to obtain the third value; Based on the first value, the second value, the third value, and the area, the first distribution data corresponding to the current pose of the image acquisition component is calculated.

5. The method according to claim 1, characterized in that, Before adjusting the image acquisition component to the target pose, the method further includes: Based on the target pose, path planning is performed to obtain the adjustment path of the image acquisition component from the current pose to the target pose; Based on the path planning results, it is determined whether the image acquisition component will collide with the movable object among the at least two target objects during the process of adjusting from the current pose to the target pose; When the image acquisition component does not collide with the movable object, the adjustment of the image acquisition component to the target pose is performed.

6. The method according to claim 1, characterized in that, Determining a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image, includes: The first distance between each of at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image are determined as follows: The motion trajectory of the current target object is determined by using multiple historical images formed from historical signals acquired by the image acquisition component. Based on the motion trajectory, determine the first expected position of the current target object relative to the center point of the target image; Identify an image region within a predetermined range centered on a first predicted location in the target image; When a current target object is identified, a first distance between the identified current target object and the center point of the target image, and a second distance between the identified current target object and a predetermined point on the boundary of the target image are determined.

7. The method according to claim 1, characterized in that, After acquiring the target image formed by the signals collected by the image acquisition component when it is in the current pose, the following is also included: When the target object is not present in the target image, a second expected position of the target object outside the target image is determined; Based on the second expected position, path planning is performed to obtain the adjustment path of the image acquisition component from the current pose toward the second expected position; Based on the path planning results, prompt information is generated regarding the adjustment direction of the image acquisition component, so that the operator can adjust the pose of the image acquisition component according to the adjustment direction using a pose adjustment manipulator.

8. The method according to claim 7, characterized in that, After performing path planning based on the second predicted position to obtain the adjustment path of the image acquisition component from the current pose toward the second predicted position, the method further includes: Feedback control resistance is provided on the pose adjustment manipulator of the image acquisition component, and the control resistance in the first operating direction of the pose adjustment manipulator is less than the control resistance in other operating directions. The first operating direction is the operating direction corresponding to the adjustment direction of the image acquisition component in the path planning result, and the other operating directions are operating directions other than the first operating direction in the preset set of operating directions.

9. The method according to claim 7, characterized in that, After generating the prompt information regarding the adjustment direction of the image acquisition component, the following is also included: During the process of adjusting the pose of the image acquisition component through the pose adjustment manipulator, the second distribution data corresponding to the pose of the image acquisition component is calculated in real time based on the image formed by the signal acquired by the image acquisition component. When the second distribution data corresponding to the pose of the image acquisition component reaches a predetermined range, a target pose is determined within the pose adjustment range of the image acquisition component, and the image acquisition component is adjusted to the target pose.

10. The method according to claim 9, characterized in that, The first distribution data and the second distribution data are calculated in the same way.

11. The method according to claim 9, characterized in that, The second distribution data corresponding to the pose of the image acquisition component is calculated in real time based on the image formed by the signals acquired by the image acquisition component, including: The image acquisition component captures the first image formed by the signal acquired at the first moment. Obtain a first distance between each of at least two target objects in the first image and the center point of the first image, and a second distance between each target object and a predetermined point on the boundary of the first image; wherein, when a target object is not in the first image, determine the first distance between the second expected position of the target object and the center point of the first image, and the second distance between the second expected position of the target object and the predetermined point on the boundary of the first image; Based on the first distance and the second distance, the second distribution data corresponding to the pose of the image acquisition component at the first moment is calculated.

12. The method according to claim 11, characterized in that, Based on the first distance and the second distance, the second distribution data corresponding to the pose of the image acquisition component at the first moment is calculated, including: Based on the first distance and the second distance, a first distribution matrix corresponding to the pose at the first moment is formed; Determine the second image formed by the signal acquired by the image acquisition component at the second time and the estimated positions of the at least two target objects, and determine the first distance between the estimated positions of the at least two target objects and the center point of the second image, and the second distance between the estimated positions of the at least two target objects and predetermined points on the boundary of the second image; Based on the first distance between the estimated positions of the at least two target objects and the center point of the second image, and the second distance between the estimated positions and predetermined points on the boundary of the second image, a second distribution matrix corresponding to the pose at the second time moment is formed. Based on the first distribution matrix and the second distribution matrix, the second distribution data corresponding to the pose of the image acquisition component at the first moment is calculated.

13. The method according to claim 12, characterized in that, The elements in the first distribution matrix also include: When the target object is occluded, the area occupied by the occluder in the first image; And / or, The pose deviation between the pose at the first moment and the reference pose, wherein the reference pose is determined in advance based on the target object in a stationary state among the at least two target objects.

14. An adjustment device for an image acquisition component, characterized in that, include: The acquisition unit is used to acquire the target image formed by the signals collected by the image acquisition component when it is in the current pose; The first determining unit is configured to determine a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image; the predetermined point is the point on the boundary of the target image that is closest to the target object. The first calculation unit is used to calculate the first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, and the second distance between each target object and a predetermined point on the boundary of the target image. The second determining unit is used to determine a target pose within the pose adjustment range of the image acquisition component when the first distribution data is less than or equal to a preset threshold, wherein the first distribution data corresponding to the target pose is greater than the first distribution data corresponding to the current pose. An adjustment unit is used to adjust the image acquisition component to the target pose; Specifically, based on a first distance between each of at least two target objects in the target image and the center point of the target image, and a second distance between each target object and a predetermined point on the boundary of the target image, the first distribution data corresponding to the current pose of the image acquisition component is calculated, including: When the target object is occluded, obtain the area occupied by the occluding object in the target image; Based on the first distance between each of the at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, and the area, the first distribution data corresponding to the current pose of the image acquisition component is calculated.

15. The apparatus according to claim 14, characterized in that, The first computing unit includes: The first acquisition subunit is used to acquire a reference pose, wherein the reference pose is determined based on the target object that is stationary among the at least two target objects; The first determining subunit is used to determine the pose deviation between the current pose of the image acquisition component and the reference pose; The first calculation subunit is used to calculate the first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, and the pose deviation.

16. The apparatus according to claim 15, characterized in that, The first computing subunit includes: The second acquisition subunit is used to acquire the area occupied by the occlusion in the target image when the target object is occluded; The second calculation subunit is used to calculate the first distribution data corresponding to the current pose of the image acquisition component based on the first distance between each of the at least two target objects in the target image and the center point of the target image, the second distance between each target object and a predetermined point on the boundary of the target image, the pose deviation, and the area.

17. An electronic device, characterized in that, include: A memory and a processor, the processor and the memory being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to implement the steps of the method according to any one of claims 1 to 13.

18. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed, implement the steps of the method according to any one of claims 1 to 13.

Images