Casing Alignment Method, Device, Computer Equipment, Storage Medium

Abstract

The present application relates to a casing alignment method, device, computer device, storage medium, and computer program product. The method includes: determining a target fixed point of at least one casing to be aligned in a target action area in a target working scenario. Determining a virtual fixed point of each alignment component according to a target in at least one alignment component in the target working scenario. Virtually fusing the target fixed point, the virtual fixed point, and the target working scenario, and displaying the target fixed point on the casing to be aligned and the virtual fixed point on the alignment component through an augmented reality device to guide the alignment of the casing to be aligned with the alignment component. In this way, the alignment accuracy of the casing to be aligned is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular, to a casing alignment method, device, computer device, storage medium, and computer program product. Background Art

[0002] With the development of instrument technology, in order to perform repair operations on a tiny abnormal area, a tiny incision needs to be made on the surface of the target object, and then a casing is placed in the incision so that the manipulator can insert the instrument for the repair operation into the abnormal area of the target object through the channel formed by the casing to complete the repair operation.

[0003] Before the instrument reaches the channel formed by the casing, it is necessary to align the manipulator with the casing, that is, align the fixed point on the manipulator with the fixed point on the casing. In traditional technologies, the alignment operation is often completed by an operator manually controlling the movement of the manipulator.

[0004] However, during the process of the operator moving the manipulator, the operator does not know the real-time position of the fixed point on the manipulator. Therefore, it is difficult to accurately align the manipulator with the casing, that is, there is a problem of low accuracy in casing alignment. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a casing alignment method, device, computer device, computer-readable storage medium, and computer program product that can improve the accuracy rate of casing alignment.

[0006] In a first aspect, the present application provides a casing alignment method. The method includes:

[0007] Determine the target fixed point of at least one casing to be aligned in the target working scenario in the target action area;

[0008] According to the target in at least one alignment component in the target working scenario, determine the virtual fixed point of each alignment component;

[0009] Perform virtual fusion on the target fixed point, the virtual fixed point, and the target working scenario, and display the target fixed point on the casing to be aligned and the virtual fixed point on the alignment component through an augmented reality device to guide the alignment of the casing to be aligned with the alignment component.

[0010] In a second aspect, the present application further provides a casing alignment device. The device includes:

[0011] A first determination module, configured to determine the target fixed point of at least one casing to be aligned in the target working scenario

[0012] A second determination module, configured to determine virtual fixed points of each of the alignment components according to targets in at least one alignment component in the target working scenario;

[0013] A display module, configured to perform virtual fusion on the target fixed point, the virtual fixed points, and the target working scenario, and display the target fixed point on the to-be-aligned sleeve through an augmented reality device, and display the virtual fixed points on the alignment components, so as to guide the alignment of the to-be-aligned sleeve with the alignment components.

[0014] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0015] Determine a target fixed point of at least one to-be-aligned sleeve in a target working scenario in a target action area;

[0016] Determine virtual fixed points of each of the alignment components according to targets in at least one alignment component in the target working scenario;

[0017] Perform virtual fusion on the target fixed point, the virtual fixed points, and the target working scenario, and display the target fixed point on the to-be-aligned sleeve through an augmented reality device, and display the virtual fixed points on the alignment components, so as to guide the alignment of the to-be-aligned sleeve with the alignment components.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0019] Determine a target fixed point of at least one to-be-aligned sleeve in a target working scenario in a target action area;

[0020] Determine virtual fixed points of each of the alignment components according to targets in at least one alignment component in the target working scenario;

[0021] Perform virtual fusion on the target fixed point, the virtual fixed points, and the target working scenario, and display the target fixed point on the to-be-aligned sleeve through an augmented reality device, and display the virtual fixed points on the alignment components, so as to guide the alignment of the to-be-aligned sleeve with the alignment components.

[0022] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0023] Determine the target fixed points of at least one sleeve to be aligned in the target working scenario in the target action area;

[0024] Determine the virtual fixed points of each alignment component according to the targets in at least one alignment component in the target working scenario;

[0025] Perform virtual fusion on the target fixed points, the virtual fixed points and the target working scenario, and display the target fixed points on the sleeve to be aligned and the virtual fixed points on the alignment component through an augmented reality device to guide the alignment of the sleeve to be aligned with the alignment component.

[0026] The above sleeve alignment method, device, computer device, storage medium and computer program product realize the accurate positioning of the target fixed points in the target working scenario by determining the target fixed points of at least one sleeve to be aligned in the target working scenario in the target action area, and truthfully reflect the position information of the sleeve to be aligned. According to the targets in at least one alignment component in the target working scenario, the accurate positioning of the virtual fixed points of each alignment component in the target working scenario can be realized, and the position information of the alignment component is accurately and effectively reflected. Perform virtual fusion on the target fixed points, the virtual fixed points and the target working scenario, and display the target fixed points on the sleeve to be aligned and the virtual fixed points on the alignment component through an augmented reality device to guide the alignment of the sleeve to be aligned with the alignment component. In this way, by real-time displaying the positions of the target fixed points and the virtual fixed points in the augmented reality device, the alignment state of the sleeve to be aligned and the alignment component in the target working scenario can be reproduced in a timely and accurate manner, greatly improving the accuracy of the alignment of the sleeve to be aligned. Description of the Drawings

[0027] Figure 1 It is an application environment diagram of the sleeve alignment method in an embodiment;

[0028] Figure 2 It is a flowchart of the sleeve alignment method in an embodiment;

[0029] Figure 3 It is a schematic diagram of the target working scenario in an embodiment;

[0030] Figure 4A It is a schematic diagram of the shape of the target in an embodiment;

[0031] Figure 4B It is a schematic diagram of the shape of the target in another embodiment;

[0032] Figure 5 It is a schematic diagram of the target coordinate system in an embodiment;

[0033] Figure 6Schematic diagram of the display of the virtual fixed point and the target fixed point in an augmented reality device in an embodiment;

[0034] Figure 7 Flow schematic diagram of the steps for determining the target fixed point in an embodiment;

[0035] Figure 8 Flow schematic diagram of the steps for determining the target fixed point in another embodiment;

[0036] Figure 9 Schematic diagram of the situation where the tail of the alignment sleeve intersects with the target action area in an embodiment;

[0037] Figure 10 Flow schematic diagram of the steps for determining the target contour points in an embodiment;

[0038] Figure 11 Flow schematic diagram of the steps for determining the target fixed point in another embodiment;

[0039] Figure 12 Flow schematic diagram of the steps for determining the virtual fixed point in an embodiment;

[0040] Figure 13 Flow schematic diagram of the steps for determining the virtual fixed point in another embodiment;

[0041] Figure 14 Schematic diagram of the conversion relationship between three coordinate systems in an embodiment;

[0042] Figure 15 Flow schematic diagram of the steps for determining the virtual fixed point in another embodiment;

[0043] Figure 16 Schematic diagram of the target working scenario in another embodiment;

[0044] Figure 17 Schematic diagram of the pairing process in an embodiment;

[0045] Figure 18 Flow schematic diagram of monitoring the pairing operation in an embodiment;

[0046] Figure 19 Flow schematic diagram of the sleeve alignment method in another embodiment;

[0047] Figure 20 Schematic diagram of the unit used for implementing the sleeve alignment method in an embodiment;

[0048] Figure 21 Structural block diagram of the sleeve alignment device in an embodiment;

[0049] Figure 22Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0050] To make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The casing alignment method provided by the embodiments of the present application can be applied to, for example Figure 1 the application environment shown in the figure. Among them, the alignment component 102 communicates with the computer device 106 through the network, and the augmented reality device 104 communicates with the computer device 106 through the network. The data storage system can store the data that the computer device 106 needs to process. The data storage system can be integrated in the computer device 106, or can be placed in the cloud or on other network computer devices. The computer device 106 determines the target fixed points of at least one casing to be aligned in the target working scenario in the target action area. The computer device 106 determines the virtual fixed points of each of the alignment components 102 according to the targets in at least one alignment component 102 in the target working scenario. The target fixed point, the virtual fixed point and the target working scenario are virtually fused, and the target fixed point is displayed on the casing to be aligned through the augmented reality device 104, and the virtual fixed point is displayed on the alignment component 102 to guide the casing to be aligned with the alignment component 102. Among them, the computer device can be a terminal or a server. Among them, the terminal can be, but is not limited to, various personal computers, console devices, laptop computers, smart phones, tablet portable wearable devices. The portable wearable device can be a head-mounted device. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0052] In one embodiment, as Figure 2 shown in the figure, a casing alignment method is provided. Taking the method applied to Figure 1 the computer device 106 in the figure as an example, the method includes the following steps:

[0053] Step S202, determine the target fixed points of at least one casing to be aligned in the target working scenario in the target action area.

[0054] Among them, the target working scenario is a scenario including alignment components, casings to be aligned, and target objects, such as Figure 3As shown in the figure. Among them, the alignment component can be one end of the robotic arm aligned with the sleeve to be aligned, or the entire robotic arm, and there is no specific limitation. Among them, the sleeve to be aligned can be a trocar, which is used to form a tubular working channel in the target action area of the target object. Among them, the augmented reality device is used to observe the movement of the alignment component in the virtual scene in real time, and this virtual scene is a scene that can truly restore the target working scene. Among them, the target action area is the abdominal cavity area of the target object.

[0055] Specifically, in the target working scene, a punching process is pre-performed on the target action area of the target object. The computer device obtains the environmental image captured by the camera device for the target working scene, or the computer device obtains the target image captured by the camera device for the sleeve to be aligned in the target working scene. Among them, the environmental image contains the detailed information of the sleeve to be aligned and the alignment component, and the target image only contains the detailed information of the sleeve to be aligned. The computer device determines the position of a target fixed point implemented in the target working scene based on one of the environmental image or the target image. Among them, the camera device can be directly installed on the augmented reality device or independently set in the target working scene, and there is no specific limitation.

[0056] Among them, the detailed information of the sleeve to be aligned and the alignment component contained in the environmental image is the structural shape and size information of the alignment component, the structural shape and size information of the sleeve to be aligned, the relative position relationship between the sleeve to be aligned and the alignment component, the spatial position of the alignment component, and the position of the sleeve to be aligned in the target action area. In addition, the environmental image also contains the positions of other devices in the target working scene other than the alignment component and the sleeve to be aligned and the structural shape and size information of other devices, etc.

[0057] It should be noted that the target image is obtained by separately photographing only the area where the sleeve to be aligned is located in the target working scene, which can ensure more and more accurate detailed information of the sleeve to be aligned. Thus, the position of the target fixed point in the target working scene can be effectively determined. The environmental image is taken of the entire target working scene and contains the detailed information of the sleeve to be aligned and the alignment component, which can simplify the process of the shooting work and simplify the process of obtaining the pre-image.

[0058] For example, an operator wears the augmented reality device, and at least one camera device is deployed in the augmented reality device. At least one environmental image of the target work scene is captured by at least one camera device, or at least one fixed point of at least one sleeve to be aligned in the target work scene is captured by at least one camera device to obtain at least one target image. The augmented reality device sends at least one target image or at least one environmental image to the computer device. The computer device determines the position of the target fixed point in the target image based on the position information of the target fixed point in at least one target image. Or the computer device determines the position of the target fixed point in the environmental image based on the position information of the target fixed point in at least one environmental image.

[0059] Step S204, determine the virtual fixed points of each of the alignment components according to the targets in at least one alignment component in the target work scene.

[0060] Among them, the target is a kind of picture, and the target carries coding information, and the coding information is used to determine the alignment component corresponding to the target. Among them, the shape of a kind of target is as Figure 4A shown. The target can be composed of black and white squares. Among them, the black block diagram is used as the background, and the four vertices of the target are used as the four foci. The coding information is determined based on the position and quantity of the white squares in the target, and the coding information can be represented in binary. Among them, the shape of another kind of target is as Figure 4B shown. A preset number of circular reflective stickers are pasted on the black block diagram, and the preset number is generally between 4 and 9. When the camera device captures the target, the circular reflective stickers will appear bright white, and the coding information is determined according to the quantity and position of the circular reflective stickers. Among them, the virtual fixed point is a point at a preset height on the alignment component.

[0061] Specifically, the camera device captures a to-be-processed image of the alignment component with a target pasted thereon in the target work scene. The computer device obtains the to-be-processed image sent by the camera device and determines the positions of at least one virtual fixed point in the target work scene based on the position information of the virtual fixed point in the to-be-processed image. Or the computer device obtains the environmental image sent by the camera device and determines the positions of at least one virtual fixed point in the target work scene based on the position information of the virtual fixed point in the environmental image.

[0062] For example, a computer device obtains preset position information corresponding to each alignment component. The preset position information represents the position information of the virtual fixed point corresponding to the alignment component in the target coordinate system. The preset position information is determined before the alignment component leaves the factory. The computer device captures at least one alignment component in the target working scenario through at least one camera device to obtain at least one image to be processed. The computer device obtains the image to be processed sent by the camera device, and determines the virtual fixed points of the alignment components in the target working scenario based on the position information of the virtual fixed points in the image to be processed and the preset position information corresponding to each virtual fixed point respectively. Alternatively, the computer device obtains at least one environmental image, and determines the virtual fixed points of the alignment components in the target working scenario based on the position information of the virtual fixed points in the environmental image and the preset position information corresponding to each virtual fixed point respectively. The target coordinate system is as Figure 5 shown, the plane where the x-axis and y-axis of the target coordinate system are located is on the surface of the alignment component, and the z-axis of the target coordinate system is perpendicular to the plane where the x-axis and y-axis are located.

[0063] Step S206: Virtually fuse the target fixed point, the virtual fixed point, and the target working scenario, and display the target fixed point on the sleeve to be aligned and the virtual fixed point on the alignment component through an augmented reality device to guide the sleeve to be aligned to be aligned with the alignment component.

[0064] Specifically, the computer device sends the target fixed point, the virtual fixed point, and the scenario information corresponding to the target working scenario to the augmented reality device for virtual fusion to construct a virtual space, and the positions of all points in the virtual space are the same as the positions of all points in the target working scenario. The augmented reality device displays the target fixed point corresponding to the sleeve to be aligned on the sleeve to be aligned, and displays the virtual fixed point corresponding to the alignment component in the alignment component to guide the sleeve to be aligned and the alignment component to be aligned.

[0065] In the virtual space, the display of the virtual fixed point and the target fixed point is as Figure 6 shown. In the virtual space, the virtual sleeve is represented by a cylinder, the virtual fixed point in the virtual space is represented by a sphere connected to the cylinder. The virtual sleeve is the representation form of the sleeve in the simulated alignment state, and the virtual fixed point and the real fixed point coincide in the alignment state. The target fixed point in the virtual space can be displayed as spheres of different colors. In the virtual space, the virtual fixed point and the target fixed point in the process of alignment will be displayed in the form of glowing spheres. In this way, the augmented reality device can transmit glowing spheres of different colors to the operator's eyes to achieve fusion with the target working scenario in the real space.

[0066] The above-mentioned casing alignment method realizes the precise positioning of at least one target fixed point of the casing to be aligned in the target working scenario in the target action area, truthfully reflecting the position information of the casing to be aligned. According to the targets in at least one alignment component in the target working scenario, the precise positioning of the virtual fixed points of each alignment component in the target working scenario can be realized, accurately and effectively reflecting the position information of the alignment components. The target fixed point, the virtual fixed point and the target working scenario are virtually fused, and the target fixed point is displayed on the casing to be aligned through an augmented reality device, and the virtual fixed point is displayed on the alignment component to guide the alignment of the casing to be aligned with the alignment component. In this way, by real-time displaying the positions of the target fixed point and the virtual fixed point in the augmented reality device, the alignment state of the casing to be aligned and the alignment component in the target working scenario can be reproduced in a timely and accurate manner, greatly improving the alignment accuracy of the casing to be aligned.

[0067] In one embodiment, as Figure 7 shown, the determination of at least one target fixed point of the casing to be aligned in the target action area in the target working scenario includes:

[0068] Step S702, obtaining a target image obtained by image acquisition of at least one casing to be aligned in the target action area.

[0069] Among them, the target image shows the casing to be aligned and the target action area, and the center of the intersection area between the casing to be aligned and the target action area is regarded as the true fixed point.

[0070] Specifically, when a camera device is deployed in the augmented reality device, the target fixed point of at least one casing to be aligned in the target working scenario in the target action area is photographed by the camera device to obtain at least one target image. When multiple camera devices are deployed in the augmented reality device, each camera device respectively photographs a casing to be aligned to obtain a target image corresponding to the casing to be aligned.

[0071] It should be noted that the positions of all pixel points in the target image are positions in the image coordinate system.

[0072] Step S704, based on the target image, determining the positions of at least one target fixed point in the world coordinate system through the first coordinate system conversion process.

[0073] Among them, the first coordinate system conversion process is used to determine the position of the target fixed point in the world coordinate system. This first coordinate system conversion process is processed through at least one coordinate system conversion process. For example, the image coordinate system is converted into the camera coordinate system, and then the camera coordinate system is converted into the world coordinate system, or the image coordinate system is directly converted into the world coordinate system, which is not specifically limited. The world coordinate system is a coordinate system representing the real three-dimensional space, and it can be the coordinate system of the target working scene in the actual three-dimensional space.

[0074] Specifically, the computer device performs segmentation processing on the target image to obtain a mask corresponding to the sleeve to be aligned in two dimensions. This mask is a grayscale value result. For example, the brightness value on the sleeve to be aligned in the mask is 255, and the brightness value on the non-sleeve to be aligned is 0. The computer device determines the position information of the target fixed point in two dimensions based on the coordinate information corresponding to the mask, and determines the position of at least one target fixed point in the world coordinate system through the first coordinate system conversion process based on the position information of the target fixed point in two dimensions.

[0075] For example, the computer device segments the sleeve to be aligned in the target image through a segmentation model to obtain a two-dimensional mask. This segmentation model can be constructed based on a neural network. For each sleeve to be aligned, the computer device determines each initial contour point representing the contour of the corresponding sleeve to be aligned through this mask. Based on the two-dimensional coordinate information of the multiple initial contour points respectively corresponding to each sleeve to be aligned, the position of at least one target fixed point in the camera coordinate system is determined. The computer device determines the position of at least one target fixed point in the world coordinate system based on the position of at least one target fixed point in the camera coordinate system.

[0076] In this embodiment, the target image corresponding to the sleeve to be aligned can truthfully reflect the position of the target fixed point on the target image. In this way, through the first coordinate system conversion process, the position of the target fixed point in the world coordinate system can be accurately located in a timely manner. At the same time, in this embodiment, only an image needs to be acquired without additional modeling processing, which greatly simplifies the data processing steps.

[0077] In one embodiment, as Figure 8 shown, determining at least one target fixed point in the world coordinate system through the first coordinate system conversion process based on the target image includes:

[0078] Step S802, based on the target image, determine multiple initial contour points respectively corresponding to each sleeve to be aligned.

[0079] Specifically, the computer device segments each sleeve to be aligned in the target image through a segmentation model, obtaining a two-dimensional mask corresponding to each sleeve to be aligned. The segmentation model can be constructed based on a neural network. For each sleeve to be aligned, the computer device determines each initial contour point representing the contour of the corresponding sleeve to be aligned through the mask. Among them, in the case where the tail of the sleeve to be aligned intersects the target action area, such as Figure 9 as shown

[0080] It should be noted that one mask corresponds to one sleeve to be aligned, that is, one mask corresponds to one target fixed point.

[0081] Step S804: For each sleeve to be aligned, based on each initial contour point corresponding to the corresponding sleeve to be aligned, perform screening to obtain target contour points corresponding to the corresponding sleeve to be aligned.

[0082] Specifically, for each sleeve to be aligned, the computer device determines adjacent contour points adjacent to the corresponding sleeve to be aligned. Based on the positions of the adjacent contour points adjacent to the corresponding initial contour points, determine the position change results of the corresponding contour points. For each sleeve to be aligned, the computer device, based on multiple position change results corresponding to the corresponding sleeve to be aligned, through a screening condition, screens out target contour points corresponding to the corresponding sleeve to be aligned from multiple initial contour points.

[0083] Among them, the screening condition is the condition for screening out inflection points with rapid position changes, and this inflection point is the target contour point.

[0084] It should be noted that the position change results of each initial contour point can intuitively show whether the contour area formed by all initial contour points can match the contour of the sleeve to be aligned. In addition, through the screening condition, target contour points with rapid position changes can be accurately screened out. Based on the target contour points obtained by this screening condition, the shape characteristics of the intersecting area can be obtained, which is beneficial to determining the position of the target fixed point.

[0085] Step S806: For each sleeve to be aligned, based on the target contour points corresponding to the corresponding sleeve to be aligned, through ellipse fitting processing, determine the position of the target fixed point corresponding to the corresponding sleeve to be aligned in the camera coordinate system.

[0086] Specifically, for each sleeve to be aligned, the computer device determines a contour line containing the target contour points based on the target contour points corresponding to the corresponding sleeve to be aligned, and performs three-dimensional space conversion on each contour point of the two-dimensional lower contour line to obtain three-dimensional contour points to be processed in the camera coordinate system. The computer device determines the position of the target fixed point corresponding to the corresponding sleeve to be aligned through ellipse fitting processing based on the positions of multiple contour points to be processed.

[0087] It should be noted that the sleeve to be aligned and the target action area can be regarded as an ellipse or a circle. By determining the contour points of the target, the contour line passing through the major axis or minor axis of the ellipse can be obtained. Based on this, the center of the ellipse in the three-dimensional space can be fitted based on this contour line, and then the position of the target fixed point can be determined.

[0088] For example, for each sleeve to be aligned, the computer device connects the target contour points corresponding to the corresponding sleeve to be aligned to form a contour line. The computer device maps the two-dimensional contour line into the three-dimensional space through the binocular stereo matching algorithm to obtain the three-dimensional contour points to be processed in the camera coordinates. For each sleeve to be aligned, based on the contour points to be processed corresponding to the corresponding sleeve to be aligned, the position of the ellipse center is determined through three-dimensional ellipse fitting. The computer device uses the position of the ellipse center as the position of the target fixed point corresponding to the corresponding sleeve to be aligned.

[0089] Among them, the binocular stereo matching algorithm is used to sense the 3D (3 Dimensions, three-dimensional) position information of the feature points.

[0090] Step S808, for each sleeve to be aligned, based on the position of the target fixed point corresponding to the corresponding sleeve to be aligned in the camera coordinates, determine the position of the target fixed point corresponding to the corresponding aligned sleeve in the world coordinate system.

[0091] Specifically, the computer device determines the positional relationship between the camera coordinate system and the world coordinate system. For each sleeve to be aligned, based on the position of the target fixed point corresponding to the corresponding sleeve to be aligned in the camera coordinates, through this positional relationship, determine the position of the target fixed point corresponding to the corresponding aligned sleeve in the world coordinate system.

[0092] For example, the computer device determines the positional relationship T between the camera coordinate system and the world coordinate system based on the environmental image corresponding to the target working scenario through the SLAM (Simultaneous Localization And Mapping) algorithm. For each sleeve to be aligned, the product of the position q c of the target fixed point corresponding to the corresponding sleeve to be aligned in the camera coordinates and the positional relationship T is used as the position q w of the target fixed point corresponding to the corresponding aligned sleeve in the world coordinate system.

[0093] Among them, the SLAM algorithm (Simultaneous Localization And Mapping) is used for spatial mapping and positioning technology and can obtain the positional relationship between the camera and the world coordinate system in real time.

[0094] In this embodiment, for each sleeve to be aligned, by determining the initial contour points corresponding to the contour of the sleeve to be aligned for screening, the target contour points of the sleeve to be aligned on the target action area can be quickly obtained, so that the contour of the sleeve to be aligned on the target action area can be accurately fitted. Furthermore, the position of the target fixed point can be accurately determined, ensuring the accuracy of the position of the target fixed point.

[0095] In one embodiment, as Figure 10 shown, for each sleeve to be aligned, based on screening of each initial contour point corresponding to the corresponding sleeve to be aligned, obtaining the target contour points corresponding to the corresponding sleeve to be aligned includes:

[0096] Step S1002, for each initial contour point corresponding to each sleeve to be aligned, determine two adjacent contour points adjacent to the corresponding initial contour point.

[0097] Among them, for each initial contour point, the two adjacent contour points corresponding to the corresponding initial contour point are respectively the points before and after the corresponding initial contour point.

[0098] Step S1004, for each initial contour point corresponding to each sleeve to be aligned, take the corresponding initial contour point as the vertex of the angle, and based on the corresponding initial contour point and the two adjacent contour points corresponding to the corresponding initial contour point, determine the angle corresponding to the corresponding initial contour point.

[0099] Specifically, for each initial contour point corresponding to each sleeve to be aligned, the computer device takes the corresponding initial contour point as the vertex of the angle, and based on the two adjacent contour points of the corresponding initial contour point, forms the angle corresponding to the corresponding initial contour point. The computer device calculates the angle corresponding to the corresponding initial contour point based on the position of the corresponding initial contour point and the positions of the two adjacent contour points of the corresponding initial contour point.

[0100] Step S1006, based on the angles respectively corresponding to each initial contour point, perform angle screening on each initial contour point to obtain the intermediate contour points corresponding to the corresponding sleeve to be aligned.

[0101] Among them, the angle screening is to screen out the intermediate contour points whose angles are within the preset angle range.

[0102] Specifically, the computer device obtains the angles respectively corresponding to each initial contour point, and compares each angle with the preset angle range. The computer device takes the initial contour points within the preset angle range as the intermediate contour points corresponding to the corresponding sleeve to be aligned.

[0103] It should be noted that, as Figure 9As shown, the shape of each sleeve to be aligned on the two-dimensional image can be understood as the side view of the sleeve to be aligned, that is, it can be regarded as a rectangle (as Figure 9 shown). Among them, the preset angle range is (A - b, A + b), where A is the reference angle, which can be 90°, and b is the angle error, which can be 0.6% etc., and no specific limitation is made. That is to say, the angle screened and processed through the preset angle range is an angle approximately 90 degrees, that is, the four vertices of the sleeve to be aligned in the image are obtained (as Figure 9 shown).

[0104] Step S1008, for each sleeve to be aligned, based on the positions of the respective intermediate contour points corresponding to the corresponding sleeve to be aligned, determine the distance between any two intermediate contour points. Based on each distance, perform distance screening on each intermediate contour point to determine two intermediate contour points that are the farthest apart and located in the target action area corresponding to the corresponding sleeve to be aligned.

[0105] Specifically, for each sleeve to be aligned, based on the positions of the respective intermediate contour points corresponding to the corresponding sleeve to be aligned, determine the distance between any two intermediate contour points. The computer device screens out two intermediate contour points that are the farthest apart and located in the target action area from multiple distances. It should be noted that after distance screening, four intermediate contour points that are the farthest apart will be obtained. These four intermediate contour points form a rectangle, which can be regarded as the projection of the sleeve to be aligned on the plane. The rectangle intersects with the target action area to obtain two intermediate contour points. As Figure 9 shown by point M and point N, both of these two intermediate contour points can be regarded as vertices.

[0106] Step S1010, for each sleeve to be aligned, use these two intermediate contour points that are the farthest apart and located in the target action area as the target contour points corresponding to the corresponding sleeve to be aligned.

[0107] It should be noted that the target contour points can be regarded as the two ends of the area where the sleeve to be aligned intersects with the target action area in the target image.

[0108] It should be noted that in the process of determining the target fixed points corresponding to each sleeve to be aligned, it is only necessary to determine according to the vertices where the corresponding sleeve to be aligned intersects with the target action area. As Figure 9 shown, after determining vertex 3 and vertex 4 where the corresponding sleeve to be aligned intersects with the target action area. It is only necessary to perform ellipse fitting processing on the position information of vertex 3 and vertex 4 to obtain the position of the target fixed point corresponding to the corresponding sleeve to be aligned in the world coordinates.

[0109] In this embodiment, by performing angle screening on the angles of each initial contour point, the contour information of the sleeve to be aligned in the target image can be quickly reflected. By performing distance screening on the intermediate contour points obtained through angle screening, the intermediate contour points located in the target action area can be quickly located, and thus, the target contour points can be efficiently screened out.

[0110] In one embodiment, as Figure 11As shown, the computer device acquires a target image of a target object (e.g., an abdominal image), segments each to-be-aligned cannula (corresponding to Trocar in the figure) in the target image through a neural network model, and obtains a two-dimensional mask (corresponding to mask in the figure) corresponding to each to-be-aligned cannula. For each to-be-aligned cannula, the computer device determines each initial contour point representing the contour of the corresponding to-be-aligned cannula through this mask. (That is, corresponding to the step of "acquiring the edge contour of Trocar" in the figure). For each initial contour point corresponding to each to-be-aligned cannula, the computer device determines two adjacent contour points adjacent to the corresponding initial contour point. For each initial contour point corresponding to each to-be-aligned cannula, the computer device takes the corresponding initial contour point as the vertex of the angle, and based on the corresponding initial contour point and the two adjacent contour points corresponding to the corresponding initial contour point, determines the angle corresponding to the corresponding initial contour point. Based on the angles corresponding to each initial contour point respectively, the computer device performs angle screening on each initial contour point to obtain intermediate contour points corresponding to the corresponding to-be-aligned cannula. For each to-be-aligned cannula, based on the positions of each intermediate contour point corresponding to the corresponding to-be-aligned cannula, the computer device determines the distance between any two intermediate contour points, and based on each distance, performs distance screening on each intermediate contour point to determine two intermediate contour points that are the farthest apart and located in the target action area corresponding to the corresponding to-be-aligned cannula. For each to-be-aligned cannula, the computer device takes these two intermediate contour points that are the farthest apart and located in the target action area as the target contour points corresponding to the corresponding to-be-aligned cannula. The computer device intercepts the contour line where the tail of each to-be-aligned cannula intersects the target action area (e.g., the abdomen) based on the target contour points corresponding to each to-be-aligned cannula respectively. For each to-be-aligned cannula, the computer device maps the two-dimensional contour line corresponding to the corresponding to-be-aligned cannula into the three-dimensional space through a binocular stereo matching algorithm to obtain three-dimensional to-be-processed contour points in the camera coordinate system. For each to-be-aligned cannula, based on the to-be-processed contour points corresponding to the corresponding to-be-aligned cannula, through three-dimensional ellipse fitting (i.e., 3D ellipse fitting corresponding to the figure), the position of the ellipse center is determined. The computer device takes the position of this ellipse center as the position of the target fixed point corresponding to the corresponding to-be-aligned cannula. The computer device determines the position relationship between the camera coordinate system and the world coordinate system through the SLAM algorithm based on the environmental image corresponding to the target working scenario. For each to-be-aligned cannula, the product of the position of the target fixed point corresponding to the corresponding to-be-aligned cannula in the camera coordinate and the position relationship is taken as the position of the target fixed point corresponding to the corresponding aligned cannula in the world coordinate system.

[0111] In this embodiment, for each sleeve to be aligned, by screening through determining the initial contour points corresponding to the contour of the sleeve to be aligned, the target contour points of the sleeve to be aligned on the target action area can be quickly obtained, so that the contour of the sleeve to be aligned on the target action area can be accurately fitted. Furthermore, the position of the target fixed point can be accurately determined, ensuring the accuracy of the position of the target fixed point.

[0112] In one embodiment, as Figure 12 shown, determining the virtual fixed points of each alignment component according to the target in at least one alignment component in the target working scenario includes:

[0113] Step S1202, obtaining the image to be processed obtained by image acquisition of the target in at least one alignment component.

[0114] Specifically, when a camera device is deployed in the augmented reality device, the target in at least one alignment component in the target working scenario is photographed by the camera device to obtain at least one image to be processed. When multiple camera devices are deployed in the augmented reality device, each camera device respectively photographs the target in one alignment component to obtain the image to be processed corresponding to the sleeve to be aligned.

[0115] It should be noted that one target is deployed in one alignment component. And generally, there are at least two alignment components in the target working scenario. Therefore, the alignment component corresponding to the target one by one can be determined from multiple alignment components through the target in the image to be processed.

[0116] Step S1204, determining the position information of the target in the image to be processed, and determining the positions of at least one virtual fixed point in the world coordinate system through second coordinate system conversion processing, where the virtual fixed point is a point located at the preset height of the alignment component.

[0117] Among them, the second coordinate system conversion processing is used to determine the position of the virtual fixed point in the world coordinate system. The second coordinate system conversion processing is through the processing of at least one coordinate system conversion process. For example, the image coordinate system is converted into the camera coordinate system, and then the camera coordinate system is converted into the world coordinate system, or the image coordinate system is directly converted into the world coordinate system, which is not specifically limited. The world coordinate system represents the coordinate system in the real three-dimensional space, and can be the coordinate system of the target working scenario in the actual three-dimensional space.

[0118] Specifically, the computer device determines the position information of the target based on the image to be processed, and based on the position information of the target, identifies multiple corner points of the target to obtain the two-dimensional corner point position information in the image coordinate system. The computer device determines the position of at least one target fixed point in the world coordinate system through a second coordinate system conversion process based on the two-dimensional corner point position information in the image coordinate system.

[0119] For example, the computer device obtains the preset position information corresponding to each alignment component, and the computer device determines the position information of the target corresponding to the image to be processed. As Figure 4A In the first type of target described above, the first type of target can be regarded as a matrix of m rows and n columns composed of multiple rectangular units. The corner points are respectively the rectangular units where the first row and the first column are located, the rectangular unit where the mth row and the first column are located, the rectangular unit where the first row and the nth column are located, and the rectangular unit where the mth row and the nth column are located. The computer device identifies the two-dimensional corner point position information of the first type of target in the image coordinate system. In the case of one imaging device, based on the two-dimensional corner point position information and the preset position information corresponding to the target in the target image, the position of the virtual fixed point corresponding to the alignment component in the world coordinate system is determined. In the case of at least two imaging devices, the target conversion relationship between the target coordinate system and the world coordinate system is obtained, the coordinate information in the three-dimensional camera is determined based on the two-dimensional corner point position information, and based on the coordinate information in the three-dimensional camera, the target conversion relationship, and the preset position information corresponding to the target in the target image, the position of the virtual fixed point corresponding to the alignment component in the world coordinate system is determined.

[0120] In this embodiment, the image to be processed corresponding to the target in the alignment component can truthfully reflect the position of the virtual fixed point on the image to be processed. In this way, through the second coordinate system conversion process, the position of the virtual fixed point in the world coordinate system can be located in a timely and accurate manner. At the same time, in this embodiment, only the image needs to be acquired, without additional modeling processing, greatly simplifying the data processing steps.

[0121] In one embodiment, as Figure 13 shown, the determination of the position information of the target in the image to be processed and the determination of the position of at least one virtual fixed point in the world coordinate system through the second coordinate system conversion process include:

[0122] Step S1302, for each alignment component, obtain the preset position information of the virtual fixed point corresponding to the corresponding alignment component in the target coordinate system.

[0123] Among them, the preset position information represents the position information of the virtual fixed point corresponding to the alignment component in the target coordinate system. The preset position information is determined before the alignment component leaves the factory.

[0124] Specifically, for each alignment component, a sleeve is installed on the corresponding alignment component to obtain the installed first alignment component, and a virtual fixed point is pre-marked at a preset height on the sleeve. Multiple first alignment components are photographed by a camera device to obtain a marked image. For each first alignment component, the computer device intercepts the corresponding first alignment component at the preset height to obtain the intercepted area corresponding to the corresponding first alignment component. For each first alignment component, the computer device determines the position of the virtual fixed point corresponding to the corresponding first alignment component in the camera coordinate system based on the positions of the points in the intercepted area corresponding to the corresponding first alignment component. For each first alignment component, the computer device identifies the target corner points corresponding to the corresponding first alignment component in the marked image to obtain the corner point coordinates in the image coordinate system. For each first alignment component, the computer device determines the positional relationship between the camera coordinate system and the target coordinate system based on the corner point coordinates corresponding to the corresponding first alignment component in the image coordinate system. For each first alignment component, based on the positional relationship between the camera coordinate system and the target coordinate system and the position of the virtual fixed point corresponding to the corresponding first alignment component in the camera coordinate system, the preset position information of the virtual fixed point corresponding to the corresponding alignment component in the target coordinate system is determined.

[0125] It should be noted that the target coordinate system is consistent with the result of the alignment component. Therefore, the preset position information of the virtual fixed point of the alignment component in the target coordinate system can be used as a reference position information.

[0126] It should be noted that the camera device can be the same as or different from the imaging device. Among them, a depth camera is provided in the imaging device, and the depth camera is used to (such as, TOF (Time of Flight) camera, structured light camera, etc.) sense the depth information of the entire scene and the spatial xyz information.

[0127] For example, for each first alignment component, based on the marked image corresponding to the respective first alignment component, the computer device obtains the positions of the two-dimensional contour points corresponding to the respective first alignment component through image algorithm segmentation, and maps the positions of the two-dimensional contour points into the three-dimensional space to obtain the positions of the respective contour points in the camera coordinate system. For each first alignment component, the computer device processes the positions of the respective contour points in the camera coordinate system through ellipse fitting to obtain the position of the virtual fixed point corresponding to the respective first alignment component in the camera coordinate system. The computer device identifies multiple corner points of the target on the respective first alignment component and determines the two-dimensional corner point coordinates corresponding to the respective first alignment component. For each first alignment component, the computer device maps the two-dimensional corner point coordinates corresponding to the respective first alignment component into the three-dimensional space to obtain the corner point coordinates in the camera coordinate system. For each first alignment component, the computer device determines the positional relationship between the camera coordinate system and the target coordinate system based on the corner point coordinates in the camera coordinate system through the ICP (Iterative Closest Point) algorithm, and determines the preset position information of the virtual fixed point corresponding to the respective alignment component in the target coordinate system based on the positional relationship between the camera coordinate system and the target coordinate system, the corner point coordinates in the camera coordinate system corresponding to the respective first alignment component, and the position of the virtual fixed point corresponding to the respective first alignment component in the camera coordinate system.

[0128] Among them, the ICP algorithm obtains the transformation matrix between coordinate systems by repeatedly selecting corresponding point pairs and calculating the optimal rigid body transformation.

[0129] Step S1304: For each alignment component, based on the position information of the target corresponding to the respective alignment component in the image to be processed, determine the first corresponding relationship corresponding to the respective alignment component, where the first corresponding relationship represents the relationship between the camera coordinate system and the target coordinate system.

[0130] Specifically, in the case where there are at least two imaging devices, for each alignment component, based on the position information of the target corresponding to the respective alignment component in the image to be processed, the first corresponding relationship corresponding to the respective alignment component is obtained through the binocular stereo matching algorithm or the ICP algorithm. In the case where there is one imaging device, for each alignment component, the three-dimensional target position information corresponding to the respective alignment component is obtained, and based on the three-dimensional target position information and the position information of the target corresponding to the respective alignment component in the image to be processed, the first corresponding relationship corresponding to the respective alignment component is determined through the PnP (Perspective-n-Point) algorithm.

[0131] Step S1306: Obtain the environmental image corresponding to the target working scene environment, and based on the environmental image, determine the second corresponding relationship between the camera coordinate system and the world coordinate system.

[0132] Specifically, an environmental image corresponding to the target working scene environment is captured by a camera device. The computer device processes the environmental image using the SLAM algorithm to determine the second correspondence between the camera coordinate system and the world coordinate system.

[0133] Step S1308: For each alignment component, based on the first correspondence, the second correspondence corresponding to the respective alignment component, and the preset position information of the virtual fixed point corresponding to the respective alignment component, determine the position of the virtual fixed point corresponding to the respective alignment component in the world coordinate system.

[0134] Specifically, for each alignment component, the computer device multiplies the first correspondence, the second correspondence corresponding to the respective alignment component, and the preset position information of the virtual fixed point corresponding to the respective alignment component to obtain the position of the virtual fixed point corresponding to the respective alignment component in the world coordinate system.

[0135] It should be noted that this embodiment mainly involves the conversion between three coordinate systems, and the conversion relationship is as Figure 14 shown, that is, to convert a point in the target coordinate system to the world coordinate system, first convert the point in the target coordinate system to a point in the camera coordinate system, and then convert the point in the camera coordinate system to the world coordinate system. Among them, the transformation matrix This transformation matrix can be obtained according to the following formula:

[0136]

[0137] where is the transformation matrix involved in converting a point in the target coordinate system to the camera coordinate system, is the transformation matrix involved in converting a point in the camera coordinate system to the world coordinate system.

[0138] In this embodiment, through the position of the virtual fixed point on the image to be processed, the first correspondence between the camera coordinate system and the target coordinate system can be determined. Through the environmental image of the target working scene environment, the second correspondence between the camera coordinate system and the world coordinate system can be determined. In this way, through the first correspondence and the second correspondence, the conversion from the target coordinate system to the world coordinate system is realized. Thus, based on the preset position information of the virtual fixed point of the alignment component in the target coordinate system, the position of the virtual fixed point of the component to be aligned in the world coordinate system can be determined quickly and accurately.

[0139] In one embodiment, as Figure 15As shown, before the alignment operation, for each first alignment component, the computer device, based on the marked image corresponding to the corresponding first alignment component, obtains the positions of the two-dimensional contour points corresponding to the corresponding first alignment component through image algorithm segmentation, and maps the positions of the two-dimensional contour points to the three-dimensional space to obtain the positions of each contour point in the camera coordinate system. For each first alignment component, the computer device processes the positions of each contour point in the camera coordinate system through ellipse fitting to obtain the position of the virtual fixed point corresponding to the corresponding first alignment component in the camera coordinate system. The computer device identifies multiple corner points of the target on the corresponding first alignment component and determines the two-dimensional corner point coordinates corresponding to the corresponding first alignment component. For each first alignment component, the computer device maps the two-dimensional corner point coordinates corresponding to the corresponding first alignment component to the three-dimensional space to obtain the corner point coordinates in the camera coordinate system. For each first alignment component, the computer device determines the positional relationship between the camera coordinate system and the target coordinate system based on the corner point coordinates in the camera coordinate system through the ICP algorithm, and determines the preset position information of the virtual fixed point corresponding to the corresponding alignment component in the target coordinate system based on the positional relationship between the camera coordinate system and the target coordinate system, the corner point coordinates in the camera coordinate system corresponding to the corresponding first alignment component, and the position of the virtual fixed point corresponding to the corresponding first alignment component in the camera coordinate system.

[0140] During the alignment operation, in the case where there are at least two imaging devices, for each alignment component, based on the position information of the target corresponding to the corresponding alignment component in the image to be processed, the first corresponding relationship corresponding to the corresponding alignment component is obtained through the binocular stereo matching algorithm or the ICP algorithm. In the case where there is one imaging device, for each alignment component, the three-dimensional target position information corresponding to the corresponding alignment component is obtained, and the first corresponding relationship corresponding to the corresponding alignment component is determined through the three-dimensional target position information and the position information of the target corresponding to the corresponding alignment component in the image to be processed. The environmental image corresponding to the target working scene environment is obtained by photographing with the imaging device. The computer device performs SLAM algorithm processing on the environmental image to determine the second corresponding relationship between the camera coordinate system and the world coordinate system. For each alignment component, the computer device multiplies the first corresponding relationship, the second corresponding relationship, and the preset position information of the virtual fixed point corresponding to the corresponding alignment component to obtain the position of the virtual fixed point corresponding to the corresponding alignment component in the world coordinate system.

[0141] In this embodiment, the first corresponding relationship between the camera coordinate system and the target coordinate system can be determined based on the position of the virtual fixed point on the image to be processed. The second corresponding relationship between the camera coordinate system and the world coordinate system can be determined from the environmental image of the target working scene environment. In this way, the conversion from the target coordinate system to the world coordinate system is achieved through the first corresponding relationship and the second corresponding relationship. Therefore, based on the preset position information of the virtual fixed point of the alignment component in the target coordinate system, the position of the virtual fixed point of the component to be aligned in the world coordinate system can be determined quickly and accurately.

[0142] In one embodiment, when there are at least two alignment components and at least two sleeves to be aligned, the augmented reality device displays the target fixed point on the sleeve to be aligned and the virtual fixed point on the alignment component. After guiding the sleeve to be aligned with the alignment component, the method further includes: determining the numbers of the respective target fixed points based on a preset sorting direction. Determining the numbers of the respective alignment components based on the coding information carried by each target in the image to be processed. Determining the numbers of the respective virtual fixed points based on the numbers of the respective alignment components. Determining the target fixed points corresponding to the respective virtual fixed points based on a preset pairing principle, the numbers of the respective virtual fixed points, and the numbers of the respective target fixed points.

[0143] Among them, in the target working scene as Figure 16 shown, the sleeve to be aligned is inserted into the target position (such as a cutting hole) in the target action area. The preset sorting direction can be the direction from the front end to the rear end of the target object or the direction from the rear end to the front end of the target object, and is not specifically limited. Among them, the preset pairing principle represents the numbers of the virtual fixed points corresponding to the numbers of the respective target fixed points. For example, the preset pairing principle can be that the virtual fixed point of the R-th alignment component is paired with the R-th real fixed point.

[0144] Specifically, the computer device numbers each target fixed point in sequence according to the preset sorting direction to determine the numbers of the respective target fixed points. The computer device identifies the coding information carried by each target to determine the number of the alignment component where the target is located. For each sleeve to be aligned, the computer device uses the number of the corresponding alignment component as the number of the virtual fixed point corresponding to the alignment component. The computer device determines the target fixed points corresponding to the respective virtual fixed points based on the preset pairing principle, the numbers of the respective virtual fixed points, and the numbers of the respective target fixed points.

[0145] For example, as Figure 17As shown, the computer device determines the types of each target, and based on the types of the targets, determines the encoding information corresponding to the target types. For each target, the computer device determines the type corresponding to the corresponding target, and based on the type corresponding to the corresponding target, determines the encoding information corresponding to the corresponding target. For each target, the computer device determines the number of the alignment component where the corresponding target is located based on the encoding information corresponding to the corresponding target, and uses the number of the corresponding alignment component as the number of the virtual fixed point corresponding to the alignment component, that is, identifies which alignment component each virtual fixed point belongs to. The computer device numbers each target fixed point in sequence according to the preset sorting direction to determine the numbers of each target fixed point. The computer device determines the target fixed points corresponding to each virtual fixed point according to the preset pairing principle, and controls the virtual fixed points corresponding to each alignment component to reach the positions of the corresponding target fixed points to align the sleeve to be aligned with the alignment components.

[0146] In this embodiment, the numbers of each alignment component are determined through the encoding information carried by the targets to distinguish each alignment component. In this way, it is also possible to give a separate number to the virtual fixed point of each alignment component. In this way, according to the preset pairing principle, each virtual fixed point is mapped to the corresponding target fixed point, avoiding the interference of the movement of the current virtual fixed point on other virtual fixed points and ensuring the alignment efficiency.

[0147] In one embodiment, the method further includes: determining the movement path of the virtual fixed point based on the position of the target fixed point and the position of the virtual fixed point. Sending the movement path to the augmented reality device so that the virtual fixed point moves according to the movement path until the virtual fixed point moves to the position of the target fixed point, and the alignment operation of the sleeve to be aligned is completed.

[0148] Among them, during the operation, the number of virtual fixed points is less than or equal to the number of target fixed points.

[0149] Specifically, in the case of at least two target fixed points, for each virtual fixed point, the computer device uses the straight-line distance between the corresponding virtual fixed point and the target fixed point corresponding to the corresponding virtual fixed point as the movement path of the corresponding virtual fixed point. In the case of one target fixed point and the same number of virtual fixed points and target fixed points, the straight-line distance between the virtual fixed point and the target fixed point is directly used as the movement path of the virtual fixed point. The computer device sends the movement path to the augmented reality device for display to move according to the movement path until the virtual fixed point moves to the position of the target fixed point, and the alignment operation of the sleeve to be aligned is completed.

[0150] In this embodiment, when the target fixed point and the virtual fixed point are determined, the motion path corresponding to the virtual fixed point is displayed in the augmented reality device, enabling the operator to move the virtual fixed point in real time according to the displayed motion path, greatly improving the convenience of moving the virtual fixed point.

[0151] In one embodiment, the method further includes: when the position of the target fixed point does not shift, determining the moving angle of the virtual fixed point. When the moving angle is not the threshold angle, controlling the virtual fixed point to stop moving and sending a warning message to the augmented reality device.

[0152] Wherein, the moving angle represents the degree of deviation of the position of the virtual fixed point from the motion path.

[0153] Specifically, when the position of the target fixed point does not shift, the computer device determines the positions of the virtual fixed point at each moment within the current time period, and calculates the moving angle of the virtual fixed point at the current moment based on the positions of the virtual fixed point at each moment within the current time period. When the moving angle is not the threshold angle, controlling the virtual fixed point to stop moving and sending a warning message to the augmented reality device.

[0154] Wherein, the end moment of the current time period is the current moment, and the start moment of the current time period is any moment before the current moment. For example, the current moment is 12:00, and the current time period is the time period from 11:50 to 12:00.

[0155] In this embodiment, when the position of the target fixed point does not shift, by monitoring the moving angle of the virtual fixed point to monitor the movement of the virtual fixed point in real time, it can effectively avoid the movement of the virtual fixed point deviating from the motion path, ensuring the accuracy of the movement of the virtual fixed point.

[0156] In one embodiment, the method further includes: when the position of the target fixed point shifts, sending an update instruction to the augmented reality device and determining the offset position of the target fixed point that has shifted. Updating the position of the target fixed point to the offset position.

[0157] Specifically, for each target fixed point, after determining the position of the corresponding target fixed point, the position of the corresponding target fixed point is used as an anchor point. For each alignment component, when an operator controls the movement of the corresponding alignment component, if the computer device determines that the position of the target fixed point corresponding to the corresponding virtual fixed point is not at the position of the anchor point, it is determined that the position of the target fixed point corresponding to the corresponding virtual fixed point has shifted. In the case where the position of the target fixed point has shifted, the virtual fixed point corresponding to the shifted target fixed point is used as the virtual fixed point to be processed, the movement of the virtual fixed point to be processed is stopped, and an update instruction is sent to the augmented reality device to determine the offset position of the shifted target fixed point. The computer device updates the position of the target fixed point to the offset position and updates the movement path corresponding to the virtual fixed point to be processed.

[0158] In this embodiment, by real-time monitoring whether the position of the target fixed point has shifted, it is possible to predict the movement of the virtual fixed point, and timely avoid the incorrect movement of the virtual fixed point. Thus, the accuracy of the movement of the virtual fixed point is greatly improved.

[0159] In one embodiment, as Figure 18As shown, for each target fixed point, after determining the position of the corresponding target fixed point, the position of the corresponding target fixed point is used as an anchor point. For each virtual fixed point, when the operator moves the alignment component corresponding to the virtual fixed point, no modification is made when the position of the target fixed point corresponding to the virtual fixed point does not shift. When the position of the target fixed point corresponding to the virtual fixed point is not at the position of the anchor point, it is determined that the position of the target fixed point corresponding to the virtual fixed point has shifted. In the case where the position of the target fixed point has shifted, the virtual fixed point corresponding to the shifted target fixed point is used as the virtual fixed point to be processed, the movement of the virtual fixed point to be processed is stopped, and an update instruction is sent to the augmented reality device to determine the offset position of the shifted target fixed point. The computer device updates the position of the target fixed point to the offset position and updates the movement path corresponding to the virtual fixed point to be processed. For each virtual fixed point, when the position of the target fixed point corresponding to the virtual fixed point does not shift, the computer device determines the position of the corresponding virtual fixed point at each moment within the current time period, and based on the position of the corresponding virtual fixed point at each moment within the current time period, calculates the movement angle of the corresponding virtual fixed point at the current moment. When the movement angle is not the threshold angle, the movement of the corresponding virtual fixed point is controlled to stop, and a warning message (i.e., error warning) is sent to the augmented reality device. When the movement angle is the threshold angle, no modification is made, and the operator continues to control the movement of the corresponding virtual fixed point by operating the alignment component corresponding to the virtual fixed point until the virtual fixed point is successfully paired with the real fixed point, and a success indication is issued.

[0160] In this embodiment, when the target fixed point and the virtual fixed point are determined, the movement path corresponding to the virtual fixed point is displayed in the augmented reality device, enabling the operator to move the virtual fixed point in real time according to the displayed movement path, greatly improving the convenience of moving the virtual fixed point. In addition, by monitoring in real time whether the position of the target fixed point shifts and whether the movement angle of the virtual fixed point shifts, the incorrect movement of the virtual fixed point is timely avoided, thereby greatly improving the accuracy of moving the virtual fixed point.

[0161] In one embodiment, to facilitate a clearer understanding of the technical solution of the present application, a more detailed embodiment is provided for description. The process of this embodiment is as Figure 19 shown. This embodiment involves the interaction operation between an augmented reality device equipped with an image acquisition unit and an AR augmented reality unit and a computer device equipped with a 3D positioning unit, an image processing unit, a robotic arm adjustment control unit (i.e., alignment component adjustment control unit), and a pairing monitoring unit. Among them, the units involved are as Figure 20 shown.

[0162] Among them, the 3D positioning unit is used to measure the spatial information in front of the line of sight. Specifically, first, the center point of the target position obtained by recognition and segmentation on the image is mapped into the 3D scene to obtain its spatial XYZ coordinates; then, through the SLAM algorithm, spatial mapping and positioning are performed to obtain the conversion relationship between the camera coordinate system and the world coordinate system. The functions of the 3D positioning unit include the PnP algorithm (used to estimate the relative position relationship between the camera and the target), the binocular stereo matching algorithm, the SLAM algorithm, the ICP algorithm, and depth cameras (such as TOF cameras, structured light cameras, etc.). Among them, the functions of the AR augmented reality unit include: tracking the eyeballs with infrared rays, generating a virtual 3D model and projecting the 3D model onto the lens in the form of a virtual image, and then directly reflecting the image onto the user's retina through the glass of the glasses; setting spatial anchors, storing and persisting them in the cloud for any other AR device to query and share. The specific steps are as follows:

[0163] Step 1: The image processing unit in the computer device acquires a target image obtained by image acquisition of at least one to-be-aligned sleeve in the target action area. Based on the target image, the image processing unit determines multiple initial contour points corresponding to each to-be-aligned sleeve. For each initial contour point corresponding to each to-be-aligned sleeve, the image processing unit in the computer device determines two adjacent contour points adjacent to the corresponding initial contour point. For each initial contour point corresponding to each to-be-aligned sleeve, the image processing unit in the computer device uses the corresponding initial contour point as the vertex of the angle, and based on the corresponding initial contour point and the two adjacent contour points corresponding to the corresponding initial contour point, the image processing unit in the computer device determines the angle corresponding to the corresponding initial contour point. Based on the angles corresponding to the respective initial contour points, angle screening is performed on the respective initial contour points to obtain intermediate contour points corresponding to the corresponding to-be-aligned sleeves. For each to-be-aligned sleeve, based on the positions of the respective intermediate contour points corresponding to the corresponding to-be-aligned sleeve, the image processing unit in the computer device determines the distance between any two intermediate contour points. Based on the respective distances, distance screening is performed on the respective intermediate contour points to determine two intermediate contour points that are the farthest apart and located in the target action area corresponding to the corresponding to-be-aligned sleeve. For each to-be-aligned sleeve, the image processing unit in the computer device uses the two intermediate contour points that are the farthest apart and located in the target action area as the target contour points corresponding to the corresponding to-be-aligned sleeve. For each to-be-aligned sleeve, based on the target contour points corresponding to the corresponding to-be-aligned sleeve, through elliptical fitting processing, the image processing unit in the computer device determines the position of the target fixed point corresponding to the corresponding to-be-aligned sleeve in the camera coordinate system. For each to-be-aligned sleeve, based on the position of the target fixed point corresponding to the corresponding to-be-aligned sleeve in the camera coordinate, the 3D positioning unit in the computer device determines the position of the target fixed point corresponding to the corresponding aligned sleeve in the world coordinate system.

[0164] Step 2: The image acquisition unit of the augmented reality device acquires a to-be-processed image of the target in at least one alignment component, and sends the to-be-processed image to the image processing unit in the computer device. For each alignment component, the image processing unit in the computer device acquires the preset position information of the corresponding virtual fixed point in the target coordinate system. For each alignment component, based on the position information of the target corresponding to the corresponding alignment component in the to-be-processed image, the 3D positioning unit in the computer device determines the first correspondence corresponding to the corresponding alignment component, and this first correspondence characterizes the relationship between the camera coordinate system and the target coordinate system. Based on the environmental image corresponding to the target working scene environment, the 3D positioning unit in the computer device determines the second correspondence between the camera coordinate system and the world coordinate system. For each alignment component, based on the first correspondence, the second correspondence corresponding to the corresponding alignment component, and the preset position information of the virtual fixed point corresponding to the corresponding alignment component, the image processing unit in the computer device determines the position of the virtual fixed point corresponding to the corresponding alignment component in the world coordinate system.

[0165] Step 3: The image processing unit in the computer device sends the target fixed point, the virtual fixed point, and the scene information corresponding to the target working scene to the augmented reality device for virtual fusion to construct a virtual space, and the positions of the points in this virtual space are consistent with the positions of the points in the target working scene. The augmented reality device displays the target fixed point corresponding to the to-be-aligned sleeve on the to-be-aligned sleeve, and displays the virtual fixed point corresponding to the alignment component in the alignment component to guide the alignment of the to-be-aligned sleeve and the alignment component.

[0166] Step 4: In the case where there are at least two alignment components and at least two to-be-aligned sleeves, based on the preset sorting direction, the pairing monitoring unit in the computer device determines the numbers of the respective target fixed points. Based on the coding information carried by each target in the to-be-processed image, the pairing monitoring unit in the computer device determines the numbers of the respective alignment components. Based on the numbers of the respective alignment components, the pairing monitoring unit in the computer device determines the numbers of the respective virtual fixed points. Based on the preset pairing principle, the numbers of the respective virtual fixed points, and the numbers of the respective target fixed points, it is determined which target fixed points the respective virtual fixed points correspond to.

[0167] Step 5: For each target fixed point, after determining the position of the corresponding target fixed point, the pairing monitoring unit in the computer device uses the position of the corresponding target fixed point as an anchor point. For each virtual fixed point, when the operator moves the alignment component corresponding to the virtual fixed point, if the position of the target fixed point corresponding to the virtual fixed point does not shift, no modification is made. When the position of the target fixed point corresponding to the virtual fixed point is not at the position of the anchor point, the pairing monitoring unit in the computer device determines that the position of the target fixed point corresponding to the virtual fixed point has shifted. In the case where the position of a target fixed point has shifted, the pairing monitoring unit in the computer device uses the virtual fixed point corresponding to the shifted target fixed point as the virtual fixed point to be processed, stops the movement of the virtual fixed point to be processed, and sends an update instruction to the augmented reality device to determine the offset position of the shifted target fixed point. The pairing monitoring unit in the computer device updates the position of the target fixed point to the offset position and updates the movement path corresponding to the virtual fixed point to be processed. For each virtual fixed point, when the position of the target fixed point corresponding to the virtual fixed point does not shift, the pairing monitoring unit in the computer device determines the position of the corresponding virtual fixed point at each moment within the current time period, and based on the position of the corresponding virtual fixed point at each moment within the current time period, calculates the moving angle of the corresponding virtual fixed point at the current moment. When the moving angle is not the threshold angle, the pairing monitoring unit in the computer device controls the corresponding virtual fixed point to stop moving and sends a warning message (i.e., an error warning) to the augmented reality device. When the moving angle is the threshold angle, no modification is made, and the operator continues to control the corresponding virtual fixed point to move by operating the alignment component corresponding to the virtual fixed point until the virtual fixed point coincides with the real fixed point, and the robotic arm adjustment control unit in the computer device adjusts the posture of the robotic arm (i.e., the alignment component) to pair it with the sleeve to be aligned with the target object already placed, and keeps the target fixed point of the sleeve to be aligned at the target position. At this time, the pairing operation is completed and a success indication is issued.

[0168] In this embodiment, by determining the target fixed points of at least one sleeve to be aligned in the target working scenario within the target action area, the accurate positioning of the target fixed points in the target working scenario is achieved, and the position information of the sleeve to be aligned is truthfully reflected. According to the targets in at least one alignment component in the target working scenario, the accurate positioning of the virtual fixed points of each alignment component in the target working scenario can be achieved, accurately and effectively reflecting the position information of the alignment components. The target fixed points, the virtual fixed points, and the target working scenario are virtually fused, and the target fixed points are displayed on the sleeve to be aligned through an augmented reality device, and the virtual fixed points are displayed on the alignment components to guide the alignment of the sleeve to be aligned with the alignment components. In this way, by real-time displaying the positions of the target fixed points and the virtual fixed points in the augmented reality device, the alignment state of the sleeve to be aligned and the alignment components in the target working scenario can be reproduced timely and accurately, greatly improving the accuracy of the alignment of the sleeve to be aligned. In addition, in this application, deep learning, image segmentation, and positioning algorithms are used to improve the accuracy of identifying the target positions of the target objects and the positions of the virtual fixed points. In addition, through the 3D tracking and positioning algorithm, images are only acquired through the camera on the augmented reality device to calculate the spatial positions, reducing other external spatial positioning devices. Without other external spatial positioning devices, the cost is greatly simplified. In addition, through the augmented display technology, the visualization of the virtual fixed points enables the operator to quickly and accurately complete the alignment operation of the sleeve.

[0169] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order limit for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0170] Based on the same inventive concept, an embodiment of this application also provides a sleeve alignment device for implementing the sleeve alignment method involved above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the sleeve alignment device provided below can refer to the limitations on the sleeve alignment method in the above text and will not be elaborated here.

[0171] In one embodiment, as Figure 21As shown, a casing alignment device is provided, including: a first determination module 2102, a second determination module 2104, and a display module 2106, where:

[0172] The first determination module 2102 is configured to determine target fixed points of at least one casing to be aligned in a target action area in a target working scenario.

[0173] The second determination module 2104 is configured to determine virtual fixed points of each alignment component according to a target in at least one alignment component in the target working scenario.

[0174] The display module 2106 is configured to perform virtual fusion on the target fixed point, the virtual fixed point, and the target working scenario, and display the target fixed point on the casing to be aligned through an augmented reality device, and display the virtual fixed point on the alignment component, so as to guide the alignment of the casing to be aligned with the alignment component.

[0175] In one embodiment, the first determination module 2102 is configured to obtain a target image obtained by performing image acquisition on at least one casing to be aligned in a target action area. Based on the target image, through first coordinate system conversion processing, determine the positions of at least one target fixed point in the world coordinate system.

[0176] In one embodiment, the first determination module 2102 is configured to determine, based on the target image, a plurality of initial contour points respectively corresponding to each casing to be aligned. For each casing to be aligned, perform screening based on each initial contour point corresponding to the corresponding casing to be aligned to obtain target contour points corresponding to the corresponding casing to be aligned. For each casing to be aligned, based on the target contour points corresponding to the corresponding casing to be aligned, through ellipse fitting processing, determine the position of the target fixed point corresponding to the corresponding casing to be aligned in the camera coordinate system. For each casing to be aligned, based on the position of the target fixed point corresponding to the corresponding casing to be aligned in the camera coordinate, determine the position of the target fixed point corresponding to the corresponding aligned casing in the world coordinate system.

[0177] In one embodiment, the first determination module 2102 is configured to, for each initial contour point corresponding to each sleeve to be aligned, determine two adjacent contour points adjacent to the corresponding initial contour point. For each initial contour point corresponding to each sleeve to be aligned, using the corresponding initial contour point as the vertex of an angle, based on the corresponding initial contour point and the two adjacent contour points corresponding to the corresponding initial contour point, determine the angle corresponding to the corresponding initial contour point. Based on the angles respectively corresponding to the respective initial contour points, perform angle screening on the respective initial contour points to obtain intermediate contour points corresponding to the corresponding sleeves to be aligned. For each sleeve to be aligned, based on the positions of the respective intermediate contour points corresponding to the corresponding sleeve to be aligned, determine the distance between any two intermediate contour points, and based on the respective distances, perform distance screening on the respective intermediate contour points to determine two intermediate contour points that are the farthest apart and located in the target action area corresponding to the corresponding sleeve to be aligned. For each sleeve to be aligned, use the two intermediate contour points that are the farthest apart and located in the target action area as the target contour points corresponding to the corresponding sleeve to be aligned.

[0178] In one embodiment, the second determination module 2104 is configured to obtain a to-be-processed image obtained by performing image acquisition on a target in at least one alignment component. Determine the position information of the target in the to-be-processed image, and through second coordinate system conversion processing, determine the positions of at least one virtual fixed point in the world coordinate system, where the virtual fixed point is a point located at a preset height of the alignment component.

[0179] In one embodiment, the second determination module 2104 is configured to, for each alignment component, obtain the preset position information of the corresponding virtual fixed point in the target coordinate system. For each alignment component, based on the position information of the target corresponding to the corresponding alignment component in the to-be-processed image, determine the first correspondence relationship corresponding to the corresponding alignment component, where the first correspondence relationship represents the relationship between the camera coordinate system and the target coordinate system. Obtain an environmental image corresponding to the target working scenario environment, and based on the environmental image, determine the second correspondence relationship between the camera coordinate system and the world coordinate system. For each alignment component, based on the first correspondence relationship, the second correspondence relationship corresponding to the corresponding alignment component, and the preset position information of the corresponding virtual fixed point, determine the position of the virtual fixed point corresponding to the corresponding alignment component in the world coordinate system.

[0180] In one embodiment, the display module 2106 is further configured to determine the numbers of the respective target fixed points based on a preset sorting direction. Determine the numbers of the respective alignment components based on the coding information carried by each target in the to-be-processed image. Determine the numbers of the respective virtual fixed points based on the numbers of the respective alignment components. Based on a preset pairing principle, the numbers of the respective virtual fixed points, and the numbers of the respective target fixed points, determine the target fixed points respectively corresponding to the respective virtual fixed points.

[0181] In one embodiment, the display module 2106 is further configured to determine a movement path of the virtual fixed point based on the positions of the target fixed point and the virtual fixed point. The movement path is sent to the augmented reality device so that the virtual fixed point moves along the movement path until the virtual fixed point moves to the position of the target fixed point, and the alignment operation of the sleeve to be aligned is completed.

[0182] In one embodiment, the display module 2106 is further configured to determine a movement angle of the virtual fixed point when the position of the target fixed point does not shift. When the movement angle is not the threshold angle, the display module 2106 controls the virtual fixed point to stop moving and sends a warning message to the augmented reality device.

[0183] In one embodiment, the display module 2106 is further configured to send an update instruction to the augmented reality device when the position of the target fixed point shifts, and determine the shift position of the target fixed point that has shifted. The position of the target fixed point is updated to the shift position.

[0184] Each module in the above sleeve alignment device can be implemented in whole or in part by software, hardware, and their combination. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or stored in a memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0185] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 22 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sleeve alignment data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with an external terminal through a network connection. The computer program, when executed by the processor, implements a sleeve alignment method.

[0186] Those skilled in the art can understand, Figure 22The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0187] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0188] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0189] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0190] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0191] Those of ordinary skill in the art can understand that all or part of the processes in the above-described embodiment methods can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAMs), magnetoresistive random access memories (MRAMs), ferroelectric random access memories (FRAMs), phase change memories (PCMs), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0192] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0193] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application should be subject to the appended claims.

Claims

1. A casing alignment method, characterized in that, the method includes: acquiring a target image obtained by image acquisition of at least one casing to be aligned in a target action area that has been pre-punched, where the casing to be aligned is used to form a pipe-shaped working channel in the target action area; segmenting the target image through a segmentation model constructed based on a neural network to obtain two-dimensional masks corresponding to the casings to be aligned respectively. For each casing to be aligned, through the corresponding mask, a plurality of initial contour points corresponding to the casing to be aligned are determined; for each casing to be aligned, based on the positions of adjacent contour points adjacent to each initial contour point, the corresponding position change results are determined. Based on the plurality of position change results corresponding to the corresponding casing to be aligned, target contour points with rapid position changes are screened out from the plurality of initial contour points; for each casing to be aligned, based on the target contour points corresponding to the corresponding casing to be aligned, through ellipse fitting processing, the position of the target fixed point corresponding to the corresponding casing to be aligned in the camera coordinate system is determined; for each casing to be aligned, based on the position of the target fixed point corresponding to the corresponding casing to be aligned in the camera coordinate system, through the position relationship between the camera coordinate system and the world coordinate system, the position of the target fixed point corresponding to the corresponding aligned casing in the world coordinate system is determined; according to the targets in at least one alignment component in the target working scenario, the virtual fixed points of the alignment components are determined; virtually fusing the target fixed points, the virtual fixed points and the target working scenario, and displaying the target fixed points on the casing to be aligned and the virtual fixed points on the alignment component through an augmented reality device to guide the alignment of the casing to be aligned with the alignment component.

2. The method according to claim 1, characterized in that, the specific determination steps of the target contour points corresponding to the corresponding casing to be aligned include: for each initial contour point corresponding to each casing to be aligned, two adjacent contour points adjacent to the corresponding initial contour point are determined; for each initial contour point corresponding to each casing to be aligned, taking the corresponding initial contour point as the vertex of the angle, based on the corresponding initial contour point and the two adjacent contour points corresponding to the corresponding initial contour point, the angle corresponding to the corresponding initial contour point is determined; based on the angles corresponding to the respective initial contour points, angle screening is performed on the respective initial contour points to obtain intermediate contour points corresponding to the corresponding casing to be aligned; for each casing to be aligned, based on the positions of the respective intermediate contour points corresponding to the corresponding casing to be aligned, the distance between any two intermediate contour points is determined. Based on the respective distances, distance screening is performed on the respective intermediate contour points to determine two intermediate contour points that are the farthest apart and located in the target action area corresponding to the corresponding casing to be aligned; for each casing to be aligned, the two intermediate contour points that are the farthest apart and located in the target action area are used as the target contour points corresponding to the corresponding casing to be aligned.

3. The method according to claim 1, characterized in that, Based on the position of the target fixed point corresponding to the corresponding sleeve to be aligned in the camera coordinate system, and through the positional relationship between the camera coordinate system and the world coordinate system, determining the position of the target fixed point corresponding to the corresponding alignment sleeve in the world coordinate system includes: Based on the environmental image corresponding to the target working scenario, determining the positional relationship between the camera coordinate system and the world coordinate system through the SLAM algorithm; For each sleeve to be aligned, taking the product of the position of the target fixed point corresponding to the corresponding sleeve to be aligned in the camera coordinate system and the positional relationship as the position of the target fixed point corresponding to the corresponding alignment sleeve in the world coordinate system.

4. The method according to claim 1, wherein, determining the virtual fixed points of the respective alignment components according to the targets in at least one alignment component in the target working scenario includes: Obtaining a to-be-processed image obtained by performing image acquisition on the targets in at least one alignment component; Determining the position information of the targets in the to-be-processed image, and through second coordinate system conversion processing, determining the positions of at least one virtual fixed point in the world coordinate system, where the virtual fixed point is a point located at a preset height of the alignment component.

5. The method according to claim 4, wherein, determining the position information of the targets in the to-be-processed image, and through second coordinate system conversion processing, determining the positions of at least one virtual fixed point in the world coordinate system includes: For each alignment component, obtaining the preset position information of the corresponding virtual fixed point in the target coordinate system; For each alignment component, based on the position information of the target corresponding to the corresponding alignment component in the to-be-processed image, determining the first corresponding relationship corresponding to the corresponding alignment component, where the first corresponding relationship represents the relationship between the camera coordinate system and the target coordinate system; Obtaining an environmental image corresponding to the target working scenario environment, and based on the environmental image, determining the second corresponding relationship between the camera coordinate system and the world coordinate system; For each alignment component, based on the first corresponding relationship, the second corresponding relationship corresponding to the corresponding alignment component, and the preset position information of the corresponding virtual fixed point, determining the position of the virtual fixed point corresponding to the corresponding alignment component in the world coordinate system.

6. The method according to claim 1, wherein, In the case where there are at least two alignment components and at least two sleeves to be aligned, after displaying the target fixed points on the sleeves to be aligned and displaying the virtual fixed points on the alignment components through the augmented reality device to guide the alignment of the sleeves to be aligned with the alignment components, the method further includes: Based on the preset sorting direction, determining the numbers of the respective target fixed points; Based on the coding information carried by each of the targets in the to-be-processed image, determining the numbers of the respective alignment components; Based on the numbers of the respective alignment components, determining the numbers of the respective virtual fixed points; Based on the preset pairing principle, the numbers of the respective virtual fixed points, and the numbers of the respective target fixed points, determining the target fixed points respectively corresponding to the respective virtual fixed points.

7. The method according to claim 1, wherein, The method further includes: Determining a movement path of the virtual fixed point based on the positions of the target fixed point and the virtual fixed point; Sending the movement path to the augmented reality device, so that the virtual fixed point moves according to the movement path until the virtual fixed point moves to the position of the target fixed point, and the alignment operation of the sleeve to be aligned is completed.

8. The method according to claim 7, wherein, The method further includes: Determining a movement angle of the virtual fixed point when the position of the target fixed point does not shift; When the movement angle is not a threshold angle, controlling the virtual fixed point to stop moving and sending a warning message to the augmented reality device.

9. The method according to claim 8, wherein, The method further includes: When the position of the target fixed point shifts, sending an update instruction to the augmented reality device and determining the shift position of the target fixed point that has shifted; Updating the position of the target fixed point to the shift position.

10. A sleeve alignment device, wherein, The device includes: A first determination module, configured to obtain a target image obtained by performing image acquisition on at least one sleeve to be aligned in a target action area that has been pre-punched, where the sleeve to be aligned is used to form a pipeline-shaped working channel in the target action area; segmenting the target image through a segmentation model constructed based on a neural network to obtain two-dimensional masks respectively corresponding to the sleeves to be aligned, and for each sleeve to be aligned, determining a plurality of initial contour points respectively corresponding to the sleeve to be aligned through the corresponding mask; for each sleeve to be aligned, determining a corresponding position change result based on the positions of adjacent contour points adjacent to each initial contour point, and screening out target contour points with rapid position changes from the plurality of initial contour points based on the plurality of position change results corresponding to the corresponding sleeve to be aligned; for each sleeve to be aligned, determining the position of a target fixed point corresponding to the corresponding sleeve to be aligned in the camera coordinate system through ellipse fitting processing based on the target contour points corresponding to the corresponding sleeve to be aligned; for each sleeve to be aligned, determining the position of the target fixed point corresponding to the corresponding aligned sleeve in the world coordinate system through the position relationship between the camera coordinate system and the world coordinate system; A second determination module, configured to determine virtual fixed points of the alignment components according to the targets in at least one alignment component in the target working scenario; A display module, configured to perform virtual fusion on the target fixed point, the virtual fixed point and the target working scenario, and display the target fixed point on the sleeve to be aligned and the virtual fixed point on the alignment component through an augmented reality device to guide the alignment of the sleeve to be aligned with the alignment component.

11. A computer device, including a memory and a processor, where the memory stores a computer program, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, on which a computer program is stored, wherein, when the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images