Method for assisting in the joining of pipes, joining of pipes assistance system and controller

Abstract

The application discloses a method for assisting in aiming at a pipe, a pipe aiming assisting system and a controller. The method comprises the following steps: firstly, acquiring current video frames of a plurality of image acquisition devices and judging whether a first target pipe is detected in a display screen of the plurality of image acquisition devices; in the case that the first target pipe is detected, setting an image acquisition device corresponding to the display screen as a target image acquisition device, setting a holder corresponding to the target image acquisition device as a target holder, and controlling the target holder to enter a first tracking state. In the case that a pipe orifice section center point of the first target pipe is in the center of the display screen and a second target pipe is detected, the target holder is controlled to enter a second tracking state. The application can realize the automatic tracking function of the target pipe, keep the high-resolution pipe aiming field picture display, and realize the real-time tracking of the target pipe. Through the pipe aiming process visual assistance, the pipe aiming accuracy of the target pipe at different distances and different directions is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent assisted pipe alignment technology, and more specifically to a method, pipe alignment assistance system and controller for assisting pipe alignment. Background Technology

[0002] Concrete truck-mounted pumps are widely used in the construction industry due to their advantages such as flexible operation, wide construction range, and high pumping performance. However, due to complex construction environments and insufficient operator experience, the installation of concrete truck-mounted pumps in pipelines often takes a long time.

[0003] Currently, some truck-mounted concrete pumps are equipped with reversing auxiliary cameras (similar to reversing cameras) to monitor the position of target concrete pipes on the construction site and the vehicle's possible movement trajectory in real time. However, this technology usually only provides auxiliary line calibration for reducing pipe diameters, but does not consider the actual position of the target concrete pipe in use. It typically requires manual judgment of the relative positions of the two pipes before adjusting the vehicle body. Furthermore, existing technologies generally use fixed-focus cameras, with a fixed field of view. When the camera's field of view is small, it cannot see distant target concrete pipes; when the camera's field of view is wide, the pipe opening features are few, making high-precision pipe alignment difficult. Summary of the Invention

[0004] The purpose of this application is to provide a method, system and controller for assisting in tube alignment, in order to solve the problems in the prior art where the target tube easily exceeds the field of view during tube alignment and the field of view is too large during high-precision tube alignment.

[0005] To achieve the above objectives, the first aspect of this application provides a method for assisting in alignment, applied to an alignment assistance system. The alignment assistance system includes multiple pan-tilt units, multiple image acquisition devices, and a controller. The multiple image acquisition devices are respectively mounted on the multiple pan-tilt units, and the multiple pan-tilt units and multiple image acquisition devices communicate with the controller. The method includes:

[0006] Acquire the current video frames from multiple image acquisition devices;

[0007] Determine whether the first target tube is detected in the display screen of multiple image acquisition devices based on the current video frame;

[0008] If the first target tube is detected in the display screen, the image acquisition device corresponding to the display screen is set as the target image acquisition device, the pan-tilt unit corresponding to the target image acquisition device is set as the target pan-tilt unit, and the target pan-tilt unit is controlled to enter the first tracking state so that the center point of the tube opening of the first target tube is in the center of the display screen.

[0009] If, in the first tracking state, the center point of the cross-section of the first target tube is located in the center of the display screen, determine whether the second target tube is detected in the display screen.

[0010] If a second target tube is detected in the display screen, the target gimbal is controlled to enter the second tracking state so that the center point of the cross-section of the second target tube is in the center of the display screen.

[0011] In this embodiment of the application, the method further includes:

[0012] If the first target tube is not detected in the display screen, the target pan-tilt unit is controlled to enter the target search state until the first target tube is detected in the display screen.

[0013] In this embodiment of the application, the method further includes:

[0014] If the second target tube is not detected in the display screen during the second tracking state, the target pan-tilt unit is controlled to enter the target search state.

[0015] In this embodiment of the application, controlling the target gimbal to enter the target search state includes:

[0016] The target PTZ is controlled to rotate along a preset trajectory, and the first target tube is detected in the display screen of the image acquisition device based on the current video frame.

[0017] In this embodiment of the application, the method further includes;

[0018] If multiple candidate second target tubes are detected in the display screen, the candidate second target tube that is closest to the first target tube is identified as the second target tube.

[0019] In this embodiment of the application, determining whether a first target tube is detected in the display screen of the image acquisition device based on the current video frame includes:

[0020] Identify the target object in the current video frame;

[0021] If a target object is detected in a video frame, the target object is identified as the first target tube;

[0022] If more than two target objects are identified in a video frame, the first target tube and multiple candidate second target tubes are determined based on the location of the image acquisition device and the aspect ratio and absolute value of the target objects.

[0023] In this embodiment of the application, when it is determined that a first target tube is detected in the display screen, setting the image acquisition device corresponding to the display screen as the target image acquisition device, setting the pan-tilt unit corresponding to the target image acquisition device as the target pan-tilt unit, and controlling the target pan-tilt unit to enter the first tracking state includes making the center point of the tube opening cross-section of the first target tube located in the center of the display screen:

[0024] If the first target tube is detected in the display screen, determine the center point of the cross-section of the tube opening of the first target tube;

[0025] Control the rotation of the target gimbal so that the center point of the cross-section of the first target tube is in the center of the display screen.

[0026] In this embodiment of the application, determining the center point of the cross-section of the nozzle of the first target tube includes:

[0027] Determine the central axis of the first target tube;

[0028] A vector is formed by the center point of the central axis of the first target tube and the points on the cross-section of the first target tube;

[0029] The center point of the pipe opening of the first target pipe is determined by the projection of the vector onto the central axis of the pipe body.

[0030] In this embodiment of the application, when it is determined that a second target tube is detected in the display screen, controlling the target gimbal to enter a second tracking state includes aligning the center point of the cross-section of the second target tube's opening with the center of the display screen:

[0031] If a second target tube is detected in the display screen, determine the center point of the cross-section of the second target tube opening;

[0032] Control the rotation of the target gimbal so that the center point of the cross-section of the second target tube's opening is in the center of the display screen.

[0033] In this embodiment of the application, determining the center point of the cross-section of the second target tube's opening includes:

[0034] Determine the central axis of the second target tube;

[0035] A vector is formed by the center point of the central axis of the second target tube and the points on the cross-section of the second target tube;

[0036] The center point of the cross-section of the second target tube's opening is determined by the projection of the vector onto the central axis of the second target tube's body.

[0037] A second aspect of this application provides a controller, comprising:

[0038] The memory is configured to store instructions; and

[0039] The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method for assisting in the control described above.

[0040] A third aspect of this application provides a control assist system, comprising:

[0041] Multiple pan-tilt units are configured to adjust the orientation of the image acquisition devices.

[0042] Multiple image acquisition devices, each mounted on a separate pan-tilt unit, are configured to acquire images; and

[0043] The aforementioned controller communicates with multiple pan-tilt units and multiple image acquisition devices.

[0044] A fourth aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform the aforementioned method for assisting in the control process.

[0045] The above technical solution first acquires the current video frames of multiple image acquisition devices. Based on the current video frames, it is determined whether a first target tube is detected in the display screen of the multiple image acquisition devices. If the first target tube is detected in the display screen, the image acquisition device corresponding to the display screen is set as the target image acquisition device, and the pan-tilt unit corresponding to the target image acquisition device is set as the target pan-tilt unit. The target pan-tilt unit is then controlled to enter the first tracking state. In the first tracking state, if the center point of the cross-section of the first target tube's opening is located in the center of the display screen, it is determined whether a second target tube is detected in the display screen. If the second target tube is detected in the display screen, the target pan-tilt unit is controlled to enter the second tracking state. This application can achieve automatic tracking of the target tube, maintaining high-resolution on-site display while simultaneously tracking the target tube in real time. Through visual assistance during the tube alignment process, the alignment accuracy of the target tube at different distances and orientations is improved.

[0046] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0047] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0048] Figure 1 A flowchart illustrating a method for assisting in pipe alignment according to an embodiment of this application is shown schematically;

[0049] Figure 2 This schematic diagram illustrates a structural block diagram of a controller according to an embodiment of the present application;

[0050] Figure 3 The diagram illustrates a structural diagram of a pipe-supporting system according to an embodiment of this application.

[0051] Explanation of reference numerals in the attached figures

[0052] 1. Multiple pan-tilt units 2. Multiple image acquisition devices

[0053] 3 Controllers Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0055] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0056] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0057] Figure 1 A flowchart illustrating a method for assisting in pipe alignment according to an embodiment of this application is shown schematically. Figure 1 As shown in the figure, this application provides a method for assisting in pipe alignment, which may include the following steps.

[0058] Step 101: Obtain the current video frames from multiple image acquisition devices.

[0059] In this embodiment, "pipe alignment" refers to the process of connecting two target pipes. For example, in the field of construction machinery, pipe alignment can be the connection between a target concrete pipe and a reducing pipe, where the target concrete pipe is located on the construction site and the reducing pipe is located on the construction machinery. In one example, the construction machinery can be a concrete pump truck, and the operator drives the concrete pump truck to connect the target concrete pipe and the reducing pipe. In this embodiment, to achieve automatic tracking of the target concrete pipe, while maintaining a high-resolution on-site display of the pipe alignment, the target concrete pipe is also tracked in real time, achieving high-precision visual assistance for the pipe alignment process at different distances and orientations. Therefore, after the pipe alignment assistance system is powered on, the controller first acquires the current video frames from multiple image acquisition devices, which can be cameras installed on the construction machinery. In one example, a preset interval time can be used to acquire video frames at a certain periodic frequency. In another example, the controller can send commands, acquiring one video frame each time a command is sent. In yet another example, video frames are continuously acquired through the image acquisition devices. In this embodiment of the application, taking the first target pipe as a variable diameter pipe and the second target pipe as a target concrete pipe as an example, the variable diameter pipe is located by identifying the current video frame, thereby realizing real-time tracking of the target concrete pipe.

[0060] Step 102: Determine whether the first target tube is detected in the display screen of multiple image acquisition devices based on the current video frame.

[0061] In this embodiment, the reducing pipe and multiple image acquisition devices are all mounted on the engineering machinery. The installation positions of the multiple image acquisition devices can capture images of the target pipe. Preferably, there can be two image acquisition devices, respectively mounted on both sides of the engineering machinery to capture images of the target pipe. After the system is powered on, an algorithm determines whether the target pipe is in the display screen. Both the reducing pipe and the target concrete pipe may appear in the display screen. The image acquisition device containing the first target pipe, i.e., the reducing pipe, is selected as the source of the subsequent system display screen. Therefore, to improve the accuracy of subsequent pipe alignment, the controller first locates the position of the reducing pipe and then tracks the target concrete pipe in real time. Specifically, the controller identifies the current video frame to determine whether the display screen of the image acquisition device contains the reducing pipe, i.e., the first target pipe. First, the target object needs to be located in the current video frame. In one example, the target object can be located in the current video frame based on traditional visual features. In another example, the target object can be located in the current video frame using deep learning methods. After determining the position of the target object in the image, the display screen of the image acquisition device is determined by judging the number and position of the target objects to determine whether the reducing pipe, i.e., the first target pipe, has been detected.

[0062] Step 103: If the first target tube is detected in the display screen, set the image acquisition device corresponding to the display screen as the target image acquisition device, set the pan-tilt unit corresponding to the target image acquisition device as the target pan-tilt unit, and control the target pan-tilt unit to enter the first tracking state.

[0063] In this embodiment, when a first target tube appears in the display screen, it indicates the presence of a first target tube, i.e., a reducing tube. Therefore, the image acquisition device corresponding to the display screen showing the first target tube is set as the target image acquisition device, and the subsequent display screens during the tube-assisted process are all provided by this target image display device. Simultaneously, the pan-tilt unit corresponding to this target image display device is set as the target pan-tilt unit. The controller controls the target pan-tilt unit to enter a first tracking state. The first tracking state refers to the state where the controller controls the rotation to adjust the orientation of the image acquisition device, thereby tracking the center point of the cross-section of the first target tube's opening. Specifically, the controller first locates the center point of the cross-section of the first target tube's opening using an algorithm. Then, the center point of the cross-section of the first target tube's opening is used as the tracking object of the target pan-tilt unit, and the target pan-tilt unit is controlled to rotate and adjust the orientation of the image acquisition device to track the center point of the cross-section of the first target tube's opening, i.e., the first tracking state. Using the center point of the cross-section of the first target tube's opening as the tracking object of the target pan-tilt unit can improve tracking accuracy and sensitivity.

[0064] Step 104: If the center point of the cross-section of the first target tube is located in the center of the display screen in the first tracking state, determine whether the second target tube is detected in the display screen.

[0065] In this embodiment, in the first tracking state, the target pan-tilt unit rotates to adjust the orientation of the image acquisition device to track the center point of the cross-section of the first target pipe opening. When the center point of the cross-section of the first target pipe opening is in the center of the display screen, it indicates that the first target pipe is in the center of the display screen, i.e., the optimal position for the pipe. Then, the controller detects whether the second target pipe, i.e., the target concrete pipe, appears in the display screen.

[0066] Step 105: If a second target tube is detected in the display screen, control the target gimbal to enter the second tracking state.

[0067] In this embodiment, in the first tracking state, when the controller detects the presence of a second target pipe (i.e., a target concrete pipe) in the display screen, it indicates the existence of the second target pipe. At this time, the controller updates the tracking object to the center point of the cross-section of the second target pipe's opening, and the controller controls the target gimbal to enter the second tracking state. The second tracking state refers to the state where the controller controls the rotation of the camera to adjust its orientation, thereby tracking the center point of the cross-section of the second target pipe's opening. Specifically, the controller first locates the center point of the cross-section of the second target pipe's opening using an algorithm. Then, it uses the center point of the cross-section of the second target pipe's opening as the tracking object of the target gimbal, controlling the target gimbal to rotate and adjust the orientation of the image acquisition device to track the center point of the cross-section of the second target pipe's opening, i.e., the second tracking state. Using the center point of the cross-section of the second target pipe's opening as the tracking object of the target gimbal can improve tracking accuracy and sensitivity.

[0068] The above technical solution first acquires the current video frames of multiple image acquisition devices. Based on the current video frames, it is determined whether a first target tube is detected in the display screen of the multiple image acquisition devices. If the first target tube is detected in the display screen, the image acquisition device corresponding to the display screen is set as the target image acquisition device, and the pan-tilt unit corresponding to the target image acquisition device is set as the target pan-tilt unit. The target pan-tilt unit is then controlled to enter the first tracking state. In the first tracking state, if the center point of the cross-section of the first target tube's opening is located in the center of the display screen, it is determined whether a second target tube is detected in the display screen. If the second target tube is detected in the display screen, the target pan-tilt unit is controlled to enter the second tracking state. This application can achieve automatic tracking of the target tube, maintaining high-resolution on-site display while simultaneously tracking the target tube in real time. Through visual assistance during the tube alignment process, the alignment accuracy of the target tube at different distances and orientations is improved.

[0069] In this embodiment of the application, the method for assisting in pipe alignment may further include:

[0070] If the first target tube is not detected in the display screen, the target pan-tilt unit is controlled to enter the target search state until the first target tube is detected in the display screen.

[0071] In this embodiment, if the controller determines that the reducing tube (i.e., the first target tube) is not detected in the displayed screen, it means that the first target tube does not exist in the current screen. Therefore, the controller controls the target pan-tilt unit to enter the target search state. The target search state refers to the state in which the controller controls the target pan-tilt unit to rotate and adjust the orientation of the image acquisition device to find the first target tube. In one example, the target pan-tilt unit can be controlled to rotate along a preset trajectory while continuing to identify and detect the first target tube until it is detected in the displayed screen.

[0072] In this embodiment of the application, the method for assisting in pipe alignment may further include:

[0073] If the second target tube is not detected in the display screen during the second tracking state, the target pan-tilt unit is controlled to enter the target search state.

[0074] In this embodiment, during the second tracking state, if the displayed screen does not contain the second target tube, it indicates that the second target tube is not present in the current screen. Therefore, the controller controls the target pan-tilt unit to enter the target search state. The target search state refers to the state in which the controller controls the target pan-tilt unit turntable to rotate and adjust the orientation of the image acquisition device to find the first target tube. In one example, the target pan-tilt unit can be controlled to rotate along a preset trajectory while continuing to identify and detect the first target tube until it is detected in the displayed screen.

[0075] In this embodiment of the application, controlling the target gimbal to enter the target search state may further include:

[0076] The target PTZ is controlled to rotate along a preset trajectory, and the first target tube is detected in the display screen of the image acquisition device based on the current video frame.

[0077] In this embodiment, the target search state refers to the state in which the controller controls the target pan-tilt unit to rotate and adjust the orientation of the image acquisition device to search for the first target tube. The controller determines whether the display screen of the image acquisition device contains the variable-diameter tube, i.e., the first target tube, by identifying the current video frame. When the first target tube is not detected in the display screen of the image acquisition device, it means that the first target tube, i.e., the variable-diameter tube, does not exist in the screen. Therefore, the controller controls the target pan-tilt unit to enter the target search state and determines whether the first target tube is detected in the display screen of the image acquisition device based on the current video frame.

[0078] In this embodiment of the application, the method for assisting in pipe alignment may further include:

[0079] If multiple candidate second target tubes are detected in the display screen, the candidate second target tube that is closest to the first target tube is identified as the second target tube.

[0080] In this embodiment of the application, when multiple candidate second target pipes are detected in the display screen, the distance between the "target point" on each candidate second target pipe and the "target point" on the variable diameter pipe, i.e. the first target pipe, is calculated, and the candidate second target pipe with the closest distance is determined as the second target pipe, i.e. the target concrete pipe.

[0081] In this embodiment of the application, determining whether a first target tube is detected in the display screen of the image acquisition device based on the current video frame may include:

[0082] Identify the target object in the current video frame;

[0083] If a target object is detected in a video frame, the target object is identified as the first target tube;

[0084] If more than two target objects are identified in a video frame, the first target tube and multiple candidate second target tubes are determined based on the position of the image acquisition device and the aspect ratio and absolute value of the target objects.

[0085] In this embodiment, the controller identifies the current video frame to determine whether the display screen of the image acquisition device contains the reducer, i.e., the first target pipe. Specifically, the reducer or target concrete pipe needs to be located in the current video frame first. In one example, the reducer or target concrete pipe can be located in the current video frame based on traditional visual features, such as superpixel segmentation, which uses adjacent pixels with similar color, brightness, texture, and other features to perform regional segmentation, and then combines the preset color features of the reducer or target concrete pipe to locate it. In another example, the reducer or target concrete pipe can be located in the current video frame using deep learning methods, such as semantic segmentation algorithms to perform pixel-level classification of camera images, thereby achieving the location of the reducer or target concrete pipe. Alternatively, a target detection method can be used to directly find the minimum bounding moment of the reducer or target concrete pipe in the image captured by the image acquisition device, and return the position of the reducer or target concrete pipe in the image in the form of a bounding box (BBox).

[0086] After confirming the location of the reducer or target concrete pipe on the image, it is necessary to distinguish between them. Specifically, the number and location can be determined based on the positioning results. In one example, if only one target object exists in the image, then that target object is the first target pipe. In another example, if there are more than two target objects in the image, firstly, the left-right position of the first target pipe in the image can be determined based on the position of the image acquisition device, and then the first target pipe, i.e., the reducer, can be identified based on the aspect ratio and absolute values ​​of the width and height. In this case, apart from the unique first target pipe, the others are all candidate second target pipes.

[0087] In this embodiment of the application, when it is determined that a first target tube is detected in the display screen, setting the image acquisition device corresponding to the display screen as the target image acquisition device, setting the pan-tilt unit corresponding to the target image acquisition device as the target pan-tilt unit, and controlling the target pan-tilt unit to enter the first tracking state may include:

[0088] If the first target tube is detected in the display screen, determine the center point of the cross-section of the tube opening of the first target tube;

[0089] Control the rotation of the target gimbal so that the center point of the cross-section of the first target tube is in the center of the display screen.

[0090] In this embodiment, when a first target tube is detected in the displayed image, it indicates the presence of a first target tube (i.e., a variable-diameter tube). Therefore, the controller controls the target pan-tilt unit to enter a first tracking state. The first tracking state refers to the controller controlling the rotation of the movement to adjust the orientation of the image acquisition device, thereby tracking the center point of the cross-section of the first target tube's opening. Therefore, it is first necessary to determine the center point of the cross-section of the first target tube's opening and use it as the tracking object. Then, the controller controls the rotation of the target pan-tilt unit so that the center point of the cross-section of the first target tube's opening is located in the center of the displayed image.

[0091] In this embodiment of the application, determining the center point of the cross-section of the nozzle of the first target tube may include:

[0092] Determine the central axis of the first target tube;

[0093] A vector is formed by the center point of the central axis of the first target tube and the points on the cross-section of the first target tube;

[0094] The center point of the pipe opening of the first target pipe is determined by the projection of the vector onto the central axis of the pipe body.

[0095] In this embodiment of the application, the central axis of the first target pipe, i.e., the variable diameter pipe, can be calculated by an algorithm. Then, the point on the cross-section of the first target pipe can be obtained by an algorithm or by empirical value. The projection of the vector composed of the center point of the central axis of the pipe and the point on the cross-section of the first target pipe onto the central axis of the pipe can be calculated, thereby obtaining the center point of the cross-section of the pipe opening of the first target pipe.

[0096] In this embodiment of the application, when it is determined that a second target tube is detected in the display screen, controlling the target gimbal to enter the second tracking state may include:

[0097] If a second target tube is detected in the display screen, determine the center point of the cross-section of the second target tube opening;

[0098] Control the rotation of the target gimbal so that the center point of the cross-section of the second target tube's opening is in the center of the display screen.

[0099] In this embodiment, if a second target tube is detected in the displayed image, it indicates the presence of a first target tube (i.e., a variable-diameter tube). Therefore, the controller controls the target pan-tilt unit to enter a second tracking state. The second tracking state refers to the controller controlling the rotation of the camera to adjust its orientation, thereby tracking the center point of the cross-section of the second target tube's opening. Therefore, it is first necessary to determine the center point of the cross-section of the second target tube's opening and use it as the tracking object. Then, the controller controls the rotation of the target pan-tilt unit to center the cross-section of the second target tube's opening in the center of the displayed image.

[0100] In this embodiment of the application, determining the center point of the cross-section of the nozzle of the second target tube may include:

[0101] Determine the central axis of the second target tube;

[0102] A vector is formed by the center point of the central axis of the second target tube and the points on the cross-section of the second target tube;

[0103] The center point of the cross-section of the second target tube's opening is determined by the projection of the vector onto the central axis of the second target tube's body.

[0104] In this embodiment of the application, the central axis of the second target pipe, i.e., the variable diameter pipe, can be calculated by an algorithm. Then, the point on the cross-section of the second target pipe can be obtained by an algorithm or by empirical value. The projection of the vector composed of the center point of the central axis of the pipe and the point on the cross-section of the second target pipe onto the central axis of the pipe can be calculated, thereby obtaining the center point of the cross-section of the pipe opening of the second target pipe.

[0105] Figure 2 A schematic block diagram of a controller according to an embodiment of this application is shown. Figure 2 As shown in the figure, this application provides a controller that may include:

[0106] Memory 210 is configured to store instructions; and

[0107] The processor 220 is configured to retrieve instructions from the memory 210 and, when executing the instructions, to implement the aforementioned method for controlling the boom.

[0108] Specifically, in this embodiment of the application, the processor 220 can be configured to:

[0109] Acquire the current video frames from multiple image acquisition devices;

[0110] Determine whether the first target tube is detected in the display screen of multiple image acquisition devices based on the current video frame;

[0111] If a first target tube is detected in the display screen, the image acquisition device corresponding to the display screen is set as the target image acquisition device, the pan-tilt unit corresponding to the target image acquisition device is set as the target pan-tilt unit, and the target pan-tilt unit is controlled to enter the first tracking state so that the center point of the tube opening of the first target tube is located in the center of the display screen.

[0112] If, in the first tracking state, the center point of the cross-section of the first target tube is located in the center of the display screen, determine whether the second target tube is detected in the display screen.

[0113] If a second target tube is detected in the display screen, the target gimbal is controlled to enter the second tracking state so that the center point of the cross-section of the second target tube's opening is located in the center of the display screen.

[0114] Furthermore, the processor 220 can also be configured to:

[0115] If the first target tube is not detected in the display screen, the target pan-tilt unit is controlled to enter the target search state until the first target tube is detected in the display screen.

[0116] Furthermore, the processor 220 can also be configured to:

[0117] If the second target tube is not detected in the display screen during the second tracking state, the target pan-tilt unit is controlled to enter the target search state.

[0118] Furthermore, the processor 220 can also be configured to:

[0119] The target gimbal is controlled to rotate along a preset trajectory, and the first target tube is detected in the display screen of the image acquisition device based on the current video frame.

[0120] Furthermore, the processor 220 can also be configured to:

[0121] If multiple candidate second target tubes are detected in the display screen, the candidate second target tube that is closest to the first target tube is identified as the second target tube.

[0122] Furthermore, the processor 220 can also be configured to:

[0123] Identify the target object in the current video frame;

[0124] If a target object is detected in a video frame, the target object is identified as the first target tube, and the number of first target tubes is determined based on the pixel quality of the target object in the image and the aspect ratio of the target tube.

[0125] If more than two target objects are identified in a video frame, the first target tube and multiple candidate second target tubes are determined based on the location of the image acquisition device and the aspect ratio and absolute value of the target objects.

[0126] Furthermore, the processor 220 can also be configured to:

[0127] If the first target tube is detected in the display screen, determine the center point of the cross-section of the tube opening of the first target tube;

[0128] Control the rotation of the target gimbal so that the center point of the cross-section of the first target tube is in the center of the display screen.

[0129] Furthermore, the processor 220 can also be configured to:

[0130] Determine the central axis of the first target tube;

[0131] A vector is formed by the center point of the central axis of the first target tube and the points on the cross-section of the first target tube;

[0132] The center point of the pipe opening of the first target pipe is determined by the projection of the vector onto the central axis of the pipe body.

[0133] Furthermore, the processor 220 can also be configured to:

[0134] If a second target tube is detected in the display screen, determine the center point of the cross-section of the second target tube opening;

[0135] Control the rotation of the target gimbal so that the center point of the cross-section of the second target tube's opening is in the center of the display screen.

[0136] Furthermore, the processor 220 can also be configured to:

[0137] Determine the central axis of the second target tube;

[0138] A vector is formed by the center point of the central axis of the second target tube and the points on the cross-section of the second target tube;

[0139] The center point of the cross-section of the second target tube's opening is determined by the projection of the vector onto the central axis of the second target tube's body.

[0140] The above technical solution first acquires the current video frames of multiple image acquisition devices. Based on the current video frames, it is determined whether a first target tube is detected in the display screen of the multiple image acquisition devices. If the first target tube is detected in the display screen, the image acquisition device corresponding to the display screen is set as the target image acquisition device, and the pan-tilt unit corresponding to the target image acquisition device is set as the target pan-tilt unit. The target pan-tilt unit is then controlled to enter the first tracking state. In the first tracking state, if the center point of the cross-section of the first target tube's opening is located in the center of the display screen, it is determined whether a second target tube is detected in the display screen. If the second target tube is detected in the display screen, the target pan-tilt unit is controlled to enter the second tracking state. This application can achieve automatic tracking of the target tube, maintaining high-resolution on-site display while simultaneously tracking the target tube in real time. Through visual assistance during the tube alignment process, the alignment accuracy of the target tube at different distances and orientations is improved.

[0141] Figure 3 A schematic diagram illustrating the structure of a pair of pipe auxiliary systems according to an embodiment of this application is shown. Figure 3 As shown in the embodiments of this application, a pipe-connecting auxiliary system is also provided, which may include:

[0142] Multiple pan-tilt units 1 are configured to adjust the orientation of the image acquisition device;

[0143] Multiple image acquisition devices 2 are mounted on multiple pan-tilt units 1 and configured to acquire images; and

[0144] The aforementioned controller 3 communicates with multiple pan-tilt units 1 and multiple image acquisition devices 2.

[0145] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described method for assisting in the control process.

[0146] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0148] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0149] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0150] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0152] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0153] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0154] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for assisting in pipe alignment, characterized in that, This method is applied to a pipe alignment assistance system, which includes multiple pan-tilt units, multiple image acquisition devices, and a controller. The multiple image acquisition devices are respectively mounted on the multiple pan-tilt units, and each pan-tilt unit and image acquisition device communicates with the controller. Pipe alignment refers to the process of docking a first target pipe and a second target pipe. The method includes: Obtain the current video frames of the plurality of image acquisition devices; Based on the current video frame, determine whether the first target tube is detected in the display screen of the multiple image acquisition devices; If the first target tube is detected in the display screen, the image acquisition device corresponding to the display screen is set as the target image acquisition device, the pan-tilt unit corresponding to the target image acquisition device is set as the target pan-tilt unit, and the target pan-tilt unit is controlled to enter the first tracking state so that the center point of the tube opening of the first target tube is located in the center of the display screen. If, in the first tracking state, the center point of the cross-section of the first target tube is located in the center of the display screen, it is determined whether the second target tube is detected in the display screen; If the second target tube is detected in the display screen, the target gimbal is controlled to enter the second tracking state so that the center point of the tube opening of the second target tube is in the center of the display screen.

2. The method according to claim 1, characterized in that, The method further includes: If it is determined that the first target tube is not detected in the display screen, the target pan-tilt unit is controlled to enter the target search state until the first target tube is detected in the display screen.

3. The method according to claim 1, characterized in that, The method further includes: If the second target tube is not detected in the display screen during the second tracking state, the target pan-tilt unit is controlled to enter the target search state.

4. The method according to claim 2 or 3, characterized in that, The step of controlling the target gimbal to enter the target search state includes: The target gimbal is controlled to rotate along a preset trajectory, and the first target tube is detected in the display screen of the image acquisition device based on the current video frame.

5. The method according to claim 1, characterized in that, The method further includes; If multiple candidate second target tubes are detected in the display screen, the candidate second target tube that is closest to the first target tube is identified as the second target tube.

6. The method according to claim 1, characterized in that, The step of determining whether the first target tube is detected in the display screen of the image acquisition device based on the current video frame includes: Identify the target object in the current video frame; If a target object is detected in the video frame, the target object is identified as the first target tube; If more than two target objects are identified in the video frame, the first target tube and multiple candidate second target tubes are determined based on the position of the image acquisition device and the aspect ratio and absolute value of the target objects.

7. The method according to claim 1, characterized in that, When it is determined that the first target tube is detected in the display screen, the step of setting the image acquisition device corresponding to the display screen as the target image acquisition device, setting the pan-tilt unit corresponding to the target image acquisition device as the target pan-tilt unit, and controlling the target pan-tilt unit to enter the first tracking state, so that the center point of the tube opening cross-section of the first target tube is located in the center of the display screen, includes: If the first target tube is detected in the display screen, the center point of the tube opening cross-section of the first target tube is determined. Control the rotation of the target gimbal so that the center point of the cross-section of the first target tube is in the center of the display screen.

8. The method according to claim 7, characterized in that, Determining the center point of the cross-section of the first target tube's opening includes: Determine the central axis of the first target tube; A vector is formed by the center point of the central axis of the first target tube and the points on the cross-section of the first target tube; The center point of the cross-section of the first target tube is determined by the projection of the vector onto the central axis of the tube body.

9. The method according to claim 1, characterized in that, The step of controlling the target pan-tilt unit to enter a second tracking state when the second target tube is detected in the display screen, so that the center point of the cross-section of the second target tube's opening is located in the center of the display screen, includes: If the second target tube is detected in the display screen, the center point of the cross-section of the tube opening of the second target tube is determined; Control the rotation of the target gimbal so that the center point of the cross-section of the second target tube is in the center of the display screen.

10. The method according to claim 9, characterized in that, Determining the center point of the cross-section of the second target tube's opening includes: Determine the central axis of the second target tube; A vector is formed by the center point of the central axis of the second target tube and the points on the cross-section of the second target tube; The center point of the cross-section of the second target tube is determined by the projection of the vector onto the central axis of the tube body.

11. A controller, characterized in that, include: The memory is configured to store instructions; as well as A processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method for assisting in the switching process according to any one of claims 1 to 10.

12. A pipe-connecting auxiliary system, characterized in that, include: Multiple pan-tilt units are configured to adjust the orientation of multiple image acquisition devices; The plurality of image acquisition devices are respectively mounted on the plurality of pan-tilt units and configured to acquire images; as well as The controller according to claim 11 communicates with the plurality of gimbals and the plurality of image acquisition devices.

13. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the method for assisting in the connection of the tube according to any one of claims 1 to 10.

Images