Methods for assisting pipe alignment, vision-assisted pipe alignment systems and controllers

Abstract

This application discloses a method, a vision-assisted tube alignment system, and a controller for assisting in tube alignment. The method includes: acquiring the current video frame of an image acquisition device; determining whether the displayed screen of the current video frame contains a first target tube; if the first target tube is found in the displayed screen, adjusting the focal length of the image acquisition device to the target focal length; and generating alignment auxiliary lines on the displayed screen when the focal length of the image acquisition device is the target focal length. This application can adaptively switch the camera focal length according to the position of the target tube. Furthermore, by adding alignment auxiliary lines, it facilitates the driver's judgment of the relative position of the two tubes and whether the current movement trajectory can align the tubes, thereby achieving high-precision vision assistance in the tube alignment process at different distances, improving the accuracy and efficiency of high-precision tube alignment.

Description

Technical Field

[0001] This application relates to the field of vision-assisted alignment technology, and more specifically to a method, vision-assisted alignment system and controller for assisting alignment. Background Technology

[0002] Concrete truck-mounted pumps are widely used in the construction industry due to their advantages such as flexible operation, wide construction range, and high pumping performance. However, due to complex construction environments and insufficient operator experience, the installation of concrete truck-mounted pumps in pipelines often takes a long time.

[0003] Currently, some truck-mounted concrete pumps are equipped with reversing auxiliary cameras (similar to reversing cameras) to monitor the position of the target concrete pipes on the construction site and the possible movement trajectory of the vehicle in real time. However, in actual operation, the auxiliary pipe alignment system generally uses a fixed-focus camera, which cannot cover both near and far distances. Furthermore, the position of the auxiliary line is usually obtained through calibration, and the diverse postures of variable-diameter pipes can lead to insufficient auxiliary pipe alignment capability of the system, resulting in low pipe alignment efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a method, a vision-assisted alignment system, and a controller for assisting in alignment, in order to solve the problem that the fixed field of view during alignment in the prior art leads to low alignment efficiency.

[0005] To achieve the above objectives, the first aspect of this application provides a method for assisting in alignment, applied to a vision-assisted alignment system. The vision-assisted alignment system includes an image acquisition device and a controller, wherein the image acquisition device communicates with the controller, and the method includes:

[0006] Obtain the current video frame from the image acquisition device;

[0007] Determine whether the first target tube is contained in the displayed screen of the current video frame;

[0008] If the display screen contains the first target tube, adjust the focus of the image acquisition device to the target focus.

[0009] When the focal length of the image acquisition device is the target focal length, auxiliary lines for the tube are generated on the display screen.

[0010] In this embodiment, the image acquisition device includes a telephoto camera and a short-focus camera. Adjusting the focal length of the image acquisition device to the target focal length includes:

[0011] Determine whether the area of ​​the cross-section of the first target tube opening in the display image of the short-focus camera is greater than a first threshold.

[0012] If the area of ​​the cross-section of the first target tube opening in the display screen of the short-focus camera is greater than the first threshold, the short-focus camera is switched to a long-focus camera so that the focal length of the image acquisition device is the first target focal length.

[0013] Determine whether the area of ​​the cross-section of the first target tube opening in the display image of the telephoto camera is less than the second threshold.

[0014] If the area of ​​the cross-section of the first target tube opening in the display screen of the telephoto camera is less than the second threshold, the telephoto camera is switched to a short-focus camera so that the focal length of the image acquisition device is the second target focal length.

[0015] In this embodiment of the application, the image acquisition device includes a zoom camera, and adjusting the focal length of the image acquisition device to the target focal length includes:

[0016] Obtain the diameter of the opening of the first target tube;

[0017] The target focal length is determined based on the diameter of the first target tube opening;

[0018] Adjust the focal length of the image acquisition device to the third target focal length based on the target focal length.

[0019] In this embodiment of the application, determining the target focal length based on the diameter of the first target tube's opening includes:

[0020] The original unit pixel value of the opening of the first target tube is determined based on the imaging resolution of the image acquisition device and the diameter of the opening of the first target tube.

[0021] The target unit pixel value of the opening of the first target tube is determined based on the opening diameter of the first target tube and the target pixel value of the opening diameter of the first target tube.

[0022] The target focal length of the image acquisition device is determined based on the original unit pixel value, the target unit pixel value, and the original focal length of the image acquisition device.

[0023] In this embodiment of the application, generating guide lines for the display screen includes:

[0024] Upon receiving the first generation mode instruction, the poses of the first target tube and the second target tube are obtained;

[0025] Determine the slope and intercept of the central axis of the first target tube based on its pose;

[0026] Based on the pose of the second target tube, determine the central axis of the second target tube, the tangential auxiliary line perpendicular to the central axis on the tube opening section of the target tube, and the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening section of the second target tube.

[0027] In this embodiment of the application, generating guide lines for the display screen includes:

[0028] Upon receiving the second generation mode instruction, the poses of the first target tube and the second target tube are obtained;

[0029] Determine the center point of the cross-section of the first target tube's opening based on the pose of the first target tube;

[0030] Based on the pose of the second target tube, determine the central axis of the second target tube, the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening of the second target tube.

[0031] In this embodiment of the application, generating guide lines for the display screen includes:

[0032] Upon receiving the third generation mode instruction, the pose of the second target tube is obtained;

[0033] Based on the pose of the second target tube, determine the central axis of the second target tube, the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening of the second target tube.

[0034] A second aspect of this application provides a controller, comprising:

[0035] The memory is configured to store instructions; and

[0036] The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the aforementioned method for assisting in the control process.

[0037] A third aspect of this application provides a vision-assisted control system, comprising:

[0038] Image acquisition devices are configured to acquire images; and

[0039] The controller described above communicates with the image acquisition device.

[0040] A fourth aspect of this application provides a machine-readable storage medium storing instructions that cause a machine to perform the aforementioned auxiliary control method.

[0041] The above technical solution first acquires the current video frame of the image acquisition device, and then determines whether the displayed screen of the current video frame contains the first target tube. If the first target tube is found to be present in the displayed screen, the focal length of the image acquisition device is adjusted to the target focal length. Finally, with the focal length of the image acquisition device at the target focal length, alignment guide lines are generated on the displayed screen. This application can adaptively switch the camera focal length according to the position of the target tube. In addition, by adding alignment guide lines, it is easier for the driver to judge the relative position of the two tubes and whether the current movement trajectory can align the tubes, thereby achieving high-precision visual assistance in the alignment process at different distances, improving the accuracy and efficiency of tube alignment.

[0042] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0043] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0044] Figure 1 A flowchart illustrating a method for assisting in pipe alignment according to an embodiment of this application is shown schematically;

[0045] Figure 2 A flowchart illustrating a method for adjusting the focal length of an image acquisition device to a target focal length according to an embodiment of this application is shown schematically.

[0046] Figure 3 A flowchart illustrating a method for adjusting the focal length of an image acquisition device to a target focal length according to another embodiment of this application is shown schematically.

[0047] Figure 4 This schematic diagram illustrates a structural block diagram of a controller according to an embodiment of the present application;

[0048] Figure 5 The diagram schematically illustrates a structural diagram of a vision-assisted control system according to an embodiment of this application.

[0049] Explanation of reference numerals in the attached figures

[0050] 1 Image acquisition device 2 Controller Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0052] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0053] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0054] Figure 1 A flowchart illustrating a method for assisting in pipe alignment according to an embodiment of this application is shown schematically. Figure 1 As shown in the embodiment of this application, a method for assisting in alignment is provided, which is applied to a vision-assisted alignment system. The vision-assisted alignment system includes an image acquisition device and a controller. The image acquisition device communicates with the controller. The method may include the following steps.

[0055] Step 101: Obtain the current video frame of the image acquisition device.

[0056] In this application embodiment, pipe alignment refers to the process of connecting two target pipes. For example, in the field of construction machinery, pipe alignment can be the connection between a target concrete pipe and a reducing pipe, wherein the target concrete pipe is located on the construction site, and the reducing pipe and image acquisition equipment are located on the construction machinery. Preferably, the image acquisition equipment can be a camera. In one example, the construction machinery can be a concrete pump truck, and the operator drives the concrete pump truck to connect the target concrete pipe and the reducing pipe. Existing technology mainly draws on the reversing camera function of cars. This technology usually only provides auxiliary line calibration for the reducing pipe, but does not consider the position of the target concrete pipe in actual use. Usually, it is necessary to manually judge the relative position of the two pipes and then adjust the vehicle body. Furthermore, the existing technology uses a fixed-focus camera, and the field of view of the camera is fixed. When the field of view of the camera is small, the target concrete pipe in the distance cannot be seen; when the field of view of the camera is wide, there are few pipe opening features, which is not conducive to high-precision pipe alignment. Therefore, based on the shortcomings of the existing technology, this application proposes a method for assisting pipe alignment.

[0057] In this embodiment, taking the first target pipe as the target concrete pipe and the second target pipe as a variable diameter pipe as an example, the target concrete pipe is located by identifying the current video frame, thereby realizing adaptive switching of the focal length of the image acquisition device according to the position of the target concrete pipe. The controller first acquires the current video frame of the image acquisition device, and the position of the target concrete pipe in the picture can be quickly located by identifying the video frame based on traditional visual features and deep learning algorithms.

[0058] Step 102: Determine whether the display screen of the current video frame contains the first target tube.

[0059] In this embodiment, the controller identifies the current video frame to determine whether the displayed image of the current video frame contains the target concrete pipe, i.e., the first target pipe. Specifically, the target concrete pipe needs to be located in the current video frame first. In one example, the target concrete pipe can be located in the current video frame based on traditional visual features, such as superpixel segmentation, which uses pixels that are adjacent in position and have similar features such as color, brightness, and texture to perform regional segmentation, and then combines the preset color features of the variable diameter pipe or the target concrete pipe to locate the target concrete pipe. In another example, the target concrete pipe can be located in the current video frame using deep learning methods, such as semantic segmentation algorithms to perform pixel-level classification of camera images, thereby achieving the location of the target concrete pipe. Alternatively, a target detection method can be used to directly find the minimum bounding moment of the target concrete pipe in the image captured by the image acquisition device, and return the position of the target concrete pipe in the image in the form of a bounding box (BBox). It should be noted that since both the variable diameter pipe and the image acquisition device are installed on the engineering machinery, the variable diameter pipe is always present in the displayed image.

[0060] Step 103: If it is determined that the first target tube is present in the display screen, adjust the focal length of the image acquisition device to the target focal length.

[0061] In this embodiment, if the display screen contains a first target pipe, it indicates that the first target pipe, i.e., the target concrete pipe, exists in the display screen. At this time, both the target concrete pipe and the reducing pipe are present in the display screen, and pipe alignment can begin. To achieve high-precision visual assistance for pipe alignment at different distances, a suitable field of view is first calculated in reverse based on the size of the target concrete pipe. Then, the focal length of the image acquisition device is determined based on this suitable field of view. Finally, the focal length of the image acquisition device is adjusted to the target focal length to achieve high-precision pipe alignment. In one example, the image acquisition device includes a telephoto camera and a short-focus camera. Switching between the telephoto and short-focus cameras allows the image acquisition device to reach the target focal length. In another example, the image acquisition device includes a zoom camera. Zooming with the zoom camera allows the image acquisition device to reach the target focal length.

[0062] Step 104: When the focal length of the image acquisition device is the target focal length, generate auxiliary lines for the tube on the display screen.

[0063] In this embodiment, when the focal length of the image acquisition device is the target focal length, the controller can clearly identify the pipe opening features of the target concrete pipe and the reducing pipe, and high-precision pipe alignment can be achieved based on the pipe opening features. At this time, depending on the actual situation, different generation modes can be selected on the display screen to generate pipe alignment guide lines. In one example, selecting the first generation mode displays strong guide lines on the display screen. In another example, selecting the second generation mode displays ordinary guide lines on the display screen. In yet another example, selecting the third generation mode displays weak guide lines on the display screen.

[0064] The above technical solution first acquires the current video frame of the image acquisition device, then determines whether the displayed screen of the current video frame contains the first target tube. If the first target tube is found to be present in the displayed screen, the focal length of the image acquisition device is adjusted to the target focal length. Finally, with the focal length of the image acquisition device at the target focal length, alignment guide lines are generated on the displayed screen. This application can adaptively switch the camera focal length according to the position of the target tube. Furthermore, by adding alignment guide lines, it is easier for the driver to determine the relative position of the two tubes and whether the current movement trajectory can align the tubes, thereby achieving high-precision visual assistance in the alignment process at different distances, improving the accuracy and efficiency of high-precision alignment.

[0065] Figure 2 A flowchart illustrating a method for adjusting the focal length of an image acquisition device to a target focal length according to an embodiment of this application is shown schematically. Figure 2As shown in the embodiments of this application, the image acquisition device may include a telephoto camera and a short-focus camera, and adjusting the focal length of the image acquisition device to the target focal length may include the following steps.

[0066] Step 201: Determine whether the area of ​​the cross-section of the first target tube opening in the display screen of the short-focus camera is greater than the first threshold.

[0067] Step 202: If it is determined that the area of ​​the cross-section of the first target tube opening in the display screen of the short-focus camera is greater than the first threshold, control the short-focus camera to switch to a long-focus camera so that the focal length of the image acquisition device is the first target focal length.

[0068] Step 203: Determine whether the area of ​​the cross-section of the first target tube opening in the display screen of the telephoto camera is less than the second threshold.

[0069] Step 204: If the area of ​​the cross-section of the first target tube opening in the display screen of the telephoto camera is less than the second threshold, control the telephoto camera to switch to a short-focus camera so that the focal length of the image acquisition device is the second target focal length.

[0070] In this embodiment, the image acquisition device may include a telephoto camera and a short-focus camera. By controlling the switching between the telephoto and short-focus cameras, the focal length of the image acquisition device can be set to the target focal length. It should be noted that the default camera is a short-focus camera. In this embodiment, the size of the cross-section of the target concrete pipe's opening in the display screen of the short-focus camera is used as the basis for switching between the telephoto and short-focus cameras. The short-focus camera has a larger field of view, and the area of ​​the cross-section of the target concrete pipe's opening in the short-focus camera's display screen is smaller; conversely, the telephoto camera has a larger field of view, and the area of ​​the cross-section of the target concrete pipe's opening in the display screen of the short-focus camera is larger.

[0071] Specifically, in this embodiment, two thresholds can be set first, namely a first threshold and a second threshold. It is determined whether the area of ​​the cross-section of the first target pipe (i.e., the target concrete pipe) in the display screen of the short-focus camera is greater than the first threshold. If the area is greater, the controller switches the short-focus camera to a telephoto camera so that the focal length of the image acquisition device is the first target focal length. At this time, the field of view of the image acquisition device decreases, and the proportion of the target concrete pipe in the field of view increases. Similarly, it is determined whether the area of ​​the cross-section of the first target pipe (i.e., the target concrete pipe) in the display screen of the telephoto camera is less than the second threshold. If the area is less, the controller switches the telephoto camera to a short-focus camera so that the focal length of the image acquisition device is the second target focal length. At this time, the field of view of the image acquisition device increases, the proportion of the target concrete pipe in the field of view decreases, and the driver of the construction machinery can obtain more environmental information about the surrounding environment.

[0072] Preferably, since the switching between the two cameras will cause abrupt changes in the image, digital zoom can be applied first on the short-focus camera based on the distance between the target concrete pipe and the camera center. When the target concrete pipe is larger than a preset threshold in the original field of view of the short-focus camera, the controller controls the short-focus camera to switch to a telephoto camera. When switched to the telephoto camera, digital zoom continues to be applied based on the distance between the target concrete pipe and the camera center to obtain the optimal display field of view.

[0073] Figure 3 A flowchart illustrating a method for adjusting the focal length of an image acquisition device to a target focal length according to another embodiment of this application is shown schematically. Figure 3 As shown in the embodiments of this application, the image acquisition device may include a zoom camera, and adjusting the focal length of the image acquisition device to the target focal length may include the following steps.

[0074] Step 301: Obtain the diameter of the opening of the first target tube;

[0075] Step 302: Determine the target focal length based on the diameter of the first target tube opening;

[0076] Step 303: Adjust the focal length of the image acquisition device to the third target focal length according to the target focal length.

[0077] In this embodiment, the image acquisition device may include a zoom camera. A zoom lens is a camera lens that can change its focal length within a certain range to obtain different widths of field of view, different image sizes, and different scene ranges. A zoom lens can change the shooting range by changing the focal length without changing the shooting distance. Therefore, when the focal length is fixed, the farther the object being photographed is from the camera, the smaller its image proportion in the image; conversely, the closer the object is to the camera, the larger its image proportion in the image. The principle of camera imaging can be viewed as pinhole imaging. Based on the principle of pinhole imaging, the object size and the camera's default focal length are first calibrated. Subsequent use allows for the calculation of a suitable imaging focal length based on the expected image size of the object. Therefore, in this embodiment, the diameter of the first target pipe, i.e., the target concrete pipe, is used as the basis for calculating the target focal length to more accurately determine the optimal field of view. The controller first obtains the diameter of the target concrete pipe's opening, and then determines the target focal length based on the pipe diameter and the pinhole imaging principle. Finally, the focal length of the image acquisition device is adjusted to the target focal length, i.e., the third target focal length.

[0078] In this embodiment of the application, step 302, determining the target focal length based on the diameter of the first target tube opening, may include:

[0079] The original unit pixel value of the opening of the first target tube is determined based on the imaging resolution of the image acquisition device and the diameter of the opening of the first target tube.

[0080] The target unit pixel value of the opening of the first target tube is determined based on the opening diameter of the first target tube and the target pixel value of the opening diameter of the first target tube.

[0081] The target focal length of the image acquisition device is determined based on the original unit pixel value, the target unit pixel value, and the original focal length of the image acquisition device.

[0082] Specifically, determining the target focal length based on the diameter of the target concrete pipe (i.e., the first target pipe) and the principles of camera imaging requires first determining the original and target pixel values ​​of the target concrete pipe's opening. The original pixel value can be determined based on the imaging resolution of the image acquisition device and the diameter of the target concrete pipe's opening. The target pixel value can be determined based on the diameter of the first target pipe's opening and its target pixel value. Finally, the target focal length of the image acquisition device can be determined based on the original pixel value, the target pixel value, and the original focal length of the image acquisition device. Here, the imaging resolution, pipe diameter, and original focal length of the image acquisition device are pre-determined values, while the target pixel value of the target concrete pipe's opening diameter can be determined based on actual conditions.

[0083] In this embodiment, the generation modes of the pipe auxiliary lines include a first generation mode, a second generation mode, and a third generation mode, which correspond to a strong boot mode, a normal boot mode, and a weak boot mode, respectively. The controller can receive different generation mode instructions and generate pipe auxiliary lines of different modes according to the different generation mode instructions.

[0084] In one embodiment of this application, generating guide lines for the display screen may include:

[0085] Upon receiving the first generation mode instruction, the poses of the first target tube and the second target tube are obtained;

[0086] Determine the slope and intercept of the central axis of the first target tube based on its pose;

[0087] Based on the pose of the second target tube, determine the central axis of the second target tube, the tangential auxiliary line perpendicular to the central axis on the tube opening section of the target tube, and the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening section of the second target tube.

[0088] Specifically, during the pipe alignment process, the driver can select different guidance modes, each corresponding to a different auxiliary line generation mode. When the driver selects the strong guidance mode, the display shows three auxiliary lines: the centerline of the target concrete pipe, the centerline of the reducer pipe, a tangential auxiliary line perpendicular to the centerline on the reducer pipe's cross-section, and a tangential auxiliary line parallel to the ground on the reducer pipe's cross-section. When the driver selects the strong guidance mode, the controller receives a first generation mode command. In the first generation mode, the controller first acquires the poses of the first and second target pipes, namely the target concrete pipe and the reducer pipe. The pose refers to the position and orientation of the target concrete pipe and the reducer pipe on the image. Based on the pose of the target concrete pipe, the slope and intercept of its centerline can be determined. Based on the pose of the reducer pipe, the slope and intercept of its centerline, the tangential auxiliary line perpendicular to the centerline on the reducer pipe's cross-section, and the tangential auxiliary line parallel to the ground on the reducer pipe's cross-section can be determined respectively. Among them, the slopes of the central axis and the tangential auxiliary lines parallel to the ground on the pipe opening are correlated, and this correlation can be determined in advance.

[0089] In another embodiment of this application, generating guide lines on the display screen may include:

[0090] Upon receiving the second generation mode instruction, the poses of the first target tube and the second target tube are obtained;

[0091] Determine the center point of the cross-section of the first target tube's opening based on the pose of the first target tube;

[0092] Based on the pose of the second target tube, determine the central axis of the second target tube, the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening of the second target tube.

[0093] Specifically, when the driver selects the normal guidance mode, the display shows two auxiliary lines and one auxiliary point. The auxiliary lines are the centerline of the reducer and the tangential auxiliary line parallel to the ground on the cut surface of the reducer. The auxiliary point is the center point of the cut surface of the target concrete pipe. When the driver selects the normal guidance mode, the controller receives a second generation mode command. In the second generation mode, the controller first acquires the poses of the first and second target pipes, i.e., the reducer of the target concrete pipe. The pose refers to the position and orientation of the target concrete pipe and the reducer on the image. Based on the pose of the target concrete pipe, the center point of the cut surface of the target concrete pipe can be determined. Based on the pose of the reducer, the slope and intercept of the centerline of the reducer and the tangential auxiliary line parallel to the ground on the cut surface of the reducer can be determined. The slopes of the centerline and the tangential auxiliary line parallel to the ground on the cut surface of the reducer are correlated, and this correlation can be pre-calibrated.

[0094] In another embodiment of this application, generating guide lines on the display screen may include:

[0095] Upon receiving the third generation mode instruction, the pose of the second target tube is obtained;

[0096] Based on the pose of the second target tube, determine the central axis of the second target tube, the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening of the second target tube.

[0097] Specifically, when the driver selects the weak guidance mode, the display shows two auxiliary lines: the centerline of the reducer and a tangential auxiliary line parallel to the ground on the cut surface of the reducer. When the driver selects the normal guidance mode, the controller receives a third generation mode command. In the third generation mode, the controller first acquires the poses of the first and second target pipes, namely the target concrete pipe and the reducer. The pose refers to the position and orientation of the target concrete pipe and the reducer on the image. Based on the pose of the target concrete pipe, the center point of the cut surface of the target concrete pipe can be determined. Based on the pose of the reducer, the slope and intercept of the centerline of the reducer and the tangential auxiliary line parallel to the ground on the cut surface of the reducer can be determined. The slopes of the centerline and the tangential auxiliary line parallel to the ground on the cut surface of the reducer are correlated, and this correlation can be pre-calibrated.

[0098] Figure 4 A schematic block diagram of a controller according to an embodiment of this application is shown. Figure 4 As shown in the figure, this application provides a controller that may include:

[0099] Memory 410 is configured to store instructions; and

[0100] Processor 420 is configured to retrieve instructions from memory 410 and, when executing instructions, to implement the aforementioned methods for assisting control.

[0101] Specifically, in this embodiment of the application, the processor 420 can be configured to:

[0102] Obtain the current video frame from the image acquisition device;

[0103] Determine whether the first target tube is contained in the displayed screen of the current video frame;

[0104] If the display screen contains the first target tube, adjust the focus of the image acquisition device to the target focus.

[0105] When the focal length of the image acquisition device is the target focal length, auxiliary lines for the tube are generated on the display screen.

[0106] Furthermore, the processor 420 can also be configured as follows:

[0107] Determine whether the area of ​​the cross-section of the first target tube opening in the display image of the short-focus camera is greater than a first threshold.

[0108] If the area of ​​the cross-section of the first target tube opening in the display screen of the short-focus camera is greater than the first threshold, the short-focus camera is switched to a long-focus camera so that the focal length of the image acquisition device is the first target focal length.

[0109] Determine whether the area of ​​the cross-section of the first target tube opening in the display image of the telephoto camera is less than the second threshold.

[0110] If the area of ​​the cross-section of the first target tube opening in the display screen of the telephoto camera is less than the second threshold, the telephoto camera is switched to a short-focus camera so that the focal length of the image acquisition device is the second target focal length.

[0111] Furthermore, the processor 420 can also be configured as follows:

[0112] Obtain the diameter of the opening of the first target tube;

[0113] The target focal length is determined based on the diameter of the first target tube opening;

[0114] Adjust the focal length of the image acquisition device to the third target focal length based on the target focal length.

[0115] Furthermore, the processor 420 can also be configured as follows:

[0116] The original unit pixel value of the opening of the first target tube is determined based on the imaging resolution of the image acquisition device and the diameter of the opening of the first target tube.

[0117] The target unit pixel value of the opening of the first target tube is determined based on the opening diameter of the first target tube and the target pixel value of the opening diameter of the first target tube.

[0118] The target focal length of the image acquisition device is determined based on the original unit pixel value, the target unit pixel value, and the original focal length of the image acquisition device.

[0119] Furthermore, the processor 420 can also be configured as follows:

[0120] Upon receiving the first generation mode instruction, the poses of the first target tube and the second target tube are obtained;

[0121] Determine the slope and intercept of the central axis of the first target tube based on its pose;

[0122] Based on the pose of the second target tube, determine the central axis of the second target tube, the tangential auxiliary line perpendicular to the central axis on the tube opening section of the target tube, and the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening section of the second target tube.

[0123] Furthermore, the processor 420 can also be configured as follows:

[0124] Upon receiving the second generation mode instruction, the poses of the first target tube and the second target tube are obtained;

[0125] Determine the center point of the cross-section of the first target tube's opening based on the pose of the first target tube;

[0126] Based on the pose of the second target tube, determine the central axis of the second target tube, the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening of the second target tube.

[0127] Furthermore, the processor 420 can also be configured as follows:

[0128] Upon receiving the third generation mode instruction, the pose of the second target tube is obtained;

[0129] Based on the pose of the second target tube, determine the central axis of the second target tube, the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening of the second target tube.

[0130] The above technical solution first acquires the current video frame of the image acquisition device, then determines whether the displayed screen of the current video frame contains the first target tube. If the first target tube is found to be present in the displayed screen, the focal length of the image acquisition device is adjusted to the target focal length. Finally, with the focal length of the image acquisition device at the target focal length, alignment guide lines are generated on the displayed screen. This application can adaptively switch the camera focal length according to the position of the target tube. Furthermore, by adding alignment guide lines, it is easier for the driver to determine the relative position of the two tubes and whether the current movement trajectory can align the tubes, thereby achieving high-precision visual assistance in the alignment process at different distances, improving the accuracy and efficiency of high-precision alignment.

[0131] Figure 5 This schematic diagram illustrates a structural diagram of a vision-assisted control system according to an embodiment of this application. Figure 5 As shown in the embodiments of this application, a vision-assisted control system is also provided, which may include:

[0132] Image acquisition device 1 is configured to acquire images; and

[0133] According to the aforementioned controller 2, it communicates with the image acquisition device.

[0134] In this embodiment, the image acquisition device 1 acquires the alignment image, automatically locates the position of the target concrete pipe on the construction site based on traditional visual features and deep learning algorithms, and adaptively switches the focal length of the image acquisition device according to the position of the target concrete pipe via the controller 2. Furthermore, by adding alignment auxiliary lines, the driver can easily determine the relative position of the two pipes and whether the current movement trajectory can align the pipes, thereby achieving high-precision visual assistance in the alignment process at different distances, improving the accuracy and efficiency of high-precision alignment.

[0135] This application also provides a machine-readable storage medium storing instructions that cause a machine to perform the above-described method for assisting in the control process.

[0136] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0142] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0144] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for assisting in pipe alignment, characterized in that, A method applied to a vision-assisted tracking system, the vision-assisted tracking system including an image acquisition device and a controller, wherein the image acquisition device communicates with the controller, the method comprising: Obtain the current video frame of the image acquisition device; Determine whether the display screen of the current video frame contains the first target pipe, wherein the first target pipe is the target concrete pipe; If it is determined that the first target tube is present in the displayed image, the focal length of the image acquisition device is adjusted to the target focal length. When the focal length of the image acquisition device is the target focal length, auxiliary lines for the tube are generated on the display screen. When the image acquisition device includes a telephoto camera and a short-focus camera, adjusting the focal length of the image acquisition device to the target focal length includes: Determine whether the area of ​​the cross-section of the opening of the first target tube in the display image of the short-focus camera is greater than a first threshold. If it is determined that the area of ​​the cross-section of the first target tube opening in the display screen of the short-focus camera is greater than the first threshold, the short-focus camera is controlled to switch to the long-focus camera so that the focal length of the image acquisition device is the first target focal length. Determine whether the area of ​​the cross-section of the opening of the first target tube in the display screen of the telephoto camera is less than a second threshold. If it is determined that the area of ​​the cross-section of the first target tube opening in the display screen of the telephoto camera is less than the second threshold, the telephoto camera is controlled to switch to the short-focus camera so that the focal length of the image acquisition device is the second target focal length. The step of generating auxiliary lines for the tube on the display screen includes: Receive the generation mode instruction for the duct auxiliary line, and generate the duct auxiliary line according to the duct auxiliary line generation mode corresponding to the generation mode instruction. The generation mode instruction includes a first generation mode instruction, a second generation mode instruction and a third generation mode instruction. The duct auxiliary line generation modes corresponding to each generation mode instruction are strong boot mode, normal boot mode and weak boot mode, respectively. Generating the pipe alignment auxiliary line according to the pipe alignment auxiliary line generation mode corresponding to the generation mode instruction includes: When the pipe alignment auxiliary line generation mode is the strong guidance mode, the pipe alignment auxiliary line includes the central axis of the target concrete pipe, the central axis of the variable diameter pipe, a tangential auxiliary line perpendicular to the central axis on the cross-section of the variable diameter pipe opening, and a tangential auxiliary line parallel to the ground on the cross-section of the variable diameter pipe opening. When the pipe alignment auxiliary line generation mode is the normal guide mode, the pipe alignment auxiliary line and auxiliary point are generated on the display screen. The pipe alignment auxiliary line includes the central axis of the variable diameter pipe and a tangential auxiliary line parallel to the ground on the pipe opening cross-section of the variable diameter pipe. The auxiliary point is the center point of the pipe opening cross-section of the target concrete pipe. When the pipe alignment auxiliary line generation mode is the weak guidance mode, the pipe alignment auxiliary line includes the central axis of the variable diameter pipe and a tangential auxiliary line parallel to the ground on the pipe opening cross-section of the variable diameter pipe.

2. The method according to claim 1, characterized in that, The image acquisition device includes a zoom camera, and adjusting the focal length of the image acquisition device to the target focal length includes: Obtain the diameter of the opening of the first target tube; The target focal length is determined based on the diameter of the opening of the first target tube; The focal length of the image acquisition device is adjusted to the third target focal length according to the target focal length.

3. The method according to claim 2, characterized in that, The step of determining the target focal length based on the diameter of the opening of the first target tube includes: The original unit pixel value of the opening of the first target tube is determined based on the imaging resolution of the image acquisition device and the diameter of the opening of the first target tube. The target unit pixel value of the opening of the first target tube is determined based on the opening diameter of the first target tube and the target pixel value of the opening diameter of the first target tube. The target focal length of the image acquisition device is determined based on the original unit pixel value, the target unit pixel value, and the original focal length of the image acquisition device.

4. The method according to claim 1, characterized in that, The step of generating auxiliary lines for the tube on the display screen includes: Upon receiving a first generation mode instruction, the poses of the first target tube and the second target tube are obtained, wherein the second target tube is a variable diameter tube; The slope and intercept of the central axis of the first target tube are determined based on the pose of the first target tube. Based on the pose of the second target tube, determine the central axis of the second target tube, the tangential auxiliary line perpendicular to the central axis on the tube opening section of the second target tube, and the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening section of the second target tube.

5. The method according to claim 1, characterized in that, The step of generating auxiliary lines for the tube on the display screen includes: Upon receiving a second generation mode instruction, the poses of the first target tube and the second target tube are obtained; The center point of the cross-section of the first target tube opening is determined based on the pose of the first target tube. Based on the pose of the second target tube, determine the central axis of the second target tube, the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening section of the second target tube.

6. The method according to claim 1, characterized in that, The step of generating auxiliary lines for the tube on the display screen includes: Upon receiving the third generation mode instruction, the pose of the second target tube is obtained; Based on the pose of the second target tube, determine the central axis of the second target tube, the slope and intercept of the tangential auxiliary line parallel to the ground on the tube opening section of the second target tube.

7. A controller, characterized in that, include: The memory is configured to store instructions; as well as A processor configured to retrieve the instructions from the memory and, when executing the instructions, to implement the method for assisting in the switching process according to any one of claims 1 to 6.

8. A vision-assisted telemetry system, characterized in that, include: An image acquisition device, configured to capture images; as well as The controller according to claim 7 communicates with the image acquisition device.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform the method for assisting in the connection of the tube according to any one of claims 1 to 6.

Images