A method, device, equipment and storage medium for locating steel bar binding points
Through the combination of a binocular camera and a depth estimation model, the steel bar binding points are automatically positioned, which solves the problem of low steel bar binding efficiency in the existing technology, and realizes efficient automation of steel bar binding.
Patent Information
- Application Number
- CN202210601589.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-30
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-05-30
AI Technical Summary
In the prior art, the efficiency of steel bar binding is low, which leads to workers repeatedly bending over long-term work, which is very labor-intensive, easily causing physical strain and low efficiency.
Using a binocular camera and depth estimation model, through image acquisition, contour extraction and parallax calculation, the steel bar binding points are automatically positioned to realize robotic steel bar binding operations.
The efficiency of steel bar binding is improved, and the labor intensity is reduced through automation, and the level of automation of the process is significantly improved.
Smart Images

Figure CN114862829B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of construction technology, and in particular to a method, device, equipment and storage medium for locating steel bar binding points. Background Art
[0002] Rebar tying is one of the important processes in reinforced concrete construction technology. Its working goal is to pre-tie the steel frame for reinforced concrete components such as structural columns and cantilever beams. According to the requirements of the component shape, multiple steel bars are arranged and tied in a horizontal and vertical cross-pattern to form a component grid, usually called a "rebar skeleton", which is convenient for restraining concrete and strengthening the integrity of the component.
[0003] When performing binding operations, the steel skeleton can be arranged according to the shape of the target component. Workers manually use binding hooks or automatic steel bar binding machines to bind the unbound intersections. Since there are many intersections that need to be bound, workers need to bend over repeatedly for a long time, which has a high labor intensity and can easily cause serious physical strain problems. In addition, the efficiency of manual binding is low. Summary of the invention
[0004] The purpose of the present invention is to provide a method, device, equipment and storage medium for locating steel bar binding points in view of the above-mentioned deficiencies in the prior art, so as to solve the problem of low efficiency of manual binding in the prior art.
[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
[0006] In a first aspect, an embodiment of the present application provides a method for locating a steel bar binding point, the method comprising:
[0007] Acquire an image captured by a preset camera in a binocular camera for a steel frame, wherein the steel frame is a structure pre-built with a plurality of steel bars;
[0008] Extracting at least one target image of an untied point from the acquired image;
[0009] Using a pre-set contour extraction algorithm, contour extraction is performed on the target image of each untied point to obtain the steel bar cross contour of each untied point;
[0010] Calculate the plane pixel coordinates of each untied point according to the plane pixel coordinates of the cross profile of the steel bars;
[0011] According to the acquired image, a pre-trained depth estimation model of the binocular camera is used for processing to obtain a disparity map corresponding to the acquired image, wherein the disparity map includes: a disparity value of each pixel in the acquired image, and the disparity value of each pixel is used to characterize the pixel deviation of each pixel acquired by two cameras in the binocular camera;
[0012] The spatial coordinates of the unbound points are calculated according to the plane pixel coordinates of each unbound point and the disparity map corresponding to the acquired image.
[0013] Optionally, before using a pre-set contour extraction algorithm to extract contours from the target image of each untied point to obtain the steel bar cross contour of each untied point, the method further comprises:
[0014] Performing grayscale processing on the target image to obtain a grayscale image;
[0015] Performing median filtering on the grayscale image to obtain a filtered image;
[0016] Binarizing the filtered image to obtain a binary image;
[0017] Performing an opening operation on the binary image to obtain a preprocessed image;
[0018] The contour extraction algorithm is a binary image contour extraction algorithm. The pre-contour extraction algorithm is used to extract the contour of the target image of each untied point to obtain the cross contour of the steel bars at each untied point, including:
[0019] The binary image contour extraction algorithm is used to extract the contour of the preprocessed image to obtain the steel bar cross contour of each untied point.
[0020] Optionally, the calculating the plane pixel coordinates of each untied point according to the plane pixel coordinates of the cross profile of the steel bars comprises:
[0021] According to the plane pixel coordinates of the steel bar intersection contour, a preset area threshold is used to determine the pixel coordinates of the corner points of multiple area contours in the steel bar intersection contour;
[0022] The plane pixel coordinates of each unbound point are determined according to the pixel coordinates of the corner points of the multiple area contours.
[0023] Optionally, the depth estimation model includes: an encoder and a decoder; before the acquired image is processed using a pre-trained depth estimation model of the binocular camera to obtain a disparity map corresponding to the acquired image, the method further includes:
[0024] Acquire a plurality of sample image pairs of the binocular camera, each sample image pair comprising: two images captured by two cameras of the binocular camera for the same target object;
[0025] Using the encoder to encode the image captured by the preset camera in each sample image to obtain two groups of sample image features;
[0026] Using the decoder to decode the two groups of sample image features to obtain two groups of sample disparity maps;
[0027] Reconstructing each sample image pair according to the two groups of sample disparity maps to obtain a reconstructed sample image pair;
[0028] Calculating a loss function value according to the reconstructed sample image pair and each sample image pair;
[0029] According to the loss function value, the model parameters of the depth estimation model are adjusted, and the model training is re-performed based on the multiple sample image pairs until a preset stop iteration condition is reached.
[0030] Optionally, the encoder is used to encode the image captured by the preset camera in each sample image to obtain two groups of sample image features, including:
[0031] The encoder is used to perform encoding processing at multiple scales on the image captured by the preset camera in each sample image to obtain the two groups of sample image features, wherein each group of sample image features includes: sample image features at the multiple scales;
[0032] The step of using the decoder to decode the two groups of sample image features to obtain two groups of sample disparity maps includes:
[0033] The decoder is used to decode the sample image features of the multiple scales to obtain the two groups of sample disparity maps, each group of sample disparity maps including: sample disparity maps of the multiple scales.
[0034] Optionally, the depth estimation model further includes: a path enhancement network, and before reconstructing each sample image pair according to the two groups of sample disparity maps to obtain the reconstructed sample image pair, the method further includes:
[0035] Using the path enhancement network to perform downsampling processing on the feature maps corresponding to the sample disparity maps of the multiple scales, and fusing the downsampling result of each scale, the feature map of the same scale, and the sample disparity map of the same scale to obtain an updated sample disparity map of each scale;
[0036] The step of reconstructing each sample image pair according to the two groups of sample disparity maps to obtain a reconstructed sample image pair includes:
[0037] Each sample image pair is reconstructed according to the updated sample disparity map of each scale to obtain a reconstructed sample image pair.
[0038] Optionally, the encoder comprises: residual convolution layers of the plurality of scales, each residual convolution layer of each scale comprises a plurality of residual structures, each residual structure comprises: a convolution layer and a lightweight attention mechanism;
[0039] The encoder is used to perform encoding processing at multiple scales on the image captured by the preset camera in each sample image to obtain the two groups of sample image features, including:
[0040] According to the residual convolution layers of the multiple scales, feature extraction is performed on the image captured by the preset camera to obtain the two groups of sample image features.
[0041] In a second aspect, an embodiment of the present application provides a steel bar binding point positioning device, comprising:
[0042] An acquisition module is used to acquire an image captured by a preset camera in a binocular camera for a steel frame, wherein the steel frame is a structure pre-built with a plurality of steel bars;
[0043] A first extraction module extracts a target image of at least one untied point from the collected image;
[0044] The second extraction module uses a pre-set contour extraction algorithm to perform contour extraction on the target image of each untied point to obtain the cross contour of the steel bars at each untied point;
[0045] A first calculation module calculates the plane pixel coordinates of each untied point according to the plane pixel coordinates of the cross profile of the steel bars;
[0046] A processing module, which processes the acquired image using a pre-trained depth estimation model of the binocular camera to obtain a disparity map corresponding to the acquired image, wherein the disparity map includes: a disparity value of each pixel in the acquired image, and the disparity value of each pixel is used to characterize the pixel deviation of each pixel acquired by two cameras in the binocular camera;
[0047] The second calculation module calculates the spatial coordinates of the unbound points according to the plane pixel coordinates of each unbound point and the disparity map corresponding to the collected image.
[0048] In a third aspect, an embodiment of the present application provides a computer device, comprising: a storage medium and a processor, wherein the storage medium stores a computer program executable by the processor, and when the processor executes the computer program, a method for locating a steel bar binding point according to the first aspect above is implemented.
[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a method for locating a steel bar binding point according to the first aspect is implemented.
[0050] Compared with the prior art, this application has the following beneficial effects:
[0051] The embodiments of the present application provide a method, device, equipment and storage medium for locating steel bar binding points. The method can obtain a captured image of a steel bar frame by a preset camera in a binocular camera, wherein the steel bar frame is a structure pre-built with multiple steel bars, extract a target image of at least one unbound point from the captured image, use a pre-set contour extraction algorithm to perform contour extraction on the target image of each unbound point, obtain the steel bar cross contour of each unbound point, calculate the plane pixel coordinates of each unbound point according to the plane pixel coordinates of the steel bar cross contour, and use a pre-trained depth estimation model of the binocular camera to process the captured image to obtain a disparity map corresponding to the captured image, wherein the disparity map includes: disparity values of each pixel point in the captured image, the disparity value of each pixel point is used to characterize the pixel deviation of each pixel point captured by two cameras in the binocular camera, and calculate the spatial coordinates of the unbound point according to the plane pixel coordinates of each unbound point and the disparity map corresponding to the captured image, so that the steel bar binding robot can perform automated steel bar binding work according to the spatial coordinates of the unbound point, thereby improving the efficiency of steel bar binding. The efficiency of steel bar binding is effectively improved by replacing manpower with the automation of the machine. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0053] Figure 1 A schematic diagram of a steel bar tying robot frame provided in an embodiment of the present application;
[0054] Figure 2 A schematic diagram of a binocular vision system provided in an embodiment of the present application;
[0055] Figure 3A schematic diagram of a process for locating a reinforcement binding point provided in an embodiment of the present application;
[0056] Figure 4 A schematic diagram of extracting a target image provided by an embodiment of the present application;
[0057] Figure 5 A schematic diagram of extracting a cross profile of steel bars provided in an embodiment of the present application;
[0058] Figure 6 A schematic diagram of the spatial coordinates of an untied point provided in an embodiment of the present application;
[0059] Figure 7 A schematic diagram of a process flow of a target image preprocessing method provided in an embodiment of the present application;
[0060] Figure 8 A schematic diagram of a flow chart of a method for calculating the plane pixel coordinates of an unbound point provided in an embodiment of the present application;
[0061] Fig. 9 A schematic diagram of the plane pixel coordinates of an unbound point provided in an embodiment of the present application;
[0062] Fig.10 A flowchart of a method for training a depth estimation model provided in an embodiment of the present application;
[0063] Fig.11 A schematic diagram of a depth estimation Monodepth model provided in an embodiment of the present application;
[0064] Fig.12 A schematic diagram of a depth estimation model provided in an embodiment of the present application;
[0065] Fig.13 A schematic diagram of feature fusion provided in an embodiment of the present application;
[0066] Fig.14 A schematic diagram of a residual convolution layer provided in an embodiment of the present application;
[0067] Fig.15 A schematic diagram of a lightweight attention mechanism provided in an embodiment of the present application;
[0068] Fig.16 A schematic diagram of a steel bar binding point positioning device provided in an embodiment of the present application;
[0069] Fig.17 A schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0071] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.
[0072] Rebar tying is one of the important processes in reinforced concrete construction technology. In order to improve the efficiency of rebar tying, rebar tying can be performed by a rebar tying robot. Before the rebar tying robot performs rebar tying, it is necessary to locate the untied points of the rebar and obtain the position information of the untied points. At this time, the rebar tying robot can perform tying work on the untied rebars corresponding to the position information. Compared with manual tying of rebars, the rebar tying robot and the position information of the untied points can realize the automated operation of rebar tying, greatly improving the efficiency of rebar tying.
[0073] In the solution of the present application, steel bar tying can be achieved by a steel bar tying robot. The embodiment of the present application provides a frame of a steel bar tying robot to achieve tying of untied steel bars. Figure 1 A schematic diagram of a steel bar tying robot frame provided in an embodiment of the present application, such as Figure 1 As shown, the rebar tying robot includes two parts: a robot vision system and a robot system. First, the image of the rebar tying point is captured by the binocular camera of the robot vision system, and then the rebar tying point is located by a computer device. Subsequently, the computer device controls the robot arm to move to the untied point position, and finally the tying module completes the rebar tying.
[0074] The robot vision system can use the vision system to accurately and quickly identify and locate the steel bar binding points, which can provide better support for subsequent robot movement and binding work, so it is very important to choose a suitable vision system. The binocular vision system used in the embodiment of the present application can use a binocular camera to collect images, with rich image information and good stereo matching effect. The three-dimensional coordinates of the target object can be directly calculated through binocular stereo vision technology, with high positioning accuracy and strong anti-interference to light intensity.
[0075] Figure 2 A schematic diagram of a binocular vision system provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the hardware equipment of the binocular vision system consists of a binocular camera composed of two CMOS cameras and a fixed bracket. The binocular camera is fixed on the aluminum profile and fixed above the steel bar binding platform by hanging. The camera is adjusted to a position parallel to the steel bar binding platform. For example, the CMOS camera can be a Shoujue USB camera, which has the advantages of low cost and simple installation. Its resolution is 1280*720, focal length is 4.9mm, and refresh rate is 30FPS. Of course, it can also be other CMOS cameras, which are not specifically limited in the embodiments of this application.
[0076] Optionally, considering the range of activities of the laboratory robot, in order to avoid collision between the robot movement and the binocular camera, the binocular camera can be fixed 80 cm above the lashing platform. The effective acquisition range of the lashing plane is 114.7 cm*70.9 cm, which can effectively cover the range of activities of the robot.
[0077] The robot system consists of a robot module and a lashing module. The robot module is responsible for controlling the robot to move to the specified position, and the lashing module is responsible for lashing the lashing points.
[0078] The robot module includes the robot body, control box and computer equipment. The robot body can move to the specified position and maintain the relative posture according to the computer program. The control box is connected to the computer equipment through the USB data cable interface. The interface number is input into the operating software of the computer equipment to complete the connection with the computer equipment. The control box is connected to the robot body through the control line.
[0079] Optionally, the robot body used in the embodiment of the present application may be an AR3 six-degree-of-freedom robot, which is lightweight, easy to install, simple to operate, and low in price. Of course, it may also be other robots, and there is no specific limitation in the embodiment of the present application.
[0080] The binding module includes an automatic steel bar binding machine, a pneumatic clamp, an air pump, etc. The automatic steel bar binding machine is fixed to the end of the mechanical arm, and the pneumatic clamp is installed at its button. The double holes on the right are used to connect the air pump, and the air pump is controlled by air pressure to realize the opening and closing of the clamp. When the end of the mechanical arm moves to the specified position, that is, the position of the unbound point of the steel bar, the computer equipment issues a command to pressurize the air pump and drive the clamp to close, thereby controlling the automatic steel bar binding machine to perform the steel bar binding work and complete the binding of the unbound steel bar points. By replacing manpower with the automation of the machine, the efficiency of steel bar binding can be effectively improved.
[0081] In order to realize the positioning of the steel bar binding points, the steel bar binding robot can perform automatic steel bar binding according to the positioning information of the unbound points, thereby improving the efficiency of steel bar binding. In the scheme of the present application, a steel bar binding point positioning method is provided, which can obtain the positioning of the unbound points, that is, the position information.
[0082] The following is an explanation of a method for locating steel bar binding points provided in an embodiment of the present application through a specific example. Figure 3 A schematic diagram of a process for locating a steel bar binding point provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes:
[0083] S301, obtaining an image captured by a preset camera in a binocular camera with respect to a steel bar frame.
[0084] The steel frame is a structure pre-built using multiple steel bars.
[0085] The binocular camera may include a left view camera and a right view camera. In the embodiment of the present application, the position information of the untied points of the steel bars may be determined based on the left view of the steel bar frame taken by the left view camera.
[0086] Optionally, camera calibration can be performed before the camera captures the image. Camera calibration affects the accuracy of object measurement and is essential in the process of obtaining three-dimensional information of the object space. Binocular camera calibration is the process of calculating the relationship parameters between each camera and the cameras, where the camera intrinsic parameters depend on its own internal components, and the camera extrinsic parameters depend on the position between the two cameras. In the embodiment of the present application, by making a calibration plate, changing its position and angle under the camera, establishing the relationship between the known corner points and the corner points in the image, the camera intrinsic parameters and the camera extrinsic parameters are solved. Of course, other calibration methods can also be used, such as standard camera calibration method, camera active calibration method, camera self-calibration method, etc., which are not specifically limited in the embodiment of the present application.
[0087] The calibrated binocular camera is used to collect the image pair of the steel bar binding point, and the camera distortion correction is performed to obtain the image collected by the binocular camera for the steel bar frame. Among them, camera distortion can be called lens distortion. In practical applications, due to the limitations of the lens manufacturing process and the influence of the optical lens' own characteristics on imaging, the target object and the camera image are no longer simply scaled down in proportion, but there is a functional relationship between the field of view angle. This function is a nonlinear change, which is called lens distortion. Lens distortion will cause distortion in the photo. Therefore, when calibrating the camera, correction measures need to be taken to improve the accuracy of camera shooting.
[0088] Camera distortion can be divided into three types: radial distortion, tangential distortion, and thin prism distortion. The correction expression of camera distortion is shown in the following formula (1):
[0089]
[0090] Among them, (x l ,y l ) represents the distorted coordinates, (x, y) represents the ideal coordinates, k1, k2, k3 represent the radial distortion parameters, r represents the distance from the spatial point image coordinates to the origin, and r can be expressed as In the image coordinate system, the above formula (1) can be used to dedistort each image point, obtain the ideal image coordinates, and complete the camera distortion correction.
[0091] S302: extracting a target image of at least one unbound point from the acquired image.
[0092] The YOLOv4 algorithm can be used to obtain the position of the unbound point. The position is represented by the four values of left, top, right, and bottom of the collected image coordinate axes to represent the position of the target box, that is, the distance between the target box and the four boundaries of the original image. Figure 4 A schematic diagram of extracting a target image provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the four edges of the target image without binding points are used as extraction edges, the upper left corner coordinates (left, top) of the target image in the acquired image are used as the origin, and the target image is extracted by using the image cropping method to obtain the target image.
[0093] S303, using a pre-set contour extraction algorithm, extracting the contour of the target image of each untied point to obtain the cross contour of the steel bars at each untied point.
[0094] In an embodiment of the present application, the pre-contour extraction algorithm is the FindContours algorithm, which assumes that an image is composed of multiple pixel points (i, j), and uses f(i, j) to represent the grayscale value of the pixel point. Pixels with f(i, j) of 0 and 1 represent 0 pixels and 1 pixels, respectively.
[0095] The implementation steps of the FindContours algorithm are:
[0096] (1) Assume that the input image is F = {f(i,j)}, set the first boundary, and use the raster scanning method (i.e., scan line by line from left to right). If the gray value f(i,j)≠0 of a certain pixel point (i,j) is scanned, then execute step (2).
[0097] (2) If a certain pixel point (i, j) is scanned and meets the preset outer boundary starting point condition, then the pixel point is the outer boundary starting point, and f(i2, j2) = f(i, j-1).
[0098] (3) With pixel (i, j) as the center, set (i2, j2) as the starting point, and use the existence of non-zero pixels as the judgment condition to search in the (i, j) neighborhood in a clockwise direction. If it exists, record the first non-zero pixel as (i1, j1), and let f(i2, j2) = f(i1, j1), f(i3, j3) = f(i, j).
[0099] (4) With pixel (i3, j3) as the center and the next point of (i2, j2) as the starting point, search in the neighborhood of (i3, j3) in a counterclockwise direction based on whether there is a non-zero pixel. If there is, record the first non-zero pixel as (i4, j4).
[0100] (5) If f(i4,j4) = f(i,j) and f(i3,j3) = f(i1,j1), proceed to step (6); otherwise, let f(i3,j3) = f(i2,j2), f(i4,j4) = f(i3,j3), and return to step (4).
[0101] (6) If f(i, j)≠1, use the grating to scan from the next pixel point (i, j+1) until the pixel point in the lower right corner of the image is scanned.
[0102] Figure 5 A schematic diagram of extracting the cross profile of steel bars provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the FindContours algorithm extracts the contour of the steel bar intersection at each untied point according to the above algorithm steps and outputs it in the form of multiple plane pixel coordinates.
[0103] S304, calculating the plane pixel coordinates of each untied point according to the plane pixel coordinates of the cross profile of the steel bars.
[0104] The plane pixel coordinates of each untied point can be obtained by calculation through multiple plane pixel coordinates in the steel bar cross profile of the untied point.
[0105] S305 , processing the collected image using a pre-trained depth estimation model of a binocular camera to obtain a disparity map corresponding to the collected image.
[0106] The image captured by the preset camera in the binocular camera for the steel frame is used as the input of the pre-trained depth estimation model of the binocular camera, and the output result of the model is the disparity map corresponding to the captured image.
[0107] The disparity map corresponding to the captured image includes: the disparity value of each pixel in the captured image, and the disparity value of each pixel is used to characterize the pixel deviation of each pixel captured by two cameras in the binocular camera.
[0108] S306, calculating the spatial coordinates of the unbound points according to the plane pixel coordinates of each unbound point and the disparity map corresponding to the acquired image.
[0109] According to the plane pixel coordinates of each unbound point, the disparity value corresponding to the plane pixel coordinates of the unbound point is queried in the disparity map corresponding to the captured image, and the spatial coordinates of the unbound point can be calculated based on the disparity value.
[0110] Figure 6 A schematic diagram of the spatial coordinates of an untied point provided in an embodiment of the present application, such as Figure 6 As shown in the figure, when a certain point in space is photographed using binocular cameras placed at different positions on the same horizontal plane, there is a distance deviation between the projection points of the left and right camera imaging planes in the x-axis direction. This deviation is called parallax. Binocular ranging is to obtain parallax by matching the corrected feature points of the left and right views based on the stereo matching algorithm, and to measure the depth of the object in the spatial coordinate system using the principle of triangle similarity. After calibration, two cameras of the same model are in the same plane, f represents the focal length of the camera, P (X, Y, Z) is the spatial coordinate of the unbound point, O l , O r are the optical centers of the left and right cameras, Z l , Z r are the optical axes of the left and right cameras respectively, and B is the two optical centers O l and O r The distance between l (x l ,y l ), p r (x r ,y r ) is the projection point of point P on the left and right imaging planes.
[0111] According to the triangle similarity principle, the horizontal coordinate x of the projection point of the space point P on the two imaging planes is l and x r It can be expressed as the following formula (2) and the following formula (3):
[0112]
[0113]
[0114] The parallax value d of the projection points on the two imaging planes can be expressed as the following formula (4):
[0115]
[0116] Then the depth value Z of the spatial point P from the camera can be expressed as the following formula (5):
[0117]
[0118] After calculating the depth Z, the X and Y axis coordinates of the spatial point P can be solved according to the following formula (6):
[0119]
[0120] The relationship between the X-axis and Y-axis coordinates of the spatial point P and the pixel coordinates is shown in the following formula (7):
[0121]
[0122] Among them, u o 、v o are the horizontal and vertical coordinates of the origin of the image coordinate system in the pixel coordinate system, respectively, x 、f y are the focal lengths in the x and y axis directions respectively.
[0123] The spatial coordinates of the untied point P (X, Y, Z) can be calculated according to the above formula, where the origin of the spatial coordinates of P (X, Y, Z) is the position of the optical center of the left camera. The rebar tying robot can perform automated rebar tying based on the spatial coordinate information of the untied point.
[0124] In summary, the embodiment of the present application provides a method for locating steel bar binding points, which obtains the captured image of the steel bar frame by a preset camera in the binocular camera, extracts at least one target image of an unbound point from the captured image, and uses a pre-set contour extraction algorithm to perform contour extraction on the target image of each unbound point to obtain the steel bar cross contour of each unbound point, and calculates the plane pixel coordinates of each unbound point according to the plane pixel coordinates of the steel bar cross contour. According to the captured image, a pre-trained depth estimation model of the binocular camera is used for processing to obtain a disparity map corresponding to the captured image, and the spatial coordinates of the unbound point are calculated according to the plane pixel coordinates of each unbound point and the disparity map corresponding to the captured image, so that the steel bar binding robot can perform automated steel bar binding work according to the spatial coordinates of the unbound point, thereby improving the efficiency of steel bar binding, and effectively improving the efficiency of steel bar binding by replacing manpower with machine automation.
[0125] Furthermore, in the above Figure 3 Based on the method for locating a steel bar binding point shown in the figure, the embodiment of the present application also provides a method for preprocessing a target image. Optionally, Figure 7 A schematic diagram of a process flow of a target image preprocessing method provided in an embodiment of the present application is shown in FIG. Figure 7As shown, before the contour extraction is performed on the target image of each untied point, the target image of each untied point needs to be preprocessed. Before the above method S303, that is, the pre-contour extraction algorithm is used to extract the contour of the target image of each untied point, and before the steel bar cross contour of each untied point is obtained, the following is also included:
[0126] S701, grayscale the target image to obtain a grayscale image.
[0127] When a camera collects images, it is difficult to avoid extracting redundant feature information. At the same time, noise will be generated due to its own structural conditions and electromagnetic interference of internal components. Therefore, image preprocessing is required to eliminate redundant features and noise interference.
[0128] Image grayscale refers to the method of resolving the values of the three color components of RGB using mathematical operations. This application adopts the weighted average method, assigning different weight coefficients according to the human eye's perception of the three color components, and adding the products. The calculation formula of the weighted average method is shown in the following formula (8):
[0129] g=α*R+β*G+γ*B Formula (8)
[0130] Among them, g represents the gray value of the pixel, α, β, and γ are weight coefficients of the three color components, and the sum of α, β, and γ is 1. R, G, and B are specific values of the three components, all in the range of 0 to 255.
[0131] S702, performing median filtering on the grayscale image to obtain a filtered image.
[0132] The median filter sets the filter window to traverse the image pixels, arranges all the pixels in the window according to their grayscale values, and selects the grayscale median as the grayscale value of the pixel at the center of the window.
[0133] The expression of the median filter is shown in the following formula (9):
[0134] g(x,y)=Med{f(xk,yl)|(k,l∈W)} Formula (9)
[0135] Among them, g(x,y) and f(x,y) are the grayscale values of the output and input images respectively, and W represents the filter window.
[0136] S703, binarizing the filtered image to obtain a binarized image.
[0137] Binarization sets a grayscale value threshold, sets grayscale values below the threshold to 0, otherwise to 255, and normalizes the transformed grayscale values to reduce the processing burden. Depending on whether the threshold is fixed, the binarization method is divided into fixed threshold binarization and adaptive threshold binarization. The former has the advantages of simple operation and high efficiency, but cannot display local features; while the latter can display local features, but the calculation is cumbersome and the running speed is slow. This application only needs to extract the contours of the untied points of the steel bars, and does not require too much detailed information. Therefore, a fixed threshold binarization method is adopted, and the calculation method is shown in the following formula (10):
[0138]
[0139] Among them, g(x,y) represents the gray value after binarization, f(x,y) is the gray value of the pixel, and T is the set threshold.
[0140] S704, performing an opening operation on the binary image to obtain a preprocessed image.
[0141] The opening operation uses the operation method of first erosion operation and then dilation operation, which is one of the basic methods of image morphological processing. Erosion refers to cutting off and removing pixels close to the pixels covered by the target according to the selected geometric structure; dilation connects the surrounding pixels with the pixels covered by the target. The opening operation can remove isolated points and unnecessary burrs, but it will have different operation effects due to the influence of structural elements.
[0142] In an embodiment of the present application, the pre-contour extraction algorithm, a binary image contour extraction algorithm, may be a FindContours algorithm, which uses the binary image contour extraction algorithm to perform contour extraction on the obtained preprocessed image to obtain the cross contour of the steel bars at each untied point.
[0143] The embodiment of the present application provides a method for preprocessing a target image, which can eliminate redundant features and noise interference in the target image by graying the target image to obtain a grayscale image, performing median filtering on the grayscale image to obtain a filtered image, binarizing the filtered image to obtain a binarized image, and performing an opening operation on the binarized image to obtain a preprocessed image.
[0144] Furthermore, in the above Figure 3 Based on the method for locating a steel bar binding point shown in the figure, the embodiment of the present application also provides a method for calculating the plane pixel coordinates of an unbound point. Optionally, Figure 8 A schematic diagram of a flow chart of a method for calculating the plane pixel coordinates of an unbound point provided in an embodiment of the present application, such as Figure 8As shown, the plane pixel coordinates of the untied points can be calculated according to the pixel coordinates of the corner points of multiple areas in the cross profile of the steel bars. The above method S304 calculates the plane pixel coordinates of each untied point according to the plane pixel coordinates of the cross profile of the steel bars, including:
[0145] S801, according to the plane pixel coordinates of the steel bar cross profile, using a preset area threshold, determining the pixel coordinates of the corner points of multiple area contours in the steel bar cross profile.
[0146] For regular quadrilateral objects, the FindContours contour extraction algorithm uses the coordinates (x, y) of the four corner points to represent the quadrilateral contour. However, threaded steel bars are cross objects with irregular edges, such as Figure 5 As shown, the contour extraction function will obtain four irregular contours composed of the intersection of the steel bars and the four edges of the image. At the same time, the irregular edges will cause each contour to have multiple contour corner points, which is not conducive to the subsequent determination of the steel bar binding points. The present application uses a contour corner point selection method to screen the contour corner points closest to the intersection of the steel bars. First, the FindContours algorithm is used to extract four quadrilateral contours with irregular edges. Each contour consists of discrete contour corner points around four corners, that is, the plane pixel coordinates of the steel bar intersection contour. According to the following formulas (11) to (14), the discrete contour corner points around the corners closest to the steel bar intersection in each contour can be selected.
[0147] Top left outline:
[0148]
[0149] Top right outline:
[0150]
[0151] Bottom left outline:
[0152]
[0153] Bottom right outline:
[0154]
[0155] Among them, (x i ,y i ) is the contour corner point of each contour, δ is the area threshold, and the values of i and j are 0, 1, 2, etc.
[0156] Since the intersection of steel bars cannot strictly maintain a vertical angle in the actual environment, this application first uses the y coordinate as the screening condition in the algorithm to select the contour corner points within the regional threshold, and then uses the x coordinate as the screening condition for screening. If the x coordinate is used as the screening condition first, when the steel bars cross obliquely, the x coordinates of the contour corner points of some contours close to the intersection of the steel bars cannot guarantee to meet the screening condition, resulting in the contour corner points being screened out and causing a large error.
[0157] Substitute the corner points of each contour after the above screening into the following formula (15) to obtain the mean points of the discrete contour corner points around the corresponding corners of the steel bar intersection, that is, the pixel coordinates of the corner points of the contours of multiple areas in the steel bar intersection contour.
[0158]
[0159] Among them, (x i ,y i ) is the coordinate of the contour corner point close to the intersection of the steel bars in each contour after screening, and N is the number of contour corner points in the area that meets the x and y thresholds.
[0160] S802, determining the plane pixel coordinates of each unbound point according to the pixel coordinates of the corner points of the multiple area contours.
[0161] Fig. 9 A schematic diagram of the plane pixel coordinates of an unbound point provided in an embodiment of the present application, such as Fig. 9 As shown, the pixel coordinates of the corner points of multiple area contours are averaged to obtain the plane pixel coordinates of each unbound point.
[0162] An embodiment of the present application provides a method for calculating the plane pixel coordinates of untied points. According to the plane pixel coordinates of the cross contour of the steel bars, a preset area threshold is used to determine the pixel coordinates of the corner points of multiple area contours in the cross contour of the steel bars. According to the pixel coordinates of the corner points of the multiple area contours, the plane pixel coordinates of each untied point are determined. Then, the disparity value of each untied point can be determined through the plane pixel coordinates of each untied point, and the spatial coordinates of the untied point are calculated.
[0163] Furthermore, in the above Figure 3 Based on the method for locating a steel bar binding point shown in the embodiment of the present application, a method for training a depth estimation model is also provided. Optionally, Fig.10 A flowchart of a method for training a depth estimation model provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the depth estimation model includes: an encoder and a decoder. Before the above method S305, that is, using the pre-trained depth estimation model of the binocular camera for processing and obtaining the disparity map corresponding to the captured image, it also includes:
[0164] S1001, obtaining multiple sample image pairs of a binocular camera.
[0165] Each sample image pair includes: two images captured by two cameras in the binocular camera for the same target object.
[0166] S1002, using an encoder to encode the image captured by a preset camera in each sample image to obtain two groups of sample image features.
[0167] In an embodiment of the present application, the depth estimation model adopts the network framework of the depth estimation Monodepth algorithm, which is divided into two processes: encoding and decoding. The depth estimation model includes: an encoder and a decoder. In the encoding process, ResNet50 is used as the backbone feature extraction network. The depth estimation Monodepth algorithm adopts an unsupervised learning method and uses a convolutional neural network to achieve end-to-end depth estimation. During training, the left and right views obtained by the binocular camera are used instead of the depth map as the real background depth label of the network, which solves the problem of difficulty in collecting depth maps.
[0168] S1003: Use a decoder to decode the two groups of sample image features to obtain two groups of sample disparity maps.
[0169] During the decoding process, bilinear interpolation (Bilinear Sampling) and up-sampling (Up-Sampling) followed by convolution are used to amplify information.
[0170] S1004: reconstruct each sample image pair according to the two groups of sample disparity maps to obtain a reconstructed sample image pair.
[0171] Fig.11 A schematic diagram of a depth estimation Monodepth model provided in an embodiment of the present application, such as Fig.11 As shown, the image captured by the preset camera in each sample image is input to obtain two groups of sample disparity maps, including a group of left disparity maps based on the left view and a group of right disparity maps based on the right view. The image captured by the preset camera is processed with the group of right disparity maps to obtain a reconstructed group of right views. The image captured by another camera other than the preset camera is processed with the group of left disparity maps to obtain a reconstructed group of left views. The reconstructed sample image pairs include a reconstructed group of right views and a reconstructed group of left views.
[0172] S1005, calculating a loss function value according to the reconstructed sample image pair and each sample image pair.
[0173] The loss function can be used to obtain the loss value between the reconstructed view and the original view, thereby backpropagating the training network and converting the depth prediction problem into a regression problem.
[0174] In the embodiment of the present application, the loss function of a right view in the reconstructed set of right views and the corresponding left view in the reconstructed set of left views is shown in the following formula (16):
[0175]
[0176] in, and Represents the appearance matching loss function (AppearanceMatching Loss) of the reconstructed left view and right view, and Represents the difference smoothness loss function (Disparity Smoothness Loss) between the reconstructed left view and the right view, and Represents the left-right disparity consistency loss function (Left-Right Disparity Consistency Loss) of the reconstructed left and right views.
[0177] The appearance matching loss function of the reconstructed left view can be expressed as the following formula (17):
[0178]
[0179] They represent the captured left view and the reconstructed left view respectively, and α can be 0.85.
[0180] The different smoothness loss function of the reconstructed left view can be expressed as the following formula (18):
[0181]
[0182] in, and They represent the gradient of the left disparity map on the x-axis respectively.
[0183] Since the discontinuity of image depth often occurs in the gradient of the image, a weight term is used to make the disparity gradient larger as the image gradient increases, so as to obtain a globally smooth depth map with clear boundaries. In this algorithm, gradient penalties are applied to both sets of disparity maps.
[0184] The left and right disparity consistency loss function of the reconstructed left view can be expressed as the following formula (19):
[0185]
[0186] in, and Represent the left and right disparity maps respectively.
[0187] Since only the left view is used as the input of the model during training, but the output is the disparity of the left and right views, adding this loss function can make the two disparity maps as similar as possible, thus ensuring the consistency of the left and right disparity.
[0188] S1006, adjusting the model parameters of the depth estimation model according to the loss function value, and retraining the model based on multiple sample image pairs until a preset stop iteration condition is reached.
[0189] A method for training a depth estimation model provided in an embodiment of the present application obtains multiple sample image pairs of a binocular camera, each sample image pair includes: two images captured by two cameras in the binocular camera for the same target object, an encoder is used to encode the image captured by a preset camera in each sample image to obtain two groups of sample image features, a decoder is used to decode the two groups of sample image features to obtain two groups of sample disparity maps, each sample image pair is reconstructed according to the two groups of sample disparity maps to obtain a reconstructed sample image pair, a loss function value is calculated according to the reconstructed sample image pair and each sample image pair, model parameters of a depth estimation model are adjusted according to the loss function value, and the model is retrained based on multiple sample image pairs until a preset stop iteration condition is reached, and then the captured image can be processed by the trained depth estimation model of the binocular camera to obtain a disparity map corresponding to the captured image.
[0190] Furthermore, in the above Fig.10 Based on the method for training a depth estimation model shown, an embodiment of the present application also provides another method for training a depth estimation model. Optionally, the above method S1002 uses an encoder to encode the image captured by the preset camera in each sample image to obtain two sets of sample image features, including: using an encoder to encode the image captured by the preset camera in each sample image at multiple scales to obtain two sets of sample image features.
[0191] Each set of sample image features includes: sample image features of multiple scales.
[0192] During the encoding process, ResNet50 is used as the backbone feature extraction network. The size of the original image is halved each time it passes through the residual convolution layer resblock combination, so as to obtain multiple feature maps of different scales, that is, sample image features, until the size of the feature map is reduced to 1 / 64 of the original image.
[0193] The method S1003, using a decoder to decode two groups of sample image features to obtain two groups of sample disparity maps, includes: using a decoder to decode sample image features of multiple scales to obtain two groups of sample disparity maps.
[0194] Each set of sample disparity maps includes: sample disparity maps of multiple scales.
[0195] The feature map obtained in the encoding process is upsampled to expand its size, and then concatenated with some feature maps in the decoding process through residual connection to obtain the original size and Figure 1 / 8, 1 / 4, 1 / 2 and 4 feature maps of the same size as the original image, and then pass through a convolution layer with a convolution kernel of 3*3 to generate four disparity maps of different sizes. Among them, when splicing the last three feature maps of the decoding process separately, the disparity map obtained in the upper layer needs to be added.
[0196] The input of the depth estimation model is the left view. In the embodiment of the present application, the output is a left disparity map of the same size as the original image, and the spatial positioning of the unbound points is achieved through the left disparity map.
[0197] Furthermore, in the above Fig.10 Based on the method for training a depth estimation model shown, an embodiment of the present application also provides another method for training a depth estimation model. Fig.12 A schematic diagram of a depth estimation model provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, optionally, the depth estimation model also includes: a path aggregation network (PAN). Before the above method S1004, that is, according to the two groups of sample disparity maps, each sample image pair is reconstructed to obtain the reconstructed sample image pair, it also includes: using the path enhancement network to perform downsampling processing on the feature maps corresponding to the sample disparity maps of multiple scales at multiple scales, and fusing the downsampling results of each scale, the feature maps of the same scale, and the sample disparity maps of the same scale to obtain updated sample disparity maps of each scale.
[0198] In order to solve the problems of insufficient feature extraction capability, gradient disappearance, and large positioning error of the Monodepth algorithm, the previous disparity map upsampling structure is retained during the decoding process to extract more features for the disparity map. The PAN structure is added, and Concatenation is used instead of Addition as the feature fusion method to fuse shallow information and deep information. While improving the feature extraction capability of the network model, the ability to fuse high and low layer features is enhanced.
[0199] The standard Monodepth network adopts a top-down structure in the decoding process. Although it enhances the semantic features, a large amount of shallow positioning information is lost after multiple transmissions, which also leads to a decrease in the network's positioning ability. Therefore, this application integrates the Bottom-up Path Augmentation strategy in the PAN structure with the decoding part and adds a bottom-up feature pyramid structure. Fig.13A schematic diagram of feature fusion provided in an embodiment of the present application is shown in FIG. Fig.13 As shown in the figure, when downsampling, the feature maps are first adjusted by convolution with a convolution kernel size of 3*3, and then spliced with the upsampled information of the same level in the next layer connected horizontally, and then the information is integrated through the convolution layer to obtain the lower-level feature map. The deep feature map incorporates more strong positioning features, improves the positioning ability at multiple scales, and thus improves the utilization rate of the underlying information.
[0200] The method S1004, reconstructing each sample image pair according to the two groups of sample disparity maps to obtain a reconstructed sample image pair, includes: reconstructing each sample image pair according to the updated sample disparity map of each scale to obtain a reconstructed sample image pair.
[0201] A depth estimation model provided by an embodiment of the present application, the depth estimation model also includes: a path enhancement network, reconstructing each sample image pair according to two groups of sample disparity maps, before obtaining the reconstructed sample image pair, the method also includes: using the path enhancement network to perform multiple scale downsampling processing on the feature maps corresponding to the sample disparity maps of multiple scales, and fusing the downsampling results of each scale, the feature maps of the same scale, and the sample disparity maps of the same scale to obtain an updated sample disparity map of each scale, reconstructing each sample image pair according to the two groups of sample disparity maps to obtain a reconstructed sample image pair, including: reconstructing each sample image pair according to the updated sample disparity map of each scale to obtain a reconstructed sample image pair, while improving the feature extraction capability of the network model, enhancing the ability to fuse high and low layer features, and improving the accuracy of positioning.
[0202] Furthermore, in the above Fig.12 Based on the depth estimation model shown, an embodiment of the present application further provides an encoding method. Optionally, the encoder includes: residual convolution layers of multiple scales, each residual convolution layer of each scale includes multiple residual structures, and each residual structure includes: a convolution layer and a lightweight attention mechanism.
[0203] The use of residual modules is usually determined by the data scale and the depth of the residual network used. When the data scale is small and the residual network used is shallow, such as Resnet18 and Resnet34, a residual module consisting of two convolutional layers is usually used. When the data scale is large and the residual network used is deep, such as Resnet50, Resnet101, and Resnet152, a three-layer residual module is used. Since the data scale of the steel bar binding points used in this paper is large, the model is complex, and considering the computational cost of the network, this paper uses the Resnet50 network model in the encoding part. The residual convolution layer resblock of the model consists of a resconv residual structure that incorporates the CBAM (Convolutional Block Attention Module) lightweight attention mechanism.
[0204] Fig.14 A schematic diagram of a residual convolution layer provided in an embodiment of the present application is shown in FIG. Fig.14 As shown in the figure, in the residual structure resconv, the input feature map is first reduced in dimension by 1*1 convolution, then processed by 3*3 convolution, and then increased in dimension by 1*1 convolution. The key features are extracted by CBAM to generate a new feature map. Finally, the input result is added to the result of the residual structure, and the result is output after the batch normalization layer (Batch Normalization, BN) and activation function (elu). Adding CBAM can not only improve the network's ability to extract features, but also prevent the gradient from disappearing.
[0205] CBAM is a simple and efficient lightweight attention mechanism that uses two attention modules, channel and space, to extract key features more efficiently. The channel attention module assigns weights to channels with more key features, and the spatial attention module locates areas with richer key features, thereby improving the ability to extract key information in the network. Fig.15 A schematic diagram of a lightweight attention mechanism provided in an embodiment of the present application is shown as follows: Fig.15 As shown in the figure, after the feature map F is input into the network, it first undergoes global maximum pooling and average pooling, its size will be compressed, and the feature information will be integrated to generate two compressed feature maps. The two-layer perceptron (Multi-Layer Perceptron, MLP) performs a dimension transformation operation on the compressed feature map; secondly, the output feature is processed by the addition operation and activated by sigmoid; finally, the channel attention weight Mc is obtained, and the calculation formula is shown in the following formula (20):
[0206]
[0207] Among them, σ is the sigmoid activation function, W0 and W1 are the weights in the multi-layer perceptron MLP, and Represent the average pooling and maximum pooling features respectively.
[0208] After the feature map F passes through the channel attention module, a new feature map F′ is generated. After global average pooling and maximum pooling are performed according to the channel, the feature map is concatenated and convolved to output the spatial attention weighted coefficient M. s , and combine the new feature F′ with M s Multiply them together to generate the final feature map F″, as shown in the following formula (21):
[0209]
[0210] An encoder is used to perform encoding processing at multiple scales on the image captured by the preset camera in each sample image to obtain two sets of sample image features, including: extracting features from the image captured by the preset camera according to convolutional layers at multiple scales to obtain two sets of sample image features.
[0211] An encoding method provided in an embodiment of the present application includes an encoder comprising: residual convolution layers of multiple scales, each residual convolution layer of each scale includes multiple residual structures, each residual structure includes: a convolution layer and a lightweight attention mechanism, and an encoder is used to perform encoding processing at multiple scales on an image captured by a preset camera in each sample image to obtain two groups of sample image features, including: according to the residual convolution layers of multiple scales, feature extraction is performed on the image captured by the preset camera to obtain two groups of sample image features. Through the residual convolution layer of the lightweight attention mechanism, not only the network's ability to extract features can be improved, but also the gradient disappearance is prevented, thereby improving the accuracy of positioning.
[0212] By using the path enhancement network and the residual convolution layer of the lightweight attention mechanism, the number of network parameters is reduced, which solves the problems of insufficient feature extraction capability and low utilization of shallow information, and further improves the accuracy of positioning.
[0213] The present application provides a method for locating steel bar binding points. In image positioning, the target image of the unbound point is firstly extracted and preprocessed to eliminate noise interference and better extract the outline of the steel bar intersection. Secondly, the steel bar outline is extracted by a contour extraction algorithm. The contour corner point selection method is used to screen out the contour corner points closest to the steel bar intersection position, which is convenient for the image positioning of the steel bar binding point. In spatial positioning, an improved depth estimation model is provided. First, the CBAM attention mechanism is added to the decoding part to improve the feature extraction capability. Then, the PAN structure is introduced in the decoding part to enhance the utilization rate of shallow positioning features, further enhance the feature extraction capability, and improve the positioning accuracy of the steel bar binding point.
[0214] The following is a description of a steel bar binding point positioning device provided by the present application for execution. Its specific implementation process and technical effects are mentioned above and will not be repeated below.
[0215] Fig.16 A schematic diagram of a steel bar binding point positioning device provided in an embodiment of the present application, such as Fig.15 As shown, the steel bar binding point positioning device comprises:
[0216] The acquisition module 1601 is used to acquire an image captured by a preset camera in a binocular camera with respect to a steel frame, wherein the steel frame is a structure pre-built with a plurality of steel bars.
[0217] The first extraction module 1602 is configured to extract at least one target image of an unbound point from the collected image.
[0218] The second extraction module 1603 is used to use a pre-set contour extraction algorithm to perform contour extraction on the target image of each untied point to obtain the steel bar cross contour of each untied point.
[0219] The first calculation module 1604 is used to calculate the plane pixel coordinates of each untied point according to the plane pixel coordinates of the cross profile of the steel bars.
[0220] The processing module 1605 is used to process the acquired image using a pre-trained depth estimation model of the binocular camera to obtain a disparity map corresponding to the acquired image, wherein the disparity map includes: a disparity value of each pixel in the acquired image, and the disparity value of each pixel is used to characterize the pixel deviation of each pixel acquired by two cameras in the binocular camera.
[0221] The second calculation module 1606 is used to calculate the spatial coordinates of the unbound points according to the plane pixel coordinates of each unbound point and the disparity map corresponding to the collected image.
[0222] Optionally, the second extraction module 1603 is specifically used to grayscale the target image to obtain a grayscale image; perform median filtering on the grayscale image to obtain a filtered image; perform binarization on the filtered image to obtain a binary image; perform an open operation on the binary image to obtain a preprocessed image; the contour extraction algorithm is a binary image contour extraction algorithm, and a pre-contour extraction algorithm is used to perform contour extraction on the target image of each unbound point to obtain the cross contour of the steel bars at each unbound point, including: using a binary image contour extraction algorithm to perform contour extraction on the preprocessed image to obtain the cross contour of the steel bars at each unbound point.
[0223] Optionally, the first calculation module 1604 is specifically used to determine the pixel coordinates of the corner points of multiple area contours in the steel bar cross contour based on the plane pixel coordinates of the steel bar cross contour and using a preset area threshold; and determine the plane pixel coordinates of each untied point based on the pixel coordinates of the corner points of multiple area contours.
[0224] Optionally, the processing module 1605 is specifically used to obtain multiple sample image pairs of the binocular camera, each sample image pair including: two images captured by two cameras in the binocular camera for the same target object; using an encoder to encode the image captured by a preset camera in each sample image to obtain two sets of sample image features; using a decoder to decode the two sets of sample image features to obtain two sets of sample disparity maps; reconstructing each sample image pair according to the two sets of sample disparity maps to obtain a reconstructed sample image pair; calculating a loss function value according to the reconstructed sample image pair and each sample image pair; adjusting the model parameters of the depth estimation model according to the loss function value, and re-training the model based on multiple sample image pairs until a preset stop iteration condition is reached.
[0225] Optionally, the processing module 1605 is specifically used to use an encoder to perform encoding processing at multiple scales on the image captured by a preset camera in each sample image to obtain two groups of sample image features, wherein each group of sample image features includes: sample image features at multiple scales; use a decoder to decode the two groups of sample image features to obtain two groups of sample disparity maps, including: use a decoder to decode the sample image features at multiple scales to obtain two groups of sample disparity maps, each group of sample disparity maps includes: sample disparity maps at multiple scales.
[0226] Optionally, the processing module 1605 is specifically used to use a path enhancement network to perform downsampling processing on the feature maps corresponding to the sample disparity maps of multiple scales at multiple scales, and fuse the downsampling results of each scale, the feature maps of the same scale, and the sample disparity maps of the same scale to obtain an updated sample disparity map of each scale; reconstruct each sample image pair according to the two groups of sample disparity maps to obtain a reconstructed sample image pair, including: reconstructing each sample image pair according to the updated sample disparity map of each scale to obtain a reconstructed sample image pair.
[0227] Optionally, the processing module 1605 is specifically configured to perform feature extraction on an image captured by a preset camera according to residual convolution layers of multiple scales to obtain two groups of sample image features.
[0228] Fig.17 A schematic diagram of a computer device provided in an embodiment of the present application, wherein the computer device may be a computing device or a server with computing processing capabilities.
[0229] The computer device includes: a processor 1701 , a storage medium 1702 , and a bus 1703 . The processor 1701 and the storage medium 1702 are connected via the bus 1703 .
[0230] The storage medium 1702 is used to store programs, and the processor 1701 calls the programs stored in the storage medium 1702 to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.
[0231] Optionally, the present invention also provides a program product, such as a computer-readable storage medium, comprising a program, which is used to execute the above method embodiment when executed by a processor.
[0232] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.
[0233] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., which can store program codes.
[0234] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for locating a steel bar binding point, characterized in that: include: Acquire an image captured by a preset camera in a binocular camera for a steel frame, wherein the steel frame is a structure pre-built with a plurality of steel bars; Extracting at least one target image of an untied point from the acquired image; Using a pre-set contour extraction algorithm, contour extraction is performed on the target image of each untied point to obtain the steel bar cross contour of each untied point; Calculate the plane pixel coordinates of each untied point according to the plane pixel coordinates of the cross profile of the steel bars; According to the acquired image, a pre-trained depth estimation model of the binocular camera is used for processing to obtain a disparity map corresponding to the acquired image, wherein the disparity map includes: a disparity value of each pixel in the acquired image, and the disparity value of each pixel is used to characterize the pixel deviation of each pixel acquired by two cameras in the binocular camera; The spatial coordinates of the unbound points are calculated according to the plane pixel coordinates of each unbound point and the disparity map corresponding to the acquired image.
2. The method for locating the steel bar binding point according to claim 1, characterized in that: Before extracting the contour of the target image of each untied point by using a pre-set contour extraction algorithm to obtain the steel bar cross contour of each untied point, the method further comprises: Performing grayscale processing on the target image to obtain a grayscale image; Performing median filtering on the grayscale image to obtain a filtered image; Binarizing the filtered image to obtain a binary image; Performing an opening operation on the binary image to obtain a preprocessed image; The contour extraction algorithm is a binary image contour extraction algorithm. The pre-contour extraction algorithm is used to extract the contour of the target image of each untied point to obtain the cross contour of the steel bars at each untied point, including: The binary image contour extraction algorithm is used to extract the contour of the preprocessed image to obtain the steel bar cross contour of each untied point.
3. The method for locating steel bar binding points according to claim 1, characterized in that: The step of calculating the plane pixel coordinates of each untied point according to the plane pixel coordinates of the cross profile of the steel bars comprises: According to the plane pixel coordinates of the steel bar intersection contour, a preset area threshold is used to determine the pixel coordinates of the corner points of multiple area contours in the steel bar intersection contour; The plane pixel coordinates of each unbound point are determined according to the pixel coordinates of the corner points of the multiple area contours.
4. The method for locating steel bar binding points according to claim 1, characterized in that: The depth estimation model includes: an encoder and a decoder; before the acquired image is processed using the pre-trained depth estimation model of the binocular camera to obtain a disparity map corresponding to the acquired image, the method further includes: Acquire a plurality of sample image pairs of the binocular camera, each sample image pair comprising: two images captured by two cameras of the binocular camera for the same target object; Using the encoder to encode the image captured by the preset camera in each sample image to obtain two groups of sample image features; Using the decoder to decode the two groups of sample image features to obtain two groups of sample disparity maps; Reconstructing each sample image pair according to the two groups of sample disparity maps to obtain a reconstructed sample image pair; Calculating a loss function value according to the reconstructed sample image pair and each sample image pair; According to the loss function value, the model parameters of the depth estimation model are adjusted, and the model training is re-performed based on the multiple sample image pairs until a preset stop iteration condition is reached.
5. The method for locating steel bar binding points according to claim 4, characterized in that: The encoder is used to encode the image captured by the preset camera in each sample image to obtain two sets of sample image features, including: The encoder is used to perform encoding processing at multiple scales on the image captured by the preset camera in each sample image to obtain the two groups of sample image features, wherein each group of sample image features includes: sample image features at the multiple scales; The step of using the decoder to decode the two groups of sample image features to obtain two groups of sample disparity maps includes: The decoder is used to decode the sample image features of the multiple scales to obtain the two groups of sample disparity maps, each group of sample disparity maps including: sample disparity maps of the multiple scales.
6. The method for locating a reinforcement binding point according to claim 5, characterized in that: The depth estimation model further includes: a path enhancement network. Before reconstructing each sample image pair according to the two groups of sample disparity maps to obtain the reconstructed sample image pair, the method further includes: Using the path enhancement network to perform downsampling processing on the feature maps corresponding to the sample disparity maps of the multiple scales, and fusing the downsampling result of each scale, the feature map of the same scale, and the sample disparity map of the same scale to obtain an updated sample disparity map of each scale; The step of reconstructing each sample image pair according to the two groups of sample disparity maps to obtain a reconstructed sample image pair includes: Each sample image pair is reconstructed according to the updated sample disparity map of each scale to obtain a reconstructed sample image pair.
7. The method for locating a steel bar binding point according to claim 5, characterized in that: The encoder comprises: residual convolution layers of the plurality of scales, each residual convolution layer of each scale comprises a plurality of residual structures, each residual structure comprises: a convolution layer and a lightweight attention mechanism; The encoder is used to perform encoding processing at multiple scales on the image captured by the preset camera in each sample image to obtain the two groups of sample image features, including: According to the residual convolution layers of the multiple scales, feature extraction is performed on the image captured by the preset camera to obtain the two groups of sample image features.
8. A steel bar binding point positioning device, characterized in that: include: An acquisition module is used to acquire an image captured by a preset camera in a binocular camera for a steel frame, wherein the steel frame is a structure pre-built with a plurality of steel bars; A first extraction module extracts a target image of at least one untied point from the collected image; The second extraction module uses a pre-set contour extraction algorithm to perform contour extraction on the target image of each untied point to obtain the cross contour of the steel bars at each untied point; A first calculation module calculates the plane pixel coordinates of each untied point according to the plane pixel coordinates of the cross profile of the steel bars; A processing module, which processes the acquired image using a pre-trained depth estimation model of the binocular camera to obtain a disparity map corresponding to the acquired image, wherein the disparity map includes: a disparity value of each pixel in the acquired image, and the disparity value of each pixel is used to characterize the pixel deviation of each pixel acquired by two cameras in the binocular camera; The second calculation module calculates the spatial coordinates of the unbound points according to the plane pixel coordinates of each unbound point and the disparity map corresponding to the collected image.
9. A computer device, characterized in that: include: A storage medium and a processor, wherein the storage medium stores a computer program executable by the processor, and when the processor executes the computer program, the method for locating a steel bar binding point according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is read and executed, the method for locating the steel bar binding point described in any one of claims 1 to 7 is implemented.
