Tank car filling recognition and positioning method and device based on binocular camera, electronic equipment and computer readable storage medium

Abstract

The invention belongs to the technical field of visual positioning, and particularly relates to a tank truck filling recognition and positioning method and device based on a binocular camera and a computer readable storage medium. Comprising the steps of directly projecting point cloud information in a 3d camera depth map to an rgb camera picture to obtain corresponding pixel points according to a joint calibration result of a 3d camera and an rgb camera based on physical characteristics of the 3d camera and the rgb camera, so that pixel point matching of the 3d camera picture and the rgb camera picture is realized, and a good result is obtained in practical application. The problems that in the prior art, two hardware devices of a laser radar and a binocular camera are used, the hardware cost is high, and one-to-many or many-to-many pixel matching is carried out are solved.

Description

Technical Field

[0001] The present invention belongs to the field of visual positioning technology, and more specifically, relates to a tank truck filling identification and positioning method, device, electronic equipment and computer-readable storage medium based on a binocular camera. Background Art

[0002] Commonly used object recognition technologies are generally divided into feature-based recognition methods and neural network-based recognition methods. Especially in the tank truck loading scene in the outdoor scene of the chemical industry, in order to ensure safe production and improve recognition accuracy, neural network models are used for object recognition. Camera calibration technology includes monocular camera calibration and binocular calibration technology; among them, after the binocular camera is calibrated, the pixel points in the imaging screen of the two cameras can be matched through the "world coordinate system-camera coordinate system-image coordinate system-pixel coordinate system" and the spatial coordinate transformation between the two cameras.

[0003] Chinese invention patent CN112598729A discloses a target object recognition and positioning method that integrates laser and camera. It uses two hardware devices, laser radar and binocular camera. The laser radar is used to obtain the scanning point cloud. At the same time, the binocular camera is started to collect images and the image coordinates of the center of the target object are calculated using a neural network model. The image coordinates of the center of the target object are then converted into spatial coordinates in the world coordinate system and the distance of the center of the target object from the radar and the plane where the camera is located is calculated. In addition to the binocular camera, the laser radar is also used to obtain the spatial coordinates of the center of the target object and the distance of the center of the target object from the laser and the plane where the camera is located through the point cloud image. Finally, the two are weighted to obtain the final positioning information. This invention provides a target object recognition and positioning method that integrates laser and camera, which is mainly used in the fields of mobile grasping robots and warehousing and logistics, improves the accuracy of target positioning, and evaluates samples and calculation results.

[0004] In summary, although the existing technology has improved the accuracy of target positioning, it faces the following problems: it uses two hardware devices, lidar and binocular camera, at the same time, which has high hardware cost, long calculation time and great deployment difficulty; the content does not state the working distance and usage conditions of binocular camera and radar, and whether it is suitable for long-distance object recognition and positioning in outdoor scenes remains to be verified; there is a lack of specific methods on how to use lidar to obtain the center of the target object through point cloud images, and the accuracy and necessity of lidar in outdoor scenes remain to be verified. Summary of the invention

[0005] The present invention mainly solves the problem of positioning objects during tanker loading in outdoor scenes in the chemical industry, including a filling port (tanker) and a crane pipe (infusion device). An RGB camera and a 3D camera are used to identify objects on a color screen (RGB camera output) and use a pixel matching algorithm to obtain the point cloud coordinates xyz of the same object on a depth map (3D camera output). The RGB camera and the 3D camera are imaging cameras of different brands, types, functions and resolutions, and the 3D camera can output a depth map and a grayscale map.

[0006] The detailed technical scheme of the present invention is as follows:

[0007] S1. Set the binocular camera to a specified height, wherein the binocular camera includes an RGB camera and a 3D camera, and ensure that the positions of the two cameras remain unchanged and that the imaging images of the two cameras both display the complete crane pipe and the filling port;

[0008] S2. Use Zhang Zhengyou calibration method to calibrate the binocular camera, and use the chessboard calibration plate. The binocular camera calibration uses Zhang Zhengyou calibration method. When shooting the calibration plate, ensure that the calibration plate appears in both pictures at the same time and the pictures are clear. Perform multiple multi-angle and multi-distance shooting. The number of photos taken is between 30 and 50, which can ensure the calibration accuracy and accurate recognition results.

[0009] S3. After the camera is calibrated, the target detection Yolo algorithm is used to identify the filling port and the crane pipe, and the position of the filling port in the RGB image is obtained and recorded as area A, and the position of the crane pipe in the RGB image is recorded as area B;

[0010] S4. When a filling port or a crane pipe appears on the RGB screen, all valid point cloud values ​​in the 3D screen at this time are obtained and set as set X, and the corresponding pixel coordinates are recorded;

[0011] S5. Consider the elements in the point cloud value set X as world coordinates. The RGB image resolution is greater than the 3D camera resolution. The elements of the crane and the filling port in the point cloud value set X must have corresponding pixels in the RGB image. Removing the point cloud value matching in the non-critical area of ​​the RGB image does not affect the final result and reduces the time consumption of the algorithm to a certain extent.

[0012] S6. Transform each element in the point cloud value set X to the perspective of the RGB camera using the rotation angle and translation parameter between the RGB camera and the 3D camera, and obtain the world coordinate set Y of the RGB camera.

[0013] S7, according to the calibration result of the RGB camera, realize the conversion of the world coordinate system-camera coordinate system-image coordinate system-pixel coordinate system, and obtain the pixel coordinate set Z corresponding to the set Y;

[0014] S8. Traverse the set Z, find the pixel coordinates in area A and area B and obtain the corresponding point cloud values, average all the point cloud values ​​matched to area A and all the point cloud values ​​matched to area B, so as to obtain the point cloud values ​​of the target areas A and B, and realize the positioning of the irrigation port and the crane pipe.

[0015] Further, the S2 includes:

[0016] S21, using a standard aluminum substrate, the number of feature points is 11*8;

[0017] S22, take photos of the calibration plate at a distance of 40 cm, 50 cm, and 60 cm from the camera, and move the calibration plate horizontally each time a photo is taken, so that the calibration plate angle and camera imaging are different each time;

[0018] S23, repeat shooting N times, and obtain 2N pictures in total, of which N pictures are taken by the RGB camera and N pictures are taken by the 3D camera;

[0019] S24, extracting feature points from the collected images, and ensuring that the images from the two cameras are of the same scene during the extraction;

[0020] S25. Calibrate the binocular camera using the acquired image feature points, and obtain the intrinsic parameters, extrinsic parameters, and distortion coefficient of the RGB camera; the intrinsic parameters, extrinsic parameters, and distortion coefficient of the 3D camera; and the rotation angle and translation parameters between the RGB camera and the 3D camera through the calibration method.

[0021] Furthermore, the camera imaging formula is as follows:

[0022]

[0023] In formula (6), (X w ,Y w ,Z w ) is the world coordinate corresponding to the pixel point (u, v) on the screen; RT is the camera external parameter, which is the rotation matrix R and the translation matrix T, respectively. R represents the rotation matrix from the calibration plate position to the camera position, and T represents the translation matrix from the calibration plate position to the camera position. RT can determine the unique camera position relative to the calibration plate in space; Zc represents the depth information, which is the Z-axis coordinate of the point in the camera coordinate system when the point in the three-dimensional space is projected onto the two-dimensional image plane by the camera; f is the focal length of the camera; each pixel corresponds to a photosensitivity point, which is a rectangle with a physical size of width d x , high d y ; u0 and v0 represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system, and θ represents the angle between the horizontal and vertical edges of the camera's photosensitive plate (90° means no error).

[0024] Furthermore, the area of ​​the valid point cloud value is area A, area B and the colored area in the 3D image.

[0025] Furthermore, the key area is the area extending 10*10 from the complete crane pipe and the filling port edge, and the other areas are non-key areas.

[0026] First, the invalid point cloud value areas in all areas, that is, the areas with invalid point cloud values, are excluded, and the valid point cloud value areas, that is, the areas with valid point cloud values ​​are retained. Then, the key areas are retained from the valid point cloud value areas and the point cloud value matching of the non-key areas is excluded. The key areas are gradually obtained to increase the accuracy of the valid point cloud values.

[0027] Furthermore, the distortion coefficient includes a radial distortion coefficient and a tangential distortion coefficient:

[0028] Radial distortion is caused by the non-ideal characteristics of the lens, which causes the points on the edge of the image to deviate from where they should be when there is no distortion. It manifests itself as the circle on the edge of the image becoming an ellipse. The radial distortion formula is as follows:

[0029] r d =r(1+k1r 2 +k2r 4 +k3r 6 ) (1);

[0030] Where r is the radial distance from the center of the image to the pixel; k1k2k3 is the radial distortion coefficient; r d is the radial distance after radial distortion correction.

[0031] Tangential distortion is caused by the tilt of the camera lens relative to the imaging plane, which manifests itself as the bending of vertical or horizontal lines at the edge of the image. Tangential distortion is usually described by two parameters, denoted as p1 and p2. The tangential distortion formula is as follows:

[0032] x d =x+2p1xy+p2(r 2 +2x 2 ) d =y+2p2xy+p1(r 2 +2y 2 ) (2);

[0033] Among them, (x, y) is the original image coordinates, (x d ,y d ) is the image coordinate after tangential distortion correction, r 2 =x 2 +y 2 , p1 and p2 are tangential distortion coefficients.

[0034] Specifically, f, d t , u0, v0, θ are called camera intrinsic parameters; k1, k2, k3, p1, p2 are called distortion coefficients.

[0035] Furthermore, the transformation of the camera coordinate system includes:

[0036] (1) The rigid body transformation from the world coordinate system to the camera coordinate system includes the translation transformation T and the rotation transformation R. The corresponding homogeneous matrix composed of RT is multiplied by (X w ,Y w ,Z w ) of the homogeneous matrix:

[0037]

[0038] In formula (3), (X w ,Y w ,Z w ) is the world coordinate corresponding to the pixel point (u, v) on the screen; RT is the camera external parameter, which is the rotation matrix R and the translation matrix T respectively;

[0039] (2) Pinhole transformation from camera coordinate system to image coordinate system:

[0040]

[0041] In formula (4), f is the focal length of the camera;

[0042] (3) The image coordinate system and the pixel coordinate system are on the same plane, but the origins and measurement units of the two coordinate systems are different. The conversion from the image coordinate system to the pixel coordinate system is:

[0043]

[0044] In formula (5), each pixel corresponds to a photosensitivity point, which is a rectangle with a physical size of width d. x , high d y ; u0 and v0 represent the coordinates of the center of the camera photosensitive plate in the pixel coordinate system, and θ represents the angle between the horizontal and vertical edges of the camera photosensitive plate;

[0045] (4) The pixel coordinates in the pixel coordinate system obtained after the conversion are dedistorted according to the distortion parameters to obtain the undistorted pixel coordinates.

[0046] The present invention also includes a device for implementing a tank truck filling identification and positioning method based on a binocular camera, the device comprising:

[0047] A visual acquisition module, used to acquire input images;

[0048] A calibration module, used for calibrating a visual acquisition module, wherein the visual acquisition module includes a binocular camera;

[0049] A visual recognition module is used to identify the irrigation area and the crane area in the input image;

[0050] Coordinate calculation and conversion module, used for calculation and conversion between data in different coordinate systems;

[0051] Point cloud matching module, used to obtain the point cloud value of the binocular camera image and perform pixel matching between regions;

[0052] The visual positioning module is used to calculate the point cloud information of the irrigation area and the crane area in the input image.

[0053] An electronic device, comprising:

[0054] at least one processor; and

[0055] A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to execute the binocular camera-based tank truck filling identification and positioning method.

[0056] A machine-readable storage medium stores executable instructions, which, when executed, enable the machine to execute the binocular camera-based tank truck filling identification and positioning method.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The present invention proposes a tank truck filling identification and positioning method, device, electronic device and computer-readable storage medium based on a binocular camera. The method can be used for two imaging cameras of different specifications, has no requirements on the resolution of the camera, does not need to design the world coordinates in advance, is simple to use, and can achieve pixel coordinate matching after camera calibration, without considering one-to-many or many-to-many problems in pixel matching; the method is optimized based on camera calibration and can be used for two imaging cameras of different specifications (RGB camera and 3D camera), without the need to perform stereo correction, stereo matching, parallax estimation and other steps, thereby reducing calculation time and deployment difficulty. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flow chart of the tank truck filling identification and positioning method based on binocular camera described in the present invention.

[0060] Figure 2 This is the RGB camera output picture of the present invention.

[0061] Figure 3 It is the 3D camera output picture of the present invention. Specific embodiments

[0062] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0063] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0064] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0065] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0066] Example 1

[0067] This embodiment provides a tank truck filling identification and positioning method based on a binocular camera. Figure 2 , Figure 3 As shown, the binocular camera includes an RGB camera and a 3D camera. The two pictures are the same scene taken by the RGB camera and the 3D camera. Figure 2 The image is output by an RGB camera with a resolution of 1000*800 and slight distortion. Figure 3 The output image of the 3D camera has a resolution of 640*480. For easy recognition, the original grayscale image is colored. The specific steps are as follows: Figure 1 As shown, including:

[0068] S1. Set up a binocular camera and obtain the captured image;

[0069] Install the RGB camera and 3D camera to the specified height. After installation, ensure that their positions remain unchanged and that their images can display the complete crane pipe and filling port.

[0070] Preferably, if the vehicle height is 3.3m to 3.8m, the crane pipe base height is 4m, the crane pipe height is 1.4m, and the camera mounting height is 7m to 8m; after installation, the relative positions of the two must be kept unchanged, and the imaging images of the two can display the complete crane pipe and filling port.

[0071] S2, calibration of binocular camera;

[0072] Use Zhang Zhengyou calibration method to calibrate binocular cameras (including RGB cameras and 3D cameras), and use a checkerboard calibration plate to obtain the calibration results.

[0073] Preferably, S2 includes:

[0074] S21, using a standard aluminum substrate, the number of feature points is 11*8;

[0075] S22, take photos of the calibration plate at a distance of 40 cm, 50 cm, and 60 cm from the camera, and move the calibration plate horizontally each time a photo is taken, so that the calibration plate angle and camera imaging are different each time;

[0076] S23, repeat the shooting 40 times, and obtain 80 pictures in total, of which 40 are taken by the RGB camera and 40 are taken by the 3D camera;

[0077] S24, extracting feature points from the image collected in S23, and ensuring that the images of the two cameras are of the same scene during the extraction;

[0078] S25. Calibrate the binocular camera using the image feature points obtained in S24, and obtain the intrinsic parameters, extrinsic parameters, and distortion coefficient of the RGB camera; the intrinsic parameters, extrinsic parameters, and distortion coefficient of the 3D camera; and the rotation angle and translation parameters between the RGB camera and the 3D camera through the calibration method.

[0079] Furthermore, the camera imaging formula is as follows:

[0080]

[0081] In formula (6), (X w ,Y w ,Z w ) is the world coordinate corresponding to the pixel point (u, v) on the screen; RT is the camera external parameter, which is the rotation matrix R and the translation matrix T, respectively. R represents the rotation matrix from the calibration plate position to the camera position, and T represents the translation matrix from the calibration plate position to the camera position. RT can determine the unique camera position relative to the calibration plate in space; Zc represents the depth information, which is the Z-axis coordinate of the point in the camera coordinate system when the point in the three-dimensional space is projected onto the two-dimensional image plane by the camera; f is the focal length of the camera; each pixel corresponds to a photosensitivity point, which is a rectangle with a physical size of width d x , high d y ; u0 and v0 represent the coordinates of the center of the camera's photosensitive plate in the pixel coordinate system, and θ represents the angle between the horizontal and vertical edges of the camera's photosensitive plate (90° means no error).

[0082] Furthermore, the distortion coefficient includes a radial distortion coefficient and a tangential distortion coefficient:

[0083] Radial distortion is caused by the non-ideal characteristics of the lens, which causes the points on the edge of the image to deviate from where they should be when there is no distortion. It manifests itself as the circle on the edge of the image becoming an ellipse. The radial distortion formula is as follows:

[0084] r d =r(1+k1r 2 +k2r 4 +k3r 6 ) (1);

[0085] In formula (1), r is the radial distance from the image center to the pixel; k1k2k3 is the radial distortion coefficient; r d is the radial distance after radial distortion correction.

[0086] Tangential distortion is caused by the tilt of the camera lens relative to the imaging plane, which manifests itself as the bending of vertical or horizontal lines at the edge of the image. Tangential distortion is usually described by two parameters, denoted as p1p2. The tangential distortion formula is as follows:

[0087] x d =x+2p1xy+p2(r 2 +2x 2 ) y d =y+2p2xy+p1(r 2 +2y 2 ) (2);

[0088] In formula (2), (x, y) is the original image coordinate, (x d ,y d ) is the image coordinate after tangential distortion correction, r 2 =x 2 +y 2 .

[0089] Specifically, f, d y , u0, v0, θ are called camera intrinsic parameters; k1, k2, k3, p1, p2 are called distortion coefficients.

[0090] S3, identification of filling ports and crane pipes;

[0091] After the camera is calibrated, when a nozzle or a crane appears on the RGB screen, Figure 2 As shown, the target detection YOLO algorithm is used to identify the Guankou and the Heguan, and the position of the Guankou in the RGB image is obtained - the area in the red frame is recorded as area A, and the position of the Heguan in the RGB image - the area in the purple frame is recorded as area B.

[0092] S4, obtaining valid point cloud values ​​of the 3D image and forming a set X;

[0093] When a filling port or a crane pipe appears on the RGB screen, Figure 2 As shown; get all valid point cloud values ​​in the 3D image at this time and set them as set X, and record the corresponding pixel coordinates;

[0094] Specifically, Figure 3 for Figure 2 The corresponding colored 3D image, wherein the area of ​​valid point cloud values ​​can be the area after the erroneous point cloud is excluded from the 3D image, or can be directly identified by the recognition algorithm. In this embodiment, preferably: area A, area B and the colored area are the areas where the 3D camera can obtain point cloud values, which are called valid point cloud value areas; the black area in the 3D image except area A and area B is the location where the 3D camera cannot obtain point cloud values, which is called invalid point cloud value area.

[0095] Get Figure 3 All valid point cloud values ​​in are set as set X, and the corresponding pixel coordinates are recorded. The resolution of the 3D camera is 640*480, so the maximum number of elements in set X is 640*480. Affected by the lighting, the number of elements in set X is much smaller than 640*480.

[0096] S5, removing point cloud value matching of non-critical areas;

[0097] The elements in the point cloud value set X are regarded as world coordinates. Since the resolution of the RGB screen is 1000*800, and according to the requirements of S1, the imaging screens of both can display the complete crane pipe and the filling port, the elements (world coordinates) near the crane pipe and the filling port in the point cloud value set X must have corresponding pixels in the RGB screen.

[0098] Specifically, the distinction between key areas and non-key areas can be achieved by a recognition algorithm, such as Figure 2 As shown in the figure, the point cloud value matching of the non-critical area of ​​the RGB image is removed, which does not affect the final result and reduces the time consumption of the algorithm to a certain extent;

[0099] Among them, the key areas can be area A and area B in the 3D picture. In this embodiment, the key areas are preferably areas extending 10*10 from the edges of area A and area B, and other areas are non-key areas (areas other than the key areas), which can make the point cloud matching more accurate.

[0100] like Figure 3 As shown, the outlines of the crane pipe and the filling port can be clearly seen in the colored area, indicating that the main parts of the two can obtain corresponding point cloud values, ensuring the acquisition and matching of the point cloud values ​​of area A and area B.

[0101] S6. Calculate the world coordinate set Y of the RGB camera corresponding to the set X;

[0102] Each element in the point cloud value set X is transformed to the perspective of the RGB camera using the rotation angle and translation parameters between the RGB camera and the 3D camera, and the world coordinate set Y of the RGB camera is obtained.

[0103] S7, obtaining a pixel coordinate set Z corresponding to the set Y;

[0104] According to the calibration results of the RGB camera and the distortion coefficients involved in the camera, the transformation of the world coordinate system-camera coordinate system-image coordinate system-pixel coordinate system is realized to obtain the pixel coordinate set Z corresponding to the set Y.

[0105] Preferably, the transformation of the camera coordinate system includes:

[0106] (1) The transformation between the world coordinate system and the camera coordinate system is a rigid body transformation, including translation transformation T and rotation transformation R. The homogeneous matrix composed of RT is multiplied by (X w ,Y w ,Z w ) of the homogeneous matrix:

[0107]

[0108] In formula (3), (X w ,Y w ,Z w ) is the world coordinate corresponding to the pixel point (u, v) on the screen; RT is the camera external parameter, which are the rotation matrix R and the translation matrix T respectively.

[0109] (2) The transformation from camera coordinate system to image coordinate system is pinhole transformation:

[0110]

[0111] In formula (4), f is the focal length of the camera.

[0112] (3) The image coordinate system and the pixel coordinate system are on the same plane, but the origins and measurement units of the two coordinate systems are different. The conversion from the image coordinate system to the pixel coordinate system is:

[0113]

[0114] In formula (5), each pixel corresponds to a photosensitivity point, which is a rectangle with a physical size of width d. x , high d y ; u0 and v0 represent the coordinates of the center of the camera photosensitive plate in the pixel coordinate system, and θ represents the angle between the horizontal and vertical edges of the camera photosensitive plate.

[0115] (4) The pixel coordinates (u, v) obtained after the conversion in the pixel coordinate system are then dedistorted according to the distortion parameters to obtain the undistorted pixel coordinates (u'v').

[0116] S8, positioning of filling port and crane pipe;

[0117] Traverse the set Z, find the pixel coordinates in area A and area B and get the corresponding point cloud values, average all the point cloud values ​​matched to area A and all the point cloud values ​​matched to area B, and then get the point cloud values ​​of the target areas A and B to realize the positioning of the Guankou and the crane pipe.

[0118] Example 2

[0119] This embodiment provides a device for implementing a tank truck filling identification and positioning method based on a binocular camera, the device comprising:

[0120] A visual acquisition module, used to acquire input images;

[0121] A calibration module, used for calibrating a visual acquisition module, wherein the visual acquisition module includes a binocular camera;

[0122] A visual recognition module is used to identify the irrigation area and the crane area in the input image;

[0123] Coordinate calculation and conversion module, used for calculation and conversion between data in different coordinate systems;

[0124] Point cloud matching module, used to obtain the point cloud value of the binocular camera image and perform pixel matching between regions;

[0125] The visual positioning module is used to calculate the point cloud information of the irrigation area and the crane area in the input image.

[0126] Example 3

[0127] This embodiment also provides an electronic device, including:

[0128] at least one processor; and

[0129] A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to execute the tank truck filling identification and positioning method based on a binocular camera as described above.

[0130] In this embodiment, the electronic device may include, but is not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, and the like.

[0131] Example 4

[0132] This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the tank truck filling identification and positioning method based on a binocular camera as described above.

[0133] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0134] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0135] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0136] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0140] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A tank truck filling identification and positioning method based on binocular camera, characterized in that: include: S1. Set the binocular camera to a specified height, wherein the binocular camera includes an RGB camera and a 3D camera, and ensure that the positions of the two cameras remain unchanged and that the imaging images of the two cameras both display the complete crane pipe and the filling port; S2. Use Zhang Zhengyou calibration method to calibrate the binocular camera, and use the checkerboard calibration plate to obtain the calibration result; S3. After the camera is calibrated, when a filling port or a crane pipe appears on the RGB screen, the object detection YOLO algorithm is used to identify the filling port and the crane pipe, and the position of the filling port in the RGB screen is recorded as area A, and the position of the crane pipe in the RGB screen is recorded as area B; S4. When a filling port or a crane pipe appears on the RGB screen, all valid point cloud values ​​in the 3D screen at this time are obtained and set as set X, and the corresponding pixel coordinates are recorded; S5. Consider the elements in the point cloud value set X as world coordinates. The RGB screen resolution is greater than the 3D camera resolution. The elements near the crane pipe and the filling port in the point cloud value set X, that is, the world coordinates, must have corresponding pixels in the RGB screen, and remove the point cloud value matching in the non-critical area of ​​the RGB screen. S6. Transform each element in the point cloud value set X to the perspective of the RGB camera using the rotation angle and translation parameter between the RGB camera and the 3D camera, and obtain the world coordinate set Y of the RGB camera. S7, according to the calibration result of the RGB camera, realize the conversion of the world coordinate system-camera coordinate system-image coordinate system-pixel coordinate system, and obtain the pixel coordinate set Z corresponding to the set Y; S8. Traverse the set Z, find the pixel coordinates in area A and area B and obtain the corresponding point cloud values, average all the point cloud values ​​matched to area A and all the point cloud values ​​matched to area B, so as to obtain the average point cloud value of the target area A and B, and realize the positioning of the irrigation port and the crane pipe.

2. According to claim 1, a tank truck filling identification and positioning method based on a binocular camera is characterized in that: The S2 includes: S21, using a standard aluminum substrate, the number of feature points is 11*8; S22, take pictures with the calibration plate at a distance of 40 cm, 50 cm, and 60 cm from the camera, and move the calibration plate horizontally each time a picture is taken; S23, repeat shooting N times, and obtain 2N pictures in total, of which N pictures are taken by the RGB camera and N pictures are taken by the 3D camera; S24, extracting feature points from the collected images, and ensuring that the images from the two cameras are of the same scene during the extraction; S25. Calibrate the binocular camera using the acquired image feature points, and obtain the internal parameters, external parameters, and distortion coefficient of the RGB camera and the internal parameters, external parameters, and distortion coefficient of the 3D camera through the calibration method; and the rotation angle and translation parameters between the RGB camera and the 3D camera.

3. The method for identifying and locating tanker filling based on binocular camera according to claim 2 is characterized in that: The distortion coefficients include radial distortion coefficients and tangential distortion coefficients: The radial distortion formula is as follows: r d =r(1+k1r 2 +k2r 4 +k3r 6 ) (1); In formula (1), r is the radial distance from the image center to the pixel; k1, k2, k3 are radial distortion coefficients; rd is the radial distance after radial distortion correction; The tangential distortion formula is as follows: x d =x+2p1xy+p2(r 2 +2x 2 )y d =y+2p2xy+p1(r 2 +2y 2 )(2); In formula (2), (x, y) is the original image coordinate, (x d ,y d ) is the image coordinate after tangential distortion correction, r 2 =x 2 +y 2 , p1 and p2 are tangential distortion coefficients.

4. The method for identifying and locating tanker filling based on binocular camera according to claim 3 is characterized in that: The transformation of the camera coordinate system includes: (1) The transformation between the world coordinate system and the camera coordinate system is a rigid body transformation, including translation transformation T and rotation transformation R. The homogeneous matrix composed of RT is multiplied by (X w , Yw, Z w ) of the homogeneous matrix: In formula (3), (x w , Yw, Z w ) is the world coordinate corresponding to the pixel point (u, v) on the screen; RT is the camera external parameter, which is the rotation matrix R and the translation matrix T respectively; (2) The transformation from camera coordinate system to image coordinate system is pinhole transformation: In formula (4), f is the focal length of the camera; (3) The image coordinate system and the pixel coordinate system are on the same plane, but the origins and measurement units of the two coordinate systems are different. The conversion from the image coordinate system to the pixel coordinate system is: In formula (5), each pixel corresponds to a photosensitivity point, which is a rectangle with a physical size of width d. x , high d y ; u0 and v0 represent the coordinates of the center of the camera photosensitive plate in the pixel coordinate system, and θ represents the angle between the horizontal and vertical edges of the camera photosensitive plate; (4) The pixel coordinates in the pixel coordinate system obtained after the conversion are then dedistorted to obtain undistorted pixel coordinates.

5. The method for identifying and locating tanker filling based on binocular camera according to claim 1 is characterized in that: The areas of the effective point cloud values ​​are area A, area B and the colored area in the 3D image.

6. A tank truck filling identification and positioning method based on a binocular camera according to claim 1 or 5, characterized in that: The key area is the area extending 10*10 from the complete crane pipe and the edge of the filling port, and the other areas are non-key areas.

7. A device for realizing a tank truck filling identification and positioning method based on a binocular camera, characterized in that: The device comprises: A visual acquisition module, used to acquire input images; A calibration module, used for calibrating a visual acquisition module, wherein the visual acquisition module includes a binocular camera; A visual recognition module is used to identify the irrigation area and the crane area in the input image; Coordinate calculation and conversion module, used for calculation and conversion between data in different coordinate systems; The point cloud matching module is used to obtain the point cloud values ​​of the binocular camera image and perform pixel matching between regions. The visual positioning module is used to calculate the point cloud information of the irrigation area and the crane area in the input image.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to execute the tank truck filling identification and positioning method based on a binocular camera as described in any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores executable instructions, which, when executed, enable the machine to perform the tank truck filling identification and positioning method based on a binocular camera as described in any one of claims 1 to 6.

Images