Engineering machinery carrier positioning method based on visual identification and related equipment

Abstract

The invention discloses an engineering machinery carrier positioning method based on visual identification and related equipment, which can well solve the problem of positioning precision of remote control engineering machinery, can realize high-precision positioning of an engineering machinery carrier, and avoids influence on operation safety. The method comprises the following steps: acquiring an image set shot by an unmanned aerial vehicle, wherein the image set comprises at least one image of an engineering operation site; using a convolutional neural network model to judge whether the images in the image set contain a target object, wherein the target object is an engineering machinery carrier in the engineering operation site; if yes, generating a bounding box at a target object of the image; extracting pixel coordinates of the target object according to the bounding box; according to the pixel coordinates, the position of the unmanned aerial vehicle and parameters of the camera on the unmanned aerial vehicle, the distance between the camera on the unmanned aerial vehicle and the target object and the sight angle between the camera on the unmanned aerial vehicle and the target object are calculated.

Description

Technical Field

[0001] The present application relates to the field of remote control machinery technology, and in particular to a method for positioning an engineering machinery vehicle based on visual recognition and related equipment. Background Art

[0002] At the construction site, the positioning technology of remotely controlled construction machinery plays a great role in on-site operations. At present, the widely used positioning technology mainly relies on GPS or Beidou satellite navigation system. The satellite positioning system provides accurate location information for ground equipment by receiving satellite signals from space, thereby realizing remote monitoring and operation of construction machinery.

[0003] However, in application scenarios with high precision requirements, traditional satellite positioning technology often cannot meet the needs. In order to solve this problem, real-time dynamic differential positioning technology (RTK) came into being. RTK technology establishes a base station on the ground, receives satellite signals in real time and performs differential processing, and then sends the differential information to the mobile station that needs to be positioned, thereby achieving high-precision measurement of the mobile station's position.

[0004] Although RTK technology has improved positioning accuracy to a certain extent, its application is still subject to many restrictions in complex work site environments, especially in areas with complex terrain conditions and where GPS signals are easily blocked, such as urban high-rise building areas, deep mountain canyons, tunnels, etc. The performance of satellite positioning technology will drop significantly, and even large positioning deviations may occur. This deviation will not only affect the remote control accuracy of construction machinery, but may also have a serious impact on work safety. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides an engineering machinery vehicle positioning method and related equipment based on visual recognition.

[0006] The technical solution provided in this application is described below:

[0007] In a first aspect, the present application provides a method for positioning an engineering machinery vehicle based on visual recognition, the positioning method comprising:

[0008] Acquire a collection of images taken by a drone, wherein the collection of images includes at least one image of an engineering operation site;

[0009] Using a convolutional neural network model to determine whether an image in the image collection contains a target object, wherein the target object is an engineering machinery vehicle at the engineering operation site;

[0010] If so, generating a bounding box at the target object in the image;

[0011] Extracting pixel coordinates of the target object according to the bounding box;

[0012] Calculate the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object according to the pixel coordinates, the position of the drone and the parameters of the camera on the drone;

[0013] Converting the pixel coordinates into ground coordinates according to the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object;

[0014] Calculate the coordinates of the target object in the WGS84 coordinate system according to the ground coordinates;

[0015] The positioning of the construction machinery vehicle in the engineering operation site is achieved according to the coordinates of the target object in the WGS84 coordinate system.

[0016] Optionally, a convolutional neural network model is used to detect whether an image in the image collection contains a target object, where the target object is an engineering machinery vehicle at the engineering operation site, including:

[0017] Extracting all images in the image collection and annotating images containing engineering operation site scenes;

[0018] Preprocessing the labeled image to obtain a preprocessed image;

[0019] The preprocessed image is input into the convolutional neural network model so that the convolutional neural network model extracts feature information from the preprocessed image and determines whether the image contains a target object based on the feature information, wherein the target object is an engineering machinery vehicle at the engineering operation site.

[0020] Optionally, generating a bounding box at the target object in the image includes:

[0021] Determine four pixel coordinate points of the upper left corner, upper right corner, lower right corner and lower left corner of the target object in the image;

[0022] A bounding box is generated at the target object according to the four pixel coordinate points.

[0023] Optionally, extracting pixel coordinates of the target object according to the bounding box includes:

[0024] Determine the bottom center point coordinates of the bounding box;

[0025] The center point coordinates are used as reference points, and the pixel coordinates of the reference points are obtained.

[0026] Optionally, calculating the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object according to the position of the drone and the parameters of the camera on the drone includes:

[0027] Obtaining the location parameters of the drone, the location parameters including GPS coordinates and the height of the drone relative to the ground;

[0028] Obtaining parameters of the camera on the drone, the parameters of the camera on the drone including focal length, field of view angle, image resolution, pitch angle and yaw angle;

[0029] Calculate the ground sampling distance GSD of the image pixel according to the focal length and field of view angle of the camera on the UAV;

[0030] Calculate the actual size of the target object on the ground by using the bounding box, the pixel coordinates and the ground sampling distance;

[0031] Acquire the actual distance between the drone and the target object according to the height of the drone relative to the ground and the actual size of the target object on the ground;

[0032] Determining a target direction of the camera on the drone relative to the drone according to the pitch angle and the yaw angle;

[0033] Determine a first direction vector of the camera optical axis according to the posture of the drone and the target direction;

[0034] Calculate a second direction vector of the camera optical axis pointing to the target object through the pixel coordinates;

[0035] The sight angle between the camera on the drone and the target object is obtained according to the first direction vector and the second direction vector.

[0036] Optionally, converting the pixel coordinates into ground coordinates includes:

[0037] A coordinate transformation model is constructed according to the ground sampling distance GSD of the image pixel, the position parameters of the drone, the parameters of the camera on the drone, the distance between the camera on the drone and the target object, and the sight angle between the camera on the drone and the target object, wherein the coordinate transformation model is used to transform the image pixel coordinates into ground coordinates;

[0038] The pixel coordinates are converted into the ground coordinates according to the coordinate conversion model.

[0039] Optionally, after using the convolutional neural network model to determine whether an image in the image collection contains a target object, the method further includes:

[0040] If not, the image collection is re-detected until it is detected that an image in the image collection contains the target object.

[0041] A second aspect of the present application provides a construction machinery vehicle positioning system based on visual recognition, the positioning system comprising:

[0042] An acquisition unit, configured to acquire a collection of images taken by a drone, wherein the collection of images includes at least one image of an engineering operation site;

[0043] A judgment unit, configured to use a convolutional neural network model to detect whether an image in the image collection contains a target object, wherein the target object is an engineering machinery vehicle at the engineering operation site;

[0044] A generating unit, configured to generate a bounding box at the target object in the image when the judging unit determines that the image in the image collection contains the target object;

[0045] an extraction unit, configured to extract pixel coordinates of the target object according to the bounding box;

[0046] A first calculation unit is used to calculate the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object according to the pixel coordinates, the position of the drone and the parameters of the camera on the drone;

[0047] A conversion unit, configured to convert the pixel coordinates into ground coordinates according to a distance between the camera on the drone and the target object and a sight angle between the camera on the drone and the target object;

[0048] A second calculation unit, used for calculating the coordinates of the target object in the WGS84 coordinate system according to the ground coordinates;

[0049] A positioning unit is used to realize the positioning of the engineering machinery vehicle in the engineering operation site according to the coordinates of the target object in the WGS84 coordinate system.

[0050] The third aspect of the present application provides a construction machinery vehicle positioning device based on visual recognition, the device comprising:

[0051] Processor, memory, input-output unit, and bus;

[0052] The processor is connected to the memory, the input and output unit, and the bus;

[0053] The memory stores a program, and the processor calls the program to execute the method as described in any one of the first aspect and the second aspect.

[0054] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the method described in any one of the first and second aspects is performed.

[0055] It can be seen from the above technical solutions that this application has the following advantages:

[0056] This application uses the precise positioning information and image collection of drones in the air to establish accurate ground digital equipment and facilities and geographic information coordinates, and accurately derive the positioning of equipment and facilities. It can effectively solve the positioning accuracy problem of remote-controlled engineering machinery, and can achieve high-precision positioning of engineering machinery vehicles to avoid affecting operational safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 A schematic diagram of an embodiment of a method for positioning an engineering machinery vehicle based on visual recognition in the present application;

[0059] Figure 2 This is a schematic diagram of another embodiment of the method for positioning an engineering machinery vehicle based on visual recognition in the present application;

[0060] Figure 3 This is a schematic diagram of another embodiment of the method for positioning an engineering machinery vehicle based on visual recognition in the present application;

[0061] Figure 4 This is a schematic diagram of another embodiment of the method for positioning an engineering machinery vehicle based on visual recognition in the present application;

[0062] Figure 5 This is a schematic diagram of another embodiment of the method for positioning an engineering machinery vehicle based on visual recognition in the present application;

[0063] Figure 6 This is a schematic diagram of another embodiment of the method for positioning an engineering machinery vehicle based on visual recognition in the present application;

[0064] Figure 7 A schematic diagram of an embodiment of an engineering machinery vehicle positioning system based on visual recognition in the present application;

[0065] Figure 8 This is a schematic diagram of an embodiment of an engineering machinery vehicle positioning device based on visual recognition in the present application. DETAILED DESCRIPTION

[0066] The present application provides a method and related equipment for positioning construction machinery vehicles based on visual recognition. By accurately positioning information and collecting images in the air through drones, accurate ground digital equipment and facilities and geographic information coordinates are established, and the equipment and facilities are accurately derived and positioned. This can effectively solve the positioning accuracy problem of remotely controlled construction machinery, and can achieve high-precision positioning of construction machinery vehicles to avoid affecting operational safety.

[0067] It should be noted that the construction machinery vehicle positioning method based on visual recognition provided in this application can be applied to a terminal or a server. For example, the terminal can be a tablet computer, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses a terminal as an example for the execution subject.

[0068] See also Figure 1 The present application first provides an embodiment of a method for positioning an engineering machinery vehicle based on visual recognition, the embodiment comprising:

[0069] 101. Obtain a collection of images taken by a drone, wherein the collection of images includes at least one image of an engineering operation site;

[0070] In the embodiment of the present application, before using the drone to perform the image acquisition task, the drone needs to be deployed, specifically, including selecting the drone model, configuring the sensor, and planning the flight path and altitude of the drone to ensure that the engineering operation site can be fully covered. After the preliminary preparations are completed, the drone is controlled to perform flight shooting. During the flight, the drone will shoot images according to the preset plan. It should be noted that the setting of the camera on the drone can ensure that the clarity and resolution of the image meet the requirements of subsequent processing to ensure that high-definition images of the engineering operation site are captured.

[0071] After the drone completes the image shooting, the image will be transmitted to the ground station or cloud server for storage and processing. After receiving the image transmitted by the drone, the ground station or cloud server will integrate these images into an image collection, which contains at least one image of the engineering operation site for subsequent target detection and processing. The number of images of the engineering operation site in the image collection is not specifically limited. After obtaining the image collection shot by the drone, step 102 is executed.

[0072] 102. Using a convolutional neural network model, determining whether an image in the image collection contains a target object, wherein the target object is an engineering machinery vehicle at the engineering operation site;

[0073] In the embodiment of the present application, a convolutional neural network model is used to detect whether an image in an image collection contains a target object. Specifically, the model should have good target detection capabilities, especially for specific targets such as engineering machinery vehicles. During the model training phase, a large number of images labeled with engineering machinery vehicles will be used as training data sets, and the weights and parameters of the model will be optimized through iterative training so that it can accurately identify engineering machinery vehicles in images.

[0074] Each image in the image collection is then used as input and sent to the convolutional neural network model for processing. Before input, the image is preprocessed, such as adjusting the image size and normalizing the pixel values, to ensure that the model can process it correctly. The convolutional neural network model then performs multi-level feature extraction on the input image, capturing key information in the image through structures such as convolutional layers and pooling layers. The extracted features are sent to the fully connected layer for classification, and the model outputs a probability distribution, indicating the possibility of each category being included in the image.

[0075] Then, based on the output of the model, it can be determined whether the image contains the target object, that is, the construction machinery vehicle. When making a judgment, a threshold is usually used for judgment. When the probability of the construction machinery vehicle category output by the model exceeds this threshold, it is considered that the image contains the target object. Since there are many images in the image collection, for each image, the model will give a judgment result on whether the target object is contained. When it is determined that there are images in the image collection that contain the target object, step 103 is executed. When it is determined that there are images in the image collection that do not contain the target object, step 109 is executed: re-detect the image collection until it is detected that the image in the image collection contains the target object. The re-detected image collection is a newly captured image collection.

[0076] 103. Generate a bounding box at the target object in the image;

[0077] In an embodiment of the present application, after determining which images in the image collection contain the target object of the construction machinery vehicle, for each image containing the target object, the terminal will use a specific bounding box generation algorithm. The bounding box generation algorithm is usually based on the output of the target detection network, such as anchor box adjustment, non-maximum suppression, etc. to accurately locate the target object.

[0078] Among them, the bounding box generation algorithm calculates the coordinates of the rectangular box surrounding the target object, including the pixel coordinates of the upper left corner, lower left corner, lower right corner and upper right corner of the rectangle. These pixel coordinates define the position and size of the bounding box in the image. Then, based on the calculated pixel coordinates, the terminal draws the bounding box on the image to intuitively mark the position of the construction machinery vehicle. It should be noted that the bounding box is usually highlighted in a different color or line style to distinguish it from other elements in the image.

[0079] By generating a bounding box on the image containing the target object, the precise marking of the position of the construction machinery vehicle is achieved, which provides an important basis for the subsequent extraction of pixel coordinates, calculation of distance and angle, and final target positioning.

[0080] 104. Extract pixel coordinates of the target object according to the bounding box;

[0081] In an embodiment of the present application, in step 103, after the terminal determines that the image contains an engineering machinery vehicle through a convolutional neural network model, it generates a bounding box surrounding the target object and includes coordinate information defining its position and size.

[0082] To extract the pixel coordinates of the target object based on the bounding box, the following steps are required:

[0083] 1. The terminal reads the coordinate information of the bounding box, which usually includes the (x1, y1) coordinates of the upper left corner and the (x2, y2) coordinates of the lower right corner of the bounding box.

[0084] 2. With the coordinate information of the bounding box, the terminal can calculate the key pixel coordinates of the target object. The key pixel coordinates are the center point coordinates at the bottom of the bounding box. The center point coordinate calculation example is: (xc, yc) = ((x1+x2) / 2, (y1+y2) / 2)

[0085] Application examples:

[0086] Suppose that in an image taken by a drone, the terminal detects an engineering vehicle and generates a bounding box around the vehicle. The coordinates of the upper left corner of the bounding box are (100, 200) and the coordinates of the lower right corner are (300, 400). Then, the system can calculate the pixel coordinates of the center point of the vehicle as ((100+300) / 2, (200+400) / 2) = (200, 300). This center point coordinate can then be used in subsequent processing steps such as coordinate conversion and target positioning.

[0087] 105. Calculate the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object according to the pixel coordinates, the position of the drone and the parameters of the camera on the drone;

[0088] In the embodiment of the present application, pixel coordinates: pixel coordinates of the target object extracted in step 104 in the image, such as center point coordinates (u, v). Drone position: Global Positioning System (GPS) coordinates of the drone or position coordinates relative to a fixed point, such as (x_drone, y_drone, z_drone). Camera parameters: including the focal length f of the camera, the principal point coordinates (u0, v0), and the installation position and attitude of the camera relative to the drone. The above parameters can be used to further calculate the distance between the camera on the drone and the target object and the line of sight angle from the camera on the drone to the target object. The actual calculation will be explained in the subsequent description.

[0089] 106. Convert the pixel coordinates into ground coordinates according to the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object;

[0090] In an embodiment of the present application, converting pixel coordinates into ground coordinates involves converting from the two-dimensional pixel space of the image to the actual three-dimensional geographic space. The process of converting pixel coordinates into ground coordinates is essentially mapping the two-dimensional points in the image to the three-dimensional points in the actual geographic space. This conversion process depends on the internal and external parameters of the camera, the position and posture of the drone, the distance between the drone camera and the target object, the visual angle from the drone camera to the target object, and other ground control point information.

[0091] Specific:

[0092] 1. Camera calibration and parameter acquisition

[0093] Camera calibration: Through camera calibration, the camera's intrinsic parameter matrix is ​​obtained, including the focal length f, principal point coordinates (u0, v0) and extrinsic parameter matrix. The extrinsic parameter matrix is ​​used to describe the position and posture of the camera relative to a reference coordinate system.

[0094] Parameter acquisition: Ensure that the GPS position of the drone, attitude information (such as pitch angle, yaw angle, roll angle) and the installation position and attitude of the camera are accurate and reliable.

[0095] 2. Conversion from pixel coordinates to camera coordinate system

[0096] Using the camera's intrinsic parameter matrix, the pixel coordinates (u, v) in the image are converted into three-dimensional coordinates (Xc, Yc, Zc) in the camera coordinate system.

[0097] 3. Conversion from camera coordinate system to world coordinate system

[0098] The camera's external parameter matrix is ​​used to transform the three-dimensional coordinates (Xc, Yc, Zc) in the camera coordinate system to the three-dimensional coordinates (Xw, Yw, Zw) in the world coordinate system, such as the WGS84 coordinate system or the local geographic coordinate system. This conversion process determines the position and attitude of the camera relative to the world coordinate system.

[0099] The conversion from pixel coordinates to ground coordinates can be achieved through a series of matrix transformations. The specific formula may vary depending on the imaging model of the camera and the specific steps of the conversion, and is not specifically limited here.

[0100] Through the above steps, the pixel coordinates in the image can be accurately converted into ground coordinates, providing strong support for the precise positioning and target identification of drones in geographic space.

[0101] 107. Calculate the coordinates of the target object in the WGS84 coordinate system according to the ground coordinates;

[0102] In an embodiment of the present application, the coordinates of the target object in the ground coordinate system are converted into coordinates in the WGS84 coordinate system. The WGS84 coordinate system determines the position of any point on the earth's surface through three parameters: longitude, latitude, and altitude. Among them, longitude and latitude represent the positions in the east-west direction and the north-south direction, respectively, and altitude represents the height difference relative to the WGS84 reference ellipsoid. After the ground coordinates of the target object are obtained, further, after the coordinates of the target object in the WGS84 coordinate system are calculated based on the obtained ground coordinates, step 108 is executed.

[0103] 108. Positioning the construction machinery vehicle at the engineering operation site according to the coordinates of the target object in the WGS84 coordinate system.

[0104] In an embodiment of the present application, the positioning of the construction machinery vehicle at the engineering work site is achieved based on the coordinates of the target object in the WGS84 coordinate system, the coordinates of the target object in the WGS84 coordinate system are compared with the current position coordinates of the construction machinery vehicle, and the positional relationship of the vehicle relative to the target object is determined. The positioning result is then output to the operator or control system in the form of graphics, numerical values, or voice, so as to carry out subsequent engineering operations or navigation control.

[0105] It can be further known that the present application establishes accurate ground digital equipment and facilities and geographic information coordinates through precise positioning information and image collection by drones in the air, and accurately derives the positioning of equipment and facilities, which can well solve the positioning accuracy problem of remote-controlled engineering machinery, and can achieve high-precision positioning of engineering machinery vehicles to avoid affecting operational safety.

[0106] Please refer to Figure 2According to some embodiments of the present invention, in step 102, a convolutional neural network model is used to detect whether an image in the image collection contains a target object. The target object is an engineering machinery vehicle at the engineering operation site, and specifically may include, but is not limited to, the following:

[0107] 201. Extract all images in the image collection, and mark images containing engineering operation site scenes;

[0108] 202. Preprocess the labeled image to obtain a preprocessed image;

[0109] 203. Input the preprocessed image into the convolutional neural network model so that the convolutional neural network model extracts feature information from the preprocessed image and determines whether the image contains a target object based on the feature information, wherein the target object is an engineering machinery vehicle at the engineering operation site.

[0110] In an embodiment of the present application, in step 201, all images are extracted from the image collection. These images may come from different cameras, different time periods, and different engineering work sites. It is ensured that the extracted images are sufficiently diverse and representative so that the subsequent detection process can cover various possible situations. The extracted images are annotated, and the annotated content includes the location, type, and other relevant information of the engineering machinery vehicle.

[0111] In step 202, by preprocessing the image, the quality of the image data can be effectively improved to make it more suitable for the input requirements of the convolutional neural network model. After preprocessing, the noise, distortion and other factors that may interfere with model detection in the image can be reduced.

[0112] In step 203, the preprocessed image is passed as input to the convolutional neural network model, ensuring that the input image has the same format and features as the image used in model training. The convolutional neural network model extracts features from the input image through multiple convolutional layers, pooling layers, and activation layers. These layers can capture low-level features such as edges, textures, and shapes in the image, and gradually combine them into higher-level feature representations. Based on feature extraction, the model uses fully connected layers or other types of layers to detect whether there is a target object, i.e., an engineering machinery vehicle, in the image. Then the model outputs one or more prediction results, including information such as the location, category, and confidence of the target object.

[0113] This embodiment realizes automatic detection and recognition of construction machinery vehicles at engineering work sites through steps such as image extraction and annotation, image preprocessing, and processing and detection of convolutional neural network models, which helps to improve the efficiency and safety of engineering operations and reduce human errors and interventions.

[0114] Please refer to Figure 3 According to some embodiments of the present invention, generating a bounding box at the target object in the image in step 103 may specifically include, but is not limited to, the following:

[0115] 301. Determine four pixel coordinate points of the upper left corner, upper right corner, lower right corner and lower left corner of the target object in the image;

[0116] 302. Generate a bounding box at the target object according to the four pixel coordinate points.

[0117] In the embodiment of the present application, in step 301, after being processed by the convolutional neural network model, the target object in the image has been identified and located. The model output generally includes the category of the target object, the confidence level, and its approximate position in the image. In order to accurately describe the position of the target object, it is necessary to determine four key pixel coordinate points on its boundary: the upper left corner, the upper right corner, the lower right corner, and the lower left corner. A bounding box is generated based on the four key pixel coordinate points. The generated bounding box is a rectangular box that is used to tightly surround the target object and clearly identify its position in the image. It is defined by four coordinate points, which correspond to the four corners of the rectangle.

[0118] After determining the four key pixel coordinate points of the target object, a rectangular frame can be drawn according to these coordinate values ​​to clearly identify the position of the target object in the image.

[0119] This embodiment achieves accurate positioning and identification of the target object by determining four key pixel coordinate points of the target object in the image and generating a bounding box based on these points.

[0120] Please refer to Figure 4 According to some embodiments of the present invention, extracting the pixel coordinates of the target object according to the bounding box in step 104 may specifically include, but is not limited to, the following:

[0121] 401. Determine the coordinates of the bottom center point of the bounding box;

[0122] 402. Use the center point coordinates as a reference point, and obtain pixel coordinates of the reference point.

[0123] In the embodiment of the present application, the bounding box is a rectangular box that tightly surrounds the target object and is defined by four coordinate points: the upper left corner, the upper right corner, the lower right corner, and the lower left corner. The bottom of the bounding box refers to the bottom side of the rectangular box, and the center point is the midpoint of this side.

[0124] Specifically, the x-coordinate of the bottom center point can be obtained by calculating the average of the x-coordinates of the lower left corner and the lower right corner; the y-coordinate of the bottom center point is the larger value of the y-coordinates of the lower left corner and the lower right corner. Then the calculated center point coordinates are used as the reference point, and the pixel coordinates of the reference point are obtained. The reference point provides an intuitive and stable representation of the position of the target object in the image, and the reference point can be used for further analysis, tracking or annotation.

[0125] Once the coordinates of the bottom center point are determined, they represent the location of the reference point in the image. They can be used to determine the precise location of construction machinery vehicles, thereby supporting more efficient job scheduling and safety monitoring.

[0126] Please refer to Figure 5 According to some embodiments of the present invention, in step 105, calculating the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object according to the pixel coordinates, the position of the drone and the parameters of the camera on the drone may specifically include, but is not limited to, the following:

[0127] 501. Obtaining location parameters of the drone, where the location parameters include GPS coordinates and the height of the drone relative to the ground;

[0128] 502. Acquire parameters of the camera on the drone, where the parameters of the camera on the drone include focal length, field of view angle, image resolution, pitch angle, and yaw angle;

[0129] 503. Calculate the ground sampling distance GSD of the image pixel according to the focal length and field angle of the camera on the drone;

[0130] 504. Calculate the actual size of the target object on the ground by using the bounding box, the pixel coordinates and the ground sampling distance;

[0131] 505. Acquire an actual distance between the drone and the target object according to the height of the drone relative to the ground and the actual size of the target object on the ground;

[0132] 506. Determine a target direction of the camera on the drone relative to the drone according to the pitch angle and the yaw angle;

[0133] 507. Determine a first direction vector of a camera optical axis according to the posture of the UAV and the target direction;

[0134] 508. Calculate a second direction vector of the camera optical axis pointing to the target object using the pixel coordinates;

[0135] 509. Obtain a sight angle between the camera on the drone and the target object according to the first direction vector and the second direction vector.

[0136] In the embodiment of the present application, the location parameters of the drone are obtained in order to determine the specific location of the drone in three-dimensional space. Its longitude and latitude coordinates can be obtained through the GPS system of the drone, and its height relative to the ground can be obtained through the altitude sensor of the drone. Then the parameters of the camera on the drone are obtained in order to obtain the imaging characteristics and installation status of the camera. The focal length, field of view angle, image resolution and other parameters can be obtained from the technical specifications of the camera, and the pitch angle and yaw angle of the camera can be obtained through the attitude sensor of the drone.

[0137] After obtaining the position parameters of the drone and the parameters of the camera on the drone, the ground sampling distance GSD of the image pixels is calculated in order to determine the actual ground distance represented by each pixel in the image. The focal length and field of view of the camera can be used, combined with the height of the drone, to calculate the GSD through geometric relationships.

[0138] Next, the actual size of the target object on the ground is obtained. The actual size of the target object on the ground can be calculated by using the size of the bounding box, pixel coordinates and GSD through a proportional relationship.

[0139] Then the actual distance between the drone and the target object is obtained, so that the actual horizontal distance between the drone and the target object can be determined. When calculating, it is necessary to combine the height of the drone and the height of the target object, and use the Pythagorean theorem to calculate the actual distance.

[0140] After calculating the distance between the camera on the drone and the target object, the line of sight angle between the camera on the drone and the target object is further calculated. First, the target direction of the camera on the drone relative to the drone is determined in order to clarify the direction of the camera relative to the drone. The direction of the camera optical axis in the drone coordinate system is determined by the pitch angle and yaw angle. Then the first direction vector of the camera optical axis is determined. In combination with the attitude of the drone (such as roll, pitch, yaw) and the target direction of the camera relative to the drone, the first direction vector of the camera optical axis in the global coordinate system is calculated. Then the second direction vector of the camera optical axis pointing to the target object is calculated. Using the pixel coordinates and the intrinsic parameter matrix of the camera, the straight line from the optical center of the camera to the corresponding point of the target object in the image, that is, the second direction vector, is calculated; finally, the angle between the first direction vector and the second direction vector is calculated to obtain the line of sight angle between the camera optical axis and the target object. This angle can be used for further analysis, such as target tracking or navigation.

[0141] Please refer to Figure 6According to some embodiments of the present invention, in step 106, converting the pixel coordinates into ground coordinates according to the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object may specifically include, but not limited to, the following:

[0142] 601. Construct a coordinate transformation model according to the ground sampling distance GSD of the image pixel, the position parameters of the drone, the parameters of the camera on the drone, the distance between the camera on the drone and the target object, and the sight angle between the camera on the drone and the target object, wherein the coordinate transformation model is used to transform the image pixel coordinates into ground coordinates;

[0143] 602. Convert the pixel coordinates into the ground coordinates according to the coordinate conversion model.

[0144] In the embodiment of the present application, after constructing the coordinate conversion model, the necessary parameters are first collected: the ground sampling distance (GSD) of the image pixel: represents the actual ground distance represented by each pixel in the image. The position parameters of the drone: including the GPS coordinates and the height of the drone relative to the ground. The parameters of the camera on the drone: including the focal length, field of view, image resolution, pitch angle and yaw angle. The distance between the camera on the drone and the target object: calculated by the previous steps. The line of sight angle between the camera on the drone and the target object: also calculated by the previous steps. Then establish a coordinate conversion relationship: using the imaging principle of the camera, combined with the position and posture of the drone, and the parameters of the camera, establish a conversion relationship from the image pixel coordinates to the ground coordinates. The conversion relationship involves complex geometric and trigonometric calculations, including the camera's intrinsic parameter matrix, extrinsic parameter matrix and possible distortion correction. After that, the above conversion relationship is integrated into a coordinate conversion model, the input of which is the pixel coordinates in the image, and the output is the corresponding ground coordinates, so that the pixel coordinates can be converted into ground coordinates. This embodiment realizes the conversion of the pixel coordinates in the image into the actual coordinates on the ground by constructing a coordinate conversion model, which can facilitate the effective positioning of the engineering machinery vehicle.

[0145] See also Figure 7 The present application also provides an engineering machinery vehicle positioning system based on visual recognition, comprising:

[0146] An acquisition unit 701 is used to acquire a collection of images taken by a drone, wherein the collection of images includes at least one image of an engineering operation site;

[0147] A judgment unit 702 is used to use a convolutional neural network model to detect whether an image in the image collection contains a target object, where the target object is an engineering machinery vehicle at the engineering operation site;

[0148] A generating unit 703, configured to generate a bounding box at the target object in the image when the judging unit determines that the image in the image collection contains the target object;

[0149] An extraction unit 704 is used to extract pixel coordinates of the target object according to the bounding box;

[0150] A first calculation unit 705 is used to calculate the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object according to the pixel coordinates, the position of the drone and the parameters of the camera on the drone;

[0151] A conversion unit 706, configured to convert the pixel coordinates into ground coordinates;

[0152] A second calculation unit 707, configured to calculate the coordinates of the target object in the WGS84 coordinate system according to the ground coordinates;

[0153] The positioning unit 708 is used to realize the positioning of the engineering machinery vehicle in the engineering operation site according to the coordinates of the target object in the WGS84 coordinate system.

[0154] See also Figure 8 The present application also provides a construction machinery vehicle positioning device based on visual recognition, comprising:

[0155] Processor 801, memory 802, input and output unit 803, bus 804;

[0156] The processor 801 is connected to the memory 802, the input and output unit 803 and the bus 804;

[0157] The memory 802 stores a program, and the processor 801 calls the program to execute any of the above methods.

[0158] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above methods.

[0159] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A method for positioning an engineering machinery vehicle based on visual recognition, characterized in that: The positioning method comprises: Acquire a collection of images taken by a drone, wherein the collection of images includes at least one image of an engineering operation site; Using a convolutional neural network model to determine whether an image in the image collection contains a target object, wherein the target object is an engineering machinery vehicle at the engineering operation site; If so, generating a bounding box at the target object in the image; Extracting pixel coordinates of the target object according to the bounding box; Calculate the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object according to the pixel coordinates, the position of the drone and the parameters of the camera on the drone; Converting the pixel coordinates into ground coordinates according to the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object; Calculate the coordinates of the target object in the WGS84 coordinate system according to the ground coordinates; The positioning of the construction machinery vehicle in the engineering operation site is achieved according to the coordinates of the target object in the WGS84 coordinate system.

2. The method for positioning an engineering machinery vehicle based on visual recognition according to claim 1, characterized in that: Using a convolutional neural network model to detect whether an image in the image collection contains a target object, the target object being an engineering machinery vehicle in the engineering operation site, including: Extracting all images in the image collection and annotating images containing engineering operation site scenes; Preprocessing the labeled image to obtain a preprocessed image; The preprocessed image is input into the convolutional neural network model so that the convolutional neural network model extracts feature information from the preprocessed image and determines whether the image contains a target object based on the feature information, wherein the target object is an engineering machinery vehicle at the engineering operation site.

3. The method for positioning an engineering machinery vehicle based on visual recognition according to claim 1, characterized in that: Generating a bounding box at a target object in the image, comprising: Determine four pixel coordinate points of the upper left corner, upper right corner, lower right corner and lower left corner of the target object in the image; A bounding box is generated at the target object according to the four pixel coordinate points.

4. The method for positioning an engineering machinery vehicle based on visual recognition according to claim 1, characterized in that: Extracting pixel coordinates of the target object according to the bounding box includes: Determine the bottom center point coordinates of the bounding box; The center point coordinates are used as reference points, and the pixel coordinates of the reference points are obtained.

5. The method for positioning an engineering machinery vehicle based on visual recognition according to claim 4, characterized in that: Calculating the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object according to the position of the drone and the parameters of the camera on the drone, including: Obtaining the location parameters of the drone, the location parameters including GPS coordinates and the height of the drone relative to the ground; Obtaining parameters of the camera on the drone, the parameters of the camera on the drone including focal length, field of view angle, image resolution, pitch angle and yaw angle; Calculating the ground sampling distance GSD of the image pixel according to the focal length and field of view angle of the camera on the UAV; Calculate the actual size of the target object on the ground by using the bounding box, the pixel coordinates and the ground sampling distance; Acquire the actual distance between the drone and the target object according to the height of the drone relative to the ground and the actual size of the target object on the ground; Determining a target direction of the camera on the drone relative to the drone according to the pitch angle and the yaw angle; Determine a first direction vector of the camera optical axis according to the posture of the drone and the target direction; Calculate a second direction vector of the camera optical axis pointing to the target object through the pixel coordinates; The sight angle between the camera on the drone and the target object is obtained according to the first direction vector and the second direction vector.

6. The method for positioning an engineering machinery vehicle based on visual recognition according to claim 4, characterized in that: The pixel coordinates are converted into ground coordinates according to the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object, including: A coordinate transformation model is constructed according to the ground sampling distance GSD of the image pixel, the position parameters of the drone, the parameters of the camera on the drone, the distance between the camera on the drone and the target object, and the sight angle between the camera on the drone and the target object, wherein the coordinate transformation model is used to transform the image pixel coordinates into ground coordinates; The pixel coordinates are converted into the ground coordinates according to the coordinate conversion model.

7. The method for positioning an engineering machinery vehicle based on visual recognition according to claim 1, characterized in that: After using the convolutional neural network model to determine whether the image in the image collection contains the target object, the method further includes: If not, the image collection is re-detected until it is detected that an image in the image collection contains the target object.

8. An engineering machinery vehicle positioning system based on visual recognition, characterized in that: The positioning system comprises: An acquisition unit, configured to acquire a collection of images taken by a drone, wherein the collection of images includes at least one image of an engineering operation site; A judgment unit, configured to use a convolutional neural network model to detect whether an image in the image collection contains a target object, wherein the target object is an engineering machinery vehicle at the engineering operation site; A generating unit, configured to generate a bounding box at the target object in the image when the judging unit determines that the image in the image collection contains the target object; an extraction unit, configured to extract pixel coordinates of the target object according to the bounding box; A first calculation unit is used to calculate the distance between the camera on the drone and the target object and the sight angle between the camera on the drone and the target object according to the pixel coordinates, the position of the drone and the parameters of the camera on the drone; A conversion unit, configured to convert the pixel coordinates into ground coordinates according to a distance between the camera on the drone and the target object and a sight angle between the camera on the drone and the target object; A second calculation unit, used for calculating the coordinates of the target object in the WGS84 coordinate system according to the ground coordinates; A positioning unit is used to realize the positioning of the engineering machinery vehicle in the engineering operation site according to the coordinates of the target object in the WGS84 coordinate system.

9. A construction machinery vehicle positioning device based on visual recognition, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 7.

Images