A method, apparatus, device and storage medium for puncturing
By using a neural network model to automatically identify and determine the location of image control points, the problems of error and inefficiency in the traditional manual pricking method are solved, and efficient and accurate automated pricking of image control points is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN LUODAO TECH CO LTD
- Filing Date
- 2023-04-10
- Publication Date
- 2026-06-05
AI Technical Summary
In traditional UAV image acquisition equipment, the location of control points relies on manual marking, which is subjective and inefficient, especially when processing large amounts of images, resulting in a huge workload.
By employing a neural network model combined with collinearity equations, the location of control points is automatically identified and determined. Feature extraction, recognition, and coordinate calculation are performed through a multi-layer neural network model, reducing human intervention.
It has improved the automation level of aerial survey data processing, reduced manpower, and improved the accuracy and efficiency of determining the location of image control points.
Smart Images
Figure CN116563372B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of photogrammetry technology, and in particular relates to a method, apparatus, device and storage medium for puncturing. Background Technology
[0002] Currently, after a drone image acquisition device performs an aerial survey of a certain area to be measured, it obtains an image of that area. Then, the operator uses image processing software to mark the control points (control points) in the image (i.e., determine the position of the control points in the image) to obtain the coordinates of the control points in the image.
[0003] However, in traditional techniques, the locations of puncture points in the image to be punctured are determined by the operator based on visual judgment and manually marked, which involves a certain degree of subjectivity and uncertainty, and is therefore prone to human error. Furthermore, UAV image acquisition equipment typically involves a huge number of images for aerial surveys of a specific area, with each image containing several or even dozens of ground control points. If the puncturing task is performed manually, the workload is enormous and the processing efficiency is low. Summary of the Invention
[0004] In view of this, embodiments of this application provide a puncture method, apparatus, device, and storage medium to solve the problem that human error is easily introduced by manually marking the puncture point position in the image to be punctured in the prior art.
[0005] A first aspect of this application provides a method for pricking points, the method comprising: acquiring an image to be pricking; determining a target region in the image to be pricking, the target region including control points and control point identification images, the control point identification images being used to identify the positions of the control points; determining the control point identification images within the target region based on the features of the control point identification images using a neural network model, and determining the position coordinates of the control points in the image to be pricking based on the positional relationship between the control points and the control point identification images.
[0006] In conjunction with the first aspect, in a first possible implementation of the first aspect, determining the target region in the image of the point to be punctured includes: acquiring the actual position coordinates of the image control point and the device position coordinates of the acquisition device; determining the target region in the image of the point to be punctured based on the collinearity equation according to the actual position coordinates, the device position coordinates and their orientation; wherein, the acquisition device is the device that acquires the image of the point to be punctured, the device position coordinates are the position coordinates of the acquisition device when acquiring the image of the point to be punctured, and the actual position coordinates are the position coordinates of the image control point when measured in the field.
[0007] In conjunction with the first aspect, in the second possible implementation of the first aspect, a neural network model is used to determine the control point marker images within the target area based on the features of the control point marker images, and the position coordinates of the control points in the image of the point to be pierced are determined based on the positional relationship between the control points and the control point marker images. This includes: based on the first neural network model, performing preliminary extraction of the control point marker images within the target area based on the features of the control point marker images; based on the second neural network model, identifying the preliminary extracted control point marker images to finally determine the control point marker images; and based on the third neural network model, determining the position coordinates of the control points based on the positional relationship between the control points and the control point marker images.
[0008] In conjunction with the first aspect, in the third possible implementation of the first aspect, the neural network model is determined through training a training dataset and a target loss function; wherein, in the first neural network model, the training dataset includes a first training dataset with features of control point marker images, and the loss function is a classification loss function and a regression loss function; in the second neural network model, the training dataset includes positive samples from the first training dataset and an equal number of negative samples, and the loss function is a classification loss function; in the third neural network model, the training dataset includes positive samples from the first training dataset, and the loss function is a regression loss function or a heatmap loss function.
[0009] In conjunction with the first aspect, in the fourth possible implementation of the first aspect, positive samples are used to finally determine the control point marker image and calibrate the coordinates of the control points in the control point marker image; negative samples are used to remove the control point marker images extracted from the first neural network model.
[0010] In conjunction with the first aspect, in the fifth possible implementation of the first aspect, the control point marker image includes either an L-shaped marker image or an X-shaped marker image.
[0011] In conjunction with the first aspect, in the sixth possible implementation of the first aspect, when the control point marker image is an L-shaped marker image, the control point is located at the inner corner of the L-shaped marker image; when the control point marker image is an X-shaped marker image, the control point is located at the center point of the X-shaped marker image.
[0012] A second aspect of this application provides a pricking device, comprising: an acquisition unit for acquiring an image to be pricking; a target region determination unit for determining a target region in the image to be pricking, the target region including control points and control point identification images, the control point identification images being used to identify the positions of the control points; and a control point position coordinate determination unit for determining the control point identification images within the target region based on the features of the control point identification images using a neural network model, and determining the position coordinates of the control points in the image to be pricking based on the positional relationship between the control points and the control point identification images.
[0013] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any of the first aspects.
[0014] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any of the first aspects.
[0015] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment provides a puncturing method, device, equipment and storage medium, which provides an effective automated processing method for the puncturing process in aerial surveying work. This method combines a neural network model to realize automated puncturing, that is, automatically determine the control points in the aerial surveying image, reduce manpower and time consumption, and thus more effectively improve the automation level of aerial surveying work. Attached Figure Description
[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1 This is a schematic flowchart of a puncture method provided in an embodiment of this application;
[0018] Figure 2 This is a schematic diagram illustrating an application scenario of the collinearity equation in photogrammetry provided in this application embodiment;
[0019] Figure 3 This is a schematic flowchart of the first neural network model training method provided in the embodiments of this application;
[0020] Figure 4This is a schematic diagram of the imaging images of control point markers in different target areas provided in the embodiments of this application;
[0021] Figure 5 This is a schematic flowchart of the second neural network model training method provided in the embodiments of this application;
[0022] Figure 6 This is a schematic diagram of an image of another feature object with a similar shape to the control point marker provided in an embodiment of this application;
[0023] Figure 7 This is a schematic flowchart of the third neural network model training method provided in the embodiments of this application;
[0024] Figure 8 This is a schematic flowchart of the puncturing method based on various neural network models provided in the embodiments of this application;
[0025] Figure 9 This is a schematic diagram showing the location of key points provided in the embodiments of this application;
[0026] Figure 10 This is a schematic diagram of the display interface provided in an embodiment of this application;
[0027] Figure 11 This is a schematic diagram of the puncture device provided in the embodiments of this application;
[0028] Figure 12 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation
[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0030] The technical solutions provided in this application will be explained in detail below with reference to specific embodiments.
[0031] In the field of UAV photogrammetry, UAV image acquisition equipment is needed to conduct aerial surveys of the area to be surveyed in order to obtain aerial images of that area.
[0032] Currently, drone photogrammetry typically includes aerial survey fieldwork and aerial survey officework.
[0033] In aerial surveying fieldwork, to ensure the accuracy of aerial survey results, a certain number of ground control points (GCPs) need to be established in the area to be surveyed (such as an administrative region). The GCPs should evenly cover the area, and their number depends on the size of the area, the degree of terrain undulation, the complexity of the topography, and the required measurement accuracy. After the GCPs are established, field operators mark the locations of the GCPs for identification during office work. Then, aerial survey images of the area are acquired using UAV image acquisition equipment, and the three-dimensional coordinates of each GCP in its corresponding coordinate system are verified in the field using Real-time Kinematic (RTK) detection equipment. It should be understood that the coordinate system used for field measurement of GCPs using RTK detection equipment in aerial surveying fieldwork refers to the coordinate system used for calculations in the aerial surveying office work.
[0034] In aerial surveying office work, image processing software is needed to perform point refining (i.e., determining the position of ground control points in the aerial survey image) on the aerial survey images acquired during the fieldwork, in order to obtain the coordinates of the ground control points in the aerial survey image. Then, using the coordinates of the ground control points in the aerial survey image as the calculation benchmark, an absolute coordinate reference is provided for subsequent aerial triangulation calculations, and the aerial triangulation results are corrected. Therefore, the quality of the point refining process directly affects the accuracy of the subsequent aerial triangulation calculation results, and thus the accuracy of the final aerial survey results.
[0035] Typically, in aerial surveying work, when using image processing software to add points to aerial survey images (i.e. images to be added), the image processing software, after acquiring the image to be added, determines the point position based on the user's manual operation in the image to be added, and records the coordinates of the point position as the coordinates of the control point in the corresponding image to be added.
[0036] However, in traditional techniques, the locations of puncture points in the image to be punctured are determined by the operator based on visual judgment and manually marked, which involves a certain degree of subjectivity and uncertainty, and is therefore prone to human error. Furthermore, UAV image acquisition equipment typically involves a huge number of images for aerial surveys of a specific area; if the puncture task is performed manually, the workload is enormous and the processing efficiency is low.
[0037] Based on this, the present application provides a method for puncturing points. This method combines a neural network model to achieve automated puncturing (i.e., automated determination of control points in aerial survey images). The obtained control point positions are highly accurate, reducing manpower and time consumption, and greatly improving the automation level of aerial survey processing.
[0038] The puncturing method provided in this application can be applied to electronic devices. These electronic devices can be smartphones, tablets, computers running operating systems, or smart hardware devices, etc., and this embodiment does not impose specific limitations on them.
[0039] Figure 1 A schematic flowchart illustrating a puncture method provided in an embodiment of this application, such as... Figure 1 As shown, the puncture method includes the following steps S1-S3.
[0040] S1. The electronic device acquires an image of the point to be punctured.
[0041] The image to be punctured refers to the image of the area to be tested that is acquired after the UAV image acquisition equipment has completed the aerial survey of the area to be tested. The image includes the control points in the area to be tested.
[0042] When an electronic device acquires an image of a point to be punctured, it stores the image in a memory card after the drone image acquisition device acquires the image. When the electronic device needs to perform a puncturing task, it can access the stored information of the drone image acquisition device, i.e., the image of the point to be punctured, through a card reader or other reading device.
[0043] S2. The electronic device determines the target area in the image of the point to be punctured.
[0044] Since the images of the target area to be punctured acquired by the UAV image acquisition equipment are usually huge, in order to accurately identify the position of the control points in the image to be punctured, the electronic equipment needs to reduce the range of the image to be punctured to obtain the target area including the control points and the control point marker images, and then perform further processing on the target area to improve the processing speed and accuracy of puncturing.
[0045] Electronic devices can view the process of reducing the range of the image to be punctured as the process of target detection on a larger image. That is, the approximate location of the control point is detected in the image to be punctured and extracted as the target region. Then, the extracted target region is further analyzed and processed, thereby greatly reducing the detection range.
[0046] In some embodiments, the target area can be determined based on the actual position coordinates of the image control point and the device position coordinates and orientation of the image acquisition device. The image acquisition device is the device that acquires images of the point to be punctured; the device position coordinates are the position coordinates of the acquisition device when acquiring the image of the point to be punctured; and the actual position coordinates and orientation are the position coordinates of the image control point and its exterior orientation element information during field measurement. For example, the area to be measured can be determined by the following method: The electronic device first acquires the actual position coordinates of the image control point and the device position coordinates of the acquisition device. The actual position coordinates of the image control point can be obtained by field measurement using an RTK detection device during aerial surveying fieldwork. The device position coordinates of the image acquisition device can be based on the exterior orientation element information recorded in the UAV image acquisition device. The exterior orientation element information is used to locate the position parameters of the UAV acquisition device, and the distortion parameters in the exterior orientation element information can be used to perform image correction on the acquired image of the point to be punctured. Then, the electronic device uses the collinearity equation formula in photogrammetry, see [link to relevant documentation]. Figure 2 As shown, the approximate location of the control point is predicted by the collinearity equation, and the area around the approximate location is selected as the target region of the control point, thereby greatly reducing the retrieval range of the image to be punctured.
[0047] S3. The electronic device determines the control point marker image in the target area based on the characteristics of the control point marker image through a neural network model, and determines the position coordinates of the control point in the image to be punctured based on the positional relationship between the control point and the control point marker image.
[0048] Images to be punctured are typically acquired using drone image acquisition equipment. Due to variations in the area of the target region, the degree of terrain undulation, and the complexity of the topography, images acquired by drone image acquisition equipment are often quite complex. Therefore, in order to accurately puncture these complex images, electronic devices often need to utilize detection technologies capable of precisely identifying specific targets within the image for processing. Neural network models, especially deep learning convolutional neural network models, have a precise ability to identify target objects in image processing. Compared to traditional algorithms, deep learning, through extensive training, acquires target detection parameters that can better extract target features and exhibit more robust performance. Therefore, this paper considers introducing neural network models into the puncturing process of images to achieve automatic and accurate puncturing.
[0049] In some embodiments, the electronic device may create a neural network model (also referred to as a first neural network model) for extracting control point marker images within a target area. This neural network model can be trained using a training dataset with control point marker image features. However, due to the large number and high complexity of objects contained in each target area, when the electronic device extracts control point marker images within the target area using the neural network model, the extracted control point marker images may be images containing control points or images containing other feature objects similar to control points. Therefore, the electronic device needs to further identify the extracted control point marker images within the target area to determine whether the control point marker images are actual control point marker images.
[0050] Similarly, when electronic devices identify whether a control point marker image within a target area is an actual control point marker image, they can also use a neural network model (also known as a second neural network model) for identification. It should be understood that the second neural network model used here is different from the first neural network model used when extracting control point marker images within the target area. The second neural network model used to identify whether a control point marker image within a target area is an actual control point marker image is obtained by training a training dataset created from misidentified and poor-quality negative samples (training images with chaotic control point colors, blurred boundaries, or partial occlusion) detected by the first neural network model, and an equal number of high-quality positive samples (training images with bright control point colors, clear boundaries, and appropriate sizes). Therefore, the second neural network model can detect and determine whether the extraction result of the first neural network model is an actual control point marker image with control point marker image features, and eliminate some control point marker images with poor image quality due to occlusion, lens shake, shadows, etc., making accurate pricking difficult, thereby improving the accuracy of subsequent pricking.
[0051] The electronic device obtains the shape and position of the control point markers in the image to be punctured, as well as the control point marker image, through the first and second neural network models described above. However, during puncturing, it needs to obtain the position coordinates of key points in the control point marker image. These key point coordinates are the position coordinates of the control points in the control point marker image. For example, when the control point marker is L-shaped, the coordinates of the interior corner points of the L-shaped marker need to be obtained during puncturing. These interior corner points are the key points in the control point marker image. Therefore, the electronic device needs to further process the obtained control point marker image. It should be understood that the selection of the key point position mentioned here is based on the points measured on-site by operators using RTK detection equipment during aerial survey fieldwork. When the point selected by the operator during on-site measurement is an interior corner point, the key point is an interior corner point; when the point selected by the operator during on-site measurement is an exterior corner point, the key point is an exterior corner point. Meanwhile, control point images with different identifiers have different key points. For example, the key point corresponding to the control point of an L-shaped identifier is usually the inner corner of the L-shaped identifier; the key point corresponding to the control point of an X-shaped identifier is usually the center point of the X-shaped identifier.
[0052] After recognizing a control point (CPT) marker image, the electronic device needs to extract the coordinates of key points within the CPT marker image to perform subsequent key point detection tasks. During key point coordinate extraction, the coordinate positions of key points are selected for different CPT markers. Since many CPT markers are often located in complex backgrounds, the background of the CPT marker image is unpredictable. Therefore, traditional computer vision methods struggle to guarantee the accuracy of the extracted key point coordinates. To ensure the robustness of the extracted key point coordinates, a neural network model (also known as a third neural network model) can be used to extract the key point coordinates. It should be understood that the third neural network model used here has the same basic structure as the second neural network model used to identify whether a CPT marker image within a target area is an actual CPT marker image, but the weight parameters used in the specific recognition operation are different. The third neural network model used for keypoint coordinate extraction in control point (CPT) images is obtained by selecting keypoints from CPT images with good imaging quality extracted by the first neural network model as the dataset label file, thus obtaining the training dataset required for the third neural network model, and then training it. Therefore, the third neural network model can obtain the position coordinates of keypoints in the CPT image, and use these keypoint coordinates as the position coordinates of the CPTs in the image to be punctured.
[0053] Figure 3A schematic flowchart illustrating the first neural network model training method provided in this application embodiment is shown below. Figure 3 As shown, the specific steps include S301-S304.
[0054] S301, The electronic device determines the training dataset for training the first neural network model.
[0055] In this embodiment, the electronic device used to train the neural network model can be the same device or a different device as the electronic device used to execute the spike method.
[0056] In this embodiment, the method of introducing a neural network model can achieve more robust target detection. However, the neural network model inevitably requires a large dataset for training. When the electronic device trains the first neural network model, due to the complexity of the detection environment, in order to ensure accurate extraction of control points in complex environments, the training dataset can be selected from training datasets with control point marker features. For example, the electronic device mainly uses training images with bright colors, clear boundaries, and appropriate sizes of control point markers as the training dataset for the first neural network model. At the same time, since some control point markers may not conform to the standard shape or be occluded in the image of the target point when drawing them in the field during aerial surveying operations, the electronic device also uses some training images with small size and poor imaging quality containing control point markers as the training dataset to improve the recall rate and generalization ability of the first neural network model, thereby improving its detection accuracy. The above two different training datasets are collectively referred to as the first training dataset. Figure 4 The image shown is a schematic diagram of the image control points marked in different target regions.
[0057] S302. The electronic device extracts control point identification images from the training dataset based on the first neural network model.
[0058] In some embodiments, the electronic device extracts control point marker images of training images in the training dataset based on an untrained first neural network model to obtain detection values of the control point marker images in the training dataset.
[0059] S303. The electronic device calculates the target loss of the first neural network model when extracting the image of the control point marker through a loss function.
[0060] In some embodiments, the electronic device calibrates the training images in the training dataset to obtain the label values of the control point marker images in the training images, i.e., the true values of the control point marker images in the training images, and the detection values of the control point marker images calculated by the first neural network model. The target loss of the first neural network model in the extraction of control point marker images is calculated by combining the loss function.
[0061] It should be noted that the target loss in the first neural network model can be calculated based on the loss function corresponding to the selected neural network model. For example, if the first neural network model is a YOLO v5s neural network model, the loss function selected when calculating the target loss includes both classification loss function and regression loss function.
[0062] S304. The electronic device trains the first neural network model based on the target loss and the training dataset to obtain a first neural network model that can extract control point marker images.
[0063] In some embodiments, the electronic device inputs the real values of the training images in the training dataset and the calculated target loss into a first neural network model for backpropagation and parameter updates, thereby obtaining a first neural network model capable of extracting control point marker images.
[0064] Figure 5 A schematic flowchart illustrating the second neural network model training method provided in this application embodiment is shown below. Figure 5 As shown, the specific steps include S501-S504.
[0065] S501, The electronic device determines the training dataset for training the second neural network model.
[0066] In some embodiments, since the second neural network model is used to identify whether the control point marker image within the target region is an actual control point marker image, rather than an image containing other feature objects similar to control point markers, such as... Figure 6 The diagram shows images of other feature objects with similar shapes to the control point markers. Therefore, the electronic device uses the false recognitions and negative samples with poor imaging quality (training images with chaotic control point colors, blurred boundaries, and excessively small areas) detected by the first neural network model, along with the same number of positive samples with good imaging quality (training images with bright control point colors, clear boundaries, and appropriate sizes), to create a training dataset for false detections.
[0067] S502, The electronic device recognizes training images in the training dataset based on the second neural network model.
[0068] In some embodiments, the electronic device identifies control point marker images in the training images within the training dataset based on an untrained second neural network model, in order to obtain detection values for the control point marker images in the training dataset.
[0069] S503. The electronic device calculates the target loss of the second neural network model when recognizing control point marker images using a loss function.
[0070] In some embodiments, the electronic device calibrates the training images in the training dataset to obtain the label values of the control point marker images in the training images, i.e., the true values of the control point marker images in the training images, and the detection values calculated by the second neural network model. The target loss of the second neural network model in the recognition of control point marker images is then calculated by combining the loss function.
[0071] It should be understood that in this embodiment, the second neural network model is used to determine whether the detected result is an actual control point marker image, and to remove some control point marker images with poor imaging quality due to occlusion, lens shake, shadows, etc., making it difficult to accurately perform puncturing, thereby improving the accuracy of puncturing. This type of judgment method belongs to the classification problem in deep learning, that is, a yes or no problem. Therefore, a classification loss function can be selected to calculate the target loss.
[0072] It should be noted that in the second neural network model, the detected value or the true value of the control point marker image refers to whether the control point marker image is an actual control point marker image. If it is an actual control point marker image, it is marked as a control point marker image based on the labeling method (such as one-hot annotation, that is, using 0 and 1 to indicate whether it is a target object). If it is not an actual control point marker image, it is marked as a non-control point marker image based on the labeling method.
[0073] S504. The electronic device trains the second neural network model based on the target loss and the training dataset to obtain a second neural network model that can identify whether it is an actual control point marker image.
[0074] In some embodiments, the electronic device inputs the real values of the training images in the training dataset and the calculated target loss into the second neural network model for training. During the training process, the misidentified and poor-quality negative samples are labeled as non-control point marker images, while the same number of good-quality positive samples are labeled as control point marker images, thereby obtaining a second neural network model that can identify whether it is an actual control point marker image.
[0075] Figure 7 A schematic flowchart illustrating the third neural network model training method provided in this application embodiment is shown below. Figure 7As shown, the specific steps include S701-S704.
[0076] S701, The electronic device determines the training dataset for training the third neural network model.
[0077] In this embodiment, the third neural network model is used to extract the position coordinates of key points in the control point marker image. The electronic device uses a high-quality control point marker image detected by the first neural network model (e.g., a positive sample from the training dataset of the first neural network model, i.e., an image of correctly identified control point markers) to annotate the key points in the control point marker image using an annotation tool (such as a Python script developed using the matplotlib library). This results in the training dataset required for the third neural network model used for key point extraction. See [link to documentation]. Figure 9 As shown.
[0078] S702, the electronic device performs key point recognition on training images in the training dataset based on a third neural network model.
[0079] In some embodiments, the electronic device performs keypoint recognition on training images in the training dataset based on an untrained third neural network model to obtain the detection value of keypoints for each training image in the training dataset.
[0080] S703, the electronic device calculates the target loss of the third neural network model when extracting key points using a loss function.
[0081] In some embodiments, the electronic device calibrates the coordinates of key points in the training image to obtain the label values of the key point coordinates in the training image, i.e. the true values of the key points in the training image, and the detection values calculated by the third neural network model. The target loss of the third neural network model in key point recognition is calculated by combining the loss function.
[0082] It should be understood that in this embodiment, the third neural network model is used to extract the position coordinates of key points in the control point marker image. The structure of the third neural network model used here is completely consistent with the structure of the second neural network model. However, since the prediction of key point coordinates in the control point marker image is a regression problem, a regression loss function or a heatmap loss function is selected to calculate the target loss instead of a classification loss function.
[0083] S704, based on the target loss and training dataset, trains the third neural network model to obtain a third neural network model that can extract the position coordinates of key points in the control point marker image.
[0084] In some embodiments, the electronic device inputs the real values of the training images in the training dataset and the calculated target loss into the third neural network model for backpropagation to update the parameters, thereby obtaining a third neural network model capable of extracting the position coordinates of key points in the control point marker image.
[0085] The above describes the training process of each neural network model involved in the embodiments of this application. When applying the trained neural network models to the puncture method provided in the embodiments of this application, please refer to... Figure 8 As shown, it includes the following steps S801-S803.
[0086] S801, the electronic device is based on the first neural network model and performs preliminary extraction of the control point marker image within the target area according to the features of the control point marker image.
[0087] When selecting the first neural network model, to ensure high extraction accuracy while maximizing processing efficiency, the electronic device can choose a highly reliable neural network model. Furthermore, since the control point marker image used as the detection target in this embodiment is relatively simple, the electronic device can choose a lightweight neural network model for target detection. "Lightweight" is determined by the size and computational complexity of the neural network model; for example, SqueezeNet, MobileNet, ShuffleNet, and Xception are all lightweight neural network models. For instance, YOLO is one type of target detection neural network model. YOLO v5, which evolved from YOLO v1 to v4, is a currently widely used lightweight target detection network with fast computation speed. Moreover, YOLO v5 can be further derived into four models—YOLO v5s, YOLO v5m, YOLO v5l, and YOLO v5x—based on the network's depth and width. In this embodiment, to ensure operational efficiency, the electronic device can select the YOLO v5s neural network model with the simplest network structure and train it using the training methods described in steps S301-S304. After training, the electronic device inputs the image to be punctured into the YOLO v5s neural network model, that is, the model extracts control points from the image to be punctured, obtaining several region images including control point markers, such as... Figure 4 As shown.
[0088] S802, the electronic device, based on the second neural network model, identifies the initially extracted control point marker image and finally determines the control point marker image.
[0089] Because the region image containing the control point markers extracted by the first neural network model contains numerous and complex objects, it is prone to false detection and extraction of targets with shapes similar to the control point markers. (See [link to relevant documentation]). Figure 6 As shown. Therefore, the electronic device uses a trained second neural network model that can identify control points to identify the extracted control point marker images, thereby filtering out misidentified control point marker images and irregular, difficult-to-distinguish control point marker images.
[0090] S803, the electronic device is based on the third neural network model, and determines the position coordinates of the control points according to the positional relationship between the control points and the control point identifier image.
[0091] To complete the pricking task, after recognizing the control point marker image, the electronic device extracts the coordinates of key points in the control point marker image based on a trained third neural network model. These key point coordinates are the coordinates of the control points in the image to be pricking. (See [link]). Figure 9 As shown.
[0092] In this embodiment, when the electronic device performs the above-described puncture method, it can display the following: Figure 10 The interface shown. See also Figure 10 As shown, the interface includes the original UAV image in the central view, the image of the target area of the image control point in the upper right view, and the location of the spikes in the lower right view. Figure 10 The small black dots in the image represent the positions of the spikes predicted by the neural network model, and the coordinates of the control points can be obtained based on the positions of these spikes.
[0093] The puncturing method provided in this application embodiment can accurately identify the position coordinates of control points in the image to be punctured, reducing manpower and time consumption, and greatly improving the automation level of aerial surveying processing. This method can meet the construction requirements of various construction units, has a high control point recognition rate in various environments, does not require the creation of specific control points, and is highly practical.
[0094] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0095] Figure 11 This is a schematic diagram of a puncture device provided in an embodiment of this application, as shown below. Figure 11As shown, the device includes: an acquisition unit for acquiring an image of the point to be punctured; a target region determination unit for determining a target region in the image of the point to be punctured, the target region including control points and control point identification images, the control point identification images being used to identify the positions of the control points; and a control point position coordinate determination unit for determining the control point identification images within the target region based on the features of the control point identification images using a neural network model, and determining the position coordinates of the control points in the image of the point to be punctured based on the positional relationship between the control points and the control point identification images.
[0096] Figure 12 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 12 As shown, the electronic device 12 of this embodiment includes: a processor 120, a memory 121, and a computer program 122, such as a puncture program, stored in the memory 121 and executable on the processor 120. When the processor 120 executes the computer program 122, it implements the steps in the various puncture method embodiments described above. Alternatively, when the processor 120 executes the computer program 122, it implements the functions of each module / unit in the various device embodiments described above.
[0097] For example, the computer program 122 may be divided into one or more modules / units, which are stored in the memory 121 and executed by the processor 120 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 122 in the electronic device 12.
[0098] The electronic device 12 can be a tablet computer, desktop computer, laptop computer, handheld computer, cloud server, or other computing device. The electronic device may include, but is not limited to, a processor 120 and a memory 121. Those skilled in the art will understand that... Figure 12 This is merely an example of electronic device 12 and does not constitute a limitation on electronic device 12. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.
[0099] The processor 120 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0100] The memory 121 can be an internal storage unit of the electronic device 12, such as a hard disk or RAM of the electronic device 12. The memory 121 can also be an external storage device of the electronic device 12, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the electronic device 12. Furthermore, the memory 121 can include both internal and external storage units of the electronic device 12. The memory 121 is used to store the computer program and other programs and data required by the electronic device. The memory 121 can also be used to temporarily store data that has been output or will be output.
[0101] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0102] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0103] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0104] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0107] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.
[0108] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for pricking points, characterized in that, The method includes: Obtain the image of the point to be punctured; A target region is determined in the image of the point to be punctured. The target region includes control points and control point identification images. The control point identification images are used to identify the positions of the control points. The image of the control point is determined within the target area by using a neural network model based on the features of the control point identification image, and the position coordinates of the control point in the image of the point to be punctured are determined based on the positional relationship between the control point and the control point identification image. The step of determining the control point marker image within the target area using a neural network model based on the features of the control point marker image, and determining the position coordinates of the control point in the image of the point to be punctured based on the positional relationship between the control point and the control point marker image, includes: Based on the first neural network model, the control point marker image is initially extracted within the target area according to the features of the control point marker image; Based on the second neural network model, the initially extracted control point marker image is identified, and the control point marker image is finally determined; wherein, the second neural network model is trained by selecting the false identification and poor imaging quality negative samples detected by the first neural network model, and the same number of positive samples with good imaging quality, to form a training dataset for false detection; Based on the third neural network model, the position coordinates of the control points are determined according to the positional relationship between the control points and the control point marker images. The third neural network model is obtained by selecting key points of the control point marker images with better imaging quality extracted from the first neural network model as the dataset label file, thus obtaining the training dataset required by the third neural network model, and then training it.
2. The method according to claim 1, characterized in that, Determining the target region in the image of the point to be punctured includes: Obtain the actual position coordinates of the control points and the device position coordinates of the acquisition equipment; Based on the actual location coordinates and the device location coordinates, the target region in the image of the point to be punctured is determined using the collinearity equation. The acquisition device is a device for acquiring the image of the point to be punctured; the device position coordinates are the position coordinates of the acquisition device when acquiring the image of the point to be punctured; the actual position coordinates are the position coordinates of the control point when measured in the field.
3. The method according to claim 1, characterized in that, The neural network model is determined through training using a training dataset and a target loss function; In the first neural network model, the training dataset includes a first training dataset with features of the control point identification image, and the loss function is a classification loss function and a regression loss function. In the second neural network model, the training dataset includes positive samples from the first training dataset and an equal number of negative samples, and the loss function is a classification loss function; In the third neural network model, the training dataset includes positive samples from the first training dataset, and the loss function is a regression loss function or a heatmap loss function.
4. The method according to claim 3, characterized in that, The positive samples are used to finally determine the control point identification image and to calibrate the coordinates of the control points in the control point identification image; The negative samples are used to remove the control point marker images extracted from the first neural network model.
5. The method according to claim 1, characterized in that, The control point marker image includes either an L-shaped marker image or an X-shaped marker image.
6. The method according to claim 5, characterized in that, When the image control point is an L-shaped image, the image control point is located at the inner corner of the L-shaped image; When the control point marker image is an X-shaped marker image, the control point is located at the center point of the X-shaped marker image.
7. A pricking device, characterized in that, The device includes: The acquisition unit is used to acquire the image of the point to be punctured; A target region determination unit is used to determine a target region in the image of the point to be punctured, the target region including control points and control point identification images, the control point identification images being used to identify the positions of the control points; The control point position coordinate determination unit is used to determine the control point identification image in the target area based on the features of the control point identification image through a neural network model, and to determine the position coordinates of the control point in the image of the point to be punctured based on the positional relationship between the control point and the control point identification image. The image control point position coordinate determination unit is used for: Based on the first neural network model, the control point marker image is initially extracted within the target area according to the features of the control point marker image; Based on the second neural network model, the initially extracted control point marker image is identified, and the control point marker image is finally determined; wherein, the second neural network model is trained by selecting the false identification and poor imaging quality negative samples detected by the first neural network model, and the same number of positive samples with good imaging quality, to form a training dataset for false detection; Based on the third neural network model, the position coordinates of the control points are determined according to the positional relationship between the control points and the control point marker images. The third neural network model is obtained by selecting key points of the control point marker images with better imaging quality extracted from the first neural network model as the dataset label file, thus obtaining the training dataset required by the third neural network model, and then training it.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.