A monocular camera target detection and spatial positioning method based on three-dimensional virtual geographic scene

Abstract

The present invention discloses a method for target detection and spatial positioning of a monocular camera based on a three-dimensional virtual geographic scene, which realizes target object detection and geographic space coordinate solution under the condition of a monocular camera. It includes the following steps: S1 constructs a real-scene fusion picture based on the three-dimensional virtual geographic scene to realize the fusion display of video and three-dimensional scene. S2 then performs target detection of the real-scene fusion image containing video based on a deep learning model to obtain the pixel value of the detected target. S3 finally realizes the conversion of pixel coordinates to geographic space coordinates based on the geospatial coordinate solution method of the three-dimensional virtual geographic scene. The present invention uses a monocular camera for target monitoring and spatial positioning, has low requirements for monitoring equipment, and uses a single sensor with its unique advantages in data processing and equipment cost. It can be applied to the monitoring and supervision of suspected illegal signs of natural resources in tower base videos, and can be used for three-dimensional overlay spatial analysis with planning red lines, so as to quickly discover suspected illegal land use signs and effectively improve the automation efficiency of monitoring and supervision of suspected illegal signs.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional target detection and spatial positioning, and in particular to a monocular camera target detection and spatial positioning method based on a three-dimensional virtual geographic scene. Background Art

[0002] The rapid construction of intelligent geospatial applications such as smart cities, perceived land, and spatiotemporal information cloud platforms has promoted the deep integration of artificial intelligence technology and geographic applications. Among them, three-dimensional target detection and spatial positioning have become important directions. However, various traditional three-dimensional target detection methods mainly rely on depth cameras or multi-eye vision, which have high requirements for hardware conditions and are difficult to be promoted to smart cities, perceived land, and spatiotemporal information cloud platform applications. How to detect the target object and calculate its geospatial coordinates under the condition of a monocular camera has largely constrained the target detection algorithm from moving towards intelligent geospatial analysis and application. Existing three-dimensional target detection and spatial positioning both rely on multi-eye cameras or radar point clouds to obtain the depth information of the target object, thereby realizing positioning calculation. This method has high requirements for hardware conditions and is difficult to be applied to the integration of monocular cameras to realize three-dimensional target detection and spatial positioning applications, such as: relying on traffic control cameras to locate the position of passing vehicles in smart city systems. With the development of three-dimensional geographic information technology, the physical world can gradually be expressed by three-dimensional geographic scene twins, that is, a three-dimensional virtual geographic scene is obtained. In a three-dimensional virtual geographic scene, the terrain, landforms, and architectural features of the real physical world can be simulated, and it has multi-LOD features. The three-dimensional virtual geographic scene is a new source of information. It provides a new concept to overcome the difficulty of spatial positioning of target objects that cannot be achieved by monocular cameras. It can approximately overcome the problem that visual depth information cannot be calculated. Through the target detection and spatial positioning methods in the three-dimensional virtual geographic scene, the combination of artificial intelligence target detection technology, three-dimensional geographic information technology, and video surveillance technology is realized, breaking through the difficulty of spatial positioning of target objects that cannot be achieved by monocular cameras. It can effectively solve the "difficult positioning" problem of target objects in intelligent geographic space applications such as existing smart cities, perception of land, and spatiotemporal information cloud platforms, and has great practical value. Summary of the invention

[0003] 1. Technical problems solved

[0004] In order to solve the problem of target detection and spatial positioning in a three-dimensional virtual geographic scene, the present invention provides a monocular camera target detection and spatial positioning method based on a three-dimensional virtual geographic scene. The method first constructs a real-scene fusion scene display through the three-dimensional virtual geographic scene and real-time monitoring video images, obtains a real-scene fusion scene image, then performs target detection on the real-scene fusion image based on a deep learning model, obtains pixel values ​​of the detected targets, and finally realizes the conversion of pixel coordinates to geospatial coordinates according to the target pixel values ​​using the geospatial coordinate solution method of the three-dimensional virtual geographic scene.

[0005] 2. Technical solution

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0007] A monocular camera target detection and spatial positioning method based on a three-dimensional virtual geographic scene comprises the following steps:

[0008] 1. Construct real-scene fusion based on three-dimensional virtual geographic scenes to achieve integrated display of video and three-dimensional scenes.

[0009] (1) Outdoor scene modeling: construct a large-scale virtual 3D scene based on the required DEM data, image data and building model data. Indoor model construction: use indoor real estate surveying and mapping data to build a 3dmax model. At the same time, based on laser scanning technology, build an indoor laser point cloud model. Adjust the accuracy of the indoor model according to the laser point cloud to construct a high-precision indoor 3D virtual scene.

[0010] (2) A 3D scene virtual camera in a 3D scene has internal orientation elements and external orientation elements, of which the internal orientation elements are usually known, while the external orientation elements need to be calibrated. The external orientation elements of the 3D scene virtual camera are obtained by performing feature matching between the video image and the 3D scene image under the same field of view, and the camera is calibrated according to the position of the 3D virtual camera in the 3D scene. The virtual camera is initialized using the calibrated external orientation elements of the virtual camera.

[0011] (3) Through the real-time RTSP video stream provided by the monitoring equipment manufacturer, the video stream is decoded to obtain video frame images of different time periods. The image is projected according to the current camera posture of the 3D virtual scene to achieve the fusion of video and 3D scene, including the real scene fusion screen based on terrain image and 3D model scene. Finally, through the scene output, the video containing the target object and the fusion image of the 3D scene are obtained as the input data of the target detection model.

[0012] 2. Perform target detection on real-scene fusion images containing videos based on the deep learning model to obtain the pixel values ​​of the detected targets.

[0013] (1) To define the samples for target detection, it is necessary to clearly explain the research object and its classification, analyze and organize its recognition rules, and form a unified understanding of the research object. First, it is necessary to determine the type of the detection target and define it as positive sample data. At the same time, a large number of images that are similar or opposite to the detection target are collected, and negative sample data is determined by classifying the types.

[0014] (2) Preprocessing of training sample data to solve problems such as inconsistent data formats or data that do not meet requirements, high data noise, poor data credibility, low data quality, and confusing data naming.

[0015] ① Data standardization: The sample data input into the deep convolutional neural network for training is often massive, and there may be various problems due to different collection times, methods or paths, which will undoubtedly affect the subsequent training. Therefore, on the basis of eliminating erroneous data, it is necessary to uniformly organize and manage the data, uniformly standardize the batches of sample images, and clarify the attribute information of the data.

[0016] ② Data denoising: Data is ultimately input into the network in the form of labels. These labels are generally collected manually by a large number of people. In addition to the phenomenon of mistypes, it is impossible to check and correct them one by one. However, the quality of these labels is crucial to the effect of model learning, but it is also one of the noises that must be faced.

[0017] ③Data enhancement: In order to prevent problems such as sample non-convergence during model training or poor recognition ability of the trained model due to limited data size, the data enhancement methods commonly used in the field of deep learning can be used to amplify samples based on existing data. Common methods include flipping the original image, random pruning, color jittering, translation, and scale transformation.

[0018] ④ Data normalization: Data normalization is to scale the value of a column of numerical features in the training data set to between 0 and 1. Its purpose is to make the features of different dimensions have a certain degree of numerical comparability, while speeding up the speed of gradient descent to find the optimal solution, greatly improving the accuracy of classification.

[0019] ⑤ Label collection: Different deep learning models have different data formats for input into the network during training. The data required for YOLO model training are image (original image) and annotation (image label), which correspond to image format and XML file format respectively, and are obtained by manually collecting labels.

[0020] (3) Target detection model training: Use the Tensorflow deep learning development framework to build the darkflow geographic target recognition and spatial positioning system. The geographic target recognition stage is mainly based on the mainstream target recognition model YOLO training.

[0021] During the model training process, the parameters of the model network need to be constantly adjusted, and the basis for adjusting the parameters is mainly some parameters output by the model training process. After the training of a batch is completed, the YOLO model conducts a comprehensive analysis based on all the training conditions of the batch to obtain the comprehensive training results after this batch. The model training process takes a lot of time. In order to achieve model convergence as soon as possible and prevent over-training, according to the output parameters of the above model training process, it is necessary to adjust the important parameters in the network in a timely manner. The important parameters and their adjustment strategies are summarized as follows:

[0022] ① Learning rate: The learning rate combined with the gradient descent algorithm is mainly used to achieve rapid convergence of the model. The learning rate can be appropriately reduced when training diverges.

[0023] ② Batch: The number of images sent to the network in each iteration is called a batch. This parameter enables the network to complete a round of prediction with fewer iterations. Under the premise of a fixed maximum number of iterations, increasing the batch value will prolong the training time, but it can better find the direction of model gradient descent. Although increasing the batch is conducive to improving memory utilization, it will also cause the problem of insufficient video memory. If the video memory allows, if a NAN error prompt appears during the training process, the batch size can be appropriately increased, and the momentum parameter can be adjusted to 0.99. Usually, the selection of this value requires repeated experiments. Too large or too small is not conducive to the effective training of the model. If it is too small, it is easy for the training to not converge, and if it is too large, it will fall into a local optimum.

[0024] ③ Subdivision: Subdivision and batch usually appear in pairs, so that each batch of images is first divided into subdivisions, then fed into the network one by one, and finally all the subdivisions are packaged together into one iteration. This helps reduce the usage of video memory.

[0025] ④Learning strategy (step, scales): step represents the number of training steps, scales represents the ratio of change of learning rate, and the two are used in combination.

[0026] ⑤ Random parameter (random): The optional values ​​of random are 1 and 0, representing a switch. When the value is set to 1, each batch of images will be randomly resized to 320-640 (integer multiples of 32) size and input into the model during training. If it is set to 0, it means that all images are adjusted to the default size and input into the network.

[0027] (4) Use the model to perform target detection and output the results. Input the video image into the trained model for target detection. Use rectangular boxes of different colors to mark the detected targets, and calculate the pixel value range of the four sides of the rectangular box. Finally, return the marked image and pixel value as the result.

[0028] 3. The geospatial coordinate solution method based on the three-dimensional virtual geographic scene realizes the conversion of pixel coordinates to geospatial coordinates.

[0029] (1) Solving the inverse perspective transformation matrix

[0030] ① Define the coordinate system. Stereoscopic vision and other directions often involve four coordinate systems: world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system.

[0031] ② Convert pixel coordinate system to image coordinate system. Both pixel coordinate system and image coordinate system are on the imaging plane, but their origin and measurement units are different. The origin of image coordinate system is the intersection of camera optical axis and imaging plane. The unit of image coordinate system is mm, which is a physical unit, while the unit of pixel coordinate system is pixel. We usually describe a pixel point by several rows and columns. dx and dy indicate how many mm each column and row represent, that is, 1pixel = dx mm.

[0032] ③Convert the image coordinate system to the camera coordinate system. The camera coordinate system to the image coordinate system is a perspective projection relationship, from 2D to 3D. At this time, the unit of the projection point p is still mm, not pixel, and needs to be further converted to the pixel coordinate system.

[0033] ④Convert the camera coordinate system to the world coordinate system. c -X c Y c Z c →O w -X w Y w Z w , which belongs to rigid body transformation, that is, the object will not be deformed and only needs to be transformed through the rotation matrix R and the offset vector T.

[0034] The above four steps give the total formula from the world coordinate system to the pixel coordinate system:

[0035]

[0036] The rotation matrix R is:

[0037]

[0038] The offset vector T is:

[0039]

[0040] (2) According to the inverse perspective transformation matrix, the world coordinates of the target point are calculated by combining the pixel values ​​of the target detection results and the virtual camera extrinsic parameters obtained by constructing the three-dimensional real-scene fusion scene, which are the geographic space coordinates of the target point. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a technical flow chart of the present invention.

[0042] Figure 2 It is a schematic diagram of target detection and spatial positioning in the present invention.

[0043] Figure 3 It is a schematic diagram of the exterior orientation line elements of the virtual camera in the present invention.

[0044] Figure 4 It is a schematic diagram of projecting a video in a three-dimensional scene in the present invention.

[0045] Figure 5 This is the naming convention for the standardized sample data in the present invention.

[0046] Figure 6 This is a schematic diagram of the target detection principle based on YOLO in the present invention.

[0047] Figure 7 It is a schematic diagram of coordinate system conversion based on inverse perspective transformation in the present invention.

[0048] Figure 8 It is a schematic diagram of the coordinate axis rotation transformation in the present invention.

[0049] Fig. 9 This is a diagram showing the effect of target detection and spatial positioning based on a monocular camera in the present invention. DETAILED DESCRIPTION

[0050] In order to make the implementation purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the implementation of the present invention will be clearly and completely described below in conjunction with the drawings in the implementation of the present invention. Obviously, the described implementation is a part of the implementation of the present invention, not all the implementations. Based on the implementation of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Combined with the technical flow chart, a monocular camera target detection and spatial positioning method based on a three-dimensional virtual geographic scene includes the following steps:

[0052] 1. Object detection and spatial positioning in three-dimensional scenes belong to the field of three-dimensional reconstruction. Its theory and technology are one of the most important hot issues in computer vision. Three-dimensional reconstruction based on a single image is difficult to achieve the expected effect due to the lack of sufficient geometric information. Therefore, this paper provides the geometric information of a single image by constructing a three-dimensional virtual scene that integrates a three-dimensional model with real video images.

[0053] (1) Construction of three-dimensional virtual scenes includes terrain and models. For outdoor scenes, large-scale virtual three-dimensional scenes are constructed based on the required DEM data, image data and building model data. For indoor model construction, 3dmax models are established using indoor real estate surveying and mapping data. At the same time, indoor laser point cloud models are constructed based on laser scanning technology. The accuracy of the indoor model is adjusted according to the laser point cloud to construct a high-precision indoor three-dimensional virtual scene.

[0054] (2) A 3D virtual camera in a 3D scene has internal and external orientation elements, of which the internal elements are usually known, while the external elements need to be calibrated. The external orientation elements are mainly about the orientation relationship between the photographic film and the object space coordinate system. The data that determines the geometric relationship between the camera's related coordinate system including the imaging plane and the object space coordinate system at the moment of photography is called the external orientation element of the film. The external orientation elements of the film are divided into line elements and angle elements, a total of 6. The line element is the object space coordinate of the camera's photography center, generally defined as (X S ,Y S ,Z S )( Figure 3 ). The other three are angular elements, which are used to determine the attitude of the image in the object space coordinate system. There are three typical ways to express the angular orientation of the image: Av-κ rotation system, Corner system, Corner system. By matching the video image with the 3D scene image under the same field of view, the exterior orientation elements of the 3D scene virtual camera are obtained, and the camera is calibrated according to the position of the 3D virtual camera in the 3D scene. At the same time, the virtual camera is initialized through the calibrated exterior orientation elements of the virtual camera to obtain the real posture of the camera.

[0055] (3) De-frame the video stream through the real-time RTSP video stream provided by the monitoring equipment manufacturer to obtain video frame images of different time periods. Project the image according to the current camera posture of the 3D virtual scene to achieve the fusion of video and 3D scene (such as Figure 4), including real-scene fusion images based on terrain images and 3D model scenes. Fusion based on terrain images matches the obvious features of images and videos to obtain the fusion effect of video sticking to the ground. Fusion based on 3D model scenes is to fit the model in the video and the scene to obtain the fusion effect of video sticking to the model. Finally, through the scene output, the image of the fusion of the video containing the target and the 3D scene is obtained as the input data of the target detection model.

[0056] 2. Based on the deep learning model, target detection of real-scene fusion images with videos is performed to obtain the pixel values ​​of the detected targets. Before collecting labels for image data, it is necessary to first clearly explain the research object and its classification, analyze and organize its recognition rules, and form a unified understanding of the research object. Through the acquired image data, positive and negative samples are divided, and the characteristics of the positive samples are described in detail.

[0057] (1) Definition of positive and negative samples

[0058] The most directly related factor to the model detection performance is the sample definition, that is, first of all, it is necessary to define the positive and negative samples of the identified target, so that the labeling personnel have a unified understanding of the collected targets and reduce the collection error. First, it is necessary to determine the type of the detection target and stipulate that it is a positive sample. At the same time, a large number of images similar to or opposite to the detection target are collected, and they are determined as negative samples through the delineation of the type. The number of labels of the collected data is estimated to determine whether the scale of model training convergence has been reached.

[0059] (2) Training sample data preprocessing

[0060] The scale and diversity of training samples are important factors that affect the recognition accuracy of convolutional neural network models. In order to train a deep learning model based on the YOLO algorithm that can accurately detect and identify certain targets in images, the training sample data needs to be preprocessed. Data preprocessing is any process that performs certain operations on raw data to prepare for the next step. According to experience, the problems with raw data generally include: inconsistent data format or does not meet the requirements, large data noise, poor data credibility, low data quality, confusing data naming, etc. Therefore, using the data preprocessing method in the field of deep learning, the sample data is preprocessed as follows:

[0061] ① Data standardization: The sample data input into the deep convolutional neural network for training is often massive data, which may have various problems due to different collection times, methods or paths, which will undoubtedly affect the subsequent training. Therefore, on the basis of eliminating erroneous data, it is necessary to organize and manage the data in a unified manner, standardize the batches of sample images, and clarify the attribute information of the data. The standardized data format is jpg, the number of channels is 3, the bit depth is 24, and the data specification naming is as follows: Figure 5 As shown (the xml file name corresponds to the jpg name), it can be effectively accessed by the neural network.

[0062] ② Data denoising: Data is ultimately input into the network in the form of labels. These labels are generally collected manually by a large number of people. In addition to the phenomenon of mistypes, it is impossible to check and correct them one by one. However, the quality of these labels is crucial to the effect of model learning, but it is also one of the noises that must be faced.

[0063] ③Data enhancement: In order to prevent problems such as sample non-convergence during model training or poor recognition ability of the trained model due to limited data size, the data enhancement methods commonly used in the field of deep learning can be used to amplify samples based on existing data. Common methods include flipping the original image, random pruning, color jittering, translation, and scale transformation.

[0064] ④ Data normalization: Data normalization is to scale the value of a column of numerical features in the training data set to between 0 and 1. Its purpose is to make the features of different dimensions comparable in value, while speeding up the gradient descent to find the optimal solution, greatly improving the accuracy of classification. After the data label collection is completed and before the deep learning model is trained, the original representation of the center point of the rectangular box (x c ,y c ,w,h) are normalized to between 0 and 1. Assuming that the coordinates of the upper left corner and lower right corner of the identification box of a construction site are (x1,y1) and (x2,y2) respectively, and the width and height of the image are w and h respectively, the normalized content is:

[0065] Normalized center point x c Coordinate calculation formula: ((x2+x1) / 2.0) / w

[0066] Normalized center point y c Coordinate calculation formula: ((y2+y1) / 2.0) / h

[0067] The calculation formula for the normalized target box width is: (x2-x1) / w

[0068] Normalized target box height calculation formula: (y2-y1) / h

[0069] (5) Label collection: Different deep learning models have different data formats for input into the network during training. The data required for YOLO model training are image (original image) and annotation (image label), which correspond to image format and XML file format respectively, and are obtained one by one through manual label collection. This manual label collection is mainly based on the above-mentioned construction land example to collect targets. The LabelImg tool is used to extract the real target information for each image of the scale data, that is, to obtain the pixel coordinates of the upper left and lower right corners of the rectangular box of the construction land target or the excavator target, and at the same time mark the target category, and finally obtain the XML file corresponding to each image. The content example is shown in Table 1.

[0070] Table 1 Model training output parameters

[0071]

[0072] (3) Target detection model training

[0073] Using the Tensorflow deep learning development framework, we built the darkflow geographic target recognition and spatial positioning system. The geographic target recognition stage is mainly based on the mainstream target recognition model YOLO training (such as Figure 6 ). When training the construction land model, the parameters that need to be input include network structure, initial weights and training sample files. The command is as follows:

[0074] Python flow --model cfg\yolo-2c.cfg --load bin\yolo.weights --train --gpu 1.0–dataset.\training\images --annotation.\training\annotation

[0075] flow is the program entry for model training. The gpu enabling parameter is 1.0. dataset and annotation are used to load the preprocessed monitoring image data and the xml data obtained by collecting labels, respectively. model is used to load the model network, and load is used to load the initial training weights.

[0076] On the basis of the identification of the target types (2) and type definitions (construction, excavator), it is necessary to modify some network parameters of the YOLO model according to the needs, and reassign the number of types (classes) in the network construction and the number of output results (filters) according to the label categories defined before model training. The model training also needs to read the defined label file, so the label needs to be changed in the labels.txt label file. During model training, in order to make the model converge as quickly as possible, the model is generally given an initial weight, which can be obtained from the YOLO official website according to the network version.

[0077] During the model training process, the parameters of the model network need to be adjusted continuously, and the basis for adjusting the parameters is mainly some parameters output during the model training process. After a batch of training is completed, the YOLO model conducts a comprehensive analysis based on all the training conditions of the batch to obtain the comprehensive training results after this batch. For example, a statement in the training process is:

[0078] 17889:0.457854,0.432366avg,0.000100rate,2.545074seconds, 468640images

[0079] The parameters and their meanings in this statement are shown in Table 2. The loss and avg loss contained in the statement are the values ​​obtained by comprehensively calculating the information loss of the center point coordinates, bounding box width and height, recognition confidence score, and target recognition type predicted during the training process, which are called loss function and average loss function respectively. The two can be used as the most important parameters for analyzing the effect of this training. The smaller their values, the better. It is generally believed that the model is basically converged when the average loss function is lower than 1.0. If the training samples are sufficient and the parameters are adjusted reasonably, lower information loss can be achieved and better recognition effect can be obtained.

[0080] Table 2 Model training output parameters

[0081]

[0082] The model training process takes a lot of time. In order to achieve model convergence as soon as possible and prevent overtraining, it is necessary to adjust the important parameters in the network in a timely manner according to the output parameters of the above model training process. The important parameters and their adjustment strategies are summarized as follows:

[0083] ① Learning rate: The learning rate combined with the gradient descent algorithm is mainly used to achieve rapid convergence of the model. The learning rate can be appropriately reduced when the training diverges. For example, during training, the parameter is initialized to 0.1. As the loss value of the model continues to decrease, the model begins to converge. When the loss value fluctuates up and down or fluctuates slightly, the learning rate is adjusted to 0.005. When the loss value drops below 1.0 and the fluctuation is small, the model basically converges.

[0084] ② Batch: The number of images sent to the network in each iteration is called a batch. This parameter enables the network to complete a round of prediction with fewer iterations. Under the premise of a fixed maximum number of iterations, increasing the batch value will prolong the training time, but it can better find the direction of model gradient descent. Although increasing the batch is conducive to improving memory utilization, it will also cause the problem of insufficient video memory. If the video memory allows, if a NAN error prompt appears during the training process, the batch size can be appropriately increased, and the momentum parameter can be adjusted to 0.99. Usually, the selection of this value requires repeated experiments. Too large or too small is not conducive to the effective training of the model. If it is too small, it is easy for the training to not converge, and if it is too large, it will fall into a local optimum.

[0085] ③Subdivision: Subdivision and batch usually appear in pairs, so that each batch of images is first divided into subdivision parts, and then sent to the network one by one, and finally all parts are packaged together into one iteration. This helps to reduce the usage of video memory. For example, if this parameter is 2, it means that half of the batch of images are selected and sent to the network at a time.

[0086] ④ Learning strategy (step, scales): step indicates the number of training steps, scales indicates the ratio of learning rate change, and the two are used in combination. In this model training process, the initial value of learning rate is set to 0.001, the step is set to 1000, 8000, 14000, and the scales are 10, .1, .1, which means that the learning rate is the initial 0.001 during 0-1000 iterations. During 1000-8000 iterations, the learning rate is automatically calculated to be 10 times the initial value, that is, 0.01. During 8000-14000 iterations, the learning rate is 0.1 times the current value, that is, 0.001. During 14000 to the maximum iteration, the learning rate is adjusted to 0.1 times the current value, that is, 0.0001. As the number of iterations increases, the method of reducing the learning rate can make the model learn more effectively, that is, better reduce the information loss during training.

[0087] ⑤ Random parameter (random): The optional values ​​of random are 1 and 0, representing a switch. When this value is set to 1, each batch of images will be randomly adjusted to 320-640 (integer multiples of 32) size and input into the model during training. If it is set to 0, all images will be adjusted to the default size and input into the network. The network used in this training process is the Darknet-19 network of YOLOv2, and the image size input into the network is 608*608.

[0088] (4) Use the model to perform target detection and output the results. Input the video image into the trained model for target detection. Use rectangular boxes of different colors to mark the detected targets, and calculate the pixel value range of the four sides of the rectangular box. Finally, return the marked image and pixel value as the result.

[0089] 3. The geospatial coordinate solution method based on the three-dimensional virtual geographic scene includes solving the inverse perspective transformation matrix, and realizing the conversion from the pixel coordinate system to the world coordinate system through the inverse perspective transformation matrix. The pixel coordinates in the two-dimensional screen image can be converted into geospatial coordinates and finally applied.

[0090] (1) Solving the inverse perspective transformation matrix

[0091] ① Define the coordinate system. Stereoscopic vision and other directions often involve four coordinate systems (such as Figure 7 (a)): World coordinate system, camera coordinate system, image coordinate system, pixel coordinate system. The four major coordinate systems are introduced as follows: ① World coordinate system: w -X w Y w Z w , with the center of the circle with a small dot as the origin O W , X W Horizontal right, Y W Down, Z W Determined by the right-hand rule. ② Camera coordinate system: O c -X c Y c Z c , with the optical center of the camera as the origin, the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image, and the Z-axis is the optical axis of the camera. ③ Image coordinate system: o-xy, with the center of the CCD image plane as the coordinate origin, the X-axis and Y-axis are parallel to the two vertical edges of the image plane. ④ Pixel coordinate system: uv, with the upper left corner vertex of the CCD image plane as the origin, the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system respectively.

[0092] ② Convert pixel coordinate system to image coordinate system. Both pixel coordinate system and image coordinate system are on the imaging plane, but their origins and measurement units are different. The origin of the image coordinate system is the intersection of the camera optical axis and the imaging plane (e.g. Figure 7 (b)). The unit of the image coordinate system is mm, which is a physical unit, while the unit of the pixel coordinate system is pixel. We usually describe a pixel by several rows and columns. dx and dy indicate how many mm each column and row represent respectively, that is, 1pixel = dxmm.

[0093]

[0094] ③Convert the image coordinate system to the camera coordinate system. The camera coordinate system to the image coordinate system is a perspective projection relationship, from 2D to 3D. At this time, the unit of the projection point p is still mm, not pixel, and needs to be further converted to the pixel coordinate system, such as Figure 7 As shown in (c), the following relationship can be obtained:

[0095]

[0096] After finishing, we can get:

[0097]

[0098]

[0099]

[0100] ④Convert the camera coordinate system to the world coordinate system. c -X c Y c Z c →O w -X w Y w Z w , which belongs to rigid body transformation, that is, the object will not be deformed, and only needs to be transformed through the rotation matrix R and the offset vector T. From the camera coordinate system to the world coordinate system, rotation and translation are involved (in fact, all movements can also be described by rotation matrices and translation vectors). Rotating different angles around different coordinate axes will obtain the corresponding rotation matrix, such as Figure 8 As shown, the schematic diagram of rotation θ around the Z axis:

[0101]

[0102]

[0103] Similarly, rotation around the X and Y axes And, we can get:

[0104]

[0105]

[0106] So we can get the rotation matrix R = R1R2R3, such as Figure 7 As shown in (d), the position of point P in the world coordinate system (X w ,Y w ,Z w ):

[0107]

[0108]

[0109]

[0110] The above four steps give the total formula from the world coordinate system to the pixel coordinate system:

[0111]

[0112] The rotation matrix R is:

[0113]

[0114] The offset vector T is:

[0115]

[0116] (3) According to the inverse perspective transformation matrix, the world coordinates of the target point are calculated by combining the pixel values ​​of the target detection results and the virtual camera extrinsic parameters obtained by constructing the three-dimensional real-scene fusion scene, which are the geographic space coordinates of the target point.

[0117] In summary, the present invention provides a method for target detection and spatial positioning of a monocular camera based on a three-dimensional virtual geographic scene. The method first constructs a three-dimensional virtual geographic scene, integrates multiple three-dimensional data models, including DEM data, image data, and indoor and outdoor three-dimensional model data, and then calibrates the three-dimensional virtual camera, initializes the virtual camera using the camera's external orientation elements, builds a real-scene fusion scene display according to the real-time monitoring video screen, obtains the real-scene fusion scene image, then trains deep learning samples, builds a target detection YOLO model, performs target detection on the input real-scene fusion image, obtains and outputs the pixel value of the detected target, and finally, according to the output target pixel value, converts the pixel coordinates to the geographic space coordinates through inverse perspective change, thereby realizing monocular camera target detection and spatial positioning based on the three-dimensional virtual geographic scene.

[0118] Example

[0119] In order to reflect the practical value of the method in this paper, the tower base video monitoring of suspected illegal construction in the field of natural resource monitoring and supervision is used as the basic scene of the experiment. The Jiangxinzhou area of ​​Nanjing is selected as the experimental area, and the tower camera is selected as the basic experimental video source. The basic form of constructing a three-dimensional virtual geographic environment is to construct a three-dimensional virtual geographic scene based on the 4D product method. The input data includes NASA Earth Data 12.5M spatial resolution DEM data, and high-resolution remote sensing image data selects Tiandi Map remote sensing image data. The location of the camera is collected using surveying and mapping methods, and the data is imported into the SuperMap platform. The video screen is analyzed and integrated with the three-dimensional scene to construct a three-dimensional real scene fusion scene.

[0120] This example realizes target recognition and spatial positioning of tower base monitoring images. The research application data is tower base monitoring images. The signs of suspected illegal construction are defined as: excavators, construction vehicles, construction materials, etc., which are used as positive samples for model training. By training the samples, a target detection model for construction land based on YOLO is obtained. According to the model, the target object is detected and the scene pixel value is output. Finally, the inverse perspective matrix is ​​used to solve the spatial position to realize the spatial positioning of the construction vehicle. The results are as follows Fig. 9 As shown, the muck truck is identified and located. In this example, the camera internal orientation elements and the video retrieval URL address are provided by the video manufacturer.

[0121] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A monocular camera target detection and spatial positioning method based on a three-dimensional virtual geographic scene, characterized in that: The steps include: S1 builds real-scene fusion based on 3D virtual geographic scenes, realizing the fusion display of video and 3D scenes; (1) Constructing a three-dimensional virtual geographic scene through 4D products or three-dimensional modeling methods. The scene data includes image data, DEM data, building model data and other landscape data. The building model data is 3DMAX model data or oblique photogrammetry data. The indoor 3DMAX model can be calibrated for accuracy based on laser point cloud data. (2) Setting the posture of the virtual camera in the three-dimensional virtual geographic scene and initializing the virtual camera by monitoring the geographic space coordinates of the device and the virtual camera viewing angle parameters, including the horizontal viewing angle, the numerical viewing angle, the horizontal field of view angle, and the vertical field of view angle; (3) Obtain real-time RTSP video stream data through monitoring equipment, deframe the video stream data, obtain a single-frame video image, project the single-frame image captured by a single-sided camera on a three-dimensional virtual geographic scene, and construct a real-scene fusion image; S2 performs target detection on real-scene fusion images containing videos based on a deep learning model and obtains the pixel value of the detected target. (1) Define the samples for target detection, determine the positive and negative samples for target detection training, and perform data preprocessing for the training samples, including data standardization, data noise reduction, data enhancement, and data normalization, and finally collect sample labels; (2) The model training process takes a lot of time. In order to achieve model convergence as quickly as possible and prevent overtraining, it is necessary to adjust important parameters in the network in a timely manner according to the output parameters of the above model training process. Important parameters and their adjustment strategies include learning rate, batch size, segmentation, learning strategy and random parameters. (3) Input an image of a real-scene fusion scene, use the trained model to perform target detection, use rectangular boxes of different colors to mark the detected targets, and calculate the pixel value range of the four sides of the rectangular box. Finally, the marked image and pixel value are returned as the result; S3 realizes the conversion from pixel coordinates to geospatial coordinates based on the geospatial coordinate solution method of three-dimensional virtual geographic scene. (1) First, define the world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system respectively. Then, according to the coordinate system conversion method in the camera imaging principle, use the inverse perspective transformation matrix to convert the pixel coordinate system to the world coordinate system. (2) The world coordinates of the target point are calculated using the pixel values ​​of the target detection results and the external orientation elements of the virtual camera obtained by constructing a three-dimensional real-scene fusion scene, which are the geographic space coordinates of the target point.

2. The method for monocular camera target detection and spatial positioning based on a three-dimensional virtual geographic scene according to claim 1, characterized in that: Based on the scene screenshots of real-scene fusion, the target detection model is trained through the YOLO algorithm, and the box selection detection results and their pixel values ​​are output.

Images