Method and device for presenting multi-target motion positions in three-dimensional real scene in real time

Abstract

The invention discloses a method and device for presenting multi-target motion positions in real time in a three-dimensional real scene, and the method comprises the steps: carrying out the sparse point cloud reconstruction of an image, and obtaining a sparse point cloud; performing dense point cloud reconstruction based on the sparse point cloud to obtain a three-dimensional real scene; detecting and tracking a plurality of moving targets in the image in real time; and positioning the world coordinates of each moving target, and presenting the world coordinates in a three-dimensional real scene. According to the invention, a three-dimensional live-action construction technology, a multi-moving target intelligent identification and tracking technology and a target intelligent real-time positioning technology are creatively integrated to construct a three-dimensional live-action, and high-precision coordinates of moving targets in a physical scene are dynamically solved in real time and are synchronously presented in the three-dimensional live-action. The method can be applied to a combat scene, the coordinate positions of moving targets such as important concerned personnel and vehicles are intelligently solved in real time, the coordinate positions are presented to a three-dimensional live-action command system, the field situation sensing ability is improved, and commanders are assisted in conducting command decisions.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a method, device, electronic device and computer-readable medium for presenting the moving positions of multiple targets in real time in a three-dimensional real scene. Background Art

[0002] In combat scenarios, it is necessary to calculate the coordinate positions of moving targets such as personnel and vehicles in real time and present them to the 3D real-scene command system, thereby improving the on-site situation awareness and effectively assisting commanders in making command decisions. However, at present, drone shooting platforms can only construct static 3D real scenes, and are still unable to construct dynamic 3D real scenes and present the dynamic target positions and deployment information in 3D real scenes in real time. Summary of the invention

[0003] In view of this, the present invention provides a method, device, electronic device and computer-readable medium for presenting the moving positions of multiple targets in real time in a three-dimensional real scene, in order to at least partially solve at least one of the above-mentioned technical problems.

[0004] In order to solve the above technical problems, the first aspect of the present invention provides a method for presenting the moving positions of multiple targets in real time in a three-dimensional real scene, the method comprising:

[0005] Perform sparse point cloud reconstruction on the image to obtain a sparse point cloud;

[0006] Performing dense point cloud reconstruction based on the sparse point cloud to obtain a three-dimensional real scene;

[0007] Detect and track multiple moving targets in images in real time;

[0008] The world coordinates of each moving target are located and presented in the three-dimensional real scene.

[0009] According to a preferred embodiment of the present invention, reconstructing a sparse point cloud from a captured image to obtain a sparse point cloud comprises:

[0010] Detect and match feature points in the image to obtain camera parameters and feature point coordinates;

[0011] Incremental SFM reconstruction is performed based on the camera parameters and feature point coordinates to obtain a sparse point cloud of a multi-image scene structure.

[0012] According to a preferred embodiment of the present invention, the detecting and matching of feature points in an image to obtain camera parameters and feature point coordinates includes:

[0013] Detect the feature points in the two images and match them to obtain matching feature point pairs;

[0014] Solve the camera pose according to the matching feature point pairs to obtain camera parameters;

[0015] According to the camera parameters, the three-dimensional coordinates of the feature points are obtained through triangulation and PnP;

[0016] The camera parameters and three-dimensional coordinates are simultaneously optimized nonlinearly through bundle adjustment to obtain the camera parameters and feature point coordinates.

[0017] According to a preferred embodiment of the present invention, reconstructing a dense point cloud based on the sparse point cloud to obtain a three-dimensional real scene includes:

[0018] Construct an initial seed grid based on the sparse point cloud;

[0019] The initial seed grid is optimized by region growing to generate depth maps at different viewing angles;

[0020] The depth maps from different perspectives are deeply fused to obtain a three-dimensional real scene.

[0021] According to a preferred embodiment of the present invention, multiple moving targets in an image are detected in real time through a preset network, and the preset network includes four parts: an input end, a Backbone, a Neck, and a Head output end, wherein: the Backbone includes a Focus structure, a CSP structure, and a CBAM attention mechanism, the Neck includes a BiFPN structure, and has four detection heads, and the Head output end includes a SIoU loss function.

[0022] According to a preferred embodiment of the present invention, the BiFPN structure removes the intermediate nodes of P3 and P7 in PANet, adds a jump connection line between the input node and the output node of the same scale, regards each bidirectional path as a feature network layer, and repeats the same layer multiple times.

[0023] According to a preferred embodiment of the present invention, multiple moving targets in an image are tracked by a target tracking model; wherein: the target tracking model includes: a DeepSort model, and a spatial attention module and a channel attention module connected to the output end of the DeepSort model.

[0024] According to a preferred embodiment of the present invention, the world coordinates of each moving target are located as follows:

[0025] Determine the direction of the target through the principle of perspective transformation;

[0026] Acquire the physical world target of the target through binocular vision convergence fixed-point model;

[0027] The physical visual target of the target is optimized through the trinocular vision fixed-point model to obtain the world coordinates of the target.

[0028] In order to solve the above technical problems, the second aspect of the present invention provides a device for presenting the moving positions of multiple targets in real time in a three-dimensional real scene, the device comprising:

[0029] The first reconstruction module is used to reconstruct a sparse point cloud of the image to obtain a sparse point cloud;

[0030] A second reconstruction module is used to perform dense point cloud reconstruction based on the sparse point cloud to obtain a three-dimensional real scene;

[0031] Tracking module, used to detect and track multiple moving targets in an image in real time;

[0032] The positioning and presentation module is used to locate the world coordinates of each moving target and present them in the three-dimensional real scene.

[0033] In order to solve the above technical problem, the third aspect of the present invention provides an electronic device, including:

[0034] Processor; and

[0035] A memory storing computer executable instructions, which when executed cause the processor to perform any of the methods described above.

[0036] The present invention is based on a drone shooting platform, and performs sparse point cloud reconstruction on the image taken by the drone to obtain a sparse point cloud; based on the sparse point cloud, a dense point cloud is reconstructed to obtain a three-dimensional real scene; multiple moving targets in the image are detected and tracked in real time; the world coordinates of each moving target are located, and they are presented in the three-dimensional real scene. Thus, the three-dimensional real scene is quickly constructed by innovatively and comprehensively using three-dimensional real scene construction technology, multi-moving target intelligent recognition and tracking technology, and target intelligent real-time positioning technology, and the like, and the high-precision coordinates of the moving targets in the physical scene are dynamically solved in real time, and they are synchronously presented in the three-dimensional real scene. It can be applied in combat scenarios, and the coordinate positions of moving targets such as personnel and vehicles, which are focused on intelligent real-time calculation, are presented in a three-dimensional real scene command system, which greatly improves the on-site situation awareness capability and effectively assists commanders in implementing command decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the technical problems solved by the present invention, the technical means adopted and the technical effects achieved more clearly, the specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, it should be noted that the drawings described below are only drawings of exemplary embodiments of the present invention, and those skilled in the art can obtain drawings of other embodiments based on these drawings without creative work.

[0038] Figure 1It is a flow chart of a method for presenting the moving positions of multiple targets in real time in a three-dimensional real scene according to the present invention;

[0039] Figure 2 is a schematic diagram of the structure of BiFPN in an embodiment of the present invention;

[0040] Figure 3a-3b They are respectively schematic diagrams of calculating angle cost and distance cost in the embodiments of the present invention;

[0041] Figures 4a to 4c They are schematic diagrams of the structure of the attention mechanism module in the embodiment of the present invention;

[0042] Figure 5a is a schematic diagram of determining the direction of a target by using the perspective transformation principle in an embodiment of the present invention;

[0043] Figure 5b is a schematic diagram of acquiring a physical world target of a target through a binocular vision convergence fixed-point model in an embodiment of the present invention;

[0044] Figure 6 It is a schematic diagram of the structural framework of a device for presenting the moving positions of multiple targets in real time in a three-dimensional real scene according to an embodiment of the present invention;

[0045] Figure 7 is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention. DETAILED DESCRIPTION

[0046] The exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. Although each exemplary embodiment can be implemented in a variety of specific ways, it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, these exemplary embodiments are provided to make the content of the present invention more complete and to facilitate the comprehensive communication of the inventive concept to those skilled in the art.

[0047] Under the premise of conforming to the technical concept of the present invention, the structure, performance, effect or other characteristics described in a specific embodiment may be combined with one or more other embodiments in any appropriate manner.

[0048] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art may implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0049] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to actual conditions without departing from the main purpose of the present invention.

[0050] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0051] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0052] Figure 1 This is a flow chart of a method for presenting the moving positions of multiple targets in real time in a three-dimensional real scene provided by the present invention; this method can be applied to combat scenes, intelligently and real-time solve the coordinate positions of moving targets such as personnel and vehicles, and present them to the three-dimensional real scene command system, greatly improving the on-site situation awareness capability and effectively assisting commanders in making command decisions. Figure 1 As shown, the method includes:

[0053] S1, reconstructing the sparse point cloud of the image to obtain a sparse point cloud;

[0054] In this embodiment, the image may be a video image captured in real time by a drone platform. Exemplarily, this step may include:

[0055] S11, detecting and matching feature points in the image to obtain camera parameters and feature point coordinates;

[0056] Specifically, this step may include:

[0057] S111, detecting feature points in two images and matching them to obtain matching feature point pairs;

[0058] For example, the Harris corner feature extraction operator can be used to detect feature points. Feature point matching is to determine the positional relationship between two images by finding similar feature points in the two images. The Euclidean distance between feature points can be used as the matching degree for measurement.

[0059] S112, solving the camera pose according to the matching feature point pairs to obtain camera parameters;

[0060] Optionally, the camera parameters can be obtained by using matching feature point pairs and solving the basic matrix and the essential matrix to solve the camera pose.

[0061] S113, obtaining the three-dimensional coordinates of the feature points through triangulation and PnP according to the camera parameters;

[0062] After obtaining the camera parameters, the three-dimensional coordinates of the feature points can be obtained through triangulation and PnP (Perspective-n-Point).

[0063] S114. Perform nonlinear optimization on camera parameters and three-dimensional coordinates simultaneously through bundling adjustment to obtain camera parameters and feature point coordinates.

[0064] Due to the presence of noise, the camera parameters and the three-dimensional coordinates of the feature points obtained in step S113 are not accurate enough. Therefore, it is necessary to perform nonlinear optimization on the camera parameters and the three-dimensional coordinates simultaneously through bundling adjustment.

[0065] In this embodiment, the bundling adjustment optimization process is as follows: ij is 0, indicating that the i-th point is not visible in the j-th camera; otherwise, ij 1 means visible. X i =(x w ,y w ,z w ) T represents the spatial coordinates of the i-th feature point; P j Represents the intrinsic parameters of the j-th camera and the extrinsic parameters of the world coordinate system and the j-th camera; represents the observation point of the i-th point in the j-th camera; u ij represents the projection point of the i-th point in the j-th camera. When there are n 3D points and m viewing angles, the following loss function can be constructed:

[0066]

[0067] Where θ=(P1,...,P m ,X1,...,X n ) is the quantity to be optimized. For the unconstrained nonlinear minimum optimization problem Usually the steepest descent method, Newton's method, Gauss-Newton method, and Levenberg-Marquardt method can be used to solve the problem. ij Represents the distance between the observation point and the projection point, that is:

[0068] S12. Perform incremental SFM reconstruction based on the camera parameters and feature point coordinates to obtain a sparse point cloud of a multi-image scene structure.

[0069] Incremental SFM (structure from motion) reconstruction reconstructs part of the scene and the external parameters of the two cameras from two images, then adds images, uses PnP to calculate the external parameters of the third image, and reconstructs more scenes through triangulation. Since there will be a lot of noise interference when solving the camera pose and the three-dimensional coordinates of the feature points, the subsequent calculation results will drift. Therefore, BA (Bundle Adjustment) optimization must be performed after each image is added to complete the triangulation reconstruction.

[0070] S2, reconstructing a dense point cloud based on the sparse point cloud to obtain a three-dimensional real scene;

[0071] Exemplarily, this step may include:

[0072] S21, constructing an initial seed grid according to the sparse point cloud;

[0073] The seed patch is constructed by the sparse points reconstructed by SFM.

[0074] S22, optimizing the initial seed grid by region growing to generate depth maps at different viewing angles;

[0075] Specifically, you can construct images of different resolutions through global perspective selection to handle different image resolution situations. Among them: global perspective selection refers to selecting a perspective that meets specific conditions for region growing. Global perspective selection generally meets the following conditions:

[0076] ① The images have the same content: by sharing the number of sift feature points;

[0077] ② The image has a sufficiently large parallax (wide baseline): through the angle of sight;

[0078] ③The images have similar resolution: by calculating the projection sphere radius.

[0079] After establishing the initial seed patch, region growing can use the method of gradual expansion of the seed patch to generate new patches in its neighborhood. The main steps include:

[0080] ① Establish a priority queue for the seed patches in the sparse point cloud based on confidence (photometric consistency);

[0081] ② Estimate the depth of the initial sparse seed points;

[0082] ③Perform nonlinear depth optimization on each seed point;

[0083] ④ After each optimization, if the corresponding pixel on the image has no depth value or the confidence value of the current pixel is higher than a certain range of neighboring pixels, its pixel is added to the queue.

[0084] S23, deeply fuse the depth maps from different perspectives to obtain a three-dimensional real scene.

[0085] Depth fusion refers to merging points with similar positions in three-dimensional space. The fusion process also imposes certain constraints on the points, usually including consistency constraints and visibility constraints. The consistency constraint means that the depth of the same point should be consistent under different viewing angles; the visibility constraint means that the three-dimensional points visible in the image cannot be blocked in the fused depth map.

[0086] S3, real-time detection and tracking of multiple moving targets in the image;

[0087] In this embodiment, multiple moving targets in an image can be detected in real time through a preset network. Specifically, the preset network may include four parts: input, Backbone, Neck, and Head output. Among them: the input is the same as the input of the YOLOv5 algorithm. Backbone includes Focus structure, CSP structure and CBAM attention mechanism, Neck includes BiFPN structure and has four detection heads, and the Head output includes SIoU loss function.

[0088] The preset network integrates the CBAM attention mechanism on the standard YOLOv5 algorithm. First, the CBAM attention mechanism is added to the Backbone to make the network pay more attention to the target to be detected and improve the detection effect; secondly, the original FPN+PAN method is replaced by BiFPN, and the three detection heads are changed to four detection heads, and small targets are predicted on the C2 feature map size; finally, the original CIoU loss function is replaced by the SIoU loss function to further improve the positioning accuracy of small targets and accelerate network convergence.

[0089] Specifically, there are many ways to integrate the attention mechanism into the YOLOv5 algorithm. For example, it can be introduced into the Backbone network, or into the Neck network Neck or Head output end. Through repeated experiments, it was found that the detection effect was best after adding CBAM to the C5 layer of the YOLOv5 algorithm. The present invention introduces BiFPN into the Neck stage and modifies it, replacing the original FPN+PAN structure of the YOLOv5 algorithm. The BiFPN structure removes the intermediate nodes of P3 and P7 in PANet, and adds a jump connection line between the input node and the output node of the same scale, treating each bidirectional path as a feature network layer, and repeating the same layer multiple times. Compared with PANet, BiFPN is optimized in the following aspects:

[0090] First, nodes with only one input edge are removed. If a node has only one input edge and no feature fusion, its contribution to the feature network that fuses different features is small. Therefore, the intermediate nodes of P3 and P7 in PANet are removed.

[0091] Second, a skip connection is added. A skip connection is added between input nodes and output nodes of the same scale. Since they are in the same layer, more features are integrated without increasing the computational cost too much.

[0092] Finally, unlike PANet which has only one top-down and bottom-up path, each bidirectional (top-down and bottom-up) path is regarded as a feature network layer (repeated blocks), and the same layer is repeated multiple times to achieve higher-level feature fusion. Figure 2 P2, P3, P4, P5 in the present invention), combined with the existing YOLOv5 architecture, the BiFPN structure used in the present invention is obtained as follows Figure 2 shown.

[0093] Furthermore, the CIoU loss used in the existing YOLOv5 algorithm takes into account the aspect ratio of the regression box, but only uses the aspect ratio as an influencing factor. If there are two box center points that are consistent with the original image, the aspect ratio is the same but the width and height values ​​are different, the CIoU loss may be consistent, which is inconsistent with the regression target. Based on this, the Head output end of this application replaces the CIoU loss with the SIoU loss function. SIoU takes into account the vector angle between the required regressions and redefines the penalty index, which consists of four parts: angle cost, distance cost, shape cost, and IoUcost.

[0094] like Figure 3a As shown, B is the prediction box, BGT is the true value box, and the angle cost is defined as follows:

[0095] A=1-2*sin 2 (ɑ-π / 4)

[0096] Where: ɑ=arcsin(C h / σ), C h , C W are the distance and height difference between the center points of the two frames respectively.

[0097] like Figure 3b As shown, the distance cost is defined as follows:

[0098]

[0099] in, γ=2-A.

[0100] It can be seen that when α→0, the contribution of distance cost is greatly reduced. The greater the distance cost contribution.

[0101] The shape cost is defined as follows:

[0102]

[0103] in: The value of θ defines the shape cost of each dataset and its value is close to 4.

[0104] Finally, the regression loss function of SIoU is:

[0105]

[0106] In this embodiment, multiple moving targets in an image can be tracked through a target tracking model; wherein: the target tracking model includes: a DeepSort model, and a spatial attention module and a channel attention module connected to the output end of the DeepSort model.

[0107] Specifically, the CBAM attention mechanism can be used to add cascade matching and confirmation of new trajectories based on the DeepSort model (algorithm). Among them: the CBAM attention mechanism is a module that combines space and channel attention. Figure 4a As shown in the figure, the output of the DeepSort model will first pass through a channel attention module to obtain a weighted result, and then pass through a spatial attention module to obtain the final weighted result.

[0108] Channel attention module such as Figure 4bAs shown. The input feature map is subjected to global max pooling and global average pooling based on width and height, and then to multi-layer perceptron MLP. The features output by MLP are added element-wise, and then activated by sigmoid to generate the final channel attention feature map. The channel attention featuremap is multiplied element-wise with the input feature map input featuremap to generate the input features required by the spatial attention module.

[0109] The spatial attention module is Figure 4c As shown in the figure. The features output by the channel attention module are used as the input features of this module. First, a global max pooling and global average pooling based on the channel are performed, and then the two results are concat based on the channel. Then, after a convolution operation, the dimension is reduced to one channel. Then, the spatial attention features are generated through sigmoid. Finally, the spatial attention features are multiplied by the input features of the module to obtain the final generated features.

[0110] S4. Locate the world coordinates of each moving target and present them in the three-dimensional real scene.

[0111] Exemplarily, this step may include:

[0112] S41, determine the direction of the target through the perspective transformation principle;

[0113] like Figure 5a , assuming that point Q is a measured point in the spatial field of view (world coordinate system), the point is projected onto the single-machine imaging plane, q(x,y) represents the imaging point of the camera coordinate system, and its pixel point is (u,v). Let M be the projection matrix. Since the camera system has been successfully calibrated, this part is known, then:

[0114]

[0115] In the above formula, Z c represents the coordinates of point Q in the depth direction of the camera coordinate system where the camera is located, (u, v, l) represent the homogeneous coordinates of the two projection points on the imaging surface corresponding to the pixel coordinates, and (Xm, Ym, Zm) represent the homogeneous coordinates of the spatial measured point Q in the world coordinate system. Given the (u, v, l) coordinates and the projection matrix M, a three-dimensional equation system can be constructed. In the case of only two equations, only the direction of the measured point Q between the cameras can be obtained.

[0116] S42, obtaining a physical world target of the target through a binocular vision convergence fixed-point model;

[0117] In this embodiment, the binocular vision convergence fixed-point model is a positioning model in which two cameras collect images of the measured object in the same scene from different positions to obtain the distance information of the measured target point. From the previous single-machine monitoring, it can be seen that a single calibration camera can obtain the direction of the measured target. If the two cameras can obtain the measured point at the same time, they can obtain the position information of the measured target through the intersection of directional rays.

[0118] Specifically, the schematic diagram of the binocular vision convergence fixed-point model is as follows: Figure 5b Assume that point Q is a point to be measured in the spatial field of view, and the pixel coordinates of the two points q1 and qr projected on the imaging plane of the left and right cameras have been detected as (ul, vl) and (u2r, vr), respectively. Assume that the two cameras in the model have been calibrated, and M1 and Mr are their respective projection matrices, then:

[0119]

[0120] Zcl and Zcr represent the coordinates of point Q in the depth direction of the camera coordinate system where the two cameras are located, mij represents the element values ​​in the two projection matrices M1 and Mr, (ul, vl, 1) and (u2r, vr, 1) represent the homogeneous coordinates of the two projection points on their respective imaging surfaces, and (Xm, Ym, Zm, 1) represents the homogeneous coordinates of the measured point Q in the world coordinate system. The two formulas can be combined to eliminate Zcl and Zcr in the above formula, and then obtain two sets of linear equations related to (Xm, Ym, Zm) as shown below:

[0121]

[0122] From this we can get

[0123]

[0124] In the above formula:

[0125]

[0126] S43. Optimize the physical visual target of the target through the trinocular vision fixed-point model to obtain the world coordinates of the target.

[0127] The trinocular vision fixed-point model is built on the basis of the binocular vision convergence ranging model by reasonably placing three cameras, and adjusting the position relationship between the cameras so that the optical axes of each camera form a certain angle with each other, that is, the model is composed of three sets of binocular ranging models. Assume that the projection points of any point Q in space on the imaging surfaces of the three cameras a, b and c are q1, q2, and q3 respectively. Ideally, the measurement value Q1 of the binocular ranging model composed of camera a and camera b, the measurement value Q2 of the binocular ranging model composed of camera b and camera c, and the measurement value Q3 of the binocular ranging system composed of camera a and camera c should overlap with the actual measured point Q at one point. However, in the actual ranging environment, due to the existence of errors in the binocular ranging system, the three points Q1, Q2, and Q3 do not overlap with the actual measured point Q. The actual three-dimensional spatial coordinates of these three points can be obtained through the binocular ranging algorithm.

[0128] Based on the method provided by the present invention for presenting the moving positions of multiple targets in real time in a three-dimensional real scene, a real-time dynamic three-dimensional reconstruction system of a drone can be constructed by adopting an integrated design of "light, machine, electricity, calculation, and software" with the support of hardware such as a multi-rotor drone, an optical pod, and a ground station (including a GPU graphics card). Specifically, two drones are hovered over a building, and images of the building and pedestrian scenes are simultaneously captured from two angles. With the scene image as input, a three-dimensional real scene model is first constructed, and then the multi-moving target intelligent recognition and tracking technology is used to identify and track pedestrians in the scene, and the image coordinates of pedestrians in the image are obtained. Finally, the target intelligent real-time positioning technology is used to convert the pixel coordinates of pedestrians into world coordinates, and the real-time dynamic presentation is performed in the three-dimensional real scene. Finally, a real-time dynamic three-dimensional reconstruction scene containing the dynamic position of pedestrians is output, and the moving pedestrian target positioning information is refreshed every 2 seconds.

[0129] Based on the above method for presenting the moving positions of multiple targets in real time in a three-dimensional real scene, the present invention also provides a device for presenting the moving positions of multiple targets in real time in a three-dimensional real scene. Figure 6 Schematic diagram of the structure of a device for presenting the moving positions of multiple targets in real time in a three-dimensional real scene according to the present invention. Figure 6 As shown, the device comprises:

[0130] A first reconstruction module 61 is used to perform sparse point cloud reconstruction on the image to obtain a sparse point cloud;

[0131] A second reconstruction module 62 is used to perform dense point cloud reconstruction based on the sparse point cloud to obtain a three-dimensional real scene;

[0132] A tracking module 63, used for detecting and tracking multiple moving targets in an image in real time;

[0133] The positioning and presenting module 64 is used to locate the world coordinates of each moving target and present them in the three-dimensional real scene.

[0134] In a specific implementation, the first reconstruction module 61 includes:

[0135] The detection module is used to detect and match feature points in the image to obtain camera parameters and feature point coordinates;

[0136] The SFM reconstruction module is used to perform incremental SFM reconstruction based on the camera parameters and feature point coordinates to obtain a sparse point cloud of a multi-image scene structure.

[0137] Specifically, the detection module includes:

[0138] A sub-detection unit, used to detect feature points in two images and match them to obtain matching feature point pairs;

[0139] A solving unit, used for solving the camera pose according to the matching feature point pairs to obtain camera parameters;

[0140] A calculation unit, used for obtaining the three-dimensional coordinates of the feature points through triangulation and PnP according to the camera parameters;

[0141] The optimization unit is used to simultaneously perform nonlinear optimization on the camera parameters and the three-dimensional coordinates through bundle adjustment to obtain the camera parameters and the feature point coordinates.

[0142] In a specific implementation, the second reconstruction module 62 includes:

[0143] A first construction unit is used to construct an initial seed grid according to the sparse point cloud;

[0144] A region generation unit, used to optimize the initial seed grid by region growing to generate depth maps at different viewing angles;

[0145] The depth fusion unit is used to deeply fuse depth maps from different perspectives to obtain a three-dimensional real scene.

[0146] In a specific implementation, the tracking module 63 detects multiple moving targets in the image in real time through a preset network, and the preset network includes four parts: input end, Backbone, Neck, and Head output end, wherein: Backbone includes Focus structure, CSP structure and CBAM attention mechanism, Neck includes BiFPN structure, and is a four-detection head, and the Head output end includes SIoU loss function. Wherein: the BiFPN structure removes the intermediate nodes of P3 and P7 in PANet, and adds a jump connection line between the input node and the output node of the same scale, regards each bidirectional path as a feature network layer, and repeats the same layer multiple times.

[0147] The tracking module 63 tracks multiple moving targets in the image through a target tracking model; wherein: the target tracking model includes: a DeepSort model, and a spatial attention module and a channel attention module connected to the output end of the DeepSort model.

[0148] In a specific implementation, the positioning presentation module 64 includes:

[0149] The first determination module is used to determine the direction of the target through the perspective transformation principle;

[0150] The second determination module is used to obtain the physical world target of the target through a binocular vision convergence fixed point model;

[0151] The third determination module is used to optimize the physical visual target of the target through the trinocular vision fixed-point model to obtain the world coordinates of the target.

[0152] Those skilled in the art will appreciate that the modules in the above device embodiments may be distributed in the device as described, or may be changed accordingly and distributed in one or more devices different from the above embodiments. The modules in the above embodiments may be combined into one module, or may be further split into multiple submodules.

[0153] An electronic device embodiment of the present invention is described below. The details described in the electronic device embodiment of the present invention should be regarded as a supplement to the above-mentioned method or device embodiment. Details not disclosed in the electronic device embodiment of the present invention can be implemented with reference to the above-mentioned method or device embodiment.

[0154] Figure 7 is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0155] like Figure 7As shown, the electronic device 400 of this exemplary embodiment is in the form of a general data processing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different electronic device components (including the storage unit 420 and the processing unit 410), a display unit 440, etc.

[0156] The storage unit 420 stores a computer-readable program, which may be a source program or a code of a read-only program. The program may be executed by the processing unit 410, so that the processing unit 410 performs the steps of various embodiments of the present invention. For example, the processing unit 410 may perform the following steps: Figure 1 Steps shown.

[0157] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 4201 and / or a cache storage unit 4202, and may further include a read-only storage unit (ROM) 4203. The storage unit 420 may also include a program / utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 including but not limited to: operating the electronic device, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0158] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0159] The electronic device 400 may also communicate with one or more external devices 300 (e.g., keyboard, display, network device, Bluetooth device, etc.) so that a user can interact with the electronic device 400 via these external devices 300, and / or the electronic device 400 can communicate with one or more other data processing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 450, and may also be performed through a network adapter 460 with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet). The network adapter 460 may communicate with other modules of the electronic device 400 via the bus 430. It should be understood that although Figure 7 Not shown, other hardware and / or software modules may be used in the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID electronic devices, tape drives, and data backup storage electronic devices, etc.

[0160] In summary, the present invention can be implemented by a method, apparatus, electronic device or computer-readable medium that executes a computer program. In practice, a general data processing device such as a microprocessor or a digital signal processor (DSP) can be used to implement some or all functions of the present invention.

[0161] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for presenting the moving positions of multiple targets in real time in a three-dimensional real scene, characterized in that: The method comprises: Perform sparse point cloud reconstruction on the image to obtain a sparse point cloud; Performing dense point cloud reconstruction based on the sparse point cloud to obtain a three-dimensional real scene; Detect and track multiple moving targets in images in real time; The world coordinates of each moving target are located and presented in the three-dimensional real scene.

2. The method according to claim 1, characterized in that The sparse point cloud reconstruction of the collected image to obtain the sparse point cloud comprises: Detect and match feature points in the image to obtain camera parameters and feature point coordinates; Incremental SFM reconstruction is performed based on the camera parameters and feature point coordinates to obtain a sparse point cloud of a multi-image scene structure.

3. The method according to claim 2, characterized in that The detecting and matching of feature points in the image to obtain camera parameters and feature point coordinates includes: Detect the feature points in the two images and match them to obtain matching feature point pairs; Solve the camera pose according to the matching feature point pairs to obtain camera parameters; According to the camera parameters, the three-dimensional coordinates of the feature points are obtained through triangulation and PnP; The camera parameters and three-dimensional coordinates are simultaneously optimized nonlinearly through bundle adjustment to obtain the camera parameters and feature point coordinates.

4. The method according to claim 1, characterized in that: The reconstructing a dense point cloud based on the sparse point cloud to obtain a three-dimensional real scene includes: Construct an initial seed grid based on the sparse point cloud; The initial seed grid is optimized by region growing to generate depth maps at different viewing angles; The depth maps from different perspectives are deeply fused to obtain a three-dimensional real scene.

5. The method according to claim 1, characterized in that Multiple moving targets in the image are detected in real time through a preset network. The preset network includes four parts: input end, Backbone, Neck, and Head output end. Among them: Backbone includes Focus structure, CSP structure and CBAM attention mechanism, Neck includes BiFPN structure and has four detection heads, and Head output end includes SIoU loss function.

6. The method according to claim 5, characterized in that The BiFPN structure removes the intermediate nodes of P3 and P7 in PANet, adds a jump connection line between the input node and the output node of the same scale, regards each bidirectional path as a feature network layer, and repeats the same layer multiple times.

7. The method according to claim 1, characterized in that Track multiple moving targets in an image through a target tracking model; wherein: the target tracking model includes: a DeepSort model, and a spatial attention module and a channel attention module connected to the output end of the DeepSort model.

8. The method according to claim 1, characterized in that The world coordinates of locating each moving target include: Determine the direction of the target through the principle of perspective transformation; Acquire the physical world target of the target through binocular vision convergence fixed-point model; The physical visual target of the target is optimized through the trinocular vision fixed-point model to obtain the world coordinates of the target.

9. A device for presenting the moving positions of multiple targets in real time in a three-dimensional real scene, characterized in that: The device comprises: The first reconstruction module is used to reconstruct a sparse point cloud of the image to obtain a sparse point cloud; A second reconstruction module is used to reconstruct a dense point cloud based on the sparse point cloud to obtain a three-dimensional real scene; Tracking module, used to detect and track multiple moving targets in an image in real time; The positioning and presentation module is used to locate the world coordinates of each moving target and present them in the three-dimensional real scene.

10. An electronic device, characterized in that: include: processor; as well as A memory storing computer executable instructions which, when executed, cause the processor to perform the method according to any one of claims 1-7.

Images