Camera point determination method, device and equipment based on three-dimensional map

Abstract

The present invention discloses a method, apparatus, and device for determining camera point positions based on a three-dimensional map. The method includes: determining an actual image captured by an actual camera deployed in an actual scene, and determining multiple first virtual images captured by multiple virtual cameras corresponding to the actual camera; performing feature extraction on the actual image and each first virtual image to obtain key point information of each image; determining an image matching result between the actual image and each first virtual image based on the matching result of the key point information of the actual image and each first virtual image; determining a second virtual image from each first virtual image based on the image matching result, and determining the point position determination result of the actual camera based on the second virtual image. The present invention achieves a significant improvement in the accuracy and consistency of point position determination, thereby reducing the possibility of human annotation errors; and improves the point annotation speed, realizing real-time annotation.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and equipment for determining camera point positions based on a three-dimensional map. Background Art

[0002] In recent years, the application of 3D maps in fields such as geographic information systems (GIS), urban planning, autonomous driving, and virtual reality has continued to increase. These applications require accurate annotation of camera points in 3D maps for further data analysis and decision support.

[0003] Existing point marking methods mostly rely on manual marking, which is not only inefficient but also easily affected by human factors, resulting in low marking accuracy and consistency. Summary of the Invention

[0004] The present invention provides a method, device and equipment for determining camera points based on a three-dimensional map, so as to solve the problem of low accuracy and low efficiency in marking camera points in a three-dimensional map.

[0005] According to one aspect of the present invention, a method for determining camera position based on a three-dimensional map is provided, comprising:

[0006] Determining an actual image captured by an actual camera deployed in an actual scene, and determining a plurality of first virtual images captured by a plurality of virtual cameras corresponding to the actual camera; wherein the virtual cameras are created in a three-dimensional map corresponding to the actual scene based on camera parameters of the actual camera;

[0007] Performing feature extraction on the actual image and each of the first virtual images to obtain key point information of each image;

[0008] determining an image matching result between the actual image and each of the first virtual images according to a matching result of the key point information between the actual image and each of the first virtual images;

[0009] A second virtual image is determined from each of the first virtual images according to the image matching result, and a point position determination result of the actual camera is determined according to the second virtual image.

[0010] According to another aspect of the present invention, there is provided a device for determining camera position based on a three-dimensional map, comprising:

[0011] an image determination module, configured to determine an actual image captured by an actual camera deployed in an actual scene, and to determine a plurality of first virtual images captured by a plurality of virtual cameras corresponding to the actual camera; wherein the virtual cameras are created in a three-dimensional map corresponding to the actual scene based on camera parameters of the actual camera;

[0012] A key point extraction module, configured to extract features from the actual image and each of the first virtual images to obtain key point information of each image;

[0013] an image matching module, configured to determine an image matching result between the actual image and each of the first virtual images based on a matching result of the key point information between the actual image and each of the first virtual images;

[0014] A point position determination module is configured to determine a second virtual image from each of the first virtual images according to the image matching result, and determine a point position determination result of the actual camera according to the second virtual image.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and a memory communicatively coupled to the at least one processor;

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the camera point position determination method based on a three-dimensional map as described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the camera point position determination method based on a three-dimensional map according to any embodiment of the present invention when executed.

[0019] The technical solution of this embodiment creates a corresponding virtual camera based on the actual camera, and obtains a camera point determination result based on the detection and matching results of key points in the images collected by each camera. Based on the camera point determination result, the actual camera can be marked in the three-dimensional map, and the accuracy and consistency of point determination can be significantly improved, thereby reducing the possibility of human marking errors; and the technical solution of this embodiment can increase the marking speed and realize real-time marking to adapt to real-time and large-scale data scenarios.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flowchart of a method for determining camera position based on a three-dimensional map according to an embodiment of the present invention;

[0023] Figure 2 is a flowchart of another method for determining camera position based on a three-dimensional map according to an embodiment of the present invention;

[0024] Figure 3 2 is a schematic diagram of a model architecture of a key point extraction model provided according to an embodiment of the present invention;

[0025] Figure 4 is a flowchart of another method for determining camera position based on a three-dimensional map according to an embodiment of the present invention;

[0026] Figure 5 This is an architecture diagram of a graph neural network provided according to an embodiment of the present invention;

[0027] Figure 6 is a flowchart of another method for determining camera position based on a three-dimensional map according to an embodiment of the present invention;

[0028] Figure 7 1 is a schematic structural diagram of a device for determining camera position based on a three-dimensional map according to an embodiment of the present invention;

[0029] Figure 8 It is a structural diagram of an electronic device for implementing the camera point position determination method based on a three-dimensional map according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "candidate", "target", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices.

[0032] Figure 1 The present invention provides a flowchart of a method for determining camera positions based on a three-dimensional map. This embodiment is applicable to the situation where the precise camera deployment position in the actual scene is uncertain, and the camera positions are accurately marked in a three-dimensional map corresponding to the actual scene. The method can be executed by a camera position determination device based on a three-dimensional map. The camera position determination device based on a three-dimensional map can be implemented in the form of hardware and / or software. The camera position determination device based on a three-dimensional map can be configured in a server with communication and computing capabilities. Figure 1 As shown, the method includes:

[0033] S110: Determine an actual image captured by an actual camera deployed in an actual scene, and determine a plurality of first virtual images captured by a plurality of virtual cameras corresponding to the actual camera.

[0034] The actual scene is a real platform with camera points deployed, such as a shopping mall or industrial park. To achieve monitoring coverage of these actual scenes, surveillance cameras are deployed at various locations in the scene. To facilitate monitoring of the environment in the actual scene, a corresponding three-dimensional map is created for the actual scene. This three-dimensional map is constructed based on the actual scenery of the actual scene, that is, the three-dimensional map at least includes all fixed objects in the actual scene. At the same time, to facilitate the management of the surveillance cameras in the actual scene, virtual cameras are set up in the three-dimensional map at locations corresponding to the actual cameras. However, to ensure that the field of view of the virtual camera is the same as that of the corresponding actual camera, the accuracy of the virtual camera's creation position and rotation information in the three-dimensional map must be guaranteed. Therefore, it is necessary to accurately obtain the specific point information of the actual camera, and create the virtual camera based on this accurate specific point information of the actual camera, ensuring the accuracy and efficiency of managing the virtual camera corresponding to the actual camera through the three-dimensional map.

[0035] A virtual camera is created in a 3D map corresponding to the actual scene based on the camera parameters of the actual camera. Initially, the actual camera's location information is unavailable; only rough location information is available. This rough location information represents a specific area, and the virtual camera corresponding to the actual camera is determined based on this rough location information.

[0036] Specifically, the camera parameters of the actual camera include at least the area range corresponding to the camera deployment location in the three-dimensional map and the camera internal parameters. Traversing within the area range, a plurality of candidate point information is obtained, and a plurality of virtual cameras are created based on the plurality of candidate point information and the plurality of candidate rotation angle information. At the same time, the internal parameters of the virtual cameras are the same as the internal parameters of the actual camera. The candidate rotation angle is determined based on the sampling results of the angle range in which the camera can rotate. Exemplarily, different combinations of candidate point information and candidate rotation angle information are determined based on the plurality of candidate point information and the plurality of candidate rotation angle information, and a virtual camera corresponding to each combination of candidate point information and candidate rotation angle information is created. That is, the plurality of virtual cameras created are a collection of multiple possible results of the deployment location of the actual camera.

[0037] The camera parameters of the actual camera also include the camera's IP information, which can be used to remotely access and control the actual camera to capture the actual image of the actual camera in the actual scene. The virtual images captured by each virtual camera are rendered based on its position information and rotation angle information in the 3D map. Each virtual camera captures a corresponding first virtual image.

[0038] Exemplarily, a real-world scene may include multiple real-world cameras. Therefore, a camera data table is established for all real-world cameras to manage and schedule the relevant information for each real-world camera based on this camera data table. The camera data table includes at least the camera IP information. Each real-world camera has a unique IP address, which enables remote access and control of the real-world camera, thereby capturing real-time real-world images. The camera data table also includes the area corresponding to the camera's deployment location in the 3D map. To improve computational efficiency and reduce unnecessary resource consumption, the camera is divided according to its installation area, resulting in the area within the real scene and the corresponding area within the 3D map. This area can be used to limit and narrow the virtual camera's rendering range, avoiding unnecessary global rendering and excessive computational burden. The camera data table also includes camera internal parameters. Each camera has different internal parameters, which directly influence the creation and matching of the virtual camera. Internal parameter information includes field of view, aspect ratio, focal length, and aperture information. After the camera data table is prepared, the real-time actual images of each actual camera are automatically obtained in batches through a Python script. The specific steps are as follows: Use the Python script to access the IP addresses of the actual cameras in sequence, automatically connect to the camera to retrieve the real-time image data as the actual image of the actual camera. For each actual camera in the camera data table, the script loops through all cameras, obtains each actual image in sequence and stores it in a local folder. When determining the point information of each actual camera, the corresponding actual image in the folder is obtained in sequence.

[0039] A series of movement parameters are generated for the virtual camera based on the area range of the actual camera at the current point to be determined. The movement parameters specifically include position information and rotation angle information. Based on the internal parameter information of the actual camera in the camera data table, a virtual camera corresponding to the actual camera is created in the UE (Unreal Engine) model to ensure that the camera internal parameters of the virtual camera are the same as the actual camera. Among them, there are two situations in the UE model of the three-dimensional map. One is that there are built-in points in the UE model. The built-in points are possible deployment points of the actual camera set in advance in the UE model. In this case, it is only necessary to determine the correspondence between the actual camera and the virtual camera and the viewing angle. The movement parameters of the virtual camera only need to set the viewing angle, that is, the rotation angle information. The viewing angle setting will generate different viewing angle combinations by adjusting the rotation angle around the Y axis and the rotation angle around the Z axis with a preset unit angle as the step size. The viewing angle change formula is as follows: .in, is the rotation angle vector, and Is an integer representing the number of rotation steps, a represents the preset unit angle. In another case, if there is no built-in point in the UE model, the movement parameters of the virtual camera need to be dynamically generated according to the area where the actual camera is located. Specifically, the position parameters (X, Y, Z) of the virtual camera are moved from the initial position ( ) changes in steps of preset distance units. The initial position can be any position within the actual camera area. The rotation angle around the Y axis and the rotation angle around the Z axis change by 60 degrees each time to ensure coverage of different viewing angles and positions. The position change formula is as follows: .in, is the position vector of the virtual camera, 、 、 Indicates the number of position steps, b is the preset distance unit, which can be set according to the actual scenario. The specific value is not limited here, for example, b=0.2 meters.

[0040] Since the camera data table stores the area range corresponding to the camera, and the same area range may include at least two actual cameras, in order to improve the efficiency of virtual camera rendering, the cameras are divided according to their areas, and the virtual cameras are divided according to the area range of the cameras in the camera data table to obtain a virtual camera sub-table corresponding to each area range. In each virtual camera sub-table, the virtual camera sub-table is further divided according to the camera type and camera intrinsic parameters to obtain a first virtual camera sub-table corresponding to each camera type and camera intrinsic parameter. In this first virtual camera sub-table, the first virtual images corresponding to each virtual camera are batch rendered and stored.

[0041] S120 , performing feature extraction on the actual image and each first virtual image to obtain key point information of each image.

[0042] The key points are identifiable feature points in the image, such as points of interest. Feature extraction is performed on the actual image to obtain actual key point information in the actual image, and feature extraction is performed on each first virtual image to obtain virtual key point information in each virtual image.

[0043] Exemplarily, a point of interest extraction model is pre-trained, and the actual image and each first virtual image are respectively input into the point of interest extraction model to obtain key point information in each image, wherein the key point information includes at least key point position information and may also include key point category information.

[0044] S130 : Determine an image matching result between the actual image and each first virtual image according to a matching result of key point information between the actual image and each first virtual image.

[0045] Key point information is matched between the actual key points in the actual image and the virtual key points in each first virtual image. An image matching result between the actual image and the first virtual image is determined based on the degree of key point information matching between the key points in the actual image and the first virtual image. The image matching result is the key point matching degree between the actual image and the first virtual image. A higher key point matching degree indicates a higher similarity between the actual image and the first virtual image, i.e., a higher degree of matching between the virtual camera corresponding to the first virtual image and the actual camera.

[0046] Specifically, the actual key points and virtual key points are traversally matched based on the key point position information in the actual image and the first virtual image, the number of key points with successful position matching is determined, and the image matching result between the actual image and the first virtual image is determined based on the ratio of the number of successfully matched key points to the number of actual key points in the actual image. A successful position match indicates that the difference between the position of the virtual key point and the position of the actual key point is less than a preset difference threshold. Furthermore, a further matching judgment is performed on the key points with successful position matching based on the key point type information to determine the number of key points with successful position and type matching. The image matching result between the actual image and the first virtual image is determined based on the ratio of the number of successfully matched key points with successful position and type matching to the number of actual key points in the actual image. A successful type match indicates that the type of the virtual key point is the same as the type of the actual key point.

[0047] Exemplarily, a key point matching model is pre-trained, the actual image and each first virtual image and key point information are input into the key point matching model, and the model output is an image matching result between the actual image and each first virtual image.

[0048] S140 : Determine a second virtual image from each first virtual image according to the image matching result, and determine a point position determination result of the actual camera according to the second virtual image.

[0049] According to the image matching result, the first virtual image with the highest matching degree is determined from all the first virtual images as the second virtual image, and the point position determination result of the actual camera is determined according to the movement parameters corresponding to the virtual camera corresponding to the second virtual image when it is created.

[0050] Specifically, the image matching result represents the image similarity. The first virtual images are sorted in descending order according to the image similarity, and the first virtual image ranked first is determined as the second virtual image. The position information and rotation angle information of the virtual camera corresponding to the second virtual image when it is created are determined, and the position information and rotation angle information of the virtual camera are used as the point determination result of the actual camera.

[0051] Optionally, the image matching result represents image similarity. The first virtual images are sorted in descending order based on the image similarity, and a preset number of the first virtual images that are ranked first are selected as the second virtual images. The virtual cameras corresponding to the second virtual images are determined to be the second virtual cameras. The position information of the second virtual cameras is calculated, and the position information of the second virtual camera with the largest percentage is used as the target position information of the actual camera. The second virtual camera with the highest image similarity among the second virtual cameras corresponding to the target position information is then determined to be the target virtual camera. The rotation angle information of the target virtual camera is determined to be the target rotation angle information of the actual camera. Finally, the target position information and the target rotation angle information are used as the point position determination result of the actual camera.

[0052] The technical solution of this embodiment creates a corresponding virtual camera based on the actual camera, and obtains a camera point determination result based on the detection and matching results of key points in the images collected by each camera. Based on the camera point determination result, the actual camera can be marked in the three-dimensional map, and the accuracy and consistency of point determination can be significantly improved, thereby reducing the possibility of human marking errors; and the technical solution of this embodiment can increase the point marking speed and realize real-time marking to adapt to real-time and large-scale data scenarios.

[0053] The implementation of the technical solution in this embodiment provides a more intelligent and efficient solution for 3D map point annotation, enhancing the practicality and analytical capabilities of 3D map data, promoting the application of 3D maps in various fields, and strengthening decision support and data analysis capabilities. This, in turn, promotes the development and innovation of related industries and effectively supports various applications such as autonomous driving, urban planning, and virtual reality.

[0054] Figure 2 This is a flowchart of another method for determining camera position based on a three-dimensional map provided by an embodiment of the present invention. This embodiment further refines the key point information extraction in the above embodiment. Figure 2 As shown, the method includes:

[0055] S210: Determine an actual image captured by an actual camera deployed in an actual scene, and determine a plurality of first virtual images captured by a plurality of virtual cameras corresponding to the actual camera.

[0056] S220 , performing feature extraction on the actual image and each first virtual image based on a pre-trained key point extraction model, and obtaining a key point detection result and a corresponding descriptor of each image as key point information.

[0057] The key point extraction model is a pre-trained model used to extract key point features from images. In order to improve the accuracy of key point matching, the output results of the key point extraction model include not only the detection results of the key points in the image themselves, but also the descriptors corresponding to each key point. The descriptors are used to describe the overall information of the local area where each key point is located, that is, the descriptors are used to characterize the overall characteristics of the local area where the corresponding key point is located.

[0058] Specifically, the actual image and each first virtual image are respectively input into a key point extraction model, which performs feature extraction on each image. The model outputs a detection result for each key point and a corresponding descriptor. The key point detection result includes at least key point location information, and the descriptor is a regional feature description of a local area formed by extending a preset range around the key point location information. The regional feature description can be a compressed result of the feature extraction result of the local area by the key point extraction model. Exemplarily, the key point extraction model can adopt a PointNet network model.

[0059] In a feasible embodiment, the key point extraction model includes a shared feature encoder, a key point detection decoder, and a descriptor generation decoder; the output of the shared feature encoder is the input of the key point detection decoder and the descriptor generation decoder, the output of the key point detection decoder is the key point detection result, and the output of the descriptor generation decoder is the descriptor corresponding to each key point;

[0060] Among them, the shared feature encoder includes at least one convolutional layer and at least one FasterNet block module; the key point detection decoder includes at least one convolutional layer and at least one FasterNet block module; the descriptor generation decoder includes a feature synthesis layer, at least one convolutional layer and at least one FasterNet block module.

[0061] The key point extraction model consists of three parts: a shared feature encoder, a key point detection decoder, and a descriptor generation decoder. The shared feature encoder is used to extract feature details in the image, the key point detection decoder is used to determine the key point detection results based on the results output by the shared feature encoder, and the descriptor generation decoder is used to determine the feature information of the local area where each key point is located as the descriptor output result based on the results output by the shared feature encoder and the key point position information output by the key point detection encoder.

[0062] like Figure 3The figure shows a schematic diagram of the model architecture of the key point extraction model. The feature embedding layer and two FasterNet block modules form a shared feature encoder, a FasterNet block module and a convolutional layer form a key point detection decoder, and the feature synthesis layer, two FasterNet block modules, and a convolutional layer form a descriptor generation decoder. In the key point extraction model, the initial feature embedding layer uses a 4×4 convolution kernel and a stride of 4 to quickly extract high-level image features. The shared feature encoder reduces the spatial dimension of the input image to 1 / 8, reducing subsequent computational pressure. Similarly, convolution kernels of other sizes can also be used in the feature synthesis layer for convolution and deconvolution processing to synthesize the feature extraction results output by the shared feature encoder and improve the accuracy of local area feature extraction. As for the convolution kernel sizes of other involved convolution layers, they can be adjusted according to actual conditions and are not restricted here.

[0063] This embodiment reduces model parameters and model calculation amount by using the FasterNet block structure in the key point extraction model, thereby improving the model processing speed and facilitating real-time processing of camera points.

[0064] In a feasible embodiment, when training the key point extraction model, the key point detection decoder uses a first loss function to predict key point position information;

[0065] The first loss function is determined based on the cross entropy loss of the region vectors of multiple local regions in the feature map obtained by the key point extraction model and the image region labels corresponding to the local regions; the region vectors are composed of the pixel features of the local region and the key point extraction results of the local region.

[0066] Specifically, in the key point extraction model training stage, the pixel point features in the feature map finally output by the model are expanded into multiple region vectors, each region vector represents a local region in the image, and multiple local regions can overlap. In order to improve the accuracy of the first loss function in predicting the key point position information, in addition to calculating the value of the first loss function based on the pixel point features of the local region, it also includes calculating the value of the first loss function in combination with the key point extraction result of the local region, so as to strengthen the key point feature extraction results of each local region in the calculation of the first loss function, thereby improving the accuracy of predicting the key point position information through the first loss function.

[0067] For example, in the key point extraction model training phase, the pixel features in the feature map output by the model are expanded into multiple The key point extraction results of the local area corresponding to the regional vector are determined, and the key point extraction results are also added to the regional vector to participate in the calculation of the first loss function. The key point extraction results of the local area include the key points in the area and the key points in the area. That is, multiple For example, an “unkey” category is added to the region vector to represent a local region without key points.

[0068] For each After the region vectors of size are normalized, the image region labels corresponding to the region vectors are subjected to cross entropy loss calculation, and the cross entropy loss calculation result is the calculation result of the first loss function. Similarly, the dimension of the image region labels corresponding to the region vectors is also , is determined based on the label of the sample image. The image region label corresponding to the region vector includes not only the feature information of the image region, but also whether the image region contains key points. For example, the first loss function is determined according to the following formula: ,in, Indicates the calculation of cross entropy loss for the vector and label carrying the key point information of the region, p represents each local region in the image, is the region vector of the local area output by the model, is the image region label corresponding to the local region, each label vector The dimension is , that is, whether the local area contains key point labels.

[0069] In a feasible embodiment, the key point extraction model is obtained by distillation training based on the teacher model; when the key point extraction model and the teacher model are distilled and trained, a second loss function is used in the descriptor generation decoder to align the descriptors output by the key point extraction model and the teacher model;

[0070] The second loss function is determined according to the following formula:

[0071] ;

[0072] in, represents the value of the second loss function, represents the first descriptor obtained by the key point extraction model, represents the second descriptor obtained by the teacher model, is an orthogonal matrix, which is determined according to the singular value decomposition results of the first descriptor and the second descriptor.

[0073] In this embodiment, the descriptor results output by the teacher model and the student model are aligned by introducing the orthogonal matrix R. The high-dimensional descriptor output by the teacher model is and the low-dimensional descriptor output by the student model The mapping is performed through the orthogonal matrix R to ensure the accuracy of the loss calculation based on the mapping result, thereby improving the similarity between the two. The student model is the key point extraction model. The orthogonal matrix used for its descriptor is calculated according to the following formula: U and V in, where and are all orthogonal matrices, It is a diagonal matrix, and R is determined based on the calculated U and V. .

[0074] When optimizing the model through the second loss function, the goal is to make the descriptor of the student model and the descriptor of the teacher model In passing After mapping, the alignment is as close as possible. , so that the descriptor of the student model is as close as possible to the descriptor of the teacher model.

[0075] Optionally, the descriptor generator decoder compresses the descriptors using LocalPCA technology, generating a compact multidimensional feature descriptor for each detected keypoint. Traditional PCA compression typically performs statistical analysis on the entire sample dataset. The descriptor generator decoder in this embodiment performs PCA compression on the descriptors of each sample image individually, avoiding information loss that can occur when compressing large datasets while ensuring that the descriptors for each image remain compact.

[0076] During the training process of the key point extraction model, knowledge distillation is used to improve the performance of the model. The total loss of the key point extraction model is given by and Composition, formula is: . It is used to measure the performance of the student model in the detection task, and It is used to ensure that the descriptors output by the student model are consistent with those output by the teacher model in a high-dimensional space. For example, the teacher model selects images with at least 128 key points in the Google Landmark Dataset v2 (GLDv2) dataset in SuperPoint detection for distillation training.

[0077] S230 : Determine an image matching result between the actual image and each first virtual image according to a matching result of key point information between the actual image and each first virtual image.

[0078] S240 : Determine a second virtual image from each first virtual image according to the image matching result, and determine a point position determination result of the actual camera according to the second virtual image.

[0079] The technical solution of this embodiment achieves efficient key point detection and descriptor generation through a key point extraction model. This allows for the extraction of high-quality features from both real and virtual images, improving the accuracy of key point matching based on feature extraction results. Furthermore, the FasterNet block structure is used in the key point extraction model to reduce model parameters, and LocalPCA is used to further compress descriptors. A first loss function and a second loss function are created to accelerate model training convergence. Furthermore, during training, knowledge distillation is used to learn from the teacher model, reducing model size without compromising model accuracy.

[0080] Figure 4 This is a flowchart of another method for determining camera position based on a three-dimensional map provided by an embodiment of the present invention. This embodiment further refines the image matching process in the above embodiment. Figure 4 As shown, the method includes:

[0081] S410: Determine an actual image captured by an actual camera deployed in an actual scene, and determine a plurality of first virtual images captured by a plurality of virtual cameras corresponding to the actual camera.

[0082] S420 : Perform feature extraction on the actual image and each first virtual image to obtain key point information of each image.

[0083] S430. Determine the global relationship between the key points in the actual image and each first virtual image based on a pre-trained graph neural network, and obtain a matching probability between each actual key point in the actual image and each virtual key point in each first virtual image.

[0084] The graph neural network learns and captures the global relationship between all key points in the actual image and the first virtual image, thereby achieving high-precision image feature matching. The main architecture of the graph neural network consists of a key point encoder, a graph neural network message passing layer, and a matching layer. Figure 5The figure shows the architecture diagram of the graph neural network, in which the key point encoder is used to construct the graph structure and encode features based on the key point detection results and corresponding descriptors of the actual image and the first virtual image. The graph neural network message passing layer is used to perform self-attention and cross-attention processing on the feature encoding results of the key point encoder to obtain the feature vectors of the actual image and the first virtual image. The matching layer is used to perform feature matching based on the feature vectors of the actual image and the first virtual image to obtain the matching probability between each actual key point in the actual image and each virtual key point in each first virtual image.

[0085] Specifically, the output result of the graph neural network is the matching probability between each actual key point in the actual image and each virtual key point in the first virtual image, which can be represented by a matching probability matrix. Each value in the matrix represents the similarity between each two key points, and the similarity is represented by the inner product. In addition, the matching probability matrix includes the result of two key points not matching, that is, there are special values ​​in the matrix, such as "unkey", to indicate that the corresponding two key points do not match, which is used to process matching key points that are partially occluded or missing in the image. By adding this category to the matching probability matrix, the accuracy of subsequent key point matching based on the matching probability matrix is ​​improved, avoiding the error influence of occluded or missing matching key points.

[0086] S440 , processing the matching probabilities between the target actual key points in the actual image and the virtual key points in any first virtual image, and determining the virtual key points that successfully match the target actual key points according to the processing results.

[0087] Since the matching probabilities between the actual target key points in the actual image obtained by the graph neural network and the virtual key points in any first virtual image may have different orders of magnitude, in order to ensure the accuracy of subsequent calculations, the processing operation of the matching probability includes at least a normalization operation. In addition, in order to prevent abnormal matching probability values ​​from interfering with key point matching, the processing operation also includes a non-maximum suppression operation. After processing the matching probabilities, the virtual key point with the highest matching probability with the actual target key point is determined based on the processing results. If the matching probability is greater than a preset probability threshold, it is determined that the virtual key point has successfully matched the actual target key point.

[0088] Exemplarily, a matching probability matrix is ​​obtained based on the matching probabilities between the actual keypoints of the target in the actual image and the virtual keypoints in any first virtual image. This matching probability matrix is ​​then normalized using the Sinkhorn algorithm to generate a soft assignment probability matrix. This ensures that the sum of the matching probabilities between any keypoint to be matched in the actual image and each virtual keypoint in the first virtual image in the soft assignment probability matrix is ​​1. The soft assignment probability matrix is ​​then processed using a non-maximum suppression algorithm to remove duplicate or unreliable matching keypoint pairs. Finally, any keypoint pair to be matched whose matching probability exceeds a preset probability threshold is considered a successful match.

[0089] S450 : Determine an image matching probability between the actual image and the first virtual image based on the sum of the matching probabilities between each actual key point in the actual image and the successfully matched virtual key points in the first virtual image, as an image matching result.

[0090] The matching probabilities of all successfully matched key point pairs in the actual image and the first virtual image are summed up. The summed result is the image matching probability between the actual image and the first virtual image, which is used as the image matching result. The image matching probability represents the similarity between the two images.

[0091] Exemplarily, the actual image includes 10 key points and the first virtual image includes 8 key points. After the above-mentioned key point matching, 6 pairs of successfully matched key point pairs are obtained, that is, a one-to-one correspondence is established between the 6 key points in the actual image and the 6 key points in the first virtual image. The image matching probability of the actual image and the first virtual image is determined based on the sum of the matching probability values ​​of the 6 pairs of successfully matched key point pairs.

[0092] S460: Determine a second virtual image from each first virtual image according to the image matching result, and determine a point position determination result of the actual camera according to the second virtual image.

[0093] The technical solution of this embodiment uses a graph neural network approach to match image features, capturing the global relationships between key points and achieving high-precision image matching. It effectively utilizes the attention mechanism and message passing mechanism to improve the accuracy and robustness of image matching.

[0094] Figure 6 This is a flowchart of another method for determining camera position based on a three-dimensional map provided by an embodiment of the present invention. This embodiment further refines the determination of the second virtual image in the above embodiment. Figure 6 As shown, the method includes:

[0095] S610: Determine an actual image captured by an actual camera deployed in an actual scene, and determine a plurality of first virtual images captured by a plurality of virtual cameras corresponding to the actual camera.

[0096] S620: Perform feature extraction on the actual image and each first virtual image to obtain key point information of each image.

[0097] S630: Determine an image matching result between the actual image and each first virtual image according to a matching result of key point information between the actual image and each first virtual image.

[0098] S640 : Sort the first virtual images according to the image matching result to obtain a sorting result.

[0099] In the sorting result, the first virtual image with a high degree of matching with the actual image is located in front of the first virtual image with a low degree of matching with the actual image.

[0100] The image matching result represents the similarity between the first virtual image and the actual image. The first virtual images are sorted from highest to lowest according to the similarity to obtain a sorting result of the first virtual images. That is, according to the sorting result, the similarity between the first virtual image ranked at the top and the actual image is greater than that between the first virtual image ranked at the bottom.

[0101] S650 , traverse each first virtual image based on the sorting result, and convert the virtual image coordinates of each virtual key point in the first virtual image into a world coordinate system according to the camera parameters of the virtual camera corresponding to the first virtual image to obtain each virtual world coordinate.

[0102] According to the sorting results, each first virtual image is traversed, and the matching error of the key points between each first virtual image and the actual image is calculated in turn. The second virtual image closest to the actual image is selected from the first virtual image based on the matching error. This avoids the error caused by determining the second virtual image based solely on the image matching results, and improves the accuracy of camera point determination.

[0103] Since the image matching result is determined based on the image position, when determining the matching error of the key points between the first virtual image and the actual image, the key points are projected into the world coordinate system, and the matching error is determined based on the matching situation in the world coordinate system to improve the accuracy of the matching error determination.

[0104] Specifically, virtual image coordinates are determined based on the position information of each virtual key point in the first virtual image. The virtual image coordinates are converted to a world coordinate system based on the camera parameters of a virtual camera corresponding to the first virtual image to obtain virtual world coordinates of each key point. A correspondence between the image coordinate system and the world coordinate system is determined based on the camera parameters of the virtual camera, and the image coordinates are converted to world coordinates based on the correspondence.

[0105] Optionally, based on the above embodiment, key point pairs that successfully match the actual image and the first virtual image are determined, and the virtual key points in the key point pairs are designated as reference virtual key points, while the actual key points are designated as reference actual key points. The virtual image coordinates of each reference virtual key point in the first virtual image are converted to a world coordinate system based on the camera parameters of the virtual camera corresponding to the first virtual image to obtain virtual world coordinates. This reduces redundant calculations for unmatched virtual key points and actual key points, reduces the computational effort, and thereby improves the efficiency of point determination.

[0106] In a feasible embodiment, converting the virtual image coordinates of each virtual key point in the first virtual image into a world coordinate system according to the camera parameters of the virtual camera corresponding to the first virtual image to obtain each virtual world coordinate includes:

[0107] Converting the virtual image coordinates of each virtual key point in the first virtual image into a camera coordinate system according to camera parameters of the virtual camera corresponding to the first virtual image to obtain virtual camera coordinates;

[0108] Determine the camera extrinsic parameters of the virtual camera according to the camera parameters of the virtual camera corresponding to the first virtual image and the coordinates of each virtual camera;

[0109] The virtual camera coordinates are converted to the world coordinate system according to the camera extrinsic parameters of the virtual camera to obtain the virtual world coordinates.

[0110] In the process of converting image coordinates to world coordinates, the image coordinates are first converted to camera coordinates, that is, the relationship between the image coordinate system and the camera coordinate system is first determined, and then the camera coordinates are converted to world coordinates, that is, the relationship between the camera coordinate system and the world coordinate system is determined.

[0111] Specifically, the relationship between the image coordinate system and the camera coordinate system is determined based on the camera parameters of the virtual camera, and the coordinates of each virtual key point in the virtual camera coordinate system are calculated. The virtual camera coordinates are determined according to the following formula:

[0112] ;

[0113] in, is the focal length of the virtual camera, are the coordinates of the center point in the virtual image, are the coordinates of each virtual key point in the virtual image, is the depth value of each virtual key point, and finally the camera coordinates of the virtual key point in the camera coordinate system are obtained .

[0114] The camera extrinsics of the virtual camera are determined based on the virtual camera coordinates and camera parameters of each key point. The camera extrinsics include the rotation matrix and the translation vector. Specifically, the rotation angle parameters are determined based on the camera parameters of the virtual camera: the rotation angle α around the X axis, the rotation angle β around the Y axis, and the rotation angle γ around the Z axis. The rotation matrix is ​​calculated based on the rotation angle parameters. The calculation formula is:

[0115] .

[0116] Determine the position parameters based on the camera parameters of the virtual camera , the translation vector is calculated based on the position parameters and the rotation matrix , the calculation formula is: Finally, the extrinsic parameter matrix of the virtual camera is .

[0117] According to the extrinsic parameter matrix of the virtual camera, the key point coordinates in the camera coordinate system can be converted to the world coordinate system. The formula is: ;in is the virtual camera coordinate of the i-th virtual key point, is the virtual world coordinate of the i-th virtual key point.

[0118] S660. Select a preset number of first key point pairs from the key point pairs that are successfully matched in the actual image and the first virtual image according to the image matching results, and determine the actual camera pose parameters based on the actual image coordinates of the actual key points and the virtual world coordinates of the virtual key points in the preset number of first key points.

[0119] A portion of the keypoint pairs that successfully match the actual image and the first virtual image are selected as first keypoint pairs, and the actual camera pose parameters are determined based on the virtual world coordinates of the virtual keypoints in the first keypoint pairs and the actual image coordinates of the actual keypoints. Since the actual camera pose parameters are determined based on the successfully matched first keypoint pairs, and since the first keypoint pair is successfully matched, i.e., the virtual keypoint and the actual keypoint are the same point, the virtual world coordinates of the virtual keypoints can also be considered to be the world coordinates corresponding to the actual image coordinates. Therefore, the actual camera pose parameters can be determined based on the actual image coordinates of the actual keypoints in the first keypoints and the virtual world coordinates of the virtual keypoints.

[0120] For example, 6 pairs of first key points that are successfully matched are randomly selected. ,in are the actual image coordinates of the actual keypoints, The virtual world coordinates of the corresponding virtual key points are calculated using the EPnP algorithm to calculate the actual camera pose parameters. , and its corresponding expression relationship is: The number of the selected first key pairs is determined according to the unknown parameter values ​​in the actual camera pose parameters and is not limited here.

[0121] S670: Determine the projection errors of other successfully matched second key point pairs according to the actual camera pose parameters, and determine whether the first virtual image meets the error condition according to the projection errors.

[0122] Among them, the other successfully matched second key point pairs are the remaining key point pairs except the first key point pair among all the successfully matched key point pairs in the actual image and the first virtual image. Since the actual camera pose parameters are determined under the assumption that the first virtual image matches the actual image, and the matching of the first virtual image with the actual image is determined based on the matching key point pairs, if the prerequisite for successful matching is accurate, then the actual image coordinates of the actual key points in the second key points and the virtual world coordinates of the virtual key points also satisfy the corresponding relationship of the actual camera pose parameters. If the actual image coordinates of the actual key points in the second key points and the virtual world coordinates of the virtual key points do not satisfy the corresponding relationship of the actual camera pose parameters, that is, the projection error is large, then it is considered that the prerequisite for successful matching between the actual image and the first virtual image is inaccurate, that is, the actual camera pose parameters are inaccurate.

[0123] Specifically, the virtual world coordinates of the virtual key point in the second key point pair are transformed according to the actual camera pose parameters to obtain the reference image coordinates, and the error between the reference image coordinates and the actual image coordinates of the actual key point in the second key point pair is determined. The error can be determined by the absolute value of the difference between the reference image coordinates and the actual image coordinates. The error is used as the projection error of the second key point pair, and the projection errors of all second key point pairs are determined according to this method. If the number of second key point pairs whose projection error is less than the error threshold is greater than the number threshold, it is determined that the first virtual image and the actual image are successfully matched, and the actual camera pose parameters are the pose parameters of the actual camera.

[0124] For example, based on the calculated actual camera pose parameters, the matching key points in each second key point pair are calculated. Projection error , its formula can be expressed as: .

[0125] S680: If not, continue traversing the first virtual image based on the sorting result until the error condition is met or the preset traversal number is reached.

[0126] Among them, the error condition is whether the number of key point pairs whose projection errors in the second key point pairs are less than the error threshold is greater than the number threshold. If it is less than or equal to the number threshold, it is determined that the first virtual image does not meet the error condition, that is, the first virtual image does not match the actual image, and the traversal is continued according to the sorting result of the above-mentioned first virtual images to determine the next first virtual image until the traversed first virtual image meets the error condition; or the preset traversal number is reached, that is, according to the sorting result, the preset traversal number of first virtual images do not meet the error condition. Since the image matching degree between the subsequent first virtual images and the actual image is low, no further calculation is performed, and the average value of the actual camera pose parameters calculated directly according to the preset traversal number of first virtual images is taken as the final camera point determination result.

[0127] S690: If satisfied, determine that the first virtual image is the second virtual image, and determine the point position determination result of the actual camera according to the actual camera pose parameters determined corresponding to the second virtual image.

[0128] If the number of key point pairs whose projection errors among the second key point pairs in the first virtual image are less than the error threshold is greater than the number threshold, it is determined that the first virtual image meets the error condition, that is, the first virtual image matches the actual image, and the first virtual image is determined to be the second virtual image.

[0129] The actual camera position determination result is determined based on the actual camera pose parameters corresponding to the second virtual image. The position determination result includes the actual camera's position parameters and rotation angle parameters. The actual camera's ID, camera IP address, location range, position parameters, and rotation angle parameters are stored in the database for front-end binding and display.

[0130] For example, the actual camera pose parameters are converted into UE parameters: position parameters and rotation angle parameters. The actual camera pose parameters are converted into Convert to positional parameters , rotation angle parameter The calculation formula is: ; ;in, Represents the value of the first row and first column in the rotation matrix, Represents the value of the second row and first column in the rotation matrix, Represents the value of the third row and first column in the rotation matrix, Represents the value of the third row and second column in the rotation matrix, Represents the value in the third row and third column of the rotation matrix.

[0131] The second virtual image that has been successfully matched is removed from the set of virtual images to be matched, and the next actual image to be matched is selected to continue point determination until all actual images are successfully matched.

[0132] The technical solution of this embodiment realizes the accurate determination of the actual camera pose through the projection conversion between the image coordinate system and the world coordinate system, and realizes the fine-tuning of the camera position parameters and rotation angle parameters through the final determination of the camera point position based on the actual camera pose, thereby avoiding the error problem caused by determining the camera point position based solely on the position parameters and rotation angle parameters of the virtual camera, and realizing high-precision point determination.

[0133] Figure 7 This is a schematic diagram of the structure of a camera point determination device based on a three-dimensional map provided by an embodiment of the present invention. Figure 7 As shown, the device includes:

[0134] An image determination module 710 is configured to determine an actual image captured by an actual camera deployed in an actual scene, and to determine a plurality of first virtual images captured by a plurality of virtual cameras corresponding to the actual camera; wherein the virtual cameras are created in a three-dimensional map corresponding to the actual scene based on camera parameters of the actual cameras;

[0135] A key point extraction module 720 is configured to extract features from the actual image and each of the first virtual images to obtain key point information of each image;

[0136] An image matching module 730 is configured to determine an image matching result between the actual image and each of the first virtual images based on a matching result of the key point information between the actual image and each of the first virtual images;

[0137] The point position determination module 740 is configured to determine a second virtual image from each of the first virtual images according to the image matching result, and determine a point position determination result of the actual camera according to the second virtual image.

[0138] The technical solution of this embodiment creates a corresponding virtual camera based on the actual camera, and obtains a camera point determination result based on the detection and matching results of key points in the images collected by each camera. Based on the camera point determination result, the actual camera can be marked in the three-dimensional map, and the accuracy and consistency of point determination can be significantly improved, thereby reducing the possibility of human marking errors; and the technical solution of this embodiment can increase the point marking speed and realize real-time marking to adapt to real-time and large-scale data scenarios.

[0139] Optional key point extraction module, including:

[0140] Based on a pre-trained key point extraction model, feature extraction is performed on the actual image and each of the first virtual images to obtain key point detection results and corresponding descriptors of each image as the key point information; wherein the descriptors are used to characterize the overall features of the local area where the corresponding key points are located.

[0141] Optionally, the key point extraction model includes a shared feature encoder, a key point detection decoder, and a descriptor generation decoder; the output of the shared feature encoder is the input of the key point detection decoder and the descriptor generation decoder, the output of the key point detection decoder is the key point detection result, and the output of the descriptor generation decoder is the descriptor corresponding to each key point;

[0142] Among them, the shared feature encoder includes at least one convolutional layer and at least one FasterNet block module; the key point detection decoder includes at least one convolutional layer and at least one FasterNet block module; the descriptor generation decoder includes a feature synthesis layer, at least one convolutional layer and at least one FasterNet block module.

[0143] Optionally, when training the key point extraction model, the key point detection decoder uses a first loss function to predict key point position information;

[0144] The first loss function is determined based on the cross entropy loss of the region vectors of multiple local regions in the feature map obtained by the key point extraction model and the image region labels corresponding to the local regions; the region vector is composed of the pixel features of the local region and the key point extraction results of the local region.

[0145] Optionally, the key point extraction model is obtained by distillation training based on the teacher model; when the key point extraction model and the teacher model are distilled and trained, the descriptor generation decoder uses a second loss function to align the descriptors output by the key point extraction model and the teacher model;

[0146] The second loss function is determined according to the following formula:

[0147] ;

[0148] in, represents the value of the second loss function, represents the first descriptor obtained by the key point extraction model, represents the second descriptor obtained by the teacher model, is an orthogonal matrix, and is determined according to the singular value decomposition results of the first descriptor and the second descriptor.

[0149] Optional image matching module, specifically used for:

[0150] Determining a global relationship between key points in the actual image and each of the first virtual images based on a pre-trained graph neural network to obtain a matching probability between each actual key point in the actual image and each virtual key point in each of the first virtual images;

[0151] Processing the matching probabilities between the actual key points of the target in the actual image and the virtual key points in any of the first virtual images, and determining the virtual key points that successfully match the actual key points of the target based on the processing results;

[0152] An image matching probability between the actual image and the first virtual image is determined according to the sum of the matching probabilities between each actual key point in the actual image and the successfully matched virtual key points in the first virtual image, as an image matching result.

[0153] Optional point determination module, including:

[0154] a first virtual image sorting unit, configured to sort the first virtual images according to the image matching result to obtain a sorting result; wherein, in the sorting result, first virtual images having a high degree of matching with the actual image are positioned before first virtual images having a low degree of matching with the actual image;

[0155] a virtual world coordinate determining unit, configured to traverse each first virtual image based on the sorting result, and convert the virtual image coordinates of each virtual key point in the first virtual image into a world coordinate system according to camera parameters of a virtual camera corresponding to the first virtual image, to obtain each virtual world coordinate;

[0156] an actual camera pose parameter determination unit, configured to select a preset number of first key point pairs from the key point pairs that are successfully matched in the actual image and the first virtual image according to the image matching result, and determine the actual camera pose parameters according to the actual image coordinates of the actual key points and the virtual world coordinates of the virtual key points in the preset number of first key points;

[0157] a projection error determining unit, configured to determine, based on the actual camera pose parameters, the projection errors of other successfully matched second key point pairs, and determine, based on the projection errors, whether the first virtual image satisfies an error condition;

[0158] a first judging unit, configured to, if not satisfied, continue traversing the first virtual image based on the sorting result until the error condition is satisfied or a preset traversal number is reached;

[0159] The second judgment unit is configured to determine that the first virtual image is a second virtual image if the conditions are met, and determine a point position determination result of the actual camera according to the actual camera pose parameters corresponding to the second virtual image.

[0160] Optionally, the virtual world coordinate determination unit is specifically configured to:

[0161] converting the virtual image coordinates of each virtual key point in the first virtual image into a camera coordinate system according to camera parameters of the virtual camera corresponding to the first virtual image to obtain virtual camera coordinates;

[0162] Determine the camera extrinsic parameters of the virtual camera according to the camera parameters of the virtual camera corresponding to the first virtual image and the coordinates of each virtual camera;

[0163] The virtual camera coordinates are converted into a world coordinate system according to the camera extrinsic parameters of the virtual camera to obtain virtual world coordinates.

[0164] The camera point position determination device based on a three-dimensional map provided in an embodiment of the present invention can execute the camera point position determination method based on a three-dimensional map provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0165] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0166] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0167] Figure 8 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0168] like Figure 8As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0169] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0170] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the method for determining camera point positions based on a three-dimensional map.

[0171] In some embodiments, the method for determining a camera position based on a three-dimensional map can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for determining a camera position based on a three-dimensional map described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method for determining a camera position based on a three-dimensional map via any other suitable means (e.g., via firmware).

[0172] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific reference products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0176] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes switch components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, switch components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0177] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0178] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0179] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0180] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for determining camera position based on a three-dimensional map, characterized in that: The method includes: Determining an actual image captured by an actual camera deployed in an actual scene, and determining a plurality of first virtual images captured by a plurality of virtual cameras corresponding to the actual camera; wherein the virtual camera is created in a UE model of a three-dimensional map corresponding to the actual scene based on camera parameters of the actual camera; creating the plurality of virtual cameras based on a plurality of candidate point information and a plurality of candidate rotation angle information, wherein the internal parameters of the virtual cameras are the same as the internal parameters of the actual cameras; the candidate point information is determined based on the area range corresponding to the actual camera in the three-dimensional map, and the candidate rotation angle information is determined based on the rotation angle range of the actual camera; Performing feature extraction on the actual image and each of the first virtual images to obtain key point information of each image; determining an image matching result between the actual image and each of the first virtual images according to a matching result of the key point information between the actual image and each of the first virtual images; sorting the first virtual images according to the image matching results to obtain a sorting result; wherein, in the sorting result, first virtual images with a high degree of matching with the actual image are located before first virtual images with a low degree of matching with the actual image; traversing each first virtual image based on the sorting result, and converting the virtual image coordinates of each virtual key point in the first virtual image into a world coordinate system according to the camera parameters of the virtual camera corresponding to the first virtual image to obtain each virtual world coordinate; Selecting a preset number of first key point pairs from the key point pairs that successfully match the actual image and the first virtual image according to the image matching result, and determining actual camera pose parameters according to the actual image coordinates of the actual key points and the virtual world coordinates of the virtual key points in the preset number of first key points; Determining projection errors of other successfully matched second key point pairs according to the actual camera pose parameters, and determining whether the first virtual image meets an error condition according to the projection errors; If not, continue traversing the first virtual image based on the sorting result until the error condition is met or a preset traversal number is reached; If so, the first virtual image is determined to be a second virtual image, and the point position determination result of the actual camera is determined according to the second virtual image.

2. The method according to claim 1, characterized in that Performing feature extraction on the actual image and each of the first virtual images to obtain key point information of each image includes: Based on a pre-trained key point extraction model, feature extraction is performed on the actual image and each of the first virtual images to obtain key point detection results and corresponding descriptors of each image as the key point information; wherein the descriptors are used to characterize the overall features of the local area where the corresponding key points are located.

3. The method according to claim 2, characterized in that in, The key point extraction model includes a shared feature encoder, a key point detection decoder and a descriptor generation decoder; the output of the shared feature encoder is the input of the key point detection decoder and the descriptor generation decoder, the output of the key point detection decoder is the key point detection result, and the output of the descriptor generation decoder is the descriptor corresponding to each key point; Among them, the shared feature encoder includes at least one convolutional layer and at least one FasterNet block module; the key point detection decoder includes at least one convolutional layer and at least one FasterNet block module; the descriptor generation decoder includes a feature synthesis layer, at least one convolutional layer and at least one FasterNet block module.

4. The method according to claim 3, characterized in that in, When training the key point extraction model, the key point detection decoder uses a first loss function to predict key point position information; The first loss function is determined based on the cross entropy loss of the region vectors of multiple local regions in the feature map obtained by the key point extraction model and the image region labels corresponding to the local regions; the region vector is composed of the pixel features of the local region and the key point extraction results of the local region.

5. The method according to claim 3, characterized in that in, The key point extraction model is obtained by distillation training based on the teacher model; when the key point extraction model and the teacher model are distilled and trained, the descriptor generation decoder uses a second loss function to align the descriptors output by the key point extraction model and the teacher model; The second loss function is determined according to the following formula: ； in, represents the value of the second loss function, represents the first descriptor obtained by the key point extraction model, represents the second descriptor obtained by the teacher model, is an orthogonal matrix, and is determined according to the singular value decomposition results of the first descriptor and the second descriptor.

6. The method according to claim 1, characterized in that Determining an image matching result between the actual image and each of the first virtual images according to a matching result of the key point information between the actual image and each of the first virtual images includes: Determining a global relationship between key points in the actual image and each of the first virtual images based on a pre-trained graph neural network to obtain a matching probability between each actual key point in the actual image and each virtual key point in each of the first virtual images; Processing the matching probabilities between the actual key points of the target in the actual image and the virtual key points in any of the first virtual images, and determining the virtual key points that successfully match the actual key points of the target based on the processing results; An image matching probability between the actual image and the first virtual image is determined according to the sum of the matching probabilities between each actual key point in the actual image and the successfully matched virtual key points in the first virtual image, as an image matching result.

7. The method according to claim 1, characterized in that Determining a point position determination result of the actual camera according to the second virtual image includes: The point position determination result of the actual camera is determined according to the actual camera pose parameters corresponding to the second virtual image.

8. The method according to claim 1, characterized in that Converting the virtual image coordinates of each virtual key point in the first virtual image to a world coordinate system according to camera parameters of a virtual camera corresponding to the first virtual image to obtain each virtual world coordinate includes: converting the virtual image coordinates of each virtual key point in the first virtual image into a camera coordinate system according to camera parameters of the virtual camera corresponding to the first virtual image to obtain virtual camera coordinates; Determine the camera extrinsic parameters of the virtual camera according to the camera parameters of the virtual camera corresponding to the first virtual image and the coordinates of each virtual camera; The virtual camera coordinates are converted into a world coordinate system according to the camera extrinsic parameters of the virtual camera to obtain virtual world coordinates.

9. A camera point determination device based on a three-dimensional map, characterized in that: The device includes: An image determination module is configured to determine an actual image captured by an actual camera deployed in an actual scene, and to determine a plurality of first virtual images captured by a plurality of virtual cameras corresponding to the actual camera; wherein the virtual camera is created based on camera parameters of the actual camera in a UE model of a three-dimensional map corresponding to the actual scene; the plurality of virtual cameras are created based on a plurality of candidate point information and a plurality of candidate rotation angle information, wherein the internal parameters of the virtual cameras are the same as the internal parameters of the actual cameras; the candidate point information is determined based on the area range corresponding to the actual camera in the three-dimensional map, and the candidate rotation angle information is determined based on the rotation angle range of the actual camera; A key point extraction module, configured to extract features from the actual image and each of the first virtual images to obtain key point information of each image; an image matching module, configured to determine an image matching result between the actual image and each of the first virtual images based on a matching result of the key point information between the actual image and each of the first virtual images; a point position determination module, configured to determine a second virtual image from each of the first virtual images according to the image matching result, and determine a point position determination result of the actual camera according to the second virtual image; Among them, the point determination module includes: a first virtual image sorting unit, configured to sort the first virtual images according to the image matching result to obtain a sorting result; wherein, in the sorting result, first virtual images having a high degree of matching with the actual image are positioned before first virtual images having a low degree of matching with the actual image; a virtual world coordinate determining unit, configured to traverse each first virtual image based on the sorting result, and convert the virtual image coordinates of each virtual key point in the first virtual image into a world coordinate system according to camera parameters of a virtual camera corresponding to the first virtual image, to obtain each virtual world coordinate; an actual camera pose parameter determination unit, configured to select a preset number of first key point pairs from the key point pairs that are successfully matched in the actual image and the first virtual image according to the image matching result, and determine the actual camera pose parameters according to the actual image coordinates of the actual key points and the virtual world coordinates of the virtual key points in the preset number of first key points; a projection error determining unit, configured to determine, based on the actual camera pose parameters, the projection errors of other successfully matched second key point pairs, and determine, based on the projection errors, whether the first virtual image satisfies an error condition; a first judging unit, configured to, if not satisfied, continue traversing the first virtual image based on the sorting result until the error condition is satisfied or a preset traversal number is reached; The second judgment unit is configured to determine that the first virtual image is a second virtual image if the conditions are met, and determine a point position determination result of the actual camera according to the second virtual image.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so as to enable the at least one processor to perform the camera point position determination method based on a three-dimensional map according to any one of claims 1 to 8.

Images