Positioning method and device, electronic device, and computer-readable storage medium
By using semantic consistency weight values in electronic devices to optimize the positioning method, the failure of traditional SLAM positioning method under lighting and environmental changes is solved, the position information accuracy based on neural network is improved, and the lighting intensity is achieved. Accurate positioning of images in the case of weak literature and science.
Patent Information
- Application Number
- CN202010742468.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-12-06
AI Technical Summary
Traditional SLAM positioning methods are prone to failure in the case of light changes, environmental changes and image art changes. The positioning information error of neural network-based positioning methods is large and difficult to meet the practical application needs.
By acquiring the image to be positioned for pose recognition processing, the semantic consistency weight value of the corresponding reference image is determined. If the weight value is greater than the threshold, the target pose information will be obtained based on the relevant information of the reference image and the initial pose information, and the target pose information will be obtained by optimizing and adjusting the relevant information of the reference image and the initial pose information. .
It improves the accuracy of positioning and navigation of electronic devices in various scenarios, and ensures positioning reliability in situations such as high light intensity and weak image and art.
Smart Images

Figure CN111862213B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a positioning method and device, an electronic device, and a computer-readable storage medium. Background Art
[0002] Simultaneous localization and mapping (SLAM) technology is a technology that allows electronic devices to locate themselves and build environmental maps based on sensor information. It is widely used in robots, drones, unmanned driving, virtual reality and other fields.
[0003] At present, the positioning results of traditional SLAM positioning methods are affected by changes in lighting, environment and texture, and positioning failure may occur in cases of high lighting intensity and weak image texture. Although the positioning method based on neural networks can estimate the position and posture information of electronic devices, the position and posture information estimated by the neural network has a large error with the actual position and posture information of the electronic device, which is difficult to meet the actual application needs. Summary of the invention
[0004] Embodiments of the present application provide a positioning method and device, an electronic device, and a computer-readable storage medium.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The present application provides a positioning method, which is applied to an electronic device. The method includes:
[0007] Acquire the image to be positioned, and perform posture recognition processing on the image to be positioned to obtain initial posture information of the electronic device when the image to be positioned is acquired;
[0008] Determining a semantic consistency weight value of a reference image corresponding to the image to be located; the reference image refers to an image collected by the electronic device and adjacent to the image to be located;
[0009] The semantic consistency weight value is used to characterize the similarity between the reference semantic information of the reference image and the first semantic information of the first image; the first image refers to an image matching the reference image in the scene map of the scene where the electronic device is located;
[0010] If the semantic consistency weight value of the reference image is greater than the weight threshold, the target posture information of the electronic device when acquiring the image to be located is determined based on the relevant information of the reference image and the initial posture information.
[0011] The present application provides a bit device, which is applied to an electronic device, and the device includes:
[0012] An acquisition unit, used for acquiring an image to be located;
[0013] A first processing unit is used to perform posture recognition processing on the image to be positioned to obtain initial posture information of the electronic device when the image to be positioned is collected;
[0014] A determination unit, configured to determine a semantic consistency weight value of a reference image corresponding to the image to be located; the reference image refers to an image collected by the electronic device and adjacent to the image to be located;
[0015] The semantic consistency weight value is used to characterize the similarity between the reference semantic information of the reference image and the first semantic information of the first image; the first image refers to an image matching the reference image in the scene map of the scene where the electronic device is located;
[0016] The second processing unit is used to determine the target posture information of the electronic device when acquiring the image to be located based on the relevant information of the reference image and the initial posture information if the semantic consistency weight value of the reference image is greater than the weight threshold.
[0017] An embodiment of the present application provides an electronic device, the electronic device comprising a processor and a memory for storing a computer program that can be run on the processor;
[0018] The processor and the memory are connected via a communication bus;
[0019] Wherein, when the processor runs the computer program stored in the memory, the steps of the above-mentioned positioning method are executed.
[0020] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the method provided by the embodiment of the present application.
[0021] The embodiments of the present application have the following beneficial effects:
[0022] The positioning method provided by the embodiment of the present application first obtains the image to be positioned, and performs posture recognition processing on the image to be positioned to obtain the initial posture information of the electronic device when the image to be positioned is collected; then, the semantic consistency weight value of the reference image corresponding to the image to be positioned is determined; the reference image refers to an image adjacent to the image to be positioned collected by the electronic device; wherein the semantic consistency weight value is used to characterize the similarity between the reference semantic information of the reference image and the first semantic information of the first image; the first image refers to an image matching the reference image in the scene map of the scene where the electronic device is located; finally, if it is detected that the semantic consistency weight value of the reference image is greater than the weight threshold, the target posture information of the electronic device when the image to be positioned is collected is determined based on the relevant information of the reference image and the initial posture information. In this way, when the semantic consistency weight value of the reference image is greater than the weight threshold, the electronic device can optimize and adjust the initial posture information corresponding to the image to be positioned based on the relevant information of the reference image to obtain accurate target posture information. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A schematic diagram of a positioning method provided in an embodiment of the present application Figure 1 ;
[0024] Figure 2 A schematic diagram of a positioning method provided in an embodiment of the present application Figure 2 ;
[0025] Figure 3 A schematic diagram of a training process of an exemplary image processing model provided in an embodiment of the present application;
[0026] Figure 4 A schematic diagram of a positioning method provided in an embodiment of the present application Figure 3 ;
[0027] Figure 5 A schematic diagram of reprojection error calculation provided in an embodiment of the present application;
[0028] Figure 6 A schematic diagram of a scene map construction process provided in an embodiment of the present application;
[0029] Figure 7 A schematic diagram of a positioning method provided in an embodiment of the present application Figure 4 ;
[0030] Figure 8 A schematic diagram of the structure of a positioning device provided in an embodiment of the present application;
[0031] Fig. 9A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present invention.
[0033] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as those commonly understood by those skilled in the art of the present application. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.
[0034] In response to the problems in the related art, the embodiments of the present application provide a positioning method, and the execution subject of the positioning method can be the positioning device provided in the embodiments of the present application, or an electronic device integrated with the positioning device, and the positioning device here can be implemented in hardware or software. Among them, the electronic devices include but are not limited to smart phones, augmented reality (AR) devices, intelligent robots, laptops, smart wearable devices, etc. Here, the electronic device can realize the positioning and navigation functions of the electronic device in various scenarios (such as indoor scenarios) through the positioning method provided in the embodiments of the present application.
[0035] Please refer to 1, Figure 1 A schematic diagram of a positioning method provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the positioning method includes steps 110 to 130. Among them,
[0036] Step 110: Acquire the image to be positioned, and perform posture recognition processing on the image to be positioned to obtain initial posture information of the electronic device when the image to be positioned is acquired.
[0037] In the embodiments provided in the present application, the electronic device may capture multiple continuous frames of images or a continuous video stream in the current scene through a camera, process each frame of the captured multiple continuous frames of images or the continuous video stream, and obtain the posture information of the electronic device when capturing each frame of the image, thereby determining the accurate posture information of the electronic device in the scene map of the current scene.
[0038] It should be noted that the electronic device can obtain the scene map of the current scene from the cloud server, and the electronic device can also pre-build the scene map of the scene, wherein different scenes can correspond to different scene maps; each scene map includes environmental features in the scene, such as walls, pillars, etc. Here, the scene map can be a three-dimensional point cloud map, that is, the map is represented by multiple groups of discrete point cloud data; the scene map can also be a two-dimensional map, that is, the map is represented by multiple frames of red, green and blue (RGB) images. The embodiment of the present application does not limit the type of scene map here.
[0039] In this way, when the electronic device is located in an unfamiliar environment, the electronic device can collect images of the surrounding environment and compare them with the scene map of the current scene, so as to determine the position of the electronic device in the current scene map and achieve positioning navigation.
[0040] Here, the image to be positioned mentioned in the embodiment of the present application is any one frame of the above-mentioned continuous multiple frames of images.
[0041] In a possible implementation, the electronic device may process the image to be positioned by using a pre-trained neural network model to obtain initial position information of the electronic device when the image to be positioned is acquired.
[0042] In another possible implementation, the electronic device may perform feature matching between the image to be positioned and the information in the scene map to obtain the initial position information of the electronic device when the positioning image is acquired.
[0043] The embodiment of the present application does not limit the method for determining the initial posture information of the electronic device when capturing the positioning image.
[0044] Step 120: Determine the semantic consistency weight value of the reference image corresponding to the image to be located; the reference image refers to an image captured by the electronic device and adjacent to the image to be located.
[0045] Among them, the semantic consistency weight value is used to characterize the similarity between the reference semantic information of the reference image and the first semantic information of the first image; the first image refers to an image that matches the reference image in the scene map of the scene where the electronic device is located.
[0046] In practical applications, it is inaccurate to determine the pose information collected in the scene map only through the information of one frame of the image to be located. Therefore, in an embodiment of the present application, the electronic device can also obtain a reference image that is associated with the image to be located, and determine the target pose information corresponding to the current image to be located in combination with the relevant information of the reference image.
[0047] In the embodiments provided in the present application, the electronic device can collect multiple consecutive frames of images in the current scene through a camera, and the reference image is an image in the multiple consecutive frames of images that is adjacent to the image to be located. For example, the reference image can be a frame image before the image to be located, or a frame image after the image to be located.
[0048] Here, the electronic device can determine the similarity between the reference image and the specific image in the scene map. If the similarity between the reference image and the specific image is higher, it indicates that the correlation between the reference image and the scene map is higher. Conversely, the correlation between the reference image and the scene map is lower. In this way, the initial position and pose information of the image to be located can be adjusted based on the relevant information of the reference image that has a high correlation with the scene map, which can further improve the accuracy of the position and pose information of the image to be located.
[0049] It should be noted that the specific image is the first image mentioned in step 120, and the first image is the image in the scene map that matches the reference image.
[0050] In the embodiments provided in the present application, the electronic device may use the semantic consistency weight value of the reference image to measure the similarity between the reference image and the first image.
[0051] Specifically, the electronic device can determine the reference semantic information of the reference image and the first semantic information of the first image, and then calculate the similarity between the reference semantic information and the first semantic information to obtain the semantic consistency weight value of the reference image. The semantic information refers to the category of each pixel in the image.
[0052] It can be understood that the higher the semantic consistency weight value of the reference image, the higher the correlation between the reference image and the scene map, and the lower the semantic consistency weight value of the reference image, the lower the correlation between the reference image and the scene map.
[0053] Step 130: If the semantic consistency weight value of the reference image is greater than the weight threshold, the target posture information of the electronic device when acquiring the image to be positioned is determined based on the relevant information of the reference image and the initial posture information.
[0054] In the embodiments provided in the present application, the electronic device can determine the relationship between the semantic consistency weight value of the reference image and the weight threshold. If the semantic consistency weight value of the reference image is greater than the weight threshold, it is considered that the reference image has a greater correlation with the scene map. Conversely, if the semantic consistency weight value of the reference image is less than or equal to the weight threshold, it is considered that the reference image has a smaller correlation with the scene map.
[0055] In this way, when the semantic consistency weight value of the reference image is greater than the weight threshold, the electronic device can optimize and adjust the initial pose information corresponding to the image to be positioned based on the relevant information of the reference image to obtain accurate target pose information.
[0056] In the embodiments provided in the present application, the relevant information of the reference image may include: reference pose information of the reference image, and / or key feature points of the reference image. Here, the key feature points of the reference image refer to feature points of key areas in the reference image, for example, key feature points may be points with stronger textures or points with greater color changes in the reference image.
[0057] It can be seen that the positioning method provided in the embodiment of the present application can determine the semantic consistency weight value of the reference image corresponding to the image to be positioned, and use the relevant information of the reference image with a larger semantic consistency weight value to obtain the target posture information of the electronic device when collecting the current image to be positioned, thereby improving the accuracy of posture calculation.
[0058] The following is an exemplary introduction to the process of determining the semantic consistency weight value of the reference image corresponding to the image to be located in step 120.
[0059] refer to Figure 2 A schematic diagram of a positioning method process shown Figure 2 , step 120 can be implemented by steps 1201 to 1203. Among them,
[0060] Step 1201: Obtain reference pose information and reference semantic information corresponding to a reference image.
[0061] Here, the electronic device may perform posture recognition processing and semantic segmentation processing on the reference image to obtain reference posture information of the electronic device when the reference image is collected, and reference semantic information of the reference image.
[0062] In a possible implementation, step 1201 may be implemented in the following manner:
[0063] Through the image processing model, the reference image is subjected to posture recognition processing and semantic segmentation processing to obtain the reference posture information of the electronic device when the reference image is collected, as well as the reference semantic information of the reference image.
[0064] In the embodiments provided in the present application, the image processing model may be a pre-trained neural network model, wherein the image processing model may be obtained by training the electronic device on sample images, or may be obtained by the electronic device from a cloud server that provides the image processing model.
[0065] Here, the image processing model is used to determine the posture information of the electronic device when capturing the image, and the semantic information of the image. In other words, the image processing model can be a multi-task processing model, that is, the image processing model can process the same image and obtain two different types of processing results, namely, the posture information when capturing the image, and the semantic information of the image, which can improve the efficiency of image processing.
[0066] In one possible implementation, the image processing model may include a convolutional layer, a temporal recurrent neural network, and a semantic segmentation network.
[0067] Specifically, the above-mentioned image processing model is used to perform posture recognition processing and semantic segmentation processing on the reference image to obtain the reference posture information of the electronic device when the reference image is collected, and the reference semantic information of the reference image, which can be specifically achieved by the following steps:
[0068] Step 1201a, extracting features of the reference image through a convolutional layer to obtain reference image features of the reference image;
[0069] Step 1201b: Based on the relevant information of the previous frame image adjacent to the reference image, perform posture recognition processing on the reference image features through a time recurrent neural network to obtain reference posture information of the electronic device when the reference image is collected;
[0070] Step 1201c: perform semantic segmentation processing on the reference image features through a semantic segmentation network to obtain reference semantic information of the reference image.
[0071] It can be understood that the convolution layer is used to perform convolution processing on the reference image, that is, the convolution layer can extract image features of the reference image to obtain reference image features.
[0072] A time recurrent neural network is used to determine the posture information of the electronic device when capturing an image. In the embodiment provided in the present application, the electronic device can use the relevant information of the previous frame image adjacent to the reference image and the extracted reference image features as inputs to the time recurrent neural network, and obtain the reference posture information of the electronic device when capturing the reference image after being processed by the time recurrent neural network.
[0073] In a possible implementation, the time recurrent neural network may be a long short-term memory network (Long Short-Term Memory, LSTM).
[0074] In addition, the semantic segmentation network is specifically used to determine the semantic information of the image. In the embodiment provided in the present application, the electronic device can use the reference image features obtained by the convolution layer as the input of the semantic segmentation network, and obtain the reference semantic information of the reference image after being processed by the semantic segmentation network.
[0075] It should be noted that the execution order of step 1201b and step 1201c is not specific.
[0076] Step 1202: Determine a first image matching the reference pose information from the scene map, and obtain first semantic information corresponding to the first image.
[0077] In the embodiments provided in the present application, the electronic device may determine a first image that matches a reference image based on reference pose information of the reference image.
[0078] It should be noted that, in addition to including multiple groups of point cloud data or multiple frames of scene images, the scene map may also include the posture information corresponding to each group of point cloud data or each frame of scene image. Thus, in the embodiment provided by the present application, the electronic device can search for the posture information matching the reference posture information from the scene map based on the posture information of the reference image, and determine the scene image corresponding to the found posture information as the first image.
[0079] Furthermore, after obtaining the first image, the electronic device may further obtain first semantic information of the first image. In a feasible exemplary embodiment, the scene map may also include semantic information of each set of point cloud data or each frame of image, so that the electronic device may directly obtain the first semantic information of the first image from the scene map. In another feasible example, after obtaining the first image, the electronic device may indirectly obtain the first semantic information by performing semantic segmentation processing on the first image. The embodiment of the present application does not limit the method for obtaining the first semantic information.
[0080] In a possible implementation, the scene map may be a three-dimensional point cloud map, that is, the scene map is represented by multiple sets of discrete point cloud data.
[0081] Based on this, step 1202 determines the first image matching the reference pose information from the scene map, which can be achieved by the following steps:
[0082] Step 1202a: searching the scene map for a first three-dimensional point cloud set that matches the reference pose information;
[0083] Step 1202b: Project the points in the first three-dimensional point cloud set into the coordinate system of the reference image to obtain a first image.
[0084] Specifically, the reference pose information of the reference image may include a rotation matrix R and a translation vector T of the electronic device. The electronic device may determine a first three-dimensional point cloud set P[X] matching the reference pose information (R, T) in the scene map based on the reference pose information (R, T) of the reference image. w Y w Z w] T .
[0085] Furthermore, the electronic device may reproject the first three-dimensional point cloud set into the image coordinate information of the reference image. Specifically, the coordinates of the first three-dimensional point cloud set in the reference image coordinate system may be obtained by formula (1).
[0086]
[0087] Among them, [uv 1] T is the coordinate of the first three-dimensional point cloud set in the reference image coordinate system, K is the intrinsic parameter matrix of the electronic device camera, and s is the scale information between the scene map coordinate system and the reference image coordinate system.
[0088] In this way, the electronic device can obtain the first image based on the coordinates of the first three-dimensional point cloud set in the reference image coordinate system.
[0089] Step 1203: Calculate the similarity between the reference semantic information and the first semantic information to obtain a semantic consistency weight value of the reference image.
[0090] In a possible implementation, the three-dimensional point cloud map may also store semantic information of each point in the three-dimensional point cloud map.
[0091] In this way, after obtaining the first image through steps 1202a and 1202b, the electronic device can also obtain first semantic information corresponding to the first image from the three-dimensional point cloud map.
[0092] In the embodiments provided in the present application, the electronic device can obtain the semantic consistency weight value of the reference image by calculating the Euclidean distance between the reference semantic information and the first semantic information; the electronic device can also calculate the repetition rate of the reference semantic information and the first semantic information to obtain the semantic consistency weight value of the reference image. The embodiments of the present application are not limited here.
[0093] The following is an example of the training process of the image processing model.
[0094] In the embodiment provided in the present application, the image processing model is obtained by training the image processing model to be trained, wherein the model to be trained includes: a convolutional layer to be trained, a time recurrent neural network to be trained, and a semantic segmentation network to be trained.
[0095] refer to Figure 3 A schematic diagram of a training process of an exemplary image processing model is shown; the training process may include the following steps:
[0096] The first step is to obtain multiple frames of sample images 3-1 taken continuously.
[0097] Here, the multiple sample image frames 3-1 may be images included in a continuously shot video stream, and each sample image frame of the multiple sample image frames 3-1 has temporal continuity.
[0098] The second step is to input the multiple frames of sample images 3-1 into the convolutional layer to be trained 3-2 for feature extraction according to the time sequence of the acquisition of the multiple frames of sample images 3-1, and obtain multiple sample image features 3-3.
[0099] Here, the multiple frames of sample images 3-1 acquired in the first step can be continuously input into the convolution layer for processing in the order of acquisition time, that is, the image features of each frame of sample image are extracted through the convolution layer. In this way, after the convolution layer processing, multiple sample image features 3-3 corresponding to the multiple frames of continuous sample images 3-1 are obtained.
[0100] It should be noted that there is a one-to-one correspondence between a plurality of sample images and a plurality of sample image features, that is, one sample image corresponds to one sample image feature.
[0101] The third step is to determine multiple predicted posture information 3-5 corresponding to multiple sample image features 3-3 through the time recurrent neural network 3-4 to be trained.
[0102] Here, after obtaining multiple sample image features 3-3, the multiple sample image features 3-3 corresponding to multiple consecutive frames of sample images are input into the time recurrent neural network 3-4 to be trained. The time recurrent neural network 3-4 to be trained can transfer the relevant information of the previous frame of the sample image of the current sample image to the posture prediction of the next frame of the sample image. By reasonably utilizing the relevant information between the sample images and then performing posture prediction, the calculation accuracy of the posture can be improved.
[0103] It should be noted that there is a one-to-one correspondence between the multiple sample image features and the multiple predicted pose information. It can be understood that there is also a one-to-one correspondence between the multiple sample images and the multiple predicted pose information.
[0104] Specifically, in the third step, the multiple predicted pose information corresponding to the multiple sample image features is determined by the time recurrent neural network to be trained, which can be achieved by the following steps:
[0105] Step 1: Based on the relevant information of the i-1th frame sample image, the sample features corresponding to the i-th frame sample image are processed by the time recurrent neural network to be trained to determine the predicted pose information corresponding to the i-th frame sample image and the relevant information of the i-th frame sample image; wherein i is an integer greater than 1 and less than or equal to N;
[0106] Step 2: Based on the relevant information of the sample image of the i-th frame, continue to perform posture recognition processing on the corresponding sample features of the sample image of the i+1-th frame through the time recurrent neural network to be trained, and determine the predicted posture information corresponding to the sample image of the i+1-th frame, until i=N-1, to obtain the predicted posture information corresponding to multiple sample image features respectively; N is the total number of the multiple frames of sample images.
[0107] It can be understood that when the electronic device performs posture recognition processing on the sample image features of the current i-th frame sample image through the time recurrent neural network to be trained, the relevant information of the previous frame sample image of the i-th frame sample image (i.e., the i-1-th frame sample image) and the image feature information of the i-th frame sample image can be input into the time recurrent neural network to be trained for processing to obtain the predicted posture information corresponding to the current i-th frame sample image.
[0108] Next, the electronic device continues to perform posture recognition processing on the sample image features of the i+1th frame sample image. Specifically, the electronic device can use the sample image features of the i+1th frame sample image and related information of the previous frame sample image (i.e., the i-th frame sample image) as inputs to the time recurrent neural network to be trained, and obtain the predicted posture information corresponding to the i+1th frame sample image after processing by the time recurrent neural network to be trained.
[0109] Furthermore, the electronic device continues to perform posture recognition processing on the sample image of the i+2th sample image in the above manner until the value of i is N-1. In this way, the electronic device can obtain the predicted posture information corresponding to each frame of the sample image.
[0110] The fourth step is to determine multiple predicted semantic information 3-7 corresponding to multiple sample image features 3-3 through the semantic segmentation network 3-6 to be trained.
[0111] Here, the electronic device can perform semantic segmentation processing on the sample image features 3-3 corresponding to the multiple sample images obtained in the second step through the semantic segmentation network 3-6 to be trained, and obtain predicted semantic information 3-7 corresponding to each sample image feature.
[0112] It should be noted that the execution order of the third and fourth steps is not specific.
[0113] The fifth step is to adjust the network parameters of the image processing model to be trained based on the loss of multiple predicted pose information 3-5 and the loss of multiple predicted semantic information 3-7 so that the loss of the output result of the image processing model meets the convergence condition.
[0114] In the embodiments provided in the present application, since the image processing model to be trained is an initial model, the predicted pose information and predicted semantic information obtained after the sample image is processed by the image processing model to be trained are not the real pose information and semantic information of the sample image. There are differences between the predicted pose information and the real pose information, and between the predicted semantic information and the real pose information.
[0115] Here, the loss between each predicted pose information and the actual pose information of the sample image corresponding to the predicted pose information, as well as the loss between each predicted semantic information and the actual semantic information of the sample image corresponding to the predicted semantic information can be calculated to adjust the network parameters of the image processing model to be trained, that is, adjust the network parameters of the convolutional layer to be trained, the time recurrent neural network to be trained, and the semantic segmentation network to be trained.
[0116] In the embodiments provided in the present application, the electronic device may train the image processing model to be trained by using a multi-classification loss function for semantic segmentation and a Euclidean distance loss function for pose recognition.
[0117] Furthermore, the electronic device continues to perform the same processing as the second to fourth steps on the sample image based on the adjusted image processing model to be trained, and obtains a set of output results (i.e., multiple predicted pose information and multiple predicted semantic information). When the output result meets the convergence condition, the training process of the image processing model to be trained is stopped to obtain a trained image processing model. When the output result does not meet the convergence condition, the network parameters of the image processing model to be trained are adjusted according to the loss of multiple predicted pose information and the loss of multiple predicted semantic information, and the sample image is continued to be processed in the same way as the second to fourth steps through the adjusted image processing model to be trained to obtain a set of output results, until the output result meets the convergence condition.
[0118] Here, the convergence condition may be that the loss value between each predicted pose information and its corresponding true pose information is less than a first specific value, and the loss value between each predicted semantic information and its corresponding true semantic information is less than a second specific value.
[0119] During the training process of the image processing model to be trained, the embodiment of the present application can reasonably use the relevant information between images to perform posture prediction, thereby improving the accuracy of posture calculation.
[0120] Based on the foregoing embodiment, in the embodiment provided by the present application, in step 110, the posture recognition processing of the image to be positioned is performed to obtain the initial posture information of the electronic device when the image to be positioned is collected, which can be achieved by the following methods:
[0121] The image to be positioned is subjected to posture recognition processing by the image processing model to obtain initial posture information of the electronic device when the image to be positioned is collected.
[0122] That is to say, the electronic device can use the above-mentioned image processing model to obtain the initial posture information of the image to be positioned according to the relationship between consecutive image frames, thereby improving the accuracy of the initial posture information.
[0123] Based on the above embodiments, reference Figure 4 The positioning method flow diagram shown Figure 3 In step 130, based on the relevant information of the reference image and the initial posture information, the target posture information of the electronic device when acquiring the image to be positioned is determined, which can also be achieved by the following steps:
[0124] Step 1301: obtaining the posture information to be adjusted based on the relevant information of the reference image and the initial posture information;
[0125] Step 1302: Obtain a second image with the largest semantic consistency weight value from M frames of images adjacent to the image to be located; M is an integer greater than 1;
[0126] Step 1303: Acquire a second three-dimensional point cloud set matching the second pose information corresponding to the second image from the three-dimensional semantic map;
[0127] Step 1304: performing feature point matching processing on the second three-dimensional point cloud set and the image to be located to obtain matching feature point pairs;
[0128] Step 1305: construct a target optimization equation based on the matching feature point pairs, and optimize the posture information to be adjusted based on the optimization equation to obtain the target posture information.
[0129] In the embodiment provided by the present application, after obtaining the posture information to be adjusted based on the relevant information of the reference image and the initial posture information of the image to be positioned, in order to further improve the positioning result, the electronic device can select the posture result of the image with a larger semantic consistency weight value from the M frame images adjacent to the image to be positioned, and match the feature points in the three-dimensional point cloud map with the feature points extracted from the current image to be positioned, and then obtain a more accurate posture calculation result through PnP nonlinear optimization.
[0130] It can be understood that the M frames of images adjacent to the image to be located can be images with similar image content to the image to be located, that is, images in the same scene. From the M frames of images, a second image with a larger semantic consistency weight value is selected, that is, the selected second image is the image most relevant to the scene map, and the second pose information corresponding to the second image is the most accurate.
[0131] Furthermore, the electronic device can obtain a second three-dimensional point cloud set matching the second posture information from the scene map, and perform feature point matching with the second three-dimensional point cloud set and image features of the image to be located to obtain multiple matching feature point pairs.
[0132] Below, the process of obtaining the target pose information corresponding to the image to be located through PnP nonlinear optimization is introduced in detail.
[0133] For the convenience of description, the embodiment of the present application uses P to represent a feature point in the second three-dimensional point cloud set, and uses p to represent a feature point in the image to be located. P and p can be a matching feature point pair.
[0134] The feature points in the second three-dimensional point cloud set are projected into the coordinate system of the image to be located. The coordinates of the feature point P can be expressed by formula (2):
[0135]
[0136] Among them, i represents the feature point number, [X i ,Y i ,Z i ,1] T is the homogeneous coordinate of the feature point P in the scene map coordinate system, K is the camera intrinsic parameter matrix of the electronic device, It is a homogeneous coordinate system composed of the posture information (R*, T*) to be adjusted of the image to be positioned. is the theoretical projection coordinate of the feature point P in the coordinate system of the image to be located.
[0137] However, the coordinates of the feature point p that actually matches the feature point P in the coordinate system of the image to be located are [u i v i 1] T .
[0138] Furthermore, the electronic device can calculate the theoretical coordinates With the actual coordinates [u i v i 1] T The error between is called the reprojection error. Figure 5 The schematic diagram of reprojection error calculation is shown in the figure. The point P in the three-dimensional coordinate system is projected to the two-dimensional coordinate system to form a point p', and the point in the actual two-dimensional coordinate system that matches the three-dimensional coordinate system is p. Figure 5 It can be seen that there is a reprojection error between the projected point p' and the actual matching point p.
[0139] Based on this, the electronic device can calculate the reprojection errors of all matching feature point pairs obtained in step 1304, and then construct a target optimization equation according to the reprojection errors. The target optimization equation here can be a least squares cost function, and the target optimization equation can be expressed by formula (3):
[0140]
[0141] Among them, t is the pose transformation matrix when the reprojection error of the matching feature point pair is minimized, and t is the state variable that needs to be optimized in formula (3).
[0142] Specifically, by adjusting the value of t, when the objective equation (3) is minimized, the electronic device can obtain the optimal posture information through the posture transformation matrix.
[0143] In the embodiment provided in the present application, before acquiring the image to be located in step 110, a scene map may also be acquired.
[0144] In one possible implementation, the electronic device may obtain a scene map of the current scene from a cloud server.
[0145] In another possible implementation, the electronic device may pre-construct a scene map of the current scene.
[0146] refer to Figure 6 As shown in the schematic diagram of the scene map construction process, the electronic device can construct the scene map through the following steps:
[0147] Step a, collecting multiple frames of scene images through an image acquisition device;
[0148] Step b: constructing a three-dimensional point cloud map of the scene based on multiple frames of scene images;
[0149] Step c, performing semantic segmentation processing on the multiple scene images to obtain scene semantic information corresponding to each scene image in the multiple scene images;
[0150] Step d: Based on the scene semantic information corresponding to each frame of the scene image, semantic annotation is performed on the three-dimensional objects in the three-dimensional map to obtain a scene map.
[0151] Here, the electronic device first collects multiple frames of scene images in the current scene through a visual sensor (ie, a camera). Each frame of the scene image may include RGB data and depth data, namely, RGB-D data.
[0152] Furthermore, the electronic device converts the collected RGB-D data of each frame of the scene image into point cloud data, and then splices the point cloud data of multiple frames of the scene image to obtain a three-dimensional point cloud map.
[0153] In addition, the electronic device can also perform semantic segmentation processing on the collected scene images through a semantic segmentation algorithm to obtain a semantic label for each frame of the scene image.
[0154] Furthermore, the electronic device semantically annotates the points in the three-dimensional point cloud map according to the correspondence between the scene image and the points in the three-dimensional point cloud map during the mapping process, and removes the three-dimensional points of dynamic objects in the three-dimensional point cloud map to generate a scene map including semantic information.
[0155] It can be seen that the positioning method provided in the embodiment of the present application can determine the semantic consistency weight value of the reference image corresponding to the image to be positioned, and use the relevant information of the reference image with a larger semantic consistency weight value to obtain the target posture information of the electronic device when collecting the current image to be positioned, thereby improving the accuracy of posture calculation.
[0156] Below, the embodiments of the present application are described in detail in conjunction with exemplary application scenarios.
[0157] The positioning method provided in the embodiment of the present application can be applied to electronic devices. When a user holding the electronic device needs to determine his or her location in an unfamiliar shopping mall environment, the electronic device can provide the user with accurate positioning services through the positioning method provided in the embodiment of the present application, thereby solving the pain points encountered by the user in life.
[0158] refer to Figure 7 The positioning process shown Figure 4 , the electronic device first collects a continuous video stream 7-1. Further, the electronic device inputs the collected continuous video stream 7-1 into the pre-trained image processing model 7-2, and the image processing model 7-2 processes each frame of the continuous video stream 7-1 through the LSTM network, and outputs the initial pose information 7-3 of each frame of the image according to the correlation between each frame of the image; at the same time, the image processing model 7-2 performs semantic segmentation processing on each frame of the image through the semantic segmentation network, and outputs the semantic information 7-4 of each frame of the image.
[0159] Furthermore, the electronic device can obtain the scene location of the current scene from the cloud server. Figure 7-5 .
[0160] Here, electronic devices can place scenes Figure 7-5 The three-dimensional point cloud set matching the initial pose information 7-3 of the current image frame is reprojected into the acquired image coordinate system, and the similarity between the semantic information 7-4 of the current image frame and the semantic information of the three-dimensional point cloud set reprojected into the image coordinate system is compared to obtain the semantic consistency weight value 7-6 of each frame of the image. The semantic consistency weight can be used to measure the accuracy of the calculation of the initial pose information of the current image frame. The higher the semantic consistency weight value, the higher the accuracy.
[0161] The relevant information of each frame of the image is screened through the semantic consistency weight value of the image; specifically, when the calculated semantic consistency weight value of the current image frame is large (for example, greater than 0.8), the initial posture information 7-3 representing the image frame has a higher accuracy. In this way, the relevant information corresponding to the image frame can be passed to the next frame of the image processing process. When calculating the posture information of the next frame of the image, the posture information 7-7 to be adjusted for the next frame of the image can be determined based on the relevant information of the current frame and the initial posture information of the next frame.
[0162] When the semantic consistency weight value of the current image frame is small (for example, less than 0.8), the initial pose information 7-3 representing the image frame has low accuracy, and the relevant information corresponding to the frame image is not transmitted. In this way, as time accumulates, the relevant information of the image corresponding to the high-precision pose will be continuously transmitted, and the calculation accuracy of the pose can be improved through the semantic consistency weight.
[0163] After obtaining the image's posture information to be adjusted, in order to further improve the positioning result, the electronic device can match the feature points corresponding to the posture results of the image with a larger semantic consistency weight value in the continuous video stream 7-1 in the scene map with the feature points extracted from the current image frame, and then obtain the target posture information 7-8 corresponding to the current frame through PnP nonlinear optimization. In this way, the accuracy of the posture calculation can be further improved.
[0164] Based on the foregoing embodiments, the embodiments of the present application provide a positioning device, which is applied to an electronic device; Figure 8 As shown, the device comprises:
[0165] An acquisition unit 801 is used to acquire an image to be located;
[0166] The first processing unit 802 performs posture recognition processing on the image to be positioned to obtain initial posture information of the electronic device when the image to be positioned is collected;
[0167] The determining unit 803 determines a semantic consistency weight value of a reference image corresponding to the image to be located; the reference image refers to an image collected by the electronic device and adjacent to the image to be located;
[0168] The semantic consistency weight value is used to characterize the similarity between the reference semantic information of the reference image and the first semantic information of the first image; the first image refers to an image matching the reference image in the scene map of the scene where the electronic device is located;
[0169] The second processing unit 804 is configured to determine, if the semantic consistency weight value of the reference image is greater than a weight threshold, target pose information of the electronic device when acquiring the image to be positioned based on relevant information of the reference image and the initial pose information.
[0170] In some embodiments, the acquisition unit 801 is used to acquire reference pose information and reference semantic information corresponding to the reference image;
[0171] The determination unit 803 is used to determine a first image matching the reference pose information from the scene map, and obtain first semantic information corresponding to the first image; calculate the similarity between the reference semantic information and the first semantic information, and obtain a semantic consistency weight value of the reference image.
[0172] In some embodiments, the scene map is a three-dimensional point cloud map.
[0173] The determination unit 803 is used to search for a first three-dimensional point cloud set matching the reference pose information from the scene map; and project the points in the first three-dimensional point cloud set into the coordinate system of the reference image to obtain the first image.
[0174] In some embodiments, the acquisition unit 801 is used to perform posture recognition processing and semantic segmentation processing on the reference image through an image processing model to obtain reference posture information of the electronic device when the reference image is collected, and reference semantic information of the reference image; the image processing model is used to determine the posture information of the electronic device when the image is collected and the semantic information of the image.
[0175] In some embodiments, the image processing model includes a convolutional layer, a temporal recurrent neural network, and a semantic segmentation network;
[0176] The acquisition unit 801 is specifically used to extract features of the reference image through the convolution layer to obtain reference image features of the reference image; based on relevant information of a previous frame image adjacent to the reference image, perform posture recognition processing on the reference image features through the time recurrent neural network to obtain reference posture information of the electronic device when the reference image is collected; perform semantic segmentation processing on the reference image features through the semantic segmentation network to obtain reference semantic information of the reference image.
[0177] In some embodiments, the image processing model is obtained by training an image processing model to be trained, wherein the model to be trained includes: a convolutional layer to be trained, a time recurrent neural network to be trained, and a semantic segmentation network to be trained;
[0178] The positioning device also includes a model training unit;
[0179] The model training unit is used to obtain multiple frames of sample images taken continuously; according to the time sequence of the acquisition of the multiple frames of sample images, the multiple frames of sample images are input into the convolutional layer to be trained for feature extraction to obtain multiple sample image features; through the time recurrent neural network to be trained, multiple predicted posture information corresponding to the multiple sample image features are determined; through the semantic segmentation network to be trained, multiple predicted semantic information corresponding to the multiple sample image features are determined; based on the loss of the multiple predicted posture information and the loss of the multiple predicted semantic information, the network parameters of the image processing model to be trained are adjusted so that the loss of the output result of the image processing model obtained meets the convergence condition.
[0180] In some embodiments, the model training unit is further used to perform posture recognition processing on the sample image features corresponding to the i-th frame sample image through the time recurrent neural network to be trained based on the relevant information of the i-1th frame sample image, and determine the predicted posture information corresponding to the i-th frame sample image and the relevant information of the i-th frame sample image; wherein i is an integer greater than 1 and less than or equal to N;
[0181] Based on the relevant information of the i-th frame sample image, continue to perform posture recognition processing on the corresponding sample image features of the i+1-th frame sample image through the time recurrent neural network to be trained, and determine the predicted posture information corresponding to the i+1-th frame sample image, until i=N-1, to obtain the predicted posture information corresponding to the multiple sample image features respectively; N is the total number of the multiple frames of sample images.
[0182] In some embodiments, the first processing unit 802 is used to perform posture recognition processing on the image to be positioned through the image processing model to obtain initial posture information of the electronic device when the image to be positioned is collected.
[0183] In some embodiments, the second processing unit 804 is configured to obtain the pose information to be adjusted based on the relevant information of the reference image and the initial pose information; obtain a second image with the largest semantic consistency weight value from M frames of images adjacent to the image to be positioned; M is an integer greater than 1; obtain a second three-dimensional point cloud set matching the second pose information corresponding to the second image from the scene map;
[0184] The second three-dimensional point cloud set and the image to be positioned are subjected to feature point matching processing to obtain matching feature point pairs; based on the matching feature point pairs, a target optimization equation is constructed, and based on the optimization equation, the posture information to be adjusted is optimized to obtain the target posture information.
[0185] In some embodiments, the positioning device also includes a scene map construction unit, which is used to collect multiple frames of scene images through an image acquisition device; based on the multiple frames of scene images, construct a three-dimensional point cloud map of the scene; perform semantic segmentation processing on the multiple frames of scene images to obtain scene semantic information corresponding to each frame of the multiple frames of scene images; based on the scene semantic information corresponding to each frame of the scene image, semantically annotate the three-dimensional objects in the three-dimensional map to obtain the scene map.
[0186] In some embodiments, the relevant information of the reference image includes: reference pose information of the reference image, and / or key feature points of the reference image.
[0187] Accordingly, an embodiment of the present disclosure provides an electronic device, Fig. 9 Schematic diagram of the structure of the electronic device according to the embodiment of the present application. Fig. 9 As shown, the electronic device 90 includes: a memory 901 for storing computer programs; a processor 902 for implementing the steps of the positioning method provided in the above embodiment when executing the computer program stored in the memory 901.
[0188] The electronic device 90 further includes a communication bus 903 ; the communication bus 903 is configured to implement connection and communication between these components.
[0189] The description of the above computer device and storage medium embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the computer device and storage medium embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.
[0190] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present disclosure, the size of the serial number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The serial numbers of the embodiments of the present disclosure are for description only and do not represent the advantages and disadvantages of the embodiments.
[0191] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0192] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0193] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present disclosure.
[0194] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0195] It can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiment are executed; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memory (ROM), magnetic disks or optical disks. Alternatively, if the above-mentioned integrated unit of the present disclosure is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present disclosure can be essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present disclosure. And the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.
[0196] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A positioning method, characterized in that: Applied in an electronic device, the method comprises: Acquire the image to be positioned, and perform posture recognition processing on the image to be positioned to obtain initial posture information of the electronic device when the image to be positioned is acquired; Determining a semantic consistency weight value of a reference image corresponding to the image to be located; the reference image refers to an image collected by the electronic device and adjacent to the image to be located; The semantic consistency weight value is used to characterize the similarity between the reference semantic information of the reference image and the first semantic information of the first image; the first image refers to an image matching the reference image in the scene map of the scene where the electronic device is located; If the semantic consistency weight value of the reference image is greater than the weight threshold, then based on the relevant information of the reference image and the initial posture information, determine the target posture information of the electronic device when collecting the image to be located; the relevant information of the reference image includes: reference posture information of the reference image, and / or key feature points of the reference image; Wherein, determining the target posture information of the electronic device when acquiring the image to be positioned based on the relevant information of the reference image and the initial posture information includes: Based on the relevant information of the reference image and the initial pose information, obtaining the pose information to be adjusted; Obtaining a second image with the largest semantic consistency weight value from M frames of images adjacent to the image to be located, wherein M is an integer greater than 1; Acquire a second three-dimensional point cloud set matching second pose information corresponding to the second image from the scene map; Performing feature point matching processing on the second three-dimensional point cloud set and the image to be located to obtain matching feature point pairs; Based on the matching feature point pairs, a target optimization equation is constructed, and based on the optimization equation, the posture information to be adjusted is optimized to obtain the target posture information.
2. The method according to claim 1, characterized in that The determining of the semantic consistency weight value of the reference image corresponding to the image to be located includes: Obtaining reference pose information and reference semantic information corresponding to the reference image; Determine, from the scene map, a first image that matches the reference pose information, and obtain first semantic information corresponding to the first image; The similarity between the reference semantic information and the first semantic information is calculated to obtain a semantic consistency weight value of the reference image.
3. The method according to claim 2, characterized in that The scene map is a three-dimensional point cloud map, and determining a first image matching the reference pose information from the scene map includes: Searching, from the scene map, a first three-dimensional point cloud set matching the reference pose information; Project the points in the first three-dimensional point cloud set into the coordinate system of the reference image to obtain the first image.
4. The method according to claim 2 or 3, characterized in that: The obtaining of reference pose information and reference semantic information of the reference image includes: Performing posture recognition processing and semantic segmentation processing on the reference image through an image processing model to obtain reference posture information of the electronic device when the reference image is collected, and reference semantic information of the reference image; The image processing model is used to determine the position information of the electronic device and the semantic information of the image when capturing the image.
5. The method according to claim 4, characterized in that The image processing model includes a convolutional layer, a temporal recurrent neural network and a semantic segmentation network; The image processing model is used to perform posture recognition processing and semantic segmentation processing on the reference image to obtain posture information of the electronic device when the reference image is collected, and reference semantic information of the reference image, including: Extracting features of the reference image through the convolution layer to obtain reference image features of the reference image; Based on relevant information of a previous frame image adjacent to the reference image, performing posture recognition processing on the reference image features through the time recurrent neural network to obtain reference posture information of the electronic device when the reference image is collected; The semantic segmentation network is used to perform semantic segmentation processing on the reference image features to obtain reference semantic information of the reference image.
6. The method according to claim 4, characterized in that The image processing model is obtained by training the image processing model to be trained, and the image processing model to be trained includes: a convolutional layer to be trained, a time recurrent neural network to be trained, and a semantic segmentation network to be trained; wherein, The training process of the image processing model includes: Acquire multiple frames of sample images taken continuously; According to the time sequence of the acquisition of the multiple frames of sample images, the multiple frames of sample images are input into the convolutional layer to be trained for feature extraction to obtain multiple sample image features; Determining a plurality of predicted pose information corresponding to the plurality of sample image features through the time recurrent neural network to be trained; Determining, by means of the semantic segmentation network to be trained, a plurality of predicted semantic information corresponding to the plurality of sample image features; Based on the losses of the multiple predicted pose information and the losses of the multiple predicted semantic information, the network parameters of the image processing model to be trained are adjusted so that the loss of the output result of the image processing model obtained satisfies the convergence condition.
7. The method according to claim 6, characterized in that The step of determining the plurality of predicted pose information corresponding to the plurality of sample image features through the time recurrent neural network to be trained comprises: Based on the relevant information of the i-1th frame sample image, perform posture recognition processing on the sample image features corresponding to the i-th frame sample image through the time recurrent neural network to be trained, and determine the predicted posture information corresponding to the i-th frame sample image and the relevant information of the i-th frame sample image; wherein i is an integer greater than 1 and less than or equal to N; Based on the relevant information of the sample image of the i-th frame, continue to perform posture recognition processing on the corresponding sample image features of the sample image of the i+1-th frame through the time recurrent neural network to be trained, and determine the predicted posture information corresponding to the sample image of the i+1-th frame, until i=N-1, to obtain the predicted posture information corresponding to the multiple sample image features respectively; N is the total number of the multiple frames of sample images.
8. The method according to claim 7, characterized in that The performing posture recognition processing on the image to be positioned to obtain initial posture information of the electronic device when the image to be positioned is collected includes: The image to be positioned is subjected to position recognition processing by the image processing model to obtain initial position information of the electronic device when the image to be positioned is collected.
9. The method according to any one of claims 1 to 3, characterized in that: Before acquiring the image to be located, the method further includes: Capturing multiple frames of scene images through an image acquisition device; Based on the multiple frames of scene images, construct a three-dimensional point cloud map of the scene; Performing semantic segmentation processing on the multiple frames of scene images to obtain scene semantic information corresponding to each frame of the multiple frames of scene images; Based on the scene semantic information corresponding to each frame of the scene image, the three-dimensional objects in the three-dimensional point cloud map are semantically annotated to obtain the scene map.
10. A positioning device, characterized in that: Applied in electronic equipment, the device comprises: An acquisition unit, used for acquiring an image to be located; A first processing unit is used to perform posture recognition processing on the image to be positioned to obtain initial posture information of the electronic device when the image to be positioned is collected; A determination unit, configured to determine a semantic consistency weight value of a reference image corresponding to the image to be located; the reference image refers to an image collected by the electronic device and adjacent to the image to be located; The semantic consistency weight value is used to characterize the similarity between the reference semantic information of the reference image and the first semantic information of the first image; the first image refers to an image matching the reference image in the scene map of the scene where the electronic device is located; A second processing unit is configured to determine, if the semantic consistency weight value of the reference image is greater than a weight threshold, target pose information of the electronic device when acquiring the image to be located based on relevant information of the reference image and the initial pose information; the relevant information of the reference image includes: reference pose information of the reference image, and / or key feature points of the reference image; The second processing unit is also used to obtain the pose information to be adjusted based on the relevant information of the reference image and the initial pose information; obtain the second image with the largest semantic consistency weight value from the M frame images adjacent to the image to be positioned; M is an integer greater than 1; obtain a second three-dimensional point cloud set matching the second pose information corresponding to the second image from the scene map; perform feature point matching processing on the second three-dimensional point cloud set and the image to be positioned to obtain matching feature point pairs; construct a target optimization equation based on the matching feature point pairs, and optimize the pose information to be adjusted based on the optimization equation to obtain the target pose information.
11. An electronic device, characterized in that: The electronic device comprises a processor and a memory for storing a computer program capable of running on the processor; The processor and the memory are connected via a communication bus; Wherein, when the processor runs the computer program stored in the memory, the steps of the positioning method described in any one of claims 1 to 9 are executed.
12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by a processor to implement the steps of the positioning method according to any one of claims 1 to 9.
