Visual positioning method, device and system

Abstract

The present invention provides a visual positioning method, device and system, belonging to the field of computer vision technology, the method comprising: extracting features from a current positioning image to obtain image feature information of the current positioning image; matching the image feature information with a local feature point cloud to obtain a feature point pair corresponding to the current positioning image; determining the first position information corresponding to the current positioning image based on the feature point pair and the absolute spatial position contained in the feature point pair; the current positioning image is acquired by a camera in an intelligent terminal. The visual positioning method, device and system provided by the present invention can utilize the advantages of relatively strong storage and computing resources of a server platform to complete the solution of initial position information with relatively large resource consumption in the server, and subsequently complete the dynamic environment visual positioning of the current positioning image offline in an iterative manner on the intelligent terminal, and can achieve more accurate and efficient visual positioning based on the intelligent terminal.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual positioning method, device and system. Background Art

[0002] Visual localization refers to the use of computer vision technology to obtain image information from a camera or a camera to achieve accurate positioning of a camera or mobile device. Smart terminals in related technologies (such as smartphones, tablets, etc.) have both cameras required for photogrammetry and sensors similar to robots and drones. Therefore, visual localization based on smart terminals has become a research hotspot in the field of computer vision technology.

[0003] Traditional monocular visual inertial positioning and tracking models are mainly used in the fields of virtual reality (VR), drones or robots. The research focuses of traditional monocular visual inertial positioning and tracking models include: the existing simultaneous localization and mapping (SLAM) technology usually uses dedicated hardware equipment, adopts a simple reference frame model and a uniform speed model, requires an initial input or initialization process, and the positioning result is often a relative position relative to the environment.

[0004] However, visual positioning based on smart terminals is mainly aimed at pedestrian navigation or vehicle navigation, which requires direct output of absolute spatial position and should avoid or simplify the initialization process. Therefore, how to achieve more accurate and efficient visual positioning based on smart terminals is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0005] The present invention provides a visual positioning method, device and system, which are used to solve the defect that it is difficult to achieve accurate and efficient visual positioning based on smart terminals in the prior art, and to achieve more accurate and efficient visual positioning based on smart terminals.

[0006] The present invention provides a visual positioning method, which is applied to an intelligent terminal connected to a server for communication, and the method comprises:

[0007] Performing feature extraction on the current positioning image to obtain image feature information of the current positioning image;

[0008] Matching the image feature information with the local feature point cloud to obtain feature point pairs corresponding to the current positioning image;

[0009] Based on the feature point pair and the absolute spatial position contained in the feature point pair, determining the first position information corresponding to the current positioning image;

[0010] Among them, the current positioning image is acquired by the camera in the smart terminal; the local feature point cloud is established based on the initial posture information sent by the server and the positioned image sequence; the positioned image sequence at least includes an initialization image sequence, and / or a historical positioning image; the initialization image sequence is acquired by the camera before acquiring the current positioning image; the historical positioning image carries the corresponding posture information; the initial posture information is obtained based on the initialization image sequence.

[0011] According to a visual positioning method provided by the present invention, after matching the image feature information with the local feature point cloud to obtain feature point pairs, the method further includes:

[0012] Acquire inertial measurement data and first historical posture information corresponding to a previous positioning image, wherein the inertial measurement data is collected by an inertial navigation module in the smart terminal;

[0013] Determine an affine transformation matrix between a collection time of the previous positioning image and a collection time of the current positioning image according to the inertial measurement data and the first historical position and posture information;

[0014] Transforming the image feature information of the current positioning image based on the affine transformation matrix to obtain second historical posture information corresponding to the acquisition moment of the previous positioning image;

[0015] When it is determined that the absolute value of the difference between the second historical pose information and the pose information corresponding to any one of the feature point pairs is less than a first preset threshold, determining the any one of the feature point pairs as a stationary feature point pair;

[0016] The second position and posture information corresponding to the current positioning image is determined by using the stationary feature point pair and the absolute spatial position of the stationary feature point pair.

[0017] According to a visual positioning method provided by the present invention, the method further includes: determining an affine transformation matrix between the acquisition time of the previous positioning image and the acquisition time of the current positioning image based on the inertial measurement data and the first historical posture information, and transforming the image feature information of the current positioning image based on the affine transformation matrix to obtain the second historical posture information corresponding to the acquisition time of the previous positioning image.

[0018] Based on the feature point pairs and the affine transformation matrix, it is determined that the inertial measurement data is valid.

[0019] According to a visual positioning method provided by the present invention, when it is determined that the inertial measurement data is invalid, a current positioning image carrying the corresponding first position information is added as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence;

[0020] After determining the second posture information corresponding to the current positioning image by using the stationary feature point pair and the absolute spatial position of the stationary feature point pair, the method further includes:

[0021] The current positioning image carrying the corresponding second posture information is added as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence.

[0022] According to a visual positioning method provided by the present invention, after obtaining the updated positioned image sequence, the method further includes:

[0023] Based on the updated positioned image sequence, obtaining a reprojection cumulative error;

[0024] Based on the reprojection accumulated error, the number of historical positioning images in the positioned image sequence is dynamically maintained.

[0025] According to a visual positioning method provided by the present invention, the specific steps of establishing the local feature point cloud include:

[0026] Taking at least two located images satisfying the constraint conditions in the located image sequence as key image pairs;

[0027] Performing forward intersection based on the initial pose information, the key image pair, and the pose information corresponding to the key image pair to establish the local feature point cloud;

[0028] Among them, the constraint conditions include that the number of matching points belonging to RANSAC internal points among the matching points between any two located images in the located image sequence is greater than or equal to a second preset threshold, the baseline distance determined according to the posture information of the two located images in the key image pair is greater than or equal to a third preset threshold, and the relative angle value determined according to the posture information of the two located images in the key image pair is greater than or equal to at least one of a fourth preset threshold.

[0029] The present invention provides a visual positioning method, which is applied to a server connected to a smart terminal for communication. The method comprises:

[0030] Determine any image frame in an initialization image sequence as a first initialization image frame, and obtain a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, wherein the initialization image sequence is acquired by a camera of the smart terminal;

[0031] Based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame, obtaining the posture information corresponding to the first initialization image frame and the second initialization image frame;

[0032] Triangulate the pose information corresponding to the first initialization image frame and the second initialization image frame, and store the obtained three-dimensional space points in a feature point cloud library;

[0033] Performing feature matching from the initialization image sequence using the feature point cloud library, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame;

[0034] Repeat the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, until the three-dimensional space points contained in the obtained third initialization image frame are expanded into the feature point cloud library, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded into the feature point cloud library;

[0035] The pose information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library is determined as the initial pose information in accordance with the matching order, and the initial pose information is sent to the intelligent terminal.

[0036] According to a visual positioning method provided by the present invention, the step of performing feature matching from the initialization image sequence using the feature point cloud library comprises:

[0037] Matching the image features of each initialized image frame with all cluster centers obtained by clustering the feature point cloud library to determine the target cluster center;

[0038] After performing secondary matching on the image features of each of the initialization image frames in the cluster where the center of the target cluster is located, the initialization image frame with the largest number of matching point pairs is used as the third initialization image frame.

[0039] The present invention also provides a visual positioning device, comprising:

[0040] A feature extraction module is used to extract features from the current positioning image to obtain image feature information of the current positioning image;

[0041] A feature matching module, used to match the image feature information with the local feature point cloud to obtain feature point pairs corresponding to the current positioning image;

[0042] A visual positioning module, used to determine the first position information corresponding to the current positioning image based on the feature point pair and the absolute spatial position contained in the feature point pair;

[0043] Among them, the current positioning image is acquired by a camera in the intelligent terminal; the local feature point cloud is established based on the initial posture information sent by the server and the positioned image sequence; the positioned image sequence includes at least an initialization image sequence, and / or a historical positioning image; the initialization image sequence is acquired by the camera before acquiring the current positioning image; the historical positioning image carries the corresponding posture information; the initial posture information is obtained based on the initialization image sequence.

[0044] The present invention also provides a visual positioning device, comprising:

[0045] An image frame determination module, used to determine any image frame in an initialization image sequence as a first initialization image frame, and obtain a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, wherein the initialization image sequence is acquired by a camera of a smart terminal;

[0046] A posture acquisition module, used to obtain posture information corresponding to the first initialization image frame and the second initialization image frame based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame;

[0047] A three-dimensional processing module, used for triangulating the posture information corresponding to the first initialization image frame and the second initialization image frame, and storing the obtained three-dimensional space points in a feature point cloud library;

[0048] A three-dimensional expansion module, used to perform feature matching from the initialization image sequence using the feature point cloud library, and expand the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame;

[0049] A repeated iteration module, used for repeating the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded to the feature point cloud library;

[0050] The information sending module is used to determine the posture information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library as the initial posture information in accordance with the matching order, and send the initial posture information to the intelligent terminal.

[0051] The present invention also provides a visual positioning system, comprising: an intelligent terminal and a server; the intelligent terminal and the server are in communication connection; the intelligent terminal comprises a camera, a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor; the intelligent terminal comprises a second memory, a second processor, and a second computer program stored in the second memory and executable on the second processor;

[0052] When the smart terminal executes the first computer program, the visual positioning method as described in any one of the above items is implemented; when the server executes the second computer program, the visual positioning method as described above is implemented.

[0053] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the visual positioning method as described in any one of the above is implemented.

[0054] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned visual positioning methods.

[0055] The visual positioning method, device and system provided by the present invention, after the intelligent terminal constructs a local feature point cloud by sending initial posture information based on the server, obtains a feature point pair by matching the image feature information of the current positioning image with the above-mentioned local feature point cloud, and then uses the rear intersection method to solve the posture information when the intelligent terminal took the current positioning image based on the above-mentioned feature point pair and the absolute spatial position contained in the above-mentioned feature point pair. As the first posture information corresponding to the current positioning image, it can take advantage of the relatively strong storage and computing resources of the server platform to complete the solution of the initial posture information with relatively large resource consumption in the server, and subsequently complete the dynamic environment visual positioning of the current positioning image offline in an iterative manner on the intelligent terminal, which can achieve more accurate and efficient visual positioning based on the intelligent terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 It is one of the flow charts of the visual positioning method provided by the present invention;

[0058] Figure 2 This is the second flow chart of the visual positioning method provided by the present invention;

[0059] Figure 3 is a schematic diagram of key frame selection in the visual positioning method provided by the present invention;

[0060] Figure 4 This is the third flow chart of the visual positioning method provided by the present invention;

[0061] Figure 5 This is the fourth flow chart of the visual positioning method provided by the present invention;

[0062] Figure 6 This is the fifth flow chart of the visual positioning method provided by the present invention;

[0063] Figure 7 It is one of the structural schematic diagrams of the visual positioning device provided by the present invention;

[0064] Figure 8 This is the second structural schematic diagram of the visual positioning device provided by the present invention;

[0065] Fig. 9 It is a structural schematic diagram of the visual positioning system provided by the present invention;

[0066] Fig.10 It is a schematic diagram of the physical structure of the intelligent terminal in the visual positioning system provided by the present invention;

[0067] Fig.11 It is a schematic diagram of the physical structure of the server in the visual positioning system provided by the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0070] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present invention, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0071] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0072] Figure 1 It is one of the flow charts of the visual positioning method provided by the present invention. It should be noted that the visual positioning method provided by the present invention is applied to an intelligent terminal connected to a server for communication. Figure 1 As shown, the method includes: step 101, extracting features from the current positioning image to obtain image feature information of the current positioning image;

[0073] Step 102: Match the image feature information with the local feature point cloud to obtain feature point pairs corresponding to the current positioning image;

[0074] Step 103: Determine the first position information corresponding to the current positioning image based on the feature point pair and the absolute spatial position contained in the feature point pair;

[0075] Among them, the current positioning image is collected by the camera in the intelligent terminal; the local feature point cloud is established based on the initial posture information sent by the server and the positioned image sequence; the positioned image sequence at least includes the initialization image sequence, and / or, the historical positioning image; the initialization image sequence is collected by the camera before collecting the current positioning image; the historical positioning image carries the corresponding posture information; the initial posture information is obtained based on the initialization image sequence.

[0076] It should be noted that the execution subject of the embodiment of the present invention is a visual positioning device. The visual positioning device can be set in an intelligent terminal that is connected to the server for communication, and the visual positioning device can also be the intelligent terminal.

[0077] Among them, the above-mentioned intelligent terminal can be any one of mobile terminals such as mobile phones, smart phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), navigation devices, smart bracelets, smart watches, digital cameras, etc.

[0078] For the above-mentioned intelligent terminals, a visual processor is disposed in the internal system on chip (SOC), and the external accessories thereof are at least provided with a camera that is in communication with the visual processor.

[0079] The visual positioning method provided by the present invention is described below by taking the above-mentioned visual positioning device as the above-mentioned intelligent terminal as an example.

[0080] Specifically, before the smart terminal independently carries out visual positioning in a dynamic environment, it first uses a camera to continuously shoot multiple images containing overlapping scenes, which are packaged into an initialization image sequence in the order of shooting and transmitted to the server.

[0081] The server accurately locates the initialized image sequence based on the visual positioning method of the three-dimensional scene, and transmits the solved absolute spatial pose as the initial pose information to the smart terminal.

[0082] After the intelligent terminal receives the initial posture information sent by the server, it triggers the dynamic environment visual positioning function.

[0083] After the smart terminal triggers the dynamic environment visual positioning function, the smart terminal will use each image collected by the camera as the current positioning image in turn, and complete the dynamic environment visual positioning of the current positioning image offline in an iterative manner.

[0084] The intelligent terminal can initialize the initial pose information as the starting pose when building a map, and then select at least two image frames with known pose information from the located image sequence for forward intersection to build a map of the local scene to obtain a local feature point cloud containing scale information and absolute spatial position.

[0085] It should be noted that the local feature point cloud in the embodiment of the present invention may be obtained by the intelligent terminal before performing dynamic environment visual positioning on the current positioning image.

[0086] It should be noted that when the sequence of located images is empty, the intelligent terminal has no known pose points as a reference, so the initialization image sequence needs to be used as a known point. After completing the dynamic environment visual positioning of the current positioning image, the current positioning image can be used as a known point, and on the basis of the initialization image sequence, the current positioning image and the pose information corresponding to the current positioning image can be added to the located image sequence as the located image and the pose information corresponding to the located image in sequence according to the time sequence of obtaining the current positioning image.

[0087] The smart terminal can extract feature points from the current positioning image, and match the obtained two-dimensional image feature information with the three-dimensional local feature point cloud, and use the matched feature point pairs and the absolute spatial positions recorded by the above feature point pairs in the local feature point cloud to perform rear intersection, so as to determine the posture information of the smart terminal when taking the current positioning image, as the first posture information corresponding to the current positioning image.

[0088] It should be noted that the smart terminal in the embodiment of the present invention may also include a positioning device. While the smart terminal collects the initialization image sequence through the camera, it also obtains the absolute spatial position of each initialization image frame in the initialization image sequence through the positioning device and transmits it to the server together with the initialization image sequence.

[0089] For example, the smart terminal can also be configured with positioning devices such as GNSS, WiFi and Bluetooth. Relying on its own built-in positioning method, it can provide a rough absolute spatial position for each initialization image frame when shooting the initialization image sequence. The visual processor aligns the initialization image sequence and the absolute spatial position of each initialization image frame through the timestamp, and integrates them into the initialization image sequence and transmits them to the server together, so that the server can speed up the efficiency of positioning the initialization image sequence.

[0090] The embodiment of the present invention utilizes the positioning device in the intelligent terminal to perform positioning during the initial image acquisition process before the first dynamic environment visual positioning work is carried out. After obtaining the rough absolute spatial position of the corresponding frame, it is stored in the initialization image sequence and transmitted to the server together to improve the efficiency of the server in positioning the initialization image sequence.

[0091] After the smart terminal in the embodiment of the present invention constructs a local feature point cloud by sending initial posture information based on the server, it matches the image feature information of the current positioning image with the above-mentioned local feature point cloud to obtain a feature point pair, and then based on the above-mentioned feature point pair and the absolute spatial position contained in the above-mentioned feature point pair, uses the rear intersection method to solve the posture information when the smart terminal took the current positioning image, as the first posture information corresponding to the current positioning image, it can take advantage of the relatively strong storage and computing resources of the server platform to complete the solution of the initial posture information with relatively large resource consumption in the server, and subsequently complete the dynamic environment visual positioning of the current positioning image offline in an iterative manner on the smart terminal, which can achieve more accurate and efficient visual positioning based on the smart terminal.

[0092] As an optional embodiment, after matching the image feature information with the local feature point cloud to obtain the feature point pair, the method further includes: acquiring inertial measurement data and first historical pose information corresponding to the last positioning image, wherein the inertial measurement data is collected by an inertial navigation module in the smart terminal;

[0093] Determine an affine transformation matrix between a collection time of a previous positioning image and a collection time of a current positioning image according to the inertial measurement data and the first historical position and posture information;

[0094] Transforming the image feature information of the current positioning image based on the affine transformation matrix to obtain the second historical posture information corresponding to the acquisition time of the previous positioning image;

[0095] When it is determined that the absolute value of the difference between the second historical pose information and the pose information corresponding to any feature point pair in the feature point pairs is less than a first preset threshold, determining any feature point pair as a stationary feature point pair;

[0096] The second posture information corresponding to the current positioning image is determined by using the stationary feature point pairs and the absolute spatial positions of the stationary feature point pairs.

[0097] It should be noted that in the use environment of smart terminals, there are often a large number of vehicles, pedestrians or other moving objects. A considerable number of features in the current positioning image are on moving people, which will have a significant impact on the visual positioning of the current positioning image. When performing visual positioning based on smart terminals, the influence of such objects needs to be excluded to meet the needs of use.

[0098] Therefore, in the embodiment of the present invention, the inertial measurement data collected by the inertial navigation module in the intelligent terminal is used to eliminate the feature point matching of the moving object in the current positioning image.

[0099] Specifically, the smart terminal may also be configured with an inertial navigation module. The three-axis acceleration and rotational angular velocity output by the inertial navigation module at high frequency are the main data sources of the inertial navigation algorithm.

[0100] Among them, the Inertial Navigation Module (IMU) is a device that integrates an inertial measurement unit (IMU) and a navigation algorithm. The IMU combines inertial sensors and navigation algorithms to provide accurate position, speed, and attitude information.

[0101] Inertial navigation modules are usually composed of sensors such as accelerometers, gyroscopes and magnetometers, which are used to measure the acceleration, angular velocity and geomagnetic field information of objects. The data from these sensors are input into the navigation algorithm, and through data fusion and filtering techniques, the object's trajectory, direction and rotation state can be estimated.

[0102] For example, Figure 2 FIG. 2 is a flow chart of the visual positioning method provided by the present invention. Figure 2 As shown, while the smart terminal receives the angular velocity and acceleration (i.e., inertial measurement data) fed back in real time by the inertial navigation module, it retrieves the last positioned image and the posture information when the smart terminal took the last positioned image (i.e., the first historical posture information) from the locally stored positioned image sequence.

[0103] After the intelligent terminal obtains the above-mentioned inertial measurement data and the above-mentioned first historical posture information, it calculates the translation and rotation between the acquisition time of the previous positioning image and the acquisition time of the current positioning image and the corresponding affine transformation matrix based on the above-mentioned first historical posture information and the above-mentioned first posture information.

[0104] For each feature point pair obtained by matching the two-dimensional image features of the current positioning image with the three-dimensional local feature point cloud, the image feature information corresponding to each feature point pair is transformed using the affine transformation matrix to obtain the pose inference result at the time of acquisition of the previous positioning image, that is, the second historical pose information.

[0105] After obtaining the second historical pose information, the second historical pose information may be compared with the pose information corresponding to each feature point pair, wherein the pose information corresponding to each feature point pair is determined based on the absolute spatial position included in each feature point pair.

[0106] If the absolute value of the difference between the second historical posture information and the posture information corresponding to any feature point pair is less than the first preset threshold, it can be considered that the features corresponding to the feature point pair are on a stationary object, and the feature point pair can be retained as a stationary feature point pair.

[0107] By performing rear intersection using the stationary feature point pairs and the absolute spatial positions corresponding to the stationary feature point pairs, the posture information of the smart terminal when shooting the current positioning image can be determined as the second posture information corresponding to the current positioning image.

[0108] It can be understood that, compared with the first pose information of the current positioning image, the second pose information of the current positioning image can more accurately describe the pose of the smart terminal when shooting the current positioning image.

[0109] On the contrary, if the absolute value of the difference between the second historical posture information and the posture information corresponding to any feature point pair is greater than or equal to the first preset threshold, it can be considered that the feature corresponding to the feature point pair is on the moving object.

[0110] In an embodiment of the present invention, based on the inertial measurement data collected by the inertial navigation module in the smart terminal and the first historical posture information when the smart terminal took the last positioning image, the affine transformation matrix corresponding to the collection time of the last positioning image and the collection time of the current positioning image is calculated, and then the image feature information is derived by affine transformation using the above affine transformation matrix to obtain the second historical posture information corresponding to the collection time of the last positioning image, and then the posture information corresponding to the feature point pair corresponding to the second historical posture information and the current positioning image is compared, and the feature point whose absolute value of the difference with the posture information of the second historical posture information is less than the first preset threshold is determined as a static feature point pair, and the posture information when the smart terminal took the current positioning image is determined based on the absolute spatial position of the static feature point pair as the second posture information corresponding to the current positioning image, so as to make a corresponding judgment on whether the object in the environment is static or moving, and the negative impact of the dynamic object on the accuracy of image feature point matching is avoided by excluding the features of the dynamic object, so as to improve the positioning accuracy of the smart terminal in performing visual positioning in a complex dynamic environment.

[0111] As an optional embodiment, based on the inertial measurement data and the first historical posture information, an affine transformation matrix between the acquisition time of the previous positioning image and the acquisition time of the current positioning image is determined, and the image feature information of the current positioning image is transformed based on the affine transformation matrix to obtain the second historical posture information corresponding to the acquisition time of the previous positioning image. The method also includes: determining that the inertial measurement data is valid based on the feature point pairs and the affine transformation matrix.

[0112] As an optional embodiment, after determining the second pose information corresponding to the current positioning image by using the static feature point pair and the absolute spatial position of the static feature point pair, the method further includes: adding the current positioning image carrying the corresponding second pose information as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence;

[0113] Specifically, after the intelligent terminal obtains the affine transformation matrix, it can calculate the matching ratio between the feature point pairs corresponding to the current positioning image and the above affine transformation matrix.

[0114] Among them, the judgment condition for determining whether the inertial measurement data is valid can be determined in advance through a large number of experiments in different scenarios, and can be a certain compliance ratio or a certain threshold range.

[0115] When the matching ratio of the preset number of feature point pairs to the affine transformation matrix is ​​greater than a preset ratio, it is determined that the inertial measurement data is valid.

[0116] It should be noted that the preset number and preset ratio in the embodiment of the present invention may be determined based on prior knowledge and / or actual conditions. In the embodiment of the present invention, the preset number and preset ratio are not specifically limited.

[0117] When it is determined that the inertial measurement data is valid, the image feature information of the current positioning image can be transformed based on the affine transformation matrix, and the second historical posture information corresponding to the acquisition time of the previous positioning image can be obtained as the theoretical basis for dynamic and static feature separation. Then, the second posture information corresponding to the current positioning image is determined based on the second historical posture information, and the current positioning image carrying the corresponding second posture information is added as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence.

[0118] Before introducing inertial navigation information to reduce interference from moving objects, the embodiments of the present invention use the degree of fit between feature point pairs and affine transformation matrices to verify the validity of inertial measurement data. The dynamic changes of the scene can be characterized by the relationship between image feature matching and the affine transformation matrix under epipolar geometry constraints, providing a data basis for eliminating interference from moving objects on feature matching.

[0119] As an optional embodiment, when it is determined that the inertial measurement data is invalid, the current positioning image carrying the corresponding first position information is added to the positioned image sequence as a historical positioning image to obtain an updated positioned image sequence.

[0120] Specifically, when the matching ratio between the preset number of feature point pairs and the affine transformation matrix is ​​not greater than a preset ratio, it is determined that the inertial measurement data is invalid.

[0121] When it is determined that the inertial measurement data is invalid, the current positioning image carrying the corresponding first position information may be added to the positioned image sequence as a historical positioning image to obtain an updated positioned image sequence.

[0122] As an optional embodiment, after obtaining the updated positioned image sequence, the method further includes: obtaining a reprojection cumulative error based on the updated positioned image sequence;

[0123] Based on the accumulated reprojection error, the number of historical positioning images in the positioned image sequence is dynamically maintained.

[0124] It should be noted that after the smart terminal solves the first pose information corresponding to the current positioning image, it needs to prepare for a new round of visual positioning. Therefore, before saving the current positioning image carrying the first pose information or saving the current positioning image carrying the second pose information, the smart terminal can reproject the historical positioning images in the positioned image sequence to obtain the current reprojection cumulative error.

[0125] Then, the intelligent terminal can determine the validity of the first pose information corresponding to the current positioning image or the second pose information corresponding to the current positioning image according to the reprojection accumulated error.

[0126] When the cumulative reprojection error does not meet the preset conditions, the visual positioning function of the smart terminal is interrupted, and the smart terminal and the server re-establish a communication connection so that the server can re-provide the initial posture information to the smart terminal to achieve the purpose of eliminating the cumulative reprojection error.

[0127] It should be noted that the preset conditions in the embodiments of the present invention may be determined based on actual conditions and / or prior knowledge, and the embodiments of the present invention do not specifically limit the above preset conditions.

[0128] Exemplarily, through the prior data obtained from experiments in different environments, a cumulative error model is established that can characterize the mapping relationship between the cumulative error of reprojection and the number of historical positioning images. Therefore, based on the above-mentioned cumulative error model, the number of historical positioning images in the positioned image sequence can be updated. If the current amount of data is insufficient, the current positioning image and its corresponding first position information are stored in the positioned image sequence. Otherwise, the historical positioning image that is at the front of the time sequence in the positioned image sequence is deleted, and the current positioning image and its corresponding first position information are added.

[0129] Exemplarily, a specific threshold value can be set in advance for the cumulative reprojection error. When the cumulative reprojection error is greater than or equal to this threshold value, the smart terminal and the server will re-establish a communication connection so that the server can provide the initial posture information to the smart terminal again, thereby achieving the purpose of eliminating the cumulative reprojection error. Otherwise, the current positioning image and its corresponding first posture information will be stored in the positioned image sequence.

[0130] The embodiment of the present invention optimizes the number of historical positioning images participating in iterative optimization in the positioned image sequence by acquiring the reprojection cumulative error generated when the position and posture of the image to be positioned are inferred from the positioned image, thereby saving the computing resources consumed by the smart terminal in performing rear-end resection, and when the reprojection cumulative error is large, the server intervenes to eliminate the cumulative error online, thereby improving the accuracy of visual positioning of the smart terminal.

[0131] As an optional embodiment, the specific steps of establishing a local feature point cloud include:

[0132] At least two located images satisfying the constraint conditions in the located image sequence are used as key image pairs;

[0133] Based on the initial pose information, key image pairs and the pose information corresponding to the key image pairs, a forward intersection is performed to establish a local feature point cloud;

[0134] Among them, the constraints include that the number of matching points belonging to RANSAC internal points among the matching points between any two located images in the located image sequence is greater than or equal to a second preset threshold, the baseline distance determined according to the posture information of the two located images in the key image pair is greater than or equal to a third preset threshold, and the relative angle value determined according to the posture information of the two located images in the key image pair is greater than or equal to at least one of a fourth preset threshold.

[0135] Figure 3 Schematic diagram of key frame selection in the visual positioning method provided by the present invention. Figure 3 As shown in FIG. 1 , if the distance between adjacent frames is too short, the baseline during the rear intersection will be correspondingly shorter, thereby affecting the accuracy of visual positioning.

[0136] Figure 4 FIG. 3 is a flow chart of the visual positioning method provided by the present invention. Figure 4 As shown, the intelligent terminal uses pre-set constraints to fit a key frame selection model, through which an even number of qualified located images are extracted from the located image sequence as key frames, and the key frames are combined into key image pairs.

[0137] For any set of key image pairs, the key frame 1 and key frame 2 contained therein are constructed into a forward intersection, a local feature point cloud is established, and then matched with the two-dimensional image feature information of the current positioning image, and the pose of the image is solved through rearward intersection.

[0138] Among them, the constraint condition used as the basis for key frame selection must at least meet the requirement that the number of matched feature points reaches a certain number, especially the number of RANSAC inliers in the matched feature points must be greater than or equal to the second preset threshold, so that the direct linear transform algorithm (DLT) can be used to solve the pose.

[0139] The embodiment of the present invention uses constraint conditions to select key image pairs from the located image sequence, and uses the key image pairs with moderate overlap of observed content and their corresponding pose information to construct local feature point clouds. By selecting key frames, the quality of constructing local scenes and the consumed computing resources can be optimized, further improving the visual positioning accuracy of smart terminals in dynamic environments.

[0140] If there are multiple key image pairs that meet the number of RANSAC inliers defined by the constraint condition, the baseline distance between key frame 1 and key frame 2 in each key image pair can be determined to exceed a certain threshold as an additional condition based on the spatial position information stored in the pose information of the two frames, thereby expanding the original constraint condition. If the baseline distance is greater than or equal to the third preset threshold, the key image pair is determined to be valid; otherwise, it is invalid and needs to be eliminated.

[0141] If there are multiple key image pairs that meet the number of RANSAC inliers defined by the constraint condition, the relative angle value between the two frames can be determined based on the spatial posture information stored in the pose information of key frame 1 and key frame 2 in each key image pair and needs to meet certain requirements as an additional condition to expand the original constraint condition. If the relative angle values ​​on the three axes are all greater than or equal to the fourth preset threshold, the key image pair is determined to be valid, otherwise, it is invalid and needs to be eliminated.

[0142] The embodiment of the present invention ensures that the relative angle value of the two frames meets the requirement of the fourth preset threshold value and is added to the constraint condition, so as to further ensure the quality of the forward intersection when multiple groups of key image pairs are preliminarily screened out, and avoid the defect of excessive intersection error caused by too small translation distance when pure rotation occurs.

[0143] Figure 5 FIG4 is a flow chart of the visual positioning method provided by the present invention. It should be noted that the visual positioning method provided by the present invention is applied to a server connected to a smart terminal for communication. Figure 5 As shown, the method includes: step 501, determining any image frame in the initialization image sequence as a first initialization image frame, obtaining a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, and the initialization image sequence is acquired by a camera of a smart terminal;

[0144] Step 502: based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame, obtain the posture information corresponding to the first initialization image frame and the second initialization image frame;

[0145] Step 503: triangulate the pose information corresponding to the first initialization image frame and the second initialization image frame, and store the obtained three-dimensional space points in a feature point cloud library;

[0146] Step 504: perform feature matching from the initialization image sequence using the feature point cloud library, and expand the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame;

[0147] Step 505, repeating the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded to the feature point cloud library;

[0148] Step 506: Determine the pose information corresponding to the last initialization image frame in the initialization image sequence that is expanded to the feature point cloud library as the initial pose information in accordance with the matching order, and send the initial pose information to the smart terminal.

[0149] It should be noted that the execution subject of the embodiment of the present invention is a visual positioning device. The visual positioning device can be set in a server that is communicatively connected to the smart terminal, and the visual positioning device can also be the server.

[0150] The visual positioning method provided by the present invention is described below by taking the above-mentioned visual positioning device as the above-mentioned server as an example.

[0151] Specifically, in order to shorten the initialization time, the server uses a graphics processor GPU to accelerate the feature extraction process of the initialization image sequence. Since Structure from Motion (SfM) does not require real-time performance, the more reliable and robust SIFT (Scale-Invariant Feature Transform) feature points are usually extracted from the initialization image sequence. The corresponding descriptor is also called a sift descriptor.

[0152] The server can combine the absolute spatial position (such as longitude and latitude coordinates) corresponding to each image frame in the initialization image sequence. Therefore, when matching images, the longitude and latitude coordinates of the first initialization image frame are used to perform a neighborhood search, and only images within a certain range are matched to obtain the second initialization image frame with the largest number of matching point pairs.

[0153] The pose of one of the first and second initialization image frames is set as a unit matrix, and then the E matrix is ​​estimated through the matching point pairs between them, and the pose of the other image is decomposed by the E matrix. After estimating the poses of the two images, three-dimensional space points can be generated through triangulation, and the three-dimensional space points are stored in the feature point cloud library for GPU processing of subsequent images in the initialization image sequence.

[0154] Then use the feature point cloud library to perform image matching and repeat the above process. Perform pnp pose estimation on the third initialized image frame obtained, then triangulate it to generate new three-dimensional space points, and expand the new three-dimensional space points to the feature point cloud library. Repeat this cycle until all images in the initialized image sequence have completed pose estimation, so as to obtain the feature point cloud library established by incremental SfM, and use the pose estimated by the last initialized image frame corresponding to the matching order as the initial pose information, which is sent to the smart terminal.

[0155] It is understandable that when the initialization scene is relatively complex and the number of feature points is large, the amount of descriptor data for the entire image may be much larger than the amount of image data. In this case, directly transmitting the image is more efficient.

[0156] The pixels of cameras are generally over 10 million, far exceeding the needs of general image positioning. Therefore, smart terminals only need to capture smaller images for easy transmission, and perform SIFT feature extraction and matching on the image on the server side.

[0157] The embodiment of the present invention utilizes the server to process the image resources that consume a lot of energy, and establishes a feature point cloud library by using the SfM method. It can use the sparse point cloud in the feature point cloud library that contains descriptors of spatial positions and multiple images at the same time to more accurately match the image features in the initialization image sequence, thereby improving the accuracy and efficiency of the server's calculation of the initialization posture.

[0158] As an optional embodiment, feature matching is performed from the initialization image sequence using the feature point cloud library, including: matching the image features of each initialization image frame with all cluster centers obtained by clustering the feature point cloud library to determine the target cluster center;

[0159] After performing secondary matching on the image features of each initialization image frame in the cluster where the center of the target cluster is located, the initialization image frame with the largest number of matching point pairs is used as the third initialization image frame.

[0160] Specifically, the server can use a clustering algorithm such as K-Means to cluster the SIFT features in the feature point cloud library. This algorithm uses K as a parameter to divide N objects into K clusters so that the similarity within the cluster is high and the similarity between clusters is low. There are K cluster centers, and these K cluster centers correspond to the K indexes of the feature point cloud library.

[0161] After extracting features from any initialization image frame in the initialization image sequence, the obtained image features are first compared with the centers of each cluster, and the cluster center with the highest feature similarity is taken as the target cluster center, and then the search is performed again within the cluster where the target cluster center is located.

[0162] In the comparison process, the spatial position is used to filter the feature points first: the distance between the rough position and the point cloud feature points is calculated, and only the feature points less than a certain threshold are searched. Finally, the initialization image frame with the largest number of matching point pairs is output as the third initialization image frame to expand the feature point cloud library.

[0163] The embodiment of the present invention clusters the feature point cloud library, uses the center of each cluster as the index of the feature point cloud library, and uses the index to match the image features of the initialization image frame. The cluster center is first determined and then adjusted and optimized in the corresponding cluster, which can avoid the local optimal problem of the server in mapping.

[0164] For example, Figure 6 FIG. 5 is a flow chart of the visual positioning method provided by the present invention. Figure 6 As shown, taking a mobile phone terminal as an example, the visual positioning method provided by the present invention and the interaction process between the server and the mobile phone terminal are explained.

[0165] The visual positioning method provided by the present invention can be divided into two main parts: server solution process and mobile phone solution process. The server solution process relies on the server storage and computing environment, and the mobile phone solution process mainly relies on the smart phone platform.

[0166] The server solution process includes: first, the SfM method is used to establish a three-dimensional scene with absolute spatial positions to obtain a feature point cloud library. Before continuous positioning, the initialization image sequence is taken by the mobile phone and uploaded to the server. The server accurately locates the initialization image sequence through a visual positioning method based on the three-dimensional scene, and transmits the absolute spatial pose of the image as the initial pose information to the mobile phone.

[0167] The mobile phone solution process includes: after the mobile phone receives the initial posture information from the server, the subsequent positioning is completed offline on the mobile phone platform through iteration. The local feature point cloud containing scale information and absolute spatial position is established through the initial posture information, and the image features to be located are extracted. After matching with the local feature point cloud, the known object coordinates of the matching feature points are used to intersect and determine the position of the mobile phone camera and the posture of the mobile phone when shooting.

[0168] In addition, in order to effectively reduce the impact of moving objects in dynamic scenes on the accuracy of feature point matching and improve the positioning accuracy of the visual positioning method provided by the present invention, inertial measurement data is used to eliminate feature point matching on moving objects in dynamic scenes, thereby improving the environmental adaptability of the visual positioning method provided by the present invention.

[0169] Figure 7 This is one of the structural schematic diagrams of the visual positioning device provided by the present invention. Figure 7 The visual positioning device provided by the present invention is described. The visual positioning device described below and the visual positioning method provided by the present invention described above can be referred to each other. Figure 7 As shown, the device includes: a feature extraction module 701, a feature matching module 702 and a visual positioning module 703.

[0170] The feature extraction module 701 is used to extract features from the current positioning image and obtain image feature information of the current positioning image;

[0171] The feature matching module 702 is used to match the image feature information with the local feature point cloud to obtain the feature point pairs corresponding to the current positioning image;

[0172] The visual positioning module 703 is used to determine the first position information corresponding to the current positioning image based on the feature point pairs and the absolute spatial positions contained in the feature point pairs;

[0173] Among them, the current positioning image is collected by the camera in the intelligent terminal; the local feature point cloud is established based on the initial posture information sent by the server and the positioned image sequence; the positioned image sequence at least includes the initialization image sequence, and / or, the historical positioning image; the initialization image sequence is collected by the camera before collecting the current positioning image; the historical positioning image carries the corresponding posture information; the initial posture information is obtained based on the initialization image sequence.

[0174] It should be noted that the visual positioning device in the embodiment of the present invention can be arranged in an intelligent terminal that is communicatively connected to the server, and the above-mentioned visual positioning device can also be the above-mentioned intelligent terminal.

[0175] Specifically, the feature extraction module 701, the feature matching module 702 and the visual positioning module 703 are electrically connected.

[0176] The visual positioning device in the embodiment of the present invention constructs a local feature point cloud by sending initial posture information based on the server, and then obtains a feature point pair by matching the image feature information of the current positioning image with the above-mentioned local feature point cloud, and then uses the rear intersection method to solve the posture information when the smart terminal takes the current positioning image based on the above-mentioned feature point pair and the absolute spatial position contained in the above-mentioned feature point pair. As the first posture information corresponding to the current positioning image, it can take advantage of the relatively strong storage and computing resources of the server platform to complete the solution of the initial posture information with relatively large resource consumption in the server, and subsequently complete the dynamic environment visual positioning of the current positioning image offline in an iterative manner on the smart terminal, which can achieve more accurate and efficient visual positioning based on the smart terminal.

[0177] Figure 8 This is the second structural diagram of the visual positioning device provided by the present invention. Figure 8 The visual positioning device provided by the present invention is described. The visual positioning device described below and the visual positioning method provided by the present invention described above can be referred to each other. Figure 8 As shown, the device includes: an image frame determination module 801, a posture acquisition module 802, a three-dimensional processing module 803, a three-dimensional expansion module 804, a repeated iteration module 805 and an information sending module 806.

[0178] An image frame determination module 801 is used to determine any image frame in an initialization image sequence as a first initialization image frame, and obtain a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, wherein the initialization image sequence is acquired by a camera of a smart terminal;

[0179] A posture acquisition module 802 is used to obtain posture information corresponding to the first initialization image frame and the second initialization image frame based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame;

[0180] A three-dimensional processing module 803 is used to triangulate the position information corresponding to the first initialization image frame and the second initialization image frame, and store the obtained three-dimensional space points in a feature point cloud library;

[0181] A three-dimensional expansion module 804 is used to perform feature matching from the initialization image sequence using the feature point cloud library, and expand the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame;

[0182] The iterative module 805 is used to repeat the steps of determining any image frame in the initialization image sequence as the first initialization image frame, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded to the feature point cloud library;

[0183] The information sending module 806 is used to determine the pose information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library as the initial pose information according to the matching order, and send the initial pose information to the smart terminal.

[0184] It should be noted that the visual positioning device in the embodiment of the present invention can be set in a server that is communicatively connected to the smart terminal, and the above-mentioned visual positioning device can also be the above-mentioned server.

[0185] Specifically, the image frame determination module 801, the posture acquisition module 802, the three-dimensional processing module 803, the three-dimensional expansion module 804, the repeated iteration module 805 and the information sending module 806 are electrically connected.

[0186] The visual positioning device in the embodiment of the present invention utilizes the server to process the image resources that consume a lot of energy, and establishes a feature point cloud library by using the SfM method. It can use the sparse point cloud in the feature point cloud library that contains descriptors of spatial positions and multiple images at the same time to more accurately match the image features in the initialization image sequence, thereby improving the accuracy and efficiency of the server's calculation of the initialization posture.

[0187] Fig. 9 Schematic diagram of the structure of the visual positioning system provided by the present invention. Fig. 9 The visual positioning system provided by the present invention is described. Fig. 9 As shown, the system includes: a smart terminal 901 and a server 902; the smart terminal 901 and the server 902 are in communication connection; the smart terminal 901 includes a camera, a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor; the server 902 includes a second memory, a second processor, and a second computer program stored in the second memory and executable on the second processor;

[0188] When the smart terminal 901 executes the first computer program, the visual positioning method as described in any one of the above items is implemented; when the server 902 executes the second computer program, the visual positioning method as described above is implemented.

[0189] Specifically, the visual positioning method system can be composed of one or more smart terminals 901 at the perception edge and a server 902 at a remote command center. Each smart terminal 901 establishes a communication connection with the server 902 through wireless communication technology to form a dynamic network, and maintains the access and exit of any smart terminal 901 through security verification means.

[0190] Fig.10 is a schematic diagram of the physical structure of the intelligent terminal in the visual positioning system provided by the present invention, such as Fig.10 As shown, the smart terminal 901 may include: a first processor (processor) 1010, a communication interface (CommunicationsInterface) 1020, a first memory (memory) 1030, a communication bus 1040 and a camera 1050, wherein the first processor 1010, the communication interface 1020, the first memory 1030, and the camera 1050 communicate with each other through the communication bus 1040. The first processor 1010 can call the logic instructions in the first memory 1030 to execute the visual positioning method, which includes: performing feature extraction on the current positioning image to obtain image feature information of the current positioning image; matching the image feature information with the local feature point cloud to obtain the feature point pair corresponding to the current positioning image; determining the first position information corresponding to the current positioning image based on the feature point pair and the absolute spatial position contained in the feature point pair; wherein the current positioning image is acquired by the camera in the smart terminal; the local feature point cloud is established based on the initial position information sent by the server and the positioned image sequence; the positioned image sequence includes at least the initialization image sequence, and / or, the historical positioning image; the initialization image sequence is acquired by the camera before acquiring the current positioning image; the historical positioning image carries the corresponding position information; the initial position information is acquired based on the initialization image sequence.

[0191] Fig.11 is a schematic diagram of the physical structure of the server in the visual positioning system provided by the present invention, such as Fig.11As shown, the server 902 may include: a second processor 1111, a communication interface 1120, a second memory 1130 and a communication bus 1040, wherein the second processor 1111, the communication interface 1120, and the second memory 1130 communicate with each other via the communication bus 1040. The second processor 1111 may call the logic instructions in the second memory 1130 to execute a visual positioning method, the method comprising: determining any image frame in the initialization image sequence as a first initialization image frame, obtaining a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, the initialization image sequence being acquired by a camera of a smart terminal; obtaining the posture information corresponding to the first initialization image frame and the second initialization image frame based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame; triangulating the posture information corresponding to the first initialization image frame and the second initialization image frame, and obtaining the obtained three-dimensional posture information. The spatial points are stored in a feature point cloud library; feature matching is performed from the initialization image sequence using the feature point cloud library, and the feature point cloud library is expanded with the three-dimensional spatial points contained in the obtained third initialization image frame; the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, and expanding the feature point cloud library with the three-dimensional spatial points contained in the obtained third initialization image frame, are repeated until the three-dimensional spatial points contained in all image frames in the initialization image sequence are expanded to the feature point cloud library; the pose information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library is determined as the initial pose information in the matching order, and the initial pose information is sent to the smart terminal.

[0192] In addition, the logic instructions in the first memory 1030 and / or the second memory 1130 mentioned above can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only first memory (ROM, Read-Only Memory), a random access first memory (RAM, RandomAccess Memory), a disk or an optical disk.

[0193] The specific process of the intelligent terminal executing the first computer program and the server executing the second computer program to implement the visual positioning method provided by the present invention can be referred to the contents of the above embodiments, which will not be described in detail in the embodiments of the present invention.

[0194] The visual positioning system in the embodiment of the present invention pre-takes at least two images taken by the smart terminal as an initialization image sequence and transmits them to the server to calculate the initial pose information, so that after the smart terminal receives the initial pose information transmitted from the server, a local scene is established locally on the smart terminal, and then the rear intersection method is used to calculate the first pose information when the smart terminal takes the current positioning image. It can take advantage of the relatively strong storage and computing resources of the server platform, and complete the resource-intensive solution process on the server. Subsequently, the pose of subsequent images taken is calculated in an iterative manner on the smart terminal platform, avoiding the slow solution speed when the smart terminal platform independently completes all visual combination positioning solutions, and improving the efficiency and accuracy of visual positioning through the interaction between the smart terminal and the server.

[0195] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the visual positioning method provided by the above-mentioned methods. The method may include: extracting features from the current positioning image to obtain image feature information of the current positioning image; matching the image feature information with the local feature point cloud to obtain feature point pairs corresponding to the current positioning image; determining the first position information corresponding to the current positioning image based on the feature point pairs and the absolute spatial positions contained in the feature point pairs; wherein the current positioning image is acquired by a camera in the intelligent terminal; the local feature point cloud is established based on the initial position information sent by the server and the positioned image sequence; the positioned image sequence includes at least an initialization image sequence, and / or a historical positioning image; the initialization image sequence is acquired by the camera before acquiring the current positioning image; the historical positioning image carries the corresponding position information; the initial position information is obtained based on the initialization image sequence.

[0196] The method may also include: determining any image frame in the initialization image sequence as the first initialization image frame, obtaining the second initialization image frame with the largest number of matching point pairs with the first initialization image frame, and the initialization image sequence is acquired by the camera of the intelligent terminal; based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame, obtaining the posture information corresponding to the first initialization image frame and the second initialization image frame; triangulating the posture information corresponding to the first initialization image frame and the second initialization image frame, and storing the obtained three-dimensional space points in a feature point cloud library; using the feature point cloud The library performs feature matching from the initialization image sequence, and expands the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame; repeats the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame, until the three-dimensional space points contained in all image frames in the initialization image sequence are expanded to the feature point cloud library; according to the matching order, the posture information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library is determined as the initial posture information, and the initial posture information is sent to the smart terminal.

[0197] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the visual positioning method provided by the above-mentioned methods, the method comprising: performing feature extraction on a current positioning image to obtain image feature information of the current positioning image; matching the image feature information with a local feature point cloud to obtain feature point pairs corresponding to the current positioning image; determining the first position information corresponding to the current positioning image based on the feature point pairs and the absolute spatial positions contained in the feature point pairs; wherein the current positioning image is acquired by a camera in a smart terminal; the local feature point cloud is established based on the initial position information sent by a server and a sequence of positioned images; the sequence of positioned images at least includes an initialization image sequence, and / or a historical positioning image; the initialization image sequence is acquired by the camera before acquiring the current positioning image; the historical positioning image carries the corresponding position information; the initial position information is acquired based on the initialization image sequence.

[0198] The method may also include: determining any image frame in the initialization image sequence as the first initialization image frame, obtaining the second initialization image frame with the largest number of matching point pairs with the first initialization image frame, and the initialization image sequence is acquired by the camera of the intelligent terminal; based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame, obtaining the posture information corresponding to the first initialization image frame and the second initialization image frame; triangulating the posture information corresponding to the first initialization image frame and the second initialization image frame, and storing the obtained three-dimensional space points in a feature point cloud library; using the feature point cloud The library performs feature matching from the initialization image sequence, and expands the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame; repeats the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame, until the three-dimensional space points contained in all image frames in the initialization image sequence are expanded to the feature point cloud library; according to the matching order, the posture information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library is determined as the initial posture information, and the initial posture information is sent to the smart terminal.

[0199] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0200] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual positioning method, characterized in that: Applied to an intelligent terminal connected to a server for communication, the method comprises: Performing feature extraction on the current positioning image to obtain image feature information of the current positioning image; Matching the image feature information with the local feature point cloud to obtain feature point pairs corresponding to the current positioning image; Based on the feature point pair and the absolute spatial position contained in the feature point pair, determining the first position information corresponding to the current positioning image; Wherein, the current positioning image is acquired by the camera in the smart terminal; the local feature point cloud is established based on the initial posture information sent by the server and the positioned image sequence; the positioned image sequence at least includes an initialization image sequence and / or a historical positioning image; the initialization image sequence is acquired by the camera before acquiring the current positioning image; the historical positioning image carries the corresponding posture information; the initial posture information is obtained based on the initialization image sequence; The determining, based on the feature point pair and the absolute spatial position contained in the feature point pair, the first position information corresponding to the current positioning image comprises: Based on the feature point pairs and the absolute spatial positions contained in the feature point pairs, the position and posture information when the smart terminal takes the current positioning image is calculated using a rear intersection method as the first position and posture information corresponding to the current positioning image; The specific steps of establishing the local feature point cloud include: Taking at least two located images satisfying the constraint conditions in the located image sequence as key image pairs; Performing forward intersection based on the initial pose information, the key image pair, and the pose information corresponding to the key image pair to establish the local feature point cloud; The constraint conditions include at least one of the following: the number of matching points belonging to RANSAC inliers among the matching points between any two positioned images in the positioned image sequence is greater than or equal to a second preset threshold, the baseline distance determined according to the posture information of the two positioned images in the key image pair is greater than or equal to a third preset threshold, and the relative angle value determined according to the posture information of the two positioned images in the key image pair is greater than or equal to a fourth preset threshold; After matching the image feature information with the local feature point cloud to obtain feature point pairs corresponding to the current positioning image, the method further includes: Acquire inertial measurement data and first historical posture information corresponding to a previous positioning image, wherein the inertial measurement data is collected by an inertial navigation module in the smart terminal; Determine an affine transformation matrix between a collection time of the previous positioning image and a collection time of the current positioning image according to the inertial measurement data and the first historical position and posture information; Transforming the image feature information of the current positioning image based on the affine transformation matrix to obtain second historical posture information corresponding to the acquisition moment of the previous positioning image; When it is determined that the absolute value of the difference between the second historical pose information and the pose information corresponding to any one of the feature point pairs is less than a first preset threshold, determining the any one of the feature point pairs as a stationary feature point pair; Determine second posture information corresponding to the current positioning image by using the stationary feature point pair and the absolute spatial position of the stationary feature point pair; The initial posture information is determined by the server based on the following steps: Determine any image frame in an initialization image sequence as a first initialization image frame, and obtain a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, wherein the initialization image sequence is acquired by a camera of the smart terminal; Based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame, obtaining the posture information corresponding to the first initialization image frame and the second initialization image frame; Triangulate the pose information corresponding to the first initialization image frame and the second initialization image frame, and store the obtained three-dimensional space points in a feature point cloud library; Performing feature matching from the initialization image sequence using the feature point cloud library, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame; Repeat the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, until the three-dimensional space points contained in the obtained third initialization image frame are expanded into the feature point cloud library, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded into the feature point cloud library; Determine the pose information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library as the initial pose information in a matching order; The method further includes determining an affine transformation matrix between the acquisition time of the previous positioning image and the acquisition time of the current positioning image based on the inertial measurement data and the first historical posture information, and transforming the image feature information of the current positioning image based on the affine transformation matrix to obtain the second historical posture information corresponding to the acquisition time of the previous positioning image: Determining, based on the feature point pairs and the affine transformation matrix, that the inertial measurement data is valid; When it is determined that the inertial measurement data is invalid, adding the current positioning image carrying the corresponding first position information as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence; After determining the second posture information corresponding to the current positioning image by using the stationary feature point pair and the absolute spatial position of the stationary feature point pair, the method further includes: The current positioning image carrying the corresponding second posture information is added as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence.

2. The visual positioning method according to claim 1, characterized in that: After obtaining the updated positioned image sequence, the method further includes: Based on the updated positioned image sequence, obtaining a reprojection cumulative error; Based on the reprojection accumulated error, the number of historical positioning images in the positioned image sequence is dynamically maintained.

3. A visual positioning method, characterized in that: Applied to a server in communication connection with a smart terminal, the method comprises: Determine any image frame in an initialization image sequence as a first initialization image frame, and obtain a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, wherein the initialization image sequence is acquired by a camera of the smart terminal; Based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame, obtaining the posture information corresponding to the first initialization image frame and the second initialization image frame; Triangulate the pose information corresponding to the first initialization image frame and the second initialization image frame, and store the obtained three-dimensional space points in a feature point cloud library; Performing feature matching from the initialization image sequence using the feature point cloud library, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame; Repeat the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, until the three-dimensional space points contained in the obtained third initialization image frame are expanded into the feature point cloud library, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded into the feature point cloud library; Determine the pose information corresponding to the last initialization image frame in the initialization image sequence that is expanded to the feature point cloud library as the initial pose information in a matching order, and send the initial pose information to the smart terminal so that the smart terminal can perform feature extraction on the current positioning image, obtain image feature information of the current positioning image, match the image feature information with the local feature point cloud, obtain feature point pairs corresponding to the current positioning image, and determine the first pose information corresponding to the current positioning image based on the feature point pairs and the absolute spatial positions contained in the feature point pairs; Wherein, the current positioning image is acquired by the camera in the smart terminal; the local feature point cloud is established based on the initial posture information sent by the server and the positioned image sequence; the positioned image sequence at least includes an initialization image sequence and / or a historical positioning image; the initialization image sequence is acquired by the camera before acquiring the current positioning image; the historical positioning image carries the corresponding posture information; the initial posture information is obtained based on the initialization image sequence; The intelligent terminal determines the first position information corresponding to the current positioning image based on the feature point pair and the absolute spatial position contained in the feature point pair, including: Based on the feature point pairs and the absolute spatial positions contained in the feature point pairs, the position and posture information when the smart terminal takes the current positioning image is calculated using a rear intersection method as the first position and posture information corresponding to the current positioning image; The specific steps of establishing the local feature point cloud include: Taking at least two located images satisfying the constraint conditions in the located image sequence as key image pairs; Performing forward intersection based on the initial pose information, the key image pair, and the pose information corresponding to the key image pair to establish the local feature point cloud; The constraint conditions include at least one of the following: the number of matching points belonging to RANSAC inliers among the matching points between any two positioned images in the positioned image sequence is greater than or equal to a second preset threshold, the baseline distance determined according to the posture information of the two positioned images in the key image pair is greater than or equal to a third preset threshold, and the relative angle value determined according to the posture information of the two positioned images in the key image pair is greater than or equal to a fourth preset threshold; After the smart terminal matches the image feature information with the local feature point cloud to obtain the feature point pairs corresponding to the current positioning image, the smart terminal is further configured to perform the following steps: Acquire inertial measurement data and first historical posture information corresponding to a previous positioning image, wherein the inertial measurement data is collected by an inertial navigation module in the smart terminal; Determine an affine transformation matrix between a collection time of the previous positioning image and a collection time of the current positioning image according to the inertial measurement data and the first historical position and posture information; Transforming the image feature information of the current positioning image based on the affine transformation matrix to obtain second historical posture information corresponding to the acquisition moment of the previous positioning image; When it is determined that the absolute value of the difference between the second historical pose information and the pose information corresponding to any one of the feature point pairs is less than a first preset threshold, determining the any one of the feature point pairs as a stationary feature point pair; Determine second posture information corresponding to the current positioning image by using the stationary feature point pair and the absolute spatial position of the stationary feature point pair; The initial posture information is determined by the server based on the following steps: Determine any image frame in an initialization image sequence as a first initialization image frame, and obtain a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, wherein the initialization image sequence is acquired by a camera of the smart terminal; Based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame, obtaining the posture information corresponding to the first initialization image frame and the second initialization image frame; Triangulate the pose information corresponding to the first initialization image frame and the second initialization image frame, and store the obtained three-dimensional space points in a feature point cloud library; Performing feature matching from the initialization image sequence using the feature point cloud library, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame; Repeat the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, until the three-dimensional space points contained in the obtained third initialization image frame are expanded into the feature point cloud library, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded into the feature point cloud library; Determine the pose information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library as the initial pose information in a matching order; The smart terminal determines the affine transformation matrix between the acquisition time of the previous positioning image and the acquisition time of the current positioning image according to the inertial measurement data and the first historical posture information, and the smart terminal transforms the image feature information of the current positioning image based on the affine transformation matrix to obtain the second historical posture information corresponding to the acquisition time of the previous positioning image. The smart terminal is also used to perform the following steps: Determining, based on the feature point pairs and the affine transformation matrix, that the inertial measurement data is valid; When it is determined that the inertial measurement data is invalid, adding the current positioning image carrying the corresponding first position information as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence; After determining the second posture information corresponding to the current positioning image by using the static feature point pair and the absolute spatial position of the static feature point pair, the smart terminal is further used to perform the following steps: The current positioning image carrying the corresponding second posture information is added as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence.

4. The visual positioning method according to claim 3, characterized in that: The using the feature point cloud library to perform feature matching from the initialization image sequence includes: Matching the image features of each initialized image frame with all cluster centers obtained by clustering the feature point cloud library to determine the target cluster center; After performing secondary matching on the image features of each of the initialization image frames in the cluster where the center of the target cluster is located, the initialization image frame with the largest number of matching point pairs is used as the third initialization image frame.

5. A visual positioning device, characterized in that: include: A feature extraction module is used to extract features from the current positioning image to obtain image feature information of the current positioning image; A feature matching module, used to match the image feature information with the local feature point cloud to obtain feature point pairs corresponding to the current positioning image; A visual positioning module, used to determine the first position information corresponding to the current positioning image based on the feature point pair and the absolute spatial position contained in the feature point pair; Wherein, the current positioning image is acquired by a camera in the intelligent terminal; the local feature point cloud is established based on the initial posture information sent by the server and the positioned image sequence; the positioned image sequence at least includes an initialization image sequence and / or a historical positioning image; the initialization image sequence is acquired by the camera before acquiring the current positioning image; the historical positioning image carries the corresponding posture information; the initial posture information is obtained based on the initialization image sequence; The visual positioning module determines the first position information corresponding to the current positioning image based on the feature point pair and the absolute spatial position contained in the feature point pair, including: Based on the feature point pairs and the absolute spatial positions contained in the feature point pairs, the position and posture information when the smart terminal takes the current positioning image is calculated using a rear intersection method as the first position and posture information corresponding to the current positioning image; The specific steps of establishing the local feature point cloud include: Taking at least two located images satisfying the constraint conditions in the located image sequence as key image pairs; Performing forward intersection based on the initial pose information, the key image pair, and the pose information corresponding to the key image pair to establish the local feature point cloud; The constraint conditions include at least one of the following: the number of matching points belonging to RANSAC inliers among the matching points between any two positioned images in the positioned image sequence is greater than or equal to a second preset threshold, the baseline distance determined according to the posture information of the two positioned images in the key image pair is greater than or equal to a third preset threshold, and the relative angle value determined according to the posture information of the two positioned images in the key image pair is greater than or equal to a fourth preset threshold; The visual positioning module is also used for: Acquire inertial measurement data and first historical posture information corresponding to a previous positioning image, wherein the inertial measurement data is collected by an inertial navigation module in the smart terminal; Determine an affine transformation matrix between a collection time of the previous positioning image and a collection time of the current positioning image according to the inertial measurement data and the first historical position and posture information; Transforming the image feature information of the current positioning image based on the affine transformation matrix to obtain second historical posture information corresponding to the acquisition moment of the previous positioning image; When it is determined that the absolute value of the difference between the second historical pose information and the pose information corresponding to any one of the feature point pairs is less than a first preset threshold, determining the any one of the feature point pairs as a stationary feature point pair; Determine second posture information corresponding to the current positioning image by using the stationary feature point pair and the absolute spatial position of the stationary feature point pair; The initial posture information is determined by the server based on the following steps: Determine any image frame in an initialization image sequence as a first initialization image frame, and obtain a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, wherein the initialization image sequence is acquired by a camera of the smart terminal; Based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame, obtaining the posture information corresponding to the first initialization image frame and the second initialization image frame; Triangulate the pose information corresponding to the first initialization image frame and the second initialization image frame, and store the obtained three-dimensional space points in a feature point cloud library; Performing feature matching from the initialization image sequence using the feature point cloud library, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame; Repeat the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, until the three-dimensional space points contained in the obtained third initialization image frame are expanded into the feature point cloud library, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded into the feature point cloud library; Determine the pose information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library as the initial pose information in a matching order; The device is also used for: Determining, based on the feature point pairs and the affine transformation matrix, that the inertial measurement data is valid; When it is determined that the inertial measurement data is invalid, adding the current positioning image carrying the corresponding first position information as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence; After determining the second posture information corresponding to the current positioning image by using the stationary feature point pair and the absolute spatial position of the stationary feature point pair, the method further includes: Adding the current positioning image carrying the corresponding second posture information as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence; The device is also used for: The current positioning image carrying the corresponding second posture information is added as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence.

6. A visual positioning device, characterized in that: include: An image frame determination module, used to determine any image frame in an initialization image sequence as a first initialization image frame, and obtain a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, wherein the initialization image sequence is acquired by a camera of a smart terminal; A posture acquisition module, used to obtain posture information corresponding to the first initialization image frame and the second initialization image frame based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame; A three-dimensional processing module, used for triangulating the posture information corresponding to the first initialization image frame and the second initialization image frame, and storing the obtained three-dimensional space points in a feature point cloud library; A three-dimensional expansion module, used to perform feature matching from the initialization image sequence using the feature point cloud library, and expand the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame; A repeated iteration module, used for repeating the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded to the feature point cloud library; An information sending module, used to determine the pose information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library as the initial pose information in accordance with the matching order, and send the initial pose information to the intelligent terminal, so that the intelligent terminal can perform feature extraction on the current positioning image, obtain the image feature information of the current positioning image, match the image feature information with the local feature point cloud, obtain the feature point pair corresponding to the current positioning image, and determine the first pose information corresponding to the current positioning image based on the feature point pair and the absolute spatial position contained in the feature point pair; Wherein, the current positioning image is acquired by the camera in the smart terminal; the local feature point cloud is established based on the initial posture information sent by the server and the positioned image sequence; the positioned image sequence at least includes an initialization image sequence and / or a historical positioning image; the initialization image sequence is acquired by the camera before acquiring the current positioning image; the historical positioning image carries the corresponding posture information; the initial posture information is obtained based on the initialization image sequence; The intelligent terminal determines the first position information corresponding to the current positioning image based on the feature point pair and the absolute spatial position contained in the feature point pair, including: Based on the feature point pairs and the absolute spatial positions contained in the feature point pairs, the position and posture information when the smart terminal takes the current positioning image is calculated using a rear intersection method as the first position and posture information corresponding to the current positioning image; The specific steps of establishing the local feature point cloud include: Taking at least two located images satisfying the constraint conditions in the located image sequence as key image pairs; Performing forward intersection based on the initial pose information, the key image pair, and the pose information corresponding to the key image pair to establish the local feature point cloud; The constraint conditions include at least one of the following: the number of matching points belonging to RANSAC inliers among the matching points between any two positioned images in the positioned image sequence is greater than or equal to a second preset threshold, the baseline distance determined according to the posture information of the two positioned images in the key image pair is greater than or equal to a third preset threshold, and the relative angle value determined according to the posture information of the two positioned images in the key image pair is greater than or equal to a fourth preset threshold; After the smart terminal matches the image feature information with the local feature point cloud to obtain the feature point pairs corresponding to the current positioning image, the smart terminal is further configured to perform the following steps: Acquire inertial measurement data and first historical posture information corresponding to a previous positioning image, wherein the inertial measurement data is collected by an inertial navigation module in the smart terminal; Determine an affine transformation matrix between a collection time of the previous positioning image and a collection time of the current positioning image according to the inertial measurement data and the first historical position and posture information; Transforming the image feature information of the current positioning image based on the affine transformation matrix to obtain second historical posture information corresponding to the acquisition moment of the previous positioning image; When it is determined that the absolute value of the difference between the second historical pose information and the pose information corresponding to any one of the feature point pairs is less than a first preset threshold, determining the any one of the feature point pairs as a stationary feature point pair; Determine second posture information corresponding to the current positioning image by using the stationary feature point pair and the absolute spatial position of the stationary feature point pair; The initial position information is determined by the visual positioning device based on the following steps: Determine any image frame in an initialization image sequence as a first initialization image frame, and obtain a second initialization image frame having the largest number of matching point pairs with the first initialization image frame, wherein the initialization image sequence is acquired by a camera of the smart terminal; Based on the absolute spatial position corresponding to the first initialization image frame and the absolute spatial position corresponding to the second initialization image frame, obtaining the posture information corresponding to the first initialization image frame and the second initialization image frame; Triangulate the pose information corresponding to the first initialization image frame and the second initialization image frame, and store the obtained three-dimensional space points in a feature point cloud library; Performing feature matching from the initialization image sequence using the feature point cloud library, and expanding the feature point cloud library with the three-dimensional space points contained in the obtained third initialization image frame; Repeat the above steps of determining any image frame in the initialization image sequence as the first initialization image frame, until the three-dimensional space points contained in the obtained third initialization image frame are expanded into the feature point cloud library, until the three-dimensional space points contained in all the image frames in the initialization image sequence are expanded into the feature point cloud library; Determine the pose information corresponding to the last initialization image frame in the initialization image sequence expanded to the feature point cloud library as the initial pose information in a matching order; The smart terminal determines the affine transformation matrix between the acquisition time of the previous positioning image and the acquisition time of the current positioning image according to the inertial measurement data and the first historical posture information, and the smart terminal transforms the image feature information of the current positioning image based on the affine transformation matrix to obtain the second historical posture information corresponding to the acquisition time of the previous positioning image. The smart terminal is also used to perform the following steps: Determining, based on the feature point pairs and the affine transformation matrix, that the inertial measurement data is valid; When it is determined that the inertial measurement data is invalid, adding the current positioning image carrying the corresponding first position information as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence; After determining the second posture information corresponding to the current positioning image by using the static feature point pair and the absolute spatial position of the static feature point pair, the smart terminal is further used to perform the following steps: The current positioning image carrying the corresponding second posture information is added as a historical positioning image to the positioned image sequence to obtain an updated positioned image sequence.

7. A visual positioning method system, characterized in that: include: A smart terminal and a server; the smart terminal and the server are in communication connection; the smart terminal comprises a camera, a first memory, a first processor, and a first computer program stored in the first memory and executable on the first processor; the smart terminal comprises a second memory, a second processor, and a second computer program stored in the second memory and executable on the second processor; When the smart terminal executes the first computer program, the visual positioning method as described in any one of claims 1 to 2 is implemented; when the server executes the second computer program, the visual positioning method as described in claim 3 or 4 is implemented.

Images