Visual inertial positioning motion consistency detection method, device and system and medium

Abstract

The invention discloses a visual inertial positioning motion consistency detection method, device and system and a medium, and relates to the technical field of computer vision. The method comprises the following steps: respectively acquiring feature maps of front and back frame images and a global feature map of the front frame image; determining similarity features according to the feature maps of the front and back frame images; processing the similarity features based on a self-attention model, and outputting an optical flow between front and back frame images; determining dynamic feature points according to the optical flow between the front and back frame images; and determining static feature points according to the global feature map and the dynamic feature points of the previous frame image so as to complete motion consistency detection. According to the motion consistency detection method, the matching relation between the front frame and the rear frame of the image is considered, the influence of local shielding is reduced, the calculation amount of the motion consistency detection model is reduced, the calculation efficiency is improved, and meanwhile the detection precision is kept.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual inertial positioning motion consistency detection method, a visual inertial positioning motion consistency detection device, a computer-readable storage medium and a visual inertial positioning system. Background Art

[0002] Visual inertial positioning methods have made great progress after a long period of development. However, existing visual inertial positioning methods mostly assume that the scene is static. If objects move in dynamic scenes, it will cause incomplete image information and motion blur, resulting in a break in the tracking relationship between image frames, leading to an offset in the sensor pose state estimation.

[0003] In the related art, a method is proposed to measure the action based on the angle deviation ratio of the key bone coordinates and the joint distance of each frame in the aligned teaching video and the student self-study video, calculate the action difference, generate posture adjustment suggestions based on the action difference, and generate multi-scale scores from local actions to overall actions based on the action difference and the motion pyramid model. This technical solution only considers the match between two action sequences, and does not consider the match between the previous and next frames in the action sequence. In addition, the motion pyramid model is involved in the calculation, and the calculation complexity is relatively high. In the related art, it is also proposed to divide one or more frames of the image in the acquired video segment data into blocks, extract the feature points and corresponding information of each block, and judge whether the scene has changed based on the currently extracted feature point information and the corresponding historical feature point information. This technical solution takes into account the relationship between global features, but it is difficult to solve the problem of local occlusion.

[0004] In addition, visual inertial positioning can also be achieved based on a large-scale visual text pre-training model, but the large-scale visual text pre-training model has two defects. First, it needs to rely on a large amount of text data that matches the visual information during the training process, which makes the model training process more difficult and less accurate. At the same time, the existing text information comes from the Internet, and the coverage of professional field data is very limited. The large-scale visual text pre-training model is seriously insufficient in the perception and cognition of professional images. Even if the pre-training model is fine-tuned by a small number of annotated images, the contradiction between the small number of samples and the large number of parameters of the pre-training model makes the gradient change during the fine-tuning of the pre-training model not obvious. Second, due to the lack of modeling of object motion laws and mining of action timing, the pre-training large model has limited ability to recognize the trajectory of moving objects, weak interpretability, and difficulty in dealing with the problem of local occlusion in dynamic scenes, which in turn affects the accuracy and robustness of pose estimation in visual inertial positioning. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art to at least a certain extent. To this end, the first purpose of the present invention is to propose a visual inertial positioning motion consistency detection method, respectively obtaining feature maps of the front and rear frame images and the global feature map of the front frame image; determining similarity features based on the feature maps of the front and rear frame images; processing the similarity features based on the self-attention model, and outputting the optical flow between the front and rear frame images; determining dynamic feature points based on the optical flow between the front and rear frame images; determining static feature points based on the global feature map and dynamic feature points of the front frame image to complete motion consistency detection. The detection method of the present invention performs deep modeling on the image captured by the camera, and uses the rich object knowledge provided by the large-scale pre-trained image segmentation model to supplement the optical flow prediction model based on self-attention to achieve motion consistency detection. In this way, the matching relationship between the front and rear frames of the image is considered to reduce the influence of local occlusion, and the calculation amount of the motion consistency detection model is reduced, the operation efficiency is improved, and the detection accuracy is maintained.

[0006] The second objective of the present invention is to provide a visual inertial positioning motion consistency detection device.

[0007] A third object of the present invention is to provide a computer-readable storage medium.

[0008] A fourth objective of the present invention is to provide a visual inertial positioning system.

[0009] To achieve the above-mentioned objectives, the first aspect of the present invention proposes a visual inertial positioning motion consistency detection method, which includes: respectively obtaining feature maps of previous and next frame images and a global feature map of the previous frame image; determining similarity features based on the feature maps of the previous and next frame images; processing the similarity features based on a self-attention model, and outputting the optical flow between the previous and next frame images; determining dynamic feature points based on the optical flow between the previous and next frame images; determining static feature points based on the global feature map and dynamic feature points of the previous frame image to complete motion consistency detection.

[0010] According to one embodiment of the present invention, the self-attention model includes an encoder and a decoder, which processes similarity features based on the self-attention model and outputs the optical flow between the previous and next frame images, including: encoding the similarity features based on the encoder to obtain target encoding features of the similarity features; decoding the target encoding features based on the decoder to output the optical flow between the previous and next frame images.

[0011] According to one embodiment of the present invention, the encoder includes multiple encoding layers and a feedforward neural network, and encodes similarity features based on the encoder to obtain target encoding features of the similarity features, including: grouping the similarity features according to pixels to obtain multiple groups of features; performing self-attention calculations and feedforward neural network calculations on each group of features to obtain encoding features of each group of features; and splicing the encoding features of multiple groups of features to obtain target encoding features.

[0012] According to one embodiment of the present invention, self-attention calculation and feedforward neural network calculation are performed on each group of features to obtain the coding features of each group of features, including: obtaining the query vector, key vector and value vector of each group of features in the current coding layer; calculating the attention features of each group of features in the current coding layer according to the query vector, key vector, value vector and attention features of the previous coding layer of the current coding layer; performing feedforward neural network calculation on the attention features of the current coding layer, and taking the sum of the result and the attention features of the current coding layer as the coding features of the current layer, and so on to obtain the coding features of the last layer of the encoder, and taking the coding features of the last layer as the coding features of each group of features.

[0013] According to one embodiment of the present invention, the decoder includes multiple decoding layers, the encoder and the decoder have the same number of layers, and the target coding features are decoded based on the decoder to output the optical flow between the previous and next frame images, including: calculating the pixel change based on the optical flow between the previous frame image and the previous and next frame images decoded in the previous cycle; center cropping the changed image to obtain the cropped image features; performing feedforward neural network calculation on the cropped image features to obtain the query vector of the decoder's current cycle; determining the key vector and value vector of the decoder's current cycle based on the coding features; determining the attention features of the decoding in this cycle based on the key vector, value vector and query vector of the decoder's current cycle, and so on, to obtain the decoding features of the decoder's last cycle, recorded as the optical flow between the previous and next frame images.

[0014] According to one embodiment of the present invention, similarity features are determined based on feature maps of previous and next frame images, including: determining a similarity matrix of the previous and next frame images based on all pixels in the feature maps of the previous and next frame images; and converting the similarity matrix into similarity features based on a linear layer.

[0015] According to one embodiment of the present invention, static feature points are determined based on a global feature map and dynamic feature points of a previous frame image, including: extracting a global feature map of the previous frame image based on SAM; performing semantic segmentation on the global feature map to obtain a semantic segmentation map; determining a segmentation area of ​​the dynamic feature points in the semantic segmentation map based on the dynamic feature points and the semantic segmentation map, and treating the segmentation area as a moving object; deleting all dynamic feature points in the identified moving object, and treating the remaining feature points in the semantic segmentation map as static feature points.

[0016] According to one embodiment of the present invention, dynamic feature points are determined based on the optical flow between previous and next frame images, including: arranging the optical flows of the previous and next frame images in order from large to small to obtain an optical flow sequence; and taking the feature points corresponding to the optical flow before a preset threshold in the optical flow sequence as dynamic feature points.

[0017] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a visual inertial positioning motion consistency detection device, which includes: an acquisition module, used to respectively acquire the feature maps of the previous and next frame images and the global feature map of the previous frame image; a first determination module, used to determine the similarity features based on the feature maps of the previous and next frame images; an output module, used to process the similarity features based on the self-attention model, and output the optical flow between the previous and next frame images; a second determination module, used to determine the dynamic feature points based on the optical flow between the previous and next frame images; a third determination module, used to determine the static feature points based on the global feature map and dynamic feature points of the previous frame image, so as to complete the motion consistency detection.

[0018] To achieve the above objectives, a third aspect of the present invention proposes a computer-readable storage medium on which a visual-inertial positioning motion consistency detection program is stored. When the visual-inertial positioning motion consistency detection program is executed by a processor, the aforementioned visual-inertial positioning motion consistency detection method is implemented.

[0019] To achieve the above-mentioned objectives, the fourth aspect of the present invention proposes a visual inertial positioning system, including a memory, a processor, and a visual inertial positioning motion consistency detection program stored in the memory and executable on the processor. When the processor executes the visual inertial positioning motion consistency detection program, the aforementioned visual inertial positioning motion consistency detection method is implemented.

[0020] According to the visual inertial positioning motion consistency detection method, device, system and medium of the embodiment of the present invention, the feature map of the front and rear frame images and the global feature map of the front frame image are obtained respectively; the similarity features are determined according to the feature maps of the front and rear frame images; the similarity features are processed based on the self-attention model, and the optical flow between the front and rear frame images is output; the dynamic feature points are determined according to the optical flow between the front and rear frame images; the static feature points are determined according to the global feature map and dynamic feature points of the front frame image to complete the motion consistency detection. The detection method of the present invention performs deep modeling on the image collected by the camera, and uses the rich object knowledge provided by the large-scale pre-trained image segmentation model to supplement the optical flow prediction model based on self-attention to achieve motion consistency detection. In this way, the matching relationship between the front and rear frames of the image is considered to reduce the influence of local occlusion, and the calculation amount of the motion consistency detection model is reduced, the operation efficiency is improved, and the detection accuracy is maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a method for detecting motion consistency of visual inertial positioning according to some embodiments of the present invention;

[0022] Figure 2 A block diagram of a visual inertial positioning motion consistency detection device according to some embodiments of the present invention;

[0023] Figure 3 FIG. 4 is a block diagram of a visual inertial positioning system according to some embodiments of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0025] The visual inertial positioning motion consistency detection method, device, system and medium according to embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0026] Figure 1 FIG. 1 is a flow chart of a method for detecting consistency of visual inertial positioning motion according to some embodiments of the present invention. Figure 1 The visual inertial positioning motion consistency detection method of the embodiment of the present application may include the following steps:

[0027] S110, respectively obtaining feature maps of the previous and next frame images and a global feature map of the previous frame image.

[0028] Specifically, two frames of images are acquired continuously, which are recorded as the front frame image and the back frame image respectively. The front frame image is input into the encoder of the SAM (Spatially Aware Matching) module to extract the global feature map, which contains the boundary features. The front and back frame images (front frame image and back frame image) are input into the feature extraction framework based on the Ghost network (a lightweight neural network architecture) for feature extraction. The output of the Ghost network is further connected to a convolution block with shared parameters of a multi-scale feature map. The convolution block includes four 3x3 convolution layers, and the feature map I of the output image is output. i ∈R H ×W , where H and W are the height and width of the feature map respectively. Ghost network obtains multiple ghost feature maps through several linear operations. The number of parameters of linear operations is much smaller than that of convolution operations, which can improve the efficiency of feature extraction.

[0029] S120, determining similarity features according to feature maps of previous and next frame images.

[0030] Specifically, the similarity measure between the feature vectors of all pixels in the previous and next frames is calculated based on the feature map of the previous frame image and the feature map of the next frame image, the similarity measure results of each pair of frames are organized into a similarity matrix, and then the similarity matrix is ​​converted into similarity features.

[0031] S130, processing the similarity features based on the self-attention model, and outputting the optical flow between the previous and next frame images.

[0032] Specifically, the self-attention model includes an encoder and a decoder. The similarity feature is input into the encoder for encoding to obtain the target encoding feature of the similarity feature, and then the target encoding feature of the similarity feature is input into the decoder for decoding to output the optical flow between the previous and next frame images.

[0033] S140, determining dynamic feature points according to the optical flow between the previous and next frame images.

[0034] Specifically, the optical flows between the previous and next frame images are sorted to obtain an optical flow sequence, and the feature points corresponding to the larger optical flow values ​​in the optical flow sequence are the dynamic feature points.

[0035] S150, determining static feature points according to the global feature map and dynamic feature points of the previous frame image to complete motion consistency detection.

[0036] Specifically, semantic segmentation is performed on the global feature map of the previous frame image to obtain a semantic segmentation map, and the position of the dynamic feature points in the semantic segmentation map is obtained. According to the distribution of the dynamic feature points in the semantic segmentation map, the segmentation area containing these feature points is identified, and the segmentation area is taken as a moving object. Then, all dynamic feature points in the moving object in the semantic segmentation map are deleted, and the remaining feature points in the semantic segmentation map are static feature points to complete motion consistency detection.

[0037] The detection method of the present invention performs deep modeling on the images collected by the camera, and uses the rich object knowledge provided by the large-scale pre-trained image segmentation model to supplement the self-attention-based optical flow prediction model to achieve motion consistency detection. In this way, the matching relationship between the previous and next frames of the image is considered to reduce the impact of local occlusion, and the calculation amount of the motion consistency detection model is reduced, the operation efficiency is improved, and the detection accuracy is maintained.

[0038] In some embodiments, the self-attention model includes an encoder and a decoder, which processes similarity features based on the self-attention model and outputs the optical flow between the previous and next frame images, including: encoding the similarity features based on the encoder to obtain target encoding features of the similarity features; decoding the target encoding features based on the decoder to output the optical flow between the previous and next frame images.

[0039] Specifically, the encoder includes multiple encoding layers, and the decoder also includes multiple decoding layers, and the number of layers of the encoder and the decoder is the same. The similarity feature is input into the encoder, and each encoding layer encodes the similarity feature to obtain the encoding feature of the current encoding layer, and then the encoding feature of the next layer is determined based on the encoding feature of the current layer until the encoder feature of the last layer is obtained, and the encoder feature of the last layer is the target encoding feature. The target encoding feature is input into the decoder, and the target encoding feature is decoded to obtain the decoding feature of the last cycle of the decoder, which is the optical flow between the previous and next frame images.

[0040] In some embodiments, the encoder includes multiple encoding layers and a feedforward neural network, and encodes similarity features based on the encoder to obtain target encoding features of the similarity features, including: grouping the similarity features according to pixels to obtain multiple groups of features; performing self-attention calculations and feedforward neural network calculations on each group of features to obtain encoding features of each group of features; and splicing the encoding features of multiple groups of features to obtain target encoding features.

[0041] Specifically, the encoder includes multiple encoding layers, such as the first encoding layer, the second encoding layer, ..., the kth encoding layer (the last layer). Before the similarity features are input into the encoder, they need to be grouped according to pixels. t ∈R H×W×M×D The pixels can be divided into H×W groups, and the characteristics of each group are S x ∈R M×D , where M represents the number of similarity words corresponding to each pixel, and D represents the dimension of each word. Each set of features is input into the current coding layer for self-attention calculation and feedforward neural network calculation to obtain the coding features of each set of features. Based on the coding features of the current layer, each set of features is input into the next coding layer for self-attention calculation and feedforward neural network calculation to obtain the coding features of each set of features. This cycle is repeated to obtain each set of coding features of the kth coding layer (the last layer). The coding features of each set of features output by the kth coding layer (the last layer) are spliced ​​to obtain the target coding features.

[0042] Exemplarily, the encoding features of each group of features output by the kth encoding layer (the last layer) are concatenated by the following formula:

[0043] S k =Concat[s k (1),…,s k (H×W)]

[0044] Among them, S k represents the target encoding feature, Concat[·] represents the concatenation function, s k(1) represents the encoding features of the first set of features output by the kth encoding layer, s k (H×W) represents the encoded features of the H×Wth group of features output by the kth encoding layer.

[0045] In some embodiments, self-attention calculation and feedforward neural network calculation are performed on each group of features to obtain the coding features of each group of features, including: obtaining the query vector, key vector and value vector of each group of features in the current coding layer; calculating the attention features of each group of features in the current coding layer according to the query vector, key vector, value vector and attention features of the previous coding layer of the current coding layer; performing feedforward neural network calculation on the attention features of the current coding layer, and taking the sum of the result and the attention features of the current coding layer as the coding features of the current layer, and so on, to obtain the coding features of the last layer of the encoder, and taking the coding features of the last layer as the coding features of each group of features.

[0046] Exemplarily, first initialize the feature group sequence s0 with M elements, and the expression of s0 is as follows:

[0047] s0=[S x1 , …, S xM ]

[0048] Among them, s0 represents the feature group sequence, S x1 represents the first feature group, S xM represents the Mth feature group.

[0049] Then, based on the attention features output by the previous encoding layer, the query vector, key vector, and value vector of each set of features in the current encoding layer are obtained through the following formula:

[0050]

[0051] Among them, Q i represents the query vector of each set of features at the current encoding layer, K i Represents the key vector of each set of features in the current encoding layer, V i Represents the value vector of each set of features in the current encoding layer, W Qi Represents the current encoding layer query vector Q i The learnable parameter matrix, W Ki Represents the current encoding layer key vector K i The parameter matrix that can be learned, W Vi Represents the current coding layer value vector V i The learnable parameter matrix, s i-1 Represents the attention features of each group of features output by the previous encoding layer.

[0052] Then, according to the query vector, key vector, value vector of the current coding layer and the attention features of the previous coding layer, the attention features of each set of features in the current coding layer are calculated by the following formula:

[0053]

[0054] in, Indicates the attention feature of each set of features in the current encoding layer, Attwention (Q i , K i , V i ) indicates that according to Q i K i , and V i The attention features calculated, softmax(·) represents the normalized exponential function, Q i represents the query vector of each set of features at the current encoding layer, K i Represents the key vector of each set of features in the current encoding layer, V i Represents the value vector of each set of features in the current encoding layer, d k represents the dimension of the attention weight vector in the attention head, s i-1 Represents the attention features of each group of features output by the previous encoding layer. In this way, the attention features of each group of features output by the previous encoding layer are mapped again through the feedforward layer, and the mapped features are added to the attention features of each group of features in the current encoding layer, thereby reducing feature attenuation.

[0055] Finally, the attention feature of the current coding layer is calculated by the feedforward neural network, and the sum of the result and the attention feature of the current coding layer is used as the coding feature of the current layer. The calculation formula of the coding feature of the current layer is as follows:

[0056]

[0057] Among them, s i represents the encoding features of each set of features in the current layer, FFN(·) represents the feedforward neural network calculation function, Represents the attention features of each set of features in the current encoding layer.

[0058] By analogy, the encoding features of the last layer are the encoding features of each group of features.

[0059] In some embodiments, the decoder includes multiple decoding layers, the encoder and the decoder have the same number of layers, and the target coding features are decoded based on the decoder to output the optical flow between the previous and next frame images, including: calculating the pixel change based on the optical flow between the previous frame image and the previous and next frame images decoded in the previous cycle; center cropping the changed image to obtain the cropped image features; performing feedforward neural network calculation on the cropped image features to obtain the query vector of the decoder's current cycle; determining the key vector and value vector of the decoder's current cycle based on the coding features; determining the attention features of the decoding in this cycle based on the key vector, value vector and query vector of the decoder's current cycle, and so on, to obtain the decoding features of the decoder's last cycle, recorded as the optical flow between the previous and next frame images.

[0060] Specifically, the target encoding feature obtained by the encoder is S k ∈R H×W×K×D , the target encoding feature S k The input is decoded in a multi-layer cyclic decoder to obtain the decoding features of the last cycle of the decoder, which is the optical flow between the previous and next frame images.

[0061] Exemplarily, the initialization optical flow f0(I)=0 is first defined, and then the pixel change is calculated according to the optical flow between the previous frame image and the previous and next frame images decoded in the previous cycle. The calculation formula is as follows:

[0062] g i =I+f i-1 (I)

[0063] Among them, g i represents the pixel change, I represents the previous frame image, and f i-1 (I) represents the optical flow between the previous and next frame images decoded in the previous cycle.

[0064] Then, the changed image is center cropped to obtain the cropped image features as follows:

[0065] q x = CenterCrop(g i )

[0066] Among them, g i represents the pixel change, CenterCrop(·) represents the change from g i Cut out a 16×16 square in the middle, q x Represents the cropped image features.

[0067] Then, the cropped image features are calculated using a feedforward neural network to obtain the query vector for this decoder cycle as follows:

[0068]

[0069] in, represents the query vector of the decoder in this cycle, q x represents the cropped image features, g i represents the image after pixel changes, FFN(·) represents the feedforward neural network calculation function, and PE(·) represents the position encoding.

[0070] Then, according to the target encoding feature S k The key vector and value vector of the decoder in this cycle are determined as follows:

[0071]

[0072] Among them, S k represents the target encoding feature, represents the key vector of the decoder in this cycle, represents the value vector of the decoder in this cycle, and FFN(·) represents the feedforward neural network calculation function.

[0073] Then, the attention feature of this cycle decoding is determined according to the key vector, value vector and query vector of the decoder in this cycle as follows:

[0074]

[0075] Among them, c i represents the attention feature of this cycle decoding, According to quantity and Computed attention features, represents the query vector of the decoder in this cycle, represents the key vector of the decoder in this cycle, A vector of values ​​representing the decoder's current iteration.

[0076] Finally, based on the decoding features of this cycle and the optical flow between the previous and next frame images calculated in the previous cycle, the optical flow between the previous and next frame images decoded in this cycle is calculated by the following formula:

[0077] f i (I) = Attention(Concat[c i ,q x ],S x , f i-1 (I))+f i-1 (I)

[0078] Among them, f i(I) represents the optical flow between the previous and next frame images of this cycle decoding, Attention(·) represents the attention feature calculation function, Concat[·] represents the concatenation function, c i represents the attention feature of this cycle decoding, q x represents the cropped image features, S x represents the characteristics of each group, S x ∈R M×D , f i-1 (I) represents the optical flow between the previous and next frame images decoded in the previous cycle.

[0079] By analogy, the decoded features of the last cycle of the decoder are obtained, which are recorded as the optical flow between the previous and next frame images.

[0080] In some embodiments, similarity features are determined based on feature maps of previous and next frame images, including: determining a similarity matrix of the previous and next frame images based on all pixels in the feature maps of the previous and next frame images; and converting the similarity matrix into similarity features based on a linear layer.

[0081] Specifically, the feature map of the previous and next frames contains the feature descriptors of each pixel in the image, and each pixel on the feature map is represented as a feature vector. For each pixel in the feature map, the similarity measure between all pixel feature vectors in the previous and next frames is calculated, such as the Euclidean distance and cosine similarity between all pixel feature vectors, and the similarity measure results of each pair of frames are organized into a similarity matrix S∈R H×W×H×W , and then use the linear layer to convert the similarity matrix into the similarity feature S t ∈R H×W×M×D .

[0082] In this way, converting the similarity matrix into similarity features can remove redundant features and improve computational efficiency.

[0083] In some embodiments, static feature points are determined based on a global feature map and dynamic feature points of a previous frame image, including: extracting a global feature map of the previous frame image based on SAM; performing semantic segmentation on the global feature map to obtain a semantic segmentation map; determining a segmentation area of ​​the dynamic feature points in the semantic segmentation map based on the dynamic feature points and the semantic segmentation map, and treating the segmented area as a moving object; deleting all dynamic feature points in the identified moving object, and treating the remaining feature points in the semantic segmentation map as static feature points.

[0084] Exemplarily, after identifying the dynamic feature points based on the optical flow between the previous and next frame images, the previous frame image is input into the encoder of the SAM module to extract the global feature map, and then the global feature map is semantically segmented to obtain a semantic segmentation map. The positions of the identified dynamic feature points are matched with the semantic segmentation map to determine the positions of the dynamic feature points in the semantic segmentation map. Based on the distribution of the dynamic feature points in the semantic segmentation map, the segmented regions containing these feature points are identified, and these regions are considered to be moving objects. The segmented regions of all identified moving objects are merged into a set, represented as in Indicates the number of moving objects. The dynamic feature points belonging to the moving object area are deleted from the segmentation map, and only the static feature points are retained to achieve motion consistency detection. Then the optical flow between the global feature map and the previous and next frame images can be used for visual inertial positioning through post-processing operations to obtain the trajectory state. In this way, the error caused by moving objects can be reduced in the subsequent visual inertial positioning.

[0085] It should be noted that the SAM module itself can realize semantic segmentation in the case of occlusion. In addition, by matching dynamic feature points and segmentation maps, the accuracy of locating occluded objects can be improved to a certain extent, thereby reducing the impact of local occlusion.

[0086] In some embodiments, dynamic feature points are determined based on the optical flow between previous and next frame images, including: arranging the optical flows of the previous and next frame images in order from large to small to obtain an optical flow sequence; and taking the feature points corresponding to the optical flow before a preset threshold in the optical flow sequence as dynamic feature points.

[0087] Specifically, an abnormal optical flow (relatively large optical flow) can be obtained based on the optical flow between the previous and next frame images and a preset threshold, and the feature points corresponding to the abnormal optical flow are dynamic feature points.

[0088] Exemplarily, the calculated optical flows between the previous and next frame images are arranged in descending order to obtain an optical flow sequence. The optical flow with a preset threshold before the optical flow sequence is taken, for example, the optical flow before 1% of the optical flow sequence is taken as the abnormal optical flow, and the feature points corresponding to these abnormal optical flows are dynamic feature points.

[0089] In some embodiments, after completing the consistency detection, the transformation matrix can be calculated by matching static feature points, and then the optical flow between the previous and next frame images and the global feature map of the previous frame image are combined to perform visual inertial positioning through post-processing operations to obtain the trajectory state.

[0090] Exemplarily, the point sets of the previous and next frames are extracted from the matched pairs of static feature points, and the transformation matrix is ​​calculated based on the point sets using the eight-point method or the DLT (Direct Linear Transformation) algorithm. The motion information calculated based on the optical flow between the previous and next frame images is combined with the transformation matrix matched with the static feature points to supplement the displacement estimation of the static feature point matching. The global feature map provides geometric information of the entire scene, and the global feature map is matched with the static feature points to correct and optimize the pose estimation. Through post-processing operations, such as nonlinear optimization algorithms, the above information is integrated to obtain a more accurate pose estimate. Finally, combined with the optimized pose estimate, the motion trajectory of the object is calculated and tracked to obtain the trajectory state.

[0091] In summary, the detection method of the present invention performs deep modeling on the images collected by the camera, and uses the rich object knowledge provided by the large-scale pre-trained image segmentation model to supplement the self-attention-based optical flow prediction model to achieve motion consistency detection. In this way, the matching relationship between the previous and next frames of the image is considered to reduce the impact of local occlusion, and the calculation amount of the motion consistency detection model is reduced, the operation efficiency is improved, and the detection accuracy is maintained.

[0092] Corresponding to the above-mentioned embodiment, the present application also proposes a visual inertial positioning motion consistency detection device.

[0093] Reference Figure 2 The visual inertial positioning motion consistency detection device 200 includes: an acquisition module 210, a first determination module 220, an output module 230, a second determination module 240 and a third determination module 250.

[0094] Among them, the acquisition module 210 is used to respectively acquire the feature map of the front and rear frame images and the global feature map of the front frame image. The first determination module 220 is used to determine the similarity features based on the feature maps of the front and rear frame images. The output module 230 is used to process the similarity features based on the self-attention model and output the optical flow between the front and rear frame images. The second determination module 240 is used to determine the dynamic feature points based on the optical flow between the front and rear frame images. The third determination module 250 is used to determine the static feature points based on the global feature map and dynamic feature points of the front frame image to complete the motion consistency detection.

[0095] According to one embodiment of the present invention, the self-attention model includes an encoder and a decoder, and the output module 230 is specifically used to encode the similarity feature based on the encoder to obtain the target encoding feature of the similarity feature; and decode the target encoding feature based on the decoder to output the optical flow between the previous and next frame images.

[0096] According to one embodiment of the present invention, the encoder includes multiple encoding layers and a feedforward neural network, and the output module 230 is also used to encode the similarity features based on the encoder to obtain target encoding features of the similarity features, specifically for grouping the similarity features according to pixels to obtain multiple groups of features; performing self-attention calculations and feedforward neural network calculations on each group of features to obtain encoding features of each group of features; and splicing the encoding features of multiple groups of features to obtain target encoding features.

[0097] According to one embodiment of the present invention, the output module 230 is also used to perform self-attention calculation and feedforward neural network calculation on each group of features to obtain the coding features of each group of features, specifically for obtaining the query vector, key vector and value vector of each group of features in the current coding layer; calculating the attention features of each group of features in the current coding layer according to the query vector, key vector, value vector and attention features of the previous coding layer of the current coding layer; performing feedforward neural network calculation on the attention features of the current coding layer, and taking the sum of the result and the attention features of the current coding layer as the coding features of the current layer, and so on, to obtain the coding features of the last layer of the encoder, and taking the coding features of the last layer as the coding features of each group of features.

[0098] According to one embodiment of the present invention, the decoder includes multiple decoding layers, and the number of layers of the encoder and the decoder is the same. The output module 230 is also used to decode the target coding features based on the decoder to output the optical flow between the previous and next frame images, specifically for calculating the pixel change based on the previous frame image and the optical flow between the previous and next frame images decoded in the previous cycle; center cropping the changed image to obtain the cropped image features; performing feedforward neural network calculation on the cropped image features to obtain the query vector of the decoder's current cycle; determining the key vector and value vector of the decoder's current cycle based on the coding features; determining the attention features of the decoding in this cycle based on the key vector, value vector and query vector of the decoder's current cycle, and so on, to obtain the decoding features of the decoder's last cycle, recorded as the optical flow between the previous and next frame images.

[0099] According to an embodiment of the present invention, the first determination module 220 is specifically used to determine the similarity matrix of the previous and next frame images according to all pixels in the feature maps of the previous and next frame images; and convert the similarity matrix into a similarity feature based on a linear layer.

[0100] According to one embodiment of the present invention, the third determination module 250 is specifically used to extract a global feature map of the previous frame image based on SAM; perform semantic segmentation on the global feature map to obtain a semantic segmentation map; determine the segmentation area of ​​the dynamic feature points in the semantic segmentation map based on the dynamic feature points and the semantic segmentation map, and treat the segmented area as a moving object; delete all dynamic feature points in the identified moving object, and use the remaining feature points in the semantic segmentation map as static feature points.

[0101] According to an embodiment of the present invention, the second determination module 240 is specifically used to arrange the optical flows of the previous and next frame images in order from large to small to obtain an optical flow sequence; and use the feature points corresponding to the optical flows before the preset threshold in the optical flow sequence as dynamic feature points.

[0102] It should be pointed out that the above explanation of the embodiments and beneficial effects of the visual inertial positioning motion consistency detection method is also applicable to the visual inertial positioning motion consistency detection device of the embodiment of the present invention, and will not be elaborated here in order to avoid redundancy.

[0103] Corresponding to the above embodiments, the present application also proposes a computer-readable storage medium.

[0104] The computer-readable storage medium of the present application stores a visual-inertial positioning motion consistency detection program thereon, and when the visual-inertial positioning motion consistency detection program is executed by a processor, the aforementioned visual-inertial positioning motion consistency detection method is implemented.

[0105] It should be pointed out that the above explanation of the embodiments and beneficial effects of the visual inertial positioning motion consistency detection method is also applicable to the computer-readable storage medium of the embodiment of the present invention, and will not be elaborated here in order to avoid redundancy.

[0106] Corresponding to the above embodiments, the present application also proposes a visual inertial positioning system.

[0107] See also Figure 3 As shown, the visual inertial positioning system 300 of the present application includes a memory 310, a processor 320, and a visual inertial positioning motion consistency detection program stored in the memory 310 and executable on the processor 320. When the processor 320 executes the visual inertial positioning motion consistency detection program, the aforementioned visual inertial positioning motion consistency detection method is implemented.

[0108] It should be pointed out that the above explanation of the embodiment and beneficial effects of the visual inertial positioning motion consistency detection method is also applicable to the visual inertial positioning system of the embodiment of the present invention, and will not be elaborated here in order to avoid redundancy.

[0109] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0110] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0112] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0113] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0114] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A visual inertial positioning motion consistency detection method, characterized in that: The method comprises: Obtain feature maps of the previous and next frame images and a global feature map of the previous frame image respectively; Determining similarity features according to the feature graphs of the preceding and following frame images; Processing the similarity features based on a self-attention model, and outputting the optical flow between the previous and next frame images; Determine dynamic feature points according to the optical flow between the previous and next frame images; Static feature points are determined according to the global feature map of the previous frame image and the dynamic feature points to complete motion consistency detection.

2. The visual inertial positioning motion consistency detection method according to claim 1, characterized in that: The self-attention model includes an encoder and a decoder, and processes the similarity feature based on the self-attention model to output the optical flow between the previous and next frame images, including: Encoding the similarity feature based on the encoder to obtain a target encoding feature of the similarity feature; The target coding feature is decoded based on the decoder to output the optical flow between the previous and next frame images.

3. The visual inertial positioning motion consistency detection method according to claim 2, characterized in that: The encoder includes a plurality of encoding layers and a feedforward neural network, and encoding the similarity feature based on the encoder to obtain a target encoding feature of the similarity feature includes: Grouping the similarity features according to pixels to obtain multiple groups of features; Perform self-attention calculation and feedforward neural network calculation on each set of features to obtain the encoding features of each set of features; The target coding feature is obtained by splicing the coding features of multiple groups of features.

4. The visual inertial positioning motion consistency detection method according to claim 3, characterized in that: Perform self-attention calculation and feedforward neural network calculation on each set of features to obtain the encoding features of each set of features, including: Get the query vector, key vector, and value vector of each set of features at the current encoding layer; Calculate the attention features of each set of features at the current coding layer according to the query vector, key vector, value vector and attention features of the previous coding layer of the current coding layer; Perform a feedforward neural network calculation on the attention feature of the current coding layer, and use the sum of the result and the attention feature of the current coding layer as the coding feature of the current layer. Similarly, obtain the coding feature of the last layer of the encoder, and use the coding feature of the last layer as the coding feature of each group of features.

5. The visual inertial positioning motion consistency detection method according to claim 2, characterized in that: The decoder includes a plurality of decoding layers, the encoder and the decoder have the same number of layers, and the target coding feature is decoded based on the decoder to output the optical flow between the previous and next frame images, including: Calculate pixel point changes according to the optical flow between the previous frame image and the previous and next frame images decoded in the previous cycle; Perform center cropping on the changed image to obtain cropped image features; Perform feed-forward neural network calculation on the cropped image features to obtain a query vector for the decoder in this cycle; Determine a key vector and a value vector of the decoder for this cycle according to the encoding feature; The attention features of the decoding in this cycle are determined according to the key vector, value vector and query vector of the decoder in this cycle, and so on, the decoding features of the last cycle of the decoder are obtained, which are recorded as the optical flow between the previous and next frame images.

6. The visual inertial positioning motion consistency detection method according to claim 1, characterized in that: The determining of similarity features according to the feature graphs of the preceding and following frame images comprises: Determine a similarity matrix of the previous and next frame images according to all pixels in the feature maps of the previous and next frame images; The similarity matrix is ​​converted into the similarity feature based on a linear layer.

7. The visual inertial positioning motion consistency detection method according to claim 1, characterized in that: Determining static feature points according to the global feature map of the previous frame image and the dynamic feature points includes: Extracting a global feature map of the previous frame image based on SAM; Performing semantic segmentation on the global feature map to obtain a semantic segmentation map; Determine a segmentation area of ​​the dynamic feature point in the semantic segmentation map according to the dynamic feature point and the semantic segmentation map, and use the segmentation area as a moving object; All dynamic feature points in the identified moving object are deleted, and the remaining feature points in the semantic segmentation map are used as the static feature points.

8. The visual inertial positioning motion consistency detection method according to claim 1, characterized in that: The determining of the dynamic feature points according to the optical flow between the preceding and succeeding frame images comprises: Arrange the optical flows of the preceding and following frame images in order from large to small to obtain an optical flow sequence; The feature point corresponding to the optical flow before the preset threshold in the optical flow sequence is used as the dynamic feature point.

9. A visual inertial positioning motion consistency detection device, characterized in that: The device comprises: An acquisition module, used to respectively acquire feature maps of the previous and next frame images and a global feature map of the previous frame image; A first determination module, used for determining similarity features according to feature graphs of the preceding and following frame images; An output module, used for processing the similarity feature based on a self-attention model, and outputting the optical flow between the previous and next frame images; A second determination module is used to determine the dynamic feature points according to the optical flow between the previous and next frame images; The third determination module is used to determine the static feature points according to the global feature map of the previous frame image and the dynamic feature points to complete motion consistency detection.

10. A computer-readable storage medium, characterized in that: A visual inertial positioning motion consistency detection program is stored thereon, and when the visual inertial positioning motion consistency detection program is executed by a processor, the visual inertial positioning motion consistency detection method according to any one of claims 1-8 is implemented.

11. A visual inertial positioning system, characterized in that: The invention comprises a memory, a processor and a visual inertial positioning motion consistency detection program stored in the memory and executable on the processor. When the processor executes the visual inertial positioning motion consistency detection program, the visual inertial positioning motion consistency detection method according to any one of claims 1 to 8 is implemented.

Images