Visual simultaneous localization and mapping key frame selection method

Abstract

The application is suitable for the field of computer application technology, and provides a key frame selection method for visual simultaneous localization and mapping, which comprises the following steps: determining a current to-be-processed image in any frame image in to-be-processed video data and determining corresponding multiple adjacent images of the current to-be-processed image, then performing feature extraction on the current to-be-processed image and each adjacent image respectively to obtain corresponding first and second global features, then determining a neighborhood image set of the current to-be-processed image according to the similarity between the first and second global features, then determining the information gain of the current to-be-processed image to the neighborhood image, and then determining the current to-be-processed image as a key frame image when the information gain is greater than an information gain threshold. Thus, the correlation of the key frame in the image content is improved by introducing rich texture information in the key frame selection, and the accuracy of VSLAM is improved.

Description

Technical Field

[0001] This application belongs to the field of computer application technology, and in particular relates to a keyframe selection method and apparatus for Visual Simultaneous Localization and Mapping (VSLAM). Background Technology

[0002] Mobile intelligent robot technology has experienced unprecedented development worldwide and is increasingly being applied to people's daily scientific research and life. In unknown real-world scenarios, mobile robots need to perceive and model their surrounding environment during movement, while also determining their own spatial position to solve the real-time localization problem. This process is known as Simultaneous Localization and Mapping (SLAM), with Visual Simultaneous Localization and Mapping (VSLAM) experiencing rapid development in recent years. VSLAM works by acquiring video data through image acquisition devices and sampling the video data to obtain keyframes, forming short-term data associations for visual tracking. It can also achieve real-time scene recognition through long-term data association.

[0003] In related technologies, when extracting keyframes for VSLAM, the richness of texture in image frames is usually not considered. This results in low correlation between the selected keyframes in terms of image content, especially between keyframes acquired at long intervals. Consequently, the VSLAM keyframe extraction algorithm cannot achieve long-term data correlation, leading to poor scene recognition capability and decreased VSLAM accuracy. Summary of the Invention

[0004] This application provides a keyframe selection method and apparatus for VSLAM, which can solve the problem that in visual simultaneous localization and map building, the texture richness of image frames is usually not considered, resulting in low correlation between selected keyframes in image content. In particular, the content correlation between keyframes acquired at long intervals is poor, which makes it impossible for the VSLAM keyframe extraction algorithm to achieve long-term data correlation, thus leading to poor scene recognition ability and decreased VSLAM accuracy.

[0005] In a first aspect, embodiments of this application provide a keyframe selection method for VSLAM, comprising: acquiring video data to be processed, wherein the video data to be processed includes multiple frames of images to be processed; determining any frame of the image to be processed in the video data to be processed as the current image to be processed, and determining multiple adjacent frames of images corresponding to the current image to be processed from the video data to be processed; performing feature extraction on the current image to be processed and each adjacent frame to determine a first global feature corresponding to the current image to be processed and a second global feature corresponding to each adjacent frame, wherein the first global feature and the second global feature include texture features of the image; selecting a set of neighboring images corresponding to the current image to be processed from the adjacent images according to the similarity between the first global feature and each second global feature; determining the information gain of the current image to be processed to the set of neighboring images; and determining the current image to be processed as a keyframe image when the information gain is greater than an information gain threshold.

[0006] In one possible implementation of the first aspect, determining the information gain of the current image to be processed on the set of neighboring images includes:

[0007] Determine the first Shannon entropy corresponding to the neighborhood image set, and the second Shannon entropy corresponding to the current image to be processed and the neighborhood image set;

[0008] Based on the first Shannon entropy and the second Shannon entropy, determine the information gain of the current image to be processed on the set of neighboring images.

[0009] Optionally, in another possible implementation of the first aspect, determining the information gain of the current image to be processed to the set of neighboring images based on the first Shannon entropy and the second Shannon entropy includes:

[0010] The difference between the second Shannon entropy and the first Shannon entropy is defined as the information gain.

[0011] Optionally, in another possible implementation of the first aspect, the above-described feature extraction of the current image to be processed and each adjacent frame to determine multiple first global features corresponding to the current image to be processed and multiple second global features corresponding to each adjacent frame includes:

[0012] Feature points are extracted from the current image to be processed and each adjacent image in each frame to determine multiple first feature point descriptors corresponding to the current image to be processed and multiple second feature point descriptors corresponding to each adjacent image in each frame.

[0013] Based on the preset visual vocabulary bag, each first feature point descriptor corresponding to the current image to be processed is classified to determine the first global descriptor corresponding to the current image to be processed.

[0014] Based on the preset visual vocabulary bag, each second feature point descriptor corresponding to each adjacent image in each frame is classified to determine the second global descriptor corresponding to each adjacent image in each frame.

[0015] Optionally, in another possible implementation of the first aspect, the aforementioned preset visual vocabulary bag includes M entries and M third feature point descriptors corresponding to the M entries, where M is a positive integer. Based on the preset visual vocabulary bag, each first feature point descriptor corresponding to the current image to be processed is categorized to determine the first global descriptor corresponding to the current image to be processed, including:

[0016] Based on the similarity between each first feature point descriptor and the third feature point descriptor corresponding to the i-th term, determine the number of first feature point descriptors that match the i-th term, where i is a positive integer greater than or equal to 1 and less than or equal to M;

[0017] The number of first feature point descriptors that match the i-th term is determined as the number of first feature points of the i-th term in the current image to be processed;

[0018] The first global descriptor is determined based on the number of first feature points categorized for each term in the current image to be processed.

[0019] Optionally, in another possible implementation of the first aspect, the aforementioned neighborhood image set includes N frames of neighborhood images, where N is a positive integer. Determining the first Shannon entropy corresponding to the neighborhood image set and the second Shannon entropy corresponding to the current image to be processed and the neighborhood image set includes:

[0020] Based on the similarity between each second feature point descriptor corresponding to the j-th frame neighborhood image and the third feature point descriptor corresponding to the i-th word, determine the number of second feature point descriptors in the j-th frame neighborhood image that match the i-th word, where j is a positive integer greater than or equal to 1 and less than or equal to N;

[0021] The number of second feature point descriptors that match the i-th word in the j-th frame neighborhood image is determined as the number of second feature points of the i-th word in the j-th frame neighborhood image;

[0022] The first occupancy probability corresponding to the i-th term is determined based on the number of second feature points classified for the i-th term in each frame of the neighborhood image.

[0023] Based on the first occupancy probability corresponding to the i-th term, determine the third Shannon entropy corresponding to the i-th term;

[0024] The sum of the third Shannon entropy corresponding to each term is determined as the first Shannon entropy;

[0025] The second occupancy probability corresponding to the i-th term is determined based on the number of first feature points classified for the i-th term in the current image to be processed and the number of second feature points classified for the i-th term in each neighboring image.

[0026] Based on the second occupancy probability corresponding to the i-th term, determine the fourth Shannon entropy corresponding to the i-th term;

[0027] The sum of the fourth Shannon entropy corresponding to each term is determined as the second Shannon entropy.

[0028] Optionally, in another possible implementation of the first aspect, before determining the first occupancy probability corresponding to the i-th term based on the number of second feature points classified for the i-th term in each frame of the neighborhood image, the method further includes:

[0029] The number of categories for each second feature point is normalized.

[0030] Optionally, in another possible implementation of the first aspect, the selection of a set of neighboring images corresponding to the current image to be processed from neighboring images based on the similarity between the first global feature and each of the second global features includes:

[0031] Based on the distance between the first global descriptor and each of the second global descriptors, a set of neighboring images corresponding to the current image to be processed is selected from the adjacent images.

[0032] Optionally, in another possible implementation of the first aspect, the selection of a set of neighboring images corresponding to the current image to be processed from neighboring images based on the distance between the first global descriptor and each of the second global descriptors includes:

[0033] The neighboring images corresponding to the second global descriptor whose distance from the first global descriptor is less than the distance threshold are determined as the neighborhood images corresponding to the current image to be processed.

[0034] A set of neighborhood images is formed by using the neighborhood images corresponding to the current image to be processed.

[0035] Secondly, embodiments of this application provide a VSLAM keyframe selection apparatus, comprising:

[0036] The first acquisition module is used to acquire video data to be processed, wherein the video data to be processed includes multiple frames of images to be processed; the first determination module is used to determine any frame of the image to be processed in the video data to be processed as the current image to be processed, and to determine multiple adjacent frames of images corresponding to the current image to be processed from the video data to be processed; the second determination module is used to extract features from the current image to be processed and each adjacent frame of images to determine the first global feature corresponding to the current image to be processed and the second global feature corresponding to each adjacent frame of images, wherein the first global feature and the second global feature include the texture features of the image; the first selection module is used to select a set of neighboring images corresponding to the current image to be processed from the neighboring images according to the similarity between the first global feature and each second global feature; the third determination module is used to determine the information gain of the current image to be processed to the set of neighboring images; the fourth determination module is used to determine the current image to be processed as a keyframe image when the information gain is greater than the information gain threshold.

[0037] Optionally, in one possible implementation of the second aspect, the aforementioned third determining module includes:

[0038] The first determining unit is used to determine the first Shannon entropy corresponding to the neighborhood image set, and the second Shannon entropy corresponding to the current image to be processed and the neighborhood image set;

[0039] The second determining unit is used to determine the information gain of the current image to be processed to the set of neighboring images based on the first Shannon entropy and the second Shannon entropy.

[0040] Optionally, in another possible implementation of the second aspect, the aforementioned first determining unit is specifically used for:

[0041] The difference between the second Shannon entropy and the first Shannon entropy is defined as the information gain.

[0042] Optionally, in another possible implementation of the second aspect, the second determining module mentioned above includes:

[0043] The third determining unit is used to extract feature points from the current image to be processed and each adjacent image in each frame, so as to determine multiple first feature point descriptors corresponding to the current image to be processed and multiple second feature point descriptors corresponding to each adjacent image in each frame.

[0044] The fourth determining unit is used to classify each first feature point descriptor corresponding to the current image to be processed according to the preset visual vocabulary bag, so as to determine the first global descriptor corresponding to the current image to be processed.

[0045] The fifth determining unit is used to classify each second feature point descriptor corresponding to each adjacent image in each frame according to a preset visual vocabulary bag, so as to determine the second global descriptor corresponding to each adjacent image in each frame.

[0046] Optionally, in another possible implementation of the second aspect, the aforementioned preset visual vocabulary bag includes M entries and M third feature point descriptors corresponding to the M entries, where M is a positive integer; correspondingly, the aforementioned fourth determining unit is specifically used for:

[0047] Based on the similarity between each first feature point descriptor and the third feature point descriptor corresponding to the i-th term, determine the number of first feature point descriptors that match the i-th term, where i is a positive integer greater than or equal to 1 and less than or equal to M;

[0048] The number of first feature point descriptors that match the i-th term is determined as the number of first feature points of the i-th term in the current image to be processed;

[0049] The first global descriptor is determined based on the number of first feature points categorized for each term in the current image to be processed.

[0050] Optionally, in another possible implementation of the second aspect, the aforementioned neighborhood image set includes N frames of neighborhood images, where N is a positive integer; correspondingly, the aforementioned first determining unit is specifically used for:

[0051] Based on the similarity between each second feature point descriptor corresponding to the j-th frame neighborhood image and the third feature point descriptor corresponding to the i-th word, determine the number of second feature point descriptors in the j-th frame neighborhood image that match the i-th word, where j is a positive integer greater than or equal to 1 and less than or equal to N;

[0052] The number of second feature point descriptors that match the i-th word in the j-th frame neighborhood image is determined as the number of second feature points of the i-th word in the j-th frame neighborhood image;

[0053] The first occupancy probability corresponding to the i-th term is determined based on the number of second feature points classified for the i-th term in each frame of the neighborhood image.

[0054] Based on the first occupancy probability corresponding to the i-th term, determine the third Shannon entropy corresponding to the i-th term;

[0055] The sum of the third Shannon entropy corresponding to each term is determined as the first Shannon entropy;

[0056] The second occupancy probability corresponding to the i-th term is determined based on the number of first feature points classified for the i-th term in the current image to be processed and the number of second feature points classified for the i-th term in each neighboring image.

[0057] Based on the second occupancy probability corresponding to the i-th term, determine the fourth Shannon entropy corresponding to the i-th term;

[0058] The sum of the fourth Shannon entropies corresponding to each of the aforementioned terms is determined as the second Shannon entropy.

[0059] Optionally, in another possible implementation of the second aspect, the first determining unit described above is further configured to:

[0060] The number of categories for each second feature point is normalized.

[0061] Optionally, in another possible implementation of the second aspect, the first selection module mentioned above includes:

[0062] The sixth determining unit is used to select a set of neighboring images corresponding to the current image to be processed from the adjacent images based on the distance between the first global descriptor and each second global descriptor.

[0063] Optionally, in another possible implementation of the second aspect, the sixth determining unit mentioned above is specifically used for:

[0064] The neighboring images corresponding to the second global descriptor whose distance from the first global descriptor is less than the distance threshold are determined as the neighborhood images corresponding to the current image to be processed.

[0065] A set of neighborhood images is formed by using the neighborhood images corresponding to the current image to be processed.

[0066] Thirdly, embodiments of this application provide a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the keyframe selection method of VSLAM as described above.

[0067] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the VSLAM keyframe selection method as described above.

[0068] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the VSLAM keyframe selection method described in any of the first aspects above.

[0069] The beneficial effects of this application embodiment compared with the prior art are: by performing global feature extraction on the current image to be processed and the neighboring images during key frame extraction, rich texture features of the image are introduced during key frame extraction, thereby improving the correlation of the extracted key frames in the image content, and thus improving the scene recognition capability of VSLAM and the accuracy of VSLAM. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart illustrating a keyframe selection method for VSLAM provided in an embodiment of this application;

[0072] Figure 2 This is a flowchart illustrating a keyframe selection method for VSLAM provided in another embodiment of this application;

[0073] Figure 3 This is a schematic diagram of the keyframe selection device for VSLAM provided in the embodiments of this application;

[0074] Figure 4 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0075] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0076] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0077] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0078] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0079] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0080] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0081] The following description, with reference to the accompanying drawings, details a keyframe selection method, apparatus, terminal device, storage medium, and computer program for VSLAM provided in this application.

[0082] Figure 1 A flowchart illustrating a VSLAM keyframe selection method provided in an embodiment of this application is shown.

[0083] Step 101: Obtain the video data to be processed, which includes multiple frames of images to be processed.

[0084] It should be noted that the VSLAM keyframe selection method of this application embodiment can be executed by the VSLAM keyframe selection device of this application embodiment. The VSLAM keyframe selection device of this application embodiment can be configured in any terminal device to execute the VSLAM keyframe selection method of this application embodiment. For example, the device of this application embodiment can be configured in any terminal device that can implement VSLAM, such as an intelligent robot, etc., and this application embodiment does not limit this.

[0085] The video data to be processed may refer to the video data captured and saved by the terminal device equipped with the VSLAM keyframe selection method of the present application during the VSLAM process, using its onboard image acquisition device to capture images of the target scene in the outside world.

[0086] The image to be processed can refer to any frame of the video data to be processed. In actual use, the VSLAM keyframe selection method of this application embodiment can process each frame of the image to be processed in the video data to be processed in turn to determine whether each frame of the image to be processed in the video data to be processed can be extracted as a keyframe image.

[0087] For example, when the VSLAM keyframe selection method in this application embodiment is configured in a mobile robot with VSLAM functionality, the video data to be processed can be video data collected by the mobile robot during its operation.

[0088] As one possible implementation, during the VSLAM process, the terminal device can extract keyframes at a preset frequency, so that the video data to be processed can be video data collected by the terminal device between the last keyframe extraction time and the current keyframe extraction time.

[0089] For example, if the terminal device extracts a keyframe every 5 seconds, and the last keyframe extraction time was 10:00:00, and the current keyframe extraction time is 10:00:05, then the video data to be processed can be the video data collected by the terminal device from 10:00:00 to 10:00:05.

[0090] It should be noted that the above examples are merely illustrative and should not be considered as limitations on this application. In actual use, the timing and frequency of keyframe extraction can be determined according to actual needs and specific application scenarios, and this application embodiment does not limit this.

[0091] Step 102: Determine any frame of the image to be processed in the video data to be processed as the current image to be processed, and determine the multiple adjacent frames corresponding to the current image to be processed from the video data to be processed.

[0092] The image to be processed can refer to any frame in the video data to be processed. It should be noted that when extracting keyframes from the video data to be processed, it can be determined whether each frame in the video data can be used as a keyframe. Thus, each frame in the video can be sequentially identified as the current image to be processed, and subsequent steps in this embodiment can be used to sequentially determine whether each frame in the video can be extracted as a keyframe.

[0093] Adjacent images can refer to image frames in the video data to be processed whose acquisition time is adjacent to that of the current image to be processed. For example, the multiple frames corresponding to the current image to be processed can be multiple frames in the video data to be processed that are before the current image to be processed and multiple frames that are after the current image to be processed.

[0094] In this embodiment of the application, after determining any frame of the video data to be processed as the current image to be processed, and determining whether the current image to be processed can be used as a keyframe image, multiple frames of images whose acquisition time is adjacent to the acquisition time of the current image to be processed can be obtained from the video data to be processed based on the acquisition time of the current image to be processed and the acquisition time of each frame of the video data to be processed, as multiple adjacent frames of images corresponding to the current image to be processed.

[0095] For example, the 10 frames in the video data to be processed that are adjacent to the current image to be processed and are located before the current image to be processed, and the 10 frames in the video data to be processed that are adjacent to the current image to be processed and are located after the current image to be processed, can be identified as the adjacent images corresponding to the current image to be processed.

[0096] It should be noted that the examples above are merely illustrative and should not be considered as limitations on this application. In actual use, the method and number of adjacent images selected can be determined according to actual needs and specific application scenarios, and this application does not limit this.

[0097] Step 103: Extract features from the current image to be processed and each adjacent image in each frame to determine multiple first global features corresponding to the current image to be processed and multiple second global features corresponding to each adjacent image in each frame.

[0098] The first global feature and the second global feature may include the texture features of the image.

[0099] The first global feature can be a feature vector that can describe the global features of the current image to be processed; the second global feature can be a feature vector that can describe the global features of the neighboring images of the current image to be processed.

[0100] In this embodiment, any global feature extraction algorithm can be used to extract global features from the current image to be processed and its neighboring images to determine the first global feature corresponding to the current image to be processed and the second global feature corresponding to each neighboring image. This allows the first global feature to contain rich texture features of the current image to be processed and the second global feature to contain rich texture features of the corresponding neighboring images. In turn, the image content information of the current image to be processed and its neighboring images is introduced through the first global feature and the second global feature, thereby improving the content correlation during keyframe extraction.

[0101] It should be noted that the method for determining the second global feature corresponding to each adjacent image in a frame can be the same as the method for determining the first global feature corresponding to the current image to be processed. In actual use, the global feature extraction algorithm to be used can be determined according to actual needs and specific application scenarios, and this application embodiment does not limit this.

[0102] Step 104: Select a set of neighboring images corresponding to the current image to be processed from the neighboring images based on the similarity between the first global feature and each second global feature.

[0103] The similarity between the first global feature and the second global feature can be determined based on the distance between them. For example, it can be determined based on the Euclidean distance, cosine distance, etc., between them; however, this embodiment does not limit this.

[0104] It is understandable that the greater the distance between the first global feature and the second global feature, the smaller the similarity between the first global feature and the second global feature; conversely, the smaller the distance between the first global feature and the second global feature, the greater the similarity between the first global feature and the second global feature.

[0105] It should be noted that in actual use, the specific method for determining the similarity between the first global feature and the second global feature can be selected according to actual needs and specific application scenarios. This application embodiment does not limit this.

[0106] The neighborhood image set can refer to the set of neighboring images that have a high similarity to the current image to be processed.

[0107] In the embodiments of this application, since the higher the similarity between the first global feature corresponding to the current image to be processed and the second global feature corresponding to the adjacent image, the higher the similarity between the current image to be processed and the adjacent image, the adjacent image corresponding to the second global feature whose similarity with the first global feature is greater than the similarity threshold can be determined as the neighborhood image corresponding to the current image to be processed, and then all the determined neighborhood images are used to form a neighborhood image set.

[0108] Step 105: Determine the information gain of the current image to be processed relative to the set of neighboring images.

[0109] Among them, the information gain of the current image to be processed to the neighborhood image set can be quantitatively described as the amount of information added to the new image set after the current image to be processed is added to the neighborhood image set compared with the neighborhood image set.

[0110] In this embodiment, the information entropy of the neighborhood image set can be determined based on the information entropy of each frame of the neighborhood image set. Then, the current image to be processed can be added to the neighborhood image set to generate a new image set. The information entropy of the new image set can be determined based on the information entropy of the current image to be processed and each frame of the neighborhood image set. Then, the information gain of the current image to be processed on the neighborhood image set can be determined based on the difference between the information entropy of the new image set and the information entropy of the neighborhood image set.

[0111] Step 106: When the information gain is greater than the information gain threshold, the current image to be processed is determined as the keyframe image.

[0112] In this embodiment, if the information gain of the current image to be processed to the set of neighboring images is greater than the information gain threshold, it can be determined that the current image to be processed has a large amount of information and a strong potential sub-image alignment capability, and thus can be sampled as a keyframe image; if the information gain of the current image to be processed to the set of neighboring images is less than the information gain threshold, it can be determined that the current image to be processed has a small amount of information, and the current image to be processed can be discarded and not determined as a keyframe image.

[0113] The VSLAM keyframe selection method provided in this application identifies any frame in the video data to be processed as the current image to be processed and determines its corresponding multiple neighboring frames. Then, feature extraction is performed on the current image to be processed and each neighboring frame to obtain its corresponding first and second global features. Next, the set of neighboring images of the current image to be processed is determined based on the similarity between the first and second global features. Then, the information gain of the current image to be processed relative to the neighboring images is determined. When the information gain is greater than an information gain threshold, the current image to be processed is identified as the keyframe image. Therefore, by performing global feature extraction on the current image to be processed and its neighboring images during keyframe extraction, rich texture features of the image are introduced, thereby improving the correlation of the extracted keyframes in terms of image content, thus enhancing the scene recognition capability and accuracy of VSLAM.

[0114] In one possible implementation of this application, a pre-defined visual vocabulary bag can be used to globally describe the current image to be processed and each adjacent image frame, thereby determining the global features of the current image to be processed and adjacent images, and further improving the relevance of the extracted keyframes to the image content.

[0115] The following is combined with Figure 2 The keyframe selection method for VSLAM provided in the embodiments of this application will be further explained.

[0116] Figure 2 A flowchart illustrating another VSLAM keyframe selection method provided in an embodiment of this application is shown.

[0117] Step 201: Obtain the video data to be processed, which includes multiple frames of images to be processed.

[0118] Step 202: Determine any frame of the video data to be processed as the current image to be processed, and obtain multiple adjacent frames corresponding to the current image to be processed.

[0119] The specific implementation process and principle of steps 201-202 above can be found in the detailed description of the above embodiments, and will not be repeated here.

[0120] Step 203: Extract feature points from the current image to be processed and each adjacent image frame to determine multiple first feature point descriptors corresponding to the current image to be processed and multiple second feature point descriptors corresponding to each adjacent image frame.

[0121] The first feature point descriptor can be a feature vector that can describe the local features of the current image to be processed; the second feature point descriptor can be a feature vector that can describe the local features of the current image to be processed.

[0122] It should be noted that in actual use, a suitable image feature extraction algorithm can be selected to extract features from the current image to be processed and its neighboring images according to actual needs and specific application scenarios. This application embodiment does not limit this. For example, the scale-invariant feature transform (SIFT) feature extraction algorithm can be used to extract features from the current image to be processed and its neighboring images.

[0123] In this embodiment, an image local feature extraction algorithm can be used to extract features from the current image to be processed, generating multiple first feature point descriptors corresponding to the current image to be processed. Each first feature point descriptor can describe the detailed features of a local region of the current image to be processed. Therefore, by obtaining a large number of first feature point descriptors of the current image to be processed, the global detailed features of the current image to be processed can be described. Thus, during keyframe extraction, by introducing feature points that can describe the detailed texture features of the image, the current image to be processed and adjacent images in each frame are described, improving the correlation of keyframes in image content.

[0124] It is understandable that the method for determining the second feature point descriptor corresponding to each adjacent image frame is the same as the method for determining the first feature point descriptor corresponding to the current image to be processed, and will not be elaborated here.

[0125] It should be noted that, in actual use, the number of extracted first and second feature point descriptors can be determined according to actual needs and specific application scenarios, and this application embodiment does not limit this. For example, the number of first feature point descriptors can be 500, 1000, 2000, etc.; wherein, the number of second feature point descriptors corresponding to adjacent images in each frame can be the same as the number of first feature point descriptors.

[0126] Step 204: According to the preset visual vocabulary bag, classify each first feature point descriptor corresponding to the current image to be processed to determine the first global descriptor corresponding to the current image to be processed.

[0127] The pre-defined visual bag of words can refer to a pre-established bag-of-words model that includes multiple words and their corresponding descriptors, and can describe any image through the words it contains.

[0128] Here, the first global descriptor can refer to a feature vector that can describe the global features of the current image to be processed. It should be noted that using the first global descriptor to describe the current image to be processed can describe the texture information of the current image to be processed. Thus, during keyframe extraction, the first global descriptor introduces rich texture information of the current image to be processed, thereby improving the image content correlation during keyframe extraction.

[0129] In this embodiment of the application, the feature point descriptors corresponding to each word included in the preset visual vocabulary bag can be used to represent each first feature point descriptor corresponding to the current image to be processed. Then, based on the number of first feature point descriptors matching each word, the statistical vector corresponding to the current image to be processed is determined, and the statistical vector is then determined as the first global descriptor corresponding to the current image to be processed.

[0130] As one possible implementation, the number of first feature point descriptors matching each term can be determined by the similarity between the first feature descriptor and the feature point descriptor corresponding to each term. That is, in one possible implementation of this application embodiment, the aforementioned preset visual vocabulary bag includes M terms and M third feature point descriptors corresponding to the M terms, where M is a positive integer; correspondingly, step 204 may include:

[0131] Based on the similarity between each first feature point descriptor and the third feature point descriptor corresponding to the i-th term, determine the number of first feature point descriptors that match the i-th term, where i is a positive integer greater than or equal to 1 and less than or equal to M;

[0132] The number of first feature point descriptors that match the i-th term is determined as the number of first feature points of the i-th term in the current image to be processed;

[0133] The first global descriptor is determined based on the number of first feature points categorized for each term in the current image to be processed.

[0134] The method for determining the third feature point description corresponding to the term can be the same as the method for determining the first and second feature point descriptors, and will not be elaborated here.

[0135] One possible implementation is to determine the similarity between the first and third feature point descriptors based on the distance between them. Then, for the i-th term, if the similarity between a first feature point descriptor and the corresponding third feature point descriptor for the i-th term is greater than or equal to a similarity threshold, then the first feature point descriptor matches the i-th term; if the similarity is less than the threshold, then the first feature point descriptor does not match the i-th term. This allows us to determine the number of first feature point descriptors that match the i-th term, and this number is defined as the number of first feature point classifications O for the i-th term in the current image to be processed. v (i), where v is the current image to be processed and i is the index of the term. Furthermore, using the same method, the number of the first feature points of each term in the current image to be processed can be determined.

[0136] For example, if the current image to be processed corresponds to 1000 first feature point descriptors, and M is 5000, for the first term in the preset visual vocabulary, if there are 100 first feature point descriptors whose similarity to the third feature point descriptor corresponding to the first term is greater than a similarity threshold, then it can be determined that there are 100 first feature point descriptors that match the first term, that is, the number of first feature point classifications of the current image to be processed for the first term is O. v (1) = 100.

[0137] It should be noted that in practical use, Euclidean distance, cosine distance, etc., can be used to measure the similarity between the first feature point descriptor and the second feature point descriptor. The similarity threshold can also be determined according to actual needs and specific application scenarios, and this application embodiment does not limit it in this regard. For example, the similarity threshold can be 0.8, 0.9, etc.

[0138] Step 205: Based on the preset visual vocabulary bag, classify each second feature point descriptor corresponding to each adjacent image in each frame to determine the second global descriptor corresponding to each adjacent image in each frame.

[0139] The second global descriptor can refer to a feature vector that can describe the global features of adjacent images. It should be noted that using the second global descriptor to describe adjacent images allows for the description of their texture information. Therefore, during keyframe extraction, the first and second global descriptors introduce rich texture information from the current image and its neighboring images, improving the image content correlation during keyframe extraction.

[0140] It should be noted that the method for determining the second global descriptor corresponding to adjacent images is the same as the method for determining the first global descriptor corresponding to the current image to be processed. The specific implementation process and principle can be found in the detailed description of the above embodiments, and will not be repeated here.

[0141] Step 206: Select a set of neighboring images corresponding to the current image to be processed from the neighboring images based on the distance between the first global descriptor and each second global descriptor.

[0142] In this embodiment of the application, the content similarity between the current image to be processed and each adjacent image can be measured based on the distance between the first global descriptor corresponding to the current image to be processed and the second global descriptor corresponding to each adjacent image in each frame. The adjacent images with higher similarity to the current image to be processed are determined as the neighborhood images of the current image to be processed. Then, the selected neighborhood images are used to form the neighborhood image set corresponding to the current image to be processed.

[0143] Furthermore, since the smaller the distance between the first global descriptor corresponding to the current image to be processed and the second global descriptor corresponding to the adjacent image, the higher the similarity between the current image to be processed and the adjacent image, a neighboring image of the current image to be processed can be selected from the adjacent images according to the set distance threshold. That is, in one possible implementation of this application embodiment, the above step 206 may include:

[0144] The neighboring images corresponding to the second global descriptor whose distance from the first global descriptor is less than the distance threshold are determined as the neighborhood images corresponding to the current image to be processed.

[0145] A set of neighborhood images is formed by using the neighborhood images corresponding to the current image to be processed.

[0146] In this embodiment, if the distance between the first global descriptor and the second global descriptor corresponding to a neighboring image in a certain frame is less than a distance threshold T, it can be determined that the content of the current image to be processed is highly correlated with that of the neighboring image, and thus the neighboring image can be identified as the neighborhood image corresponding to the current image to be processed. Furthermore, using the same method, all neighborhood images that are highly correlated with the content of the current image to be processed can be selected from the neighboring images, and all selected neighborhood images can be used to form a neighborhood image set.

[0147] It should be noted that in actual use, the distance used to measure the similarity between the first global descriptor and the second global descriptor, as well as the specific value of the distance threshold, can be determined according to actual needs and specific application scenarios. This application embodiment does not limit this. For example, the distance can be cosine distance, Euclidean distance, etc.

[0148] Step 207: Determine the first Shannon entropy corresponding to the neighborhood image set, and the second Shannon entropy corresponding to the current image to be processed and the neighborhood image set.

[0149] The first Shannon entropy can quantitatively describe the information content of a neighborhood image set. It should be noted that a larger first Shannon entropy indicates a larger information content in the neighborhood image set; conversely, a smaller first Shannon entropy indicates a smaller information content.

[0150] The second Shannon entropy can refer to the total Shannon entropy of the neighborhood image set and the current image to be processed after the current image to be processed is added to the neighborhood image set; the second Shannon entropy can describe the overall information content of the current image to be processed and the neighborhood image set.

[0151] As one possible implementation, the first Shannon entropy and the second Shannon entropy can be determined based on the completeness measure of each word in the preset visual vocabulary bag in the neighborhood image set, and the completeness measure of each word in the new image set generated after adding the current image to be processed to the neighborhood image set. That is, in one possible implementation of this application embodiment, the aforementioned neighborhood image set includes N frames of neighborhood images, where N is a positive integer; correspondingly, the aforementioned step 207 may include: determining the number of second feature point descriptors matching the i-th word in the j-th frame neighborhood image based on the similarity between each second feature point descriptor corresponding to the j-th frame neighborhood image and the third feature point descriptor corresponding to the i-th word, where j is a positive integer greater than or equal to 1 and less than or equal to N;

[0152] The number of second feature point descriptors that match the i-th word in the j-th frame neighborhood image is determined as the number of second feature points of the i-th word in the j-th frame neighborhood image;

[0153] The first occupancy probability corresponding to the i-th term is determined based on the number of second feature points classified for the i-th term in each frame of the neighborhood image.

[0154] Based on the first occupancy probability corresponding to the i-th term, determine the third Shannon entropy corresponding to the i-th term;

[0155] The sum of the third Shannon entropy corresponding to each term is determined as the first Shannon entropy;

[0156] The second occupancy probability corresponding to the i-th term is determined based on the number of first feature points classified for the i-th term in the current image to be processed and the number of second feature points classified for the i-th term in each neighboring image.

[0157] Based on the second occupancy probability corresponding to the i-th term, determine the fourth Shannon entropy corresponding to the i-th term;

[0158] The sum of the fourth Shannon entropy corresponding to each term is determined as the second Shannon entropy.

[0159] In this embodiment, the number of second feature points of the j-th frame neighboring image for the i-th term can be determined by determining that the current image to be processed has the same number of first feature points for the i-th term. j (i), where j represents the number of neighboring images and i represents the word index. Furthermore, using the same method, the number of second feature points categorized for the i-th word in each neighboring image frame can be determined. Then, the sum of the number of second feature points categorized for the i-th word in each neighboring image frame can be determined as the first occupancy probability P corresponding to the i-th word. i Wherein, the first occupancy probability P i It can be determined using the following formula:

[0160]

[0161] Among them, P i Let S(v) be the first occupancy probability corresponding to the i-th term, S(v) be the set of neighboring images corresponding to the current image to be processed, vj be the j-th frame neighboring image in the set of neighboring images, and j be the index of the neighboring image. j (i) represents the number of second feature points of the i-th word in the neighborhood image of the j-th frame, where i is the index of the word.

[0162] Furthermore, since the number of second feature points categorized for each term may fall within a large range, the numerical span between these categorizations may be significant. This means that directly using the number of first and second feature points categorized for each term to determine its occupancy probability could affect the accuracy of the occupancy probability. Therefore, the number of feature points categorized for each term can be normalized first. That is, in one possible implementation of this application, before determining the first occupancy probability of the i-th term based on the number of second feature points categorized for the i-th term in each neighboring image frame, the following may be included:

[0163] The number of categories for each second feature point is normalized.

[0164] In this embodiment of the application, the number of classifications of the first feature points of each term can be normalized to normalize the number of classifications of the second feature points to a preset numerical range. Then, the first occupancy probability corresponding to the term can be determined by using the normalized number of classifications of the second feature points, so as to eliminate the impact of the large numerical span between the number of classifications of the second feature points on the calculation accuracy and improve the accuracy of the determination of the first occupancy probability.

[0165] It should be noted that in actual use, the method for normalizing the number of second feature points and the preset range of normalization values ​​can be determined according to actual needs and specific application scenarios. This application embodiment does not limit this. For example, maximum value normalization can be used to normalize the number of second feature points to the range of [0,1].

[0166] Furthermore, the third Shannon entropy H corresponding to the i-th term can be determined using the following formula, based on the first occupancy probability corresponding to the i-th term. i ,Right now:

[0167] H i =-P i ·logP i -(1-P i )·log(1-P i )

[0168] Then, the sum of the third Shannon entropies corresponding to each word in the preset visual vocabulary bag can be determined as the first Shannon entropy H(S) corresponding to the neighborhood image set, that is:

[0169]

[0170] In this embodiment, the current image to be processed can be added to a neighborhood image set to generate a new image set. Then, based on the number of feature points classified for the i-th term in each frame of the new image set, the second occupancy probability corresponding to the i-th term is determined. That is, the second occupancy probability P corresponding to the i-th term... i ′ refers to the sum of the number of first feature points classified for the i-th word in the current image to be processed and the number of second feature points classified for the i-th word in each neighboring image, that is:

[0171]

[0172] It should be noted that if the number of second feature point classifications is normalized before calculating the first occupancy probability, then the same method should be used to normalize the number of first feature point classifications when determining the second occupancy probability, so as to determine the second occupancy probability of each term based on the normalized number of first feature point classifications.

[0173] Then, the second occupancy probability P′ corresponding to the i-th term can be calculated using the following formula. i Determine the fourth Shannon entropy H′ corresponding to the i-th term. i ,Right now:

[0174] H′ i =-P′ i ·logP′ i -(1-P′i )·log(1-P′ i )

[0175] Then, the sum of the fourth Shannon entropies corresponding to each entry in the preset visual vocabulary bag can be determined as the second Shannon entropy corresponding to the current image to be processed and the set of neighboring images, that is:

[0176]

[0177] Wherein, H(S) v For the second Shannon entropy, c i Let C be the i-th word in the preset visual vocabulary bag, where C is the preset visual vocabulary bag and i is the word number.

[0178] Step 208: Determine the information gain of the current image to be processed on the set of neighboring images based on the first Shannon entropy and the second Shannon entropy.

[0179] Among them, the information gain of the current image to be processed to the neighborhood image set can be quantitatively described as the amount of information added to the new image set after the current image to be processed is added to the neighborhood image set compared with the neighborhood image set.

[0180] As one possible implementation, the difference between the second Shannon entropy and the first Shannon entropy can be determined as the information gain ΔH of the current image to be processed to the set of neighboring images, i.e.: ΔH = H(S). v -H(S)

[0181] Step 209: When the information gain is greater than the information gain threshold, the current image to be processed is determined as the keyframe image.

[0182] The specific implementation process and principle of step 209 above can be found in the detailed description of the above embodiments, and will not be repeated here.

[0183] The VSLAM keyframe selection method provided in this application involves obtaining multiple adjacent images corresponding to the current image to be processed from the video data to be processed, and performing feature extraction on each of them to obtain a first feature point descriptor of the current image to be processed and a second feature point descriptor of each adjacent image. Then, according to a preset visual vocabulary bag, each first feature point descriptor corresponding to the current image to be processed is classified to determine the first global descriptor corresponding to the current image to be processed. Similarly, the second global descriptor corresponding to each adjacent image is obtained. Then, based on the distance between the first global descriptor and each second global descriptor, a neighborhood image set corresponding to the current image to be processed is selected from the adjacent images. Then, based on the first Shannon entropy corresponding to the neighborhood image set and the second Shannon entropy corresponding to the current image to be processed and the neighborhood image set, the information gain of the current image to be processed to the neighborhood image set is determined. Finally, when the information gain is greater than the information gain threshold, the current image to be processed is determined as the keyframe image. Therefore, by using a pre-defined visual vocabulary bag to globally describe the current image to be processed and its neighboring images during keyframe extraction, rich texture features of the image are introduced during keyframe extraction, thereby improving the correlation of the extracted keyframes in the image content, thus enhancing the scene recognition capability of VSLAM and improving the accuracy of VSLAM.

[0184] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0185] Corresponding to the keyframe selection method of VSLAM described in the above embodiments, Figure 3 A structural block diagram of the VSLAM keyframe selection device provided in the embodiments of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0186] Reference Figure 3 The device 30 includes:

[0187] The first acquisition module 31 is used to acquire video data to be processed, wherein the video data to be processed includes multiple frames of images to be processed;

[0188] The first determining module 32 is used to determine any frame of the image to be processed in the video data to be processed as the current image to be processed, and to determine multiple adjacent frames of the current image to be processed from the video data to be processed.

[0189] The second determining module 33 is used to extract features from the current image to be processed and each adjacent image in each frame, so as to determine the first global feature corresponding to the current image to be processed and the second global feature corresponding to each adjacent image in each frame, wherein the global feature includes the texture feature of the image.

[0190] The first selection module 34 is used to select a set of neighboring images corresponding to the current image to be processed from neighboring images based on the similarity between the first global feature and each second global feature.

[0191] The third determining module 35 is used to determine the information gain of the current image to be processed to the set of neighboring images;

[0192] The fourth determining module 36 is used to determine the current image to be processed as a keyframe image when the information gain is greater than the information gain threshold.

[0193] In practical use, the VSLAM keyframe selection device provided in this application embodiment can be configured in any terminal device to execute the aforementioned VSLAM keyframe selection method.

[0194] The VSLAM keyframe selection apparatus provided in this application determines any frame image in the video data to be processed as the current image to be processed and determines the corresponding multiple neighboring frames of the current image to be processed. Then, it extracts features from the current image to be processed and each neighboring frame to obtain corresponding first and second global features. Next, it determines the set of neighboring images of the current image to be processed based on the similarity between the first and second global features, and then determines the information gain of the current image to be processed relative to the neighboring images. When the information gain is greater than an information gain threshold, the current image to be processed is determined as the keyframe image. Therefore, by performing global feature extraction on the current image to be processed and the neighboring images during keyframe extraction, rich texture features of the image are introduced during keyframe extraction, thereby improving the correlation of the extracted keyframes in image content, thus improving the scene recognition capability of VSLAM and increasing the accuracy of VSLAM.

[0195] In one possible implementation of this application embodiment, the third determining module 35 includes:

[0196] The first determining unit is used to determine the first Shannon entropy corresponding to the neighborhood image set, and the second Shannon entropy corresponding to the current image to be processed and the neighborhood image set;

[0197] The second determining unit is used to determine the information gain of the current image to be processed to the set of neighboring images based on the first Shannon entropy and the second Shannon entropy.

[0198] Furthermore, in another possible implementation of this application embodiment, the first determining unit is specifically used for:

[0199] The difference between the second Shannon entropy and the first Shannon entropy is defined as the information gain.

[0200] Furthermore, in another possible implementation of this application embodiment, the second determining module 33 includes:

[0201] The third determining unit is used to extract feature points from the current image to be processed and each adjacent image in each frame, so as to determine multiple first feature point descriptors corresponding to the current image to be processed and multiple second feature point descriptors corresponding to each adjacent image in each frame.

[0202] The fourth determining unit is used to classify each first feature point descriptor corresponding to the current image to be processed according to the preset visual vocabulary bag, so as to determine the first global descriptor corresponding to the current image to be processed.

[0203] The fifth determining unit is used to classify each second feature point descriptor corresponding to each adjacent image in each frame according to a preset visual vocabulary bag, so as to determine the second global descriptor corresponding to each adjacent image in each frame.

[0204] Furthermore, in another possible implementation of this application embodiment, the aforementioned preset visual vocabulary bag includes M entries and M third feature point descriptors corresponding to the M entries, where M is a positive integer; correspondingly, the aforementioned fourth determining unit is specifically used for:

[0205] Based on the similarity between each first feature point descriptor and the third feature point descriptor corresponding to the i-th term, determine the number of first feature point descriptors that match the i-th term, where i is a positive integer greater than or equal to 1 and less than or equal to M;

[0206] The number of first feature point descriptors that match the i-th term is determined as the number of first feature points of the i-th term in the current image to be processed;

[0207] The first global descriptor is determined based on the number of first feature points categorized for each term in the current image to be processed.

[0208] Furthermore, in another possible implementation of this application embodiment, the aforementioned neighborhood image set includes N frames of neighborhood images, where N is a positive integer; correspondingly, the aforementioned first determining unit is specifically used for:

[0209] Based on the similarity between each second feature point descriptor corresponding to the j-th frame neighborhood image and the third feature point descriptor corresponding to the i-th word, determine the number of second feature point descriptors in the j-th frame neighborhood image that match the i-th word, where j is a positive integer greater than or equal to 1 and less than or equal to N;

[0210] The number of second feature point descriptors that match the i-th word in the j-th frame neighborhood image is determined as the number of second feature points of the i-th word in the j-th frame neighborhood image;

[0211] The first occupancy probability corresponding to the i-th term is determined based on the number of second feature points classified for the i-th term in each frame of the neighborhood image.

[0212] Based on the first occupancy probability corresponding to the i-th term, determine the third Shannon entropy corresponding to the i-th term;

[0213] The sum of the third Shannon entropy corresponding to each term is determined as the first Shannon entropy;

[0214] The second occupancy probability corresponding to the i-th term is determined based on the number of first feature points classified for the i-th term in the current image to be processed and the number of second feature points classified for the i-th term in each neighboring image.

[0215] Based on the second occupancy probability corresponding to the i-th term, determine the fourth Shannon entropy corresponding to the i-th term;

[0216] The sum of the fourth Shannon entropies corresponding to each of the aforementioned terms is determined as the second Shannon entropy.

[0217] Furthermore, in another possible implementation of this application embodiment, the first determining unit is further configured to:

[0218] The number of categories for each second feature point is normalized.

[0219] Furthermore, in another possible implementation of this application embodiment, the first selection module 34 includes:

[0220] The sixth determining unit is used to select a set of neighboring images corresponding to the current image to be processed from the adjacent images based on the distance between the first global descriptor and each second global descriptor.

[0221] Furthermore, in another possible implementation of this application embodiment, the sixth determining unit is specifically used for:

[0222] The neighboring images corresponding to the second global descriptor whose distance from the first global descriptor is less than the distance threshold are determined as the neighborhood images corresponding to the current image to be processed.

[0223] A set of neighborhood images is formed by using the neighborhood images corresponding to the current image to be processed.

[0224] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0225] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0226] To implement the above embodiments, this application also proposes a terminal device.

[0227] Figure 4 This is a schematic diagram of the structure of a terminal device according to an embodiment of this application.

[0228] like Figure 4 As shown, the terminal device 200 includes:

[0229] The system includes a memory 210 and at least one processor 220, and a bus 230 connecting different components (including the memory 210 and the processor 220). The memory 210 stores a computer program, which, when executed by the processor 220, implements the keyframe selection method of VSLAM as described in this application embodiment.

[0230] Bus 230 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0231] Terminal device 200 typically includes various electronically readable media. These media can be any available media that can be accessed by terminal device 200, including volatile and non-volatile media, removable and non-removable media.

[0232] Memory 210 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. Terminal device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 260 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 4 Not shown; usually referred to as a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 230 via one or more data media interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0233] A program / utility 280 having a set (at least one) of program modules 270 may be stored in, for example, memory 210. Such program modules 270 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 270 typically perform the functions and / or methods described in the embodiments of this application.

[0234] Terminal device 200 can also communicate with one or more external devices 290 (e.g., keyboard, pointing device, display 291, etc.), and with one or more devices that enable a user to interact with terminal device 200, and / or with any device that enables terminal device 200 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 292. Furthermore, terminal device 200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 293. As shown, network adapter 293 communicates with other modules of terminal device 200 via bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with terminal device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0235] The processor 220 performs various functional applications and data processing by running programs stored in the memory 210.

[0236] It should be noted that the implementation process and technical principles of the terminal device in this embodiment are explained in the foregoing description of the keyframe selection method of VSLAM in this application embodiment, and will not be repeated here.

[0237] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0238] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0239] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0240] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0241] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0242] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0243] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0244] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A keyframe selection method for Visual Simultaneous Localization and Mapping (VSLAM), characterized in that, include: Acquire video data to be processed, wherein the video data to be processed includes multiple frames of images to be processed; Any frame of the image to be processed in the video data to be processed is determined as the current image to be processed, and multiple adjacent frames corresponding to the current image to be processed are determined from the video data to be processed. Feature extraction is performed on the current image to be processed and the adjacent images in each frame to determine the first global feature corresponding to the current image to be processed and the second global feature corresponding to the adjacent images in each frame, wherein the first global feature and the second global feature include the texture features of the image; Based on the similarity between the first global feature and each of the second global features, a set of neighboring images corresponding to the current image to be processed is selected from the neighboring images; Determine the information gain of the current image to be processed relative to the set of neighboring images; When the information gain is greater than the information gain threshold, the current image to be processed is determined as a keyframe image; Determining the information gain of the current image to be processed relative to the set of neighboring images includes: Determine the first Shannon entropy corresponding to the neighborhood image set, and the second Shannon entropy corresponding to the current image to be processed and the neighborhood image set; Based on the first Shannon entropy and the second Shannon entropy, the information gain of the current image to be processed on the set of neighboring images is determined.

2. The method as described in claim 1, characterized in that, The step of determining the information gain of the current image to be processed to the neighborhood image set based on the first Shannon entropy and the second Shannon entropy includes: The difference between the second Shannon entropy and the first Shannon entropy is determined as the information gain.

3. The method as described in claim 1, characterized in that, The step of extracting features from the current image to be processed and the adjacent images in each frame to determine the first global feature corresponding to the current image to be processed and the second global feature corresponding to the adjacent images in each frame includes: Feature points are extracted from the current image to be processed and each of the adjacent images in each frame to determine multiple first feature point descriptors corresponding to the current image to be processed and multiple second feature point descriptors corresponding to each of the adjacent images in each frame; Based on a preset visual vocabulary bag, each of the first feature point descriptors corresponding to the current image to be processed is classified to determine the first global descriptor corresponding to the current image to be processed. Based on the preset visual vocabulary bag, each second feature point descriptor corresponding to each adjacent image in each frame is classified to determine the second global descriptor corresponding to each adjacent image in each frame.

4. The method as described in claim 3, characterized in that, The preset visual vocabulary bag includes M entries and M third feature point descriptors corresponding to the M entries, where M is a positive integer. The step of classifying each first feature point descriptor corresponding to the current image to be processed according to the preset visual vocabulary bag to determine the first global descriptor corresponding to the current image to be processed includes: Based on the similarity between each first feature point descriptor and the third feature point descriptor corresponding to the i-th term, the number of first feature point descriptors matching the i-th term is determined, where i is a positive integer greater than or equal to 1 and less than or equal to M; The number of first feature point descriptors that match the i-th term is determined as the number of first feature points of the i-th term in the current image to be processed; The first global descriptor is determined based on the number of first feature points categorized for each term in the current image to be processed.

5. The method as described in claim 4, characterized in that, The neighborhood image set includes N neighborhood images, where N is a positive integer. Determining the first Shannon entropy corresponding to the neighborhood image set and the second Shannon entropy corresponding to the current image to be processed and the neighborhood image set includes: Based on the similarity between each second feature point descriptor corresponding to the neighborhood image of the j-th frame and the third feature point descriptor corresponding to the i-th term, the number of second feature point descriptors matching the i-th term in the neighborhood image of the j-th frame is determined, where j is a positive integer greater than or equal to 1 and less than or equal to N; The number of second feature point descriptors that match the i-th term in the neighborhood image of the j-th frame is determined as the number of second feature points of the i-th term in the neighborhood image of the j-th frame; The first occupancy probability corresponding to the i-th term is determined based on the number of second feature points classified for the i-th term in each frame of the neighborhood image. Based on the first occupancy probability corresponding to the i-th term, determine the third Shannon entropy corresponding to the i-th term; The sum of the third Shannon entropies corresponding to each of the aforementioned terms is determined as the first Shannon entropy; The second occupancy probability corresponding to the i-th term is determined based on the number of first feature points classified for the i-th term in the current image to be processed and the number of second feature points classified for the i-th term in each frame of the neighborhood image. Based on the second occupancy probability corresponding to the i-th term, determine the fourth Shannon entropy corresponding to the i-th term; The sum of the fourth Shannon entropies corresponding to each of the aforementioned terms is determined as the second Shannon entropy.

6. The method as described in claim 5, characterized in that, Before determining the first occupancy probability corresponding to the i-th word based on the number of second feature points classified for the i-th word in each frame of the neighborhood image, the method further includes: The number of categories for each second feature point is normalized.

7. The method according to any one of claims 3-6, characterized in that, The step of selecting a set of neighboring images corresponding to the current image to be processed from the neighboring images based on the similarity between the first global feature and each of the second global features includes: Based on the distance between the first global descriptor and each of the second global descriptors, a set of neighboring images corresponding to the current image to be processed is selected from the neighboring images.

8. The method as described in claim 7, characterized in that, The step of selecting a set of neighboring images corresponding to the current image to be processed from the neighboring images based on the distance between the first global descriptor and each of the second global descriptors includes: The neighboring images corresponding to the second global descriptor whose distance from the first global descriptor is less than a distance threshold are determined as the neighborhood images corresponding to the current image to be processed; The neighborhood image set is constructed using the neighborhood images corresponding to the current image to be processed.

9. A VSLAM keyframe selection device, characterized in that, include: The first acquisition module is used to acquire video data to be processed, wherein the video data to be processed includes multiple frames of images to be processed; The first determining module is used to determine any frame of the image to be processed in the video data to be processed as the current image to be processed, and to determine multiple adjacent frames of the current image to be processed from the video data to be processed. The second determining module is used to extract features from the current image to be processed and the adjacent images in each frame to determine the first global feature corresponding to the current image to be processed and the second global feature corresponding to the adjacent images in each frame, wherein the first global feature and the second global feature include the texture features of the image; The first selection module is used to select a set of neighboring images corresponding to the current image to be processed from the neighboring images based on the similarity between the first global feature and each of the second global features; The third determining module is used to determine the information gain of the current image to be processed on the set of neighboring images; The fourth determining module is used to determine the current image to be processed as a keyframe image when the information gain is greater than the information gain threshold. The third determining module includes: The first determining unit is used to determine the first Shannon entropy corresponding to the neighborhood image set, and the second Shannon entropy corresponding to the current image to be processed and the neighborhood image set; The second determining unit is used to determine the information gain of the current image to be processed to the set of neighboring images based on the first Shannon entropy and the second Shannon entropy.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images