Method and device for determining key frame, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide a method and device for determining a key frame, electronic equipment and a storage medium. The method comprises: obtaining a current frame; wherein the current frame is a non-first frame; determining a target feature point of the current frame, and determining a displacement parallax between the current frame and a historical key frame; if the number of target feature points reaches a first preset number threshold, and the displacement parallax is greater than a first preset displacement parallax threshold, then determining the current frame as a key frame; when a next frame of the current frame is obtained, taking the next frame as the current frame and taking the key frame as the historical key frame, and determining whether the next frame is a key frame based on the historical key frame. The technical solution of the embodiments of the present disclosure screens the key frame from multiple angles, solves the problem that the selection of the key frame in the prior art is single and lacks judgment on the observation point coverage, which leads to the risk of the robustness of SLAM, and achieves the technical effect of effective screening of the key frame.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a method, apparatus, electronic device, and storage medium for determining keyframes. Background Technology

[0002] With the continuous advancement of computing, artificial intelligence, and sensor technologies, intelligent applications such as AR / VR, drones, and intelligent robots have flooded the market, making Simultaneous Localization and Mapping (SLAM) technology a focus of attention. When using SLAM systems to update spatial states, the selection of keyframes is particularly important.

[0003] Keyframes are typically determined by installing an inertial measurement unit (IMU) on the image acquisition device and selecting keyframes through a sliding window with IMU information, or by selecting keyframes based on common-view relationships.

[0004] Existing keyframe selection methods rely on a single criterion and lack comprehensive multi-angle assessment. The overall keyframe selection process lacks consideration of observation point coverage, leading to errors and inaccuracies in the identified keyframes. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, and storage medium for determining keyframes, achieving the technical effect of effectively and accurately determining keyframes.

[0006] In a first aspect, embodiments of this disclosure provide a method for determining keyframes, the method comprising:

[0007] Obtain the current frame; wherein the current frame is not the first frame;

[0008] Determine the target feature points of the current frame, and determine the displacement parallax between the current frame and historical keyframes;

[0009] If the number of target feature points reaches a first preset number threshold, and the displacement parallax is greater than the first preset displacement parallax threshold, then the current frame is determined to be a key frame.

[0010] Secondly, embodiments of this disclosure also provide an apparatus for determining keyframes, the apparatus comprising:

[0011] The current frame acquisition module is used to acquire the current frame; wherein, the current frame is not the first frame;

[0012] The feature point and disparity determination module is used to determine the target feature points of the current frame and the displacement disparity between the current frame and historical key frames.

[0013] The keyframe determination module determines the current frame as a keyframe if the number of target feature points reaches a first preset number threshold and the displacement parallax is greater than the first preset displacement parallax threshold.

[0014] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0015] One or more processors;

[0016] Storage device for storing one or more programs.

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining keyframes as described in any embodiment of this disclosure.

[0018] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a method for determining keyframes as described in any of the embodiments of this disclosure.

[0019] The technical solution of this disclosure embodiment obtains the current frame; determines the target feature points of the current frame, and determines the displacement disparity between the current frame and historical key frames; if the number of target feature points reaches a first preset number threshold and the displacement disparity is greater than the first preset displacement disparity threshold, then the current frame is determined as a key frame. When obtaining the next frame of the current frame, the next frame is used as the current frame and the key frame is used as a historical key frame. The determination of whether the next frame is a key frame based on the historical key frame solves the problem in the prior art that the selection criteria for key frames are singular and lacks judgment on the coverage of observation points, which leads to risks in the robustness of SLAM. It realizes effective screening of key frames and improves the accuracy and robustness of SLAM while ensuring real-time performance. Attached Figure Description

[0020] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 This is a flowchart illustrating a method for determining keyframes provided in Embodiment 1 of this disclosure;

[0022] Figure 2This is a flowchart illustrating a method for determining keyframes provided in Embodiment 2 of this disclosure;

[0023] Figure 3 This is a flowchart illustrating a method for determining keyframes provided in Embodiment 3 of this disclosure;

[0024] Figure 4 This is a structural block diagram of an apparatus for determining keyframes provided in Embodiment 4 of this disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of this disclosure. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0029] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0030] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0031] Example 1

[0032] Figure 1This is a flowchart illustrating a method for determining keyframes according to Embodiment 1 of this disclosure. This embodiment is applicable to situations where keyframes are determined and camera pose information is determined based on the keyframes. This method can be executed by a device for determining keyframes. The device can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC, or a server.

[0033] Before introducing this technical solution, an exemplary application scenario can be provided. The solution of this disclosure embodiment can be applied to spatial positioning and map building using a SLAM system, where it is necessary to acquire images through an image acquisition device to update the spatial state. Since the image acquisition device acquires a large number of image frames, storing all of them would require a large amount of storage space. To avoid this problem, keyframes can be determined from the acquired video frames, and then the camera position information of the image acquisition device can be determined based on the keyframes. This allows for tracking and positioning or spatial plane construction based on the camera position information.

[0034] like Figure 1 The method in this embodiment includes:

[0035] S110, Get the current frame.

[0036] The image acquisition device can capture video streams. The video frame that needs to be processed is designated as the current frame. The current frame can be the first frame or a non-first frame. It should be noted that if the current frame is the first frame, it means that the corresponding video frame has not yet been captured by the image acquisition device, and this video frame can be used as a keyframe.

[0037] Optionally, if the current frame is the first frame, then the current frame is determined as a key frame, and the key frame is used as the historical key frame for the next frame.

[0038] It should be noted that, for the next video frame, the current frame is a historical video frame. Similarly, if the current frame is a keyframe, then for the next video frame, the current frame is a historical keyframe. Specifically, if the current frame is the first image captured by the image acquisition device, it means that the image acquisition device has not captured any image frames before this frame. That is, the current frame is the first frame and has no historical keyframes, making it impossible to determine whether the current frame is a keyframe based on historical keyframes. In this case, the current frame can be directly used as a keyframe, and this keyframe can be used as a historical keyframe for the next frame, so that the next frame can be determined as a keyframe based on the keyframe determination scheme of this embodiment.

[0039] In this embodiment, video streams can be captured using an image acquisition device, optionally a camera device deployed on a mobile terminal. For example, when a user triggers application A installed on the terminal device and activates a control within application A, the camera on the terminal device can be invoked to capture the corresponding video stream. Application A can be a program for shooting AR videos.

[0040] S120. Determine the target feature points of the current frame and the displacement parallax between the current frame and historical keyframes.

[0041] The target feature points can be representative feature points in the current frame. Displacement parallax, as the name suggests, is the video frame parallax caused by the movement of the terminal device.

[0042] It should be noted that the image acquisition device may acquire many video frames before acquiring the current frame. Determining whether each video frame is a keyframe can be done using the technical solutions provided in the embodiments of this disclosure. In this embodiment, the keyframe closest to the acquisition time of the current frame is taken as the historical keyframe of the current frame.

[0043] It should be noted that image acquisition devices can capture video from different poses. The changes in the pose of the image acquisition device mainly involve translation and rotation. If rotation occurs, it only indicates a change in the shooting angle, but the content of the image frame remains unchanged. If the rotated image frame is used as a keyframe, it may overlap with previously acquired keyframes. Therefore, it is necessary to eliminate the rotational parallax between the current frame and historical keyframes, retaining only the displacement parallax. A significant displacement parallax between the current frame and historical keyframes indicates substantial differences in information between them.

[0044] Specifically, feature points contained in the current image frame can be extracted using some universal feature point extraction algorithms. Alternatively, the current image frame can be input into a commonly used image feature extraction model to obtain the feature points contained in the current frame as output by the feature extraction model. After downsampling the feature points contained in the current frame, the obtained feature points are used as target feature points. Furthermore, the pixel distance between pixels in the current frame and pixels in historical keyframes can be calculated using a distance formula, and the calculated pixel distance is used as the displacement disparity between the current frame and historical keyframes.

[0045] S130. If the number of target feature points reaches a first preset quantity threshold, and the displacement disparity is greater than a first preset displacement disparity threshold, then the current frame is determined to be a keyframe. Here, the first preset quantity threshold is a priori threshold set by the developers. The first preset displacement disparity threshold is a disparity threshold set by the developers to determine whether the displacement disparity between the current frame and historical keyframes meets the requirements.

[0046] In this embodiment, after determining that the current frame is a keyframe, when the next frame of the current frame is obtained, the next frame can be used as the current frame and the keyframe can be used as a historical keyframe to determine whether the next frame is a keyframe based on the historical keyframe.

[0047] Specifically, if the number of target feature points exceeds the prior threshold set by the developers, it indicates that the current frame contains a large number of feature points. Simultaneously, if the displacement parallax between the current frame and historical keyframes exceeds a preset displacement parallax threshold, it indicates a significant displacement between the current frame and historical keyframes, suggesting a substantial difference in content information between them, and that the current frame contains more feature information. In this case, the current frame is designated as a keyframe. When the next frame is obtained, the above steps can be repeated to determine whether the next video frame is a keyframe. For the next video frame, the current frame is the keyframe.

[0048] In practice, if the current frame is not a keyframe, the next frame can be processed to determine if it is a keyframe. Repeating these steps sequentially determines whether each video frame is a keyframe.

[0049] The technical solution of this embodiment obtains the current frame; determines the target feature points of the current frame; and determines the displacement disparity between the current frame and historical key frames. If the number of target feature points reaches a first preset number threshold and the displacement disparity is greater than the first preset displacement disparity threshold, then the current frame is determined as a key frame. When obtaining the next frame, the next frame is used as the current frame and the key frame is used as a historical key frame. Based on the historical key frames, it is determined whether the next frame is a key frame. Key frames are screened from multiple angles, which solves the problem that the selection criteria for key frames in the prior art are singular and lack judgment on the coverage of observation points, resulting in risks to the robustness of SLAM. The solution achieves the technical effect of effective and accurate key frame screening.

[0050] Example 2

[0051] Figure 2This is a flowchart illustrating a method for determining keyframes according to Embodiment 2 of this disclosure. Based on the foregoing embodiments, this embodiment further refines the determination of target feature points in the current frame and the displacement parallax between the current frame and historical keyframes. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0052] like Figure 2 As shown, the method specifically includes the following steps:

[0053] S210. Obtain the current frame; where the current frame is not the first frame.

[0054] S220. Determine the feature points to be processed in the current frame, and the stability value of each feature point to be processed.

[0055] Here, the feature points to be processed refer to all feature points contained in the current image frame. The stability value of the feature points to be processed is used to characterize whether the feature point is stable enough, and the stability is reflected in whether it appears multiple times in several consecutive video frames.

[0056] Specifically, a feature point extraction algorithm can be used to extract feature points from the current frame, and these extracted feature points can be used as the feature points to be processed in the current frame. When extracting feature points from the current frame using the feature point extraction algorithm, a stability score can be obtained for each feature point, and each score can be used as the stability value of the corresponding feature point to be processed.

[0057] S230. Based on the stability value, downsample the feature points to be processed to obtain the target feature points of the current frame.

[0058] Downsampling refers to the process of removing some feature points from the frame. The removed feature points can then be used to determine whether the current frame is a keyframe.

[0059] In this embodiment, the downsampling process can specifically be as follows: determine the region to be processed centered on the feature point to be retained with the highest current stability value, and delete the feature points to be processed in the region to be processed to obtain the target feature points to be removed from the stability values ​​of the feature points to be retained; repeat the process of determining the region to be processed centered on the feature point to be retained with the highest current stability value, and deleting the feature points to be processed in the region to be processed to obtain the target feature points to be removed from the stability values ​​of the feature points to be retained, until the current frame does not contain any target feature points with stability values.

[0060] The feature points to be retained can be those retained after downsampling of the current frame. In specific processing, the feature point with the highest stability value can be selected as the feature point to be retained. The processing region is a region defined by a certain size, centered on the feature point to be retained. For example, if the processing region is circular, then the feature point to be retained is the center of the circle, and the radius of the circle can be a preset value.

[0061] Specifically, the stability values ​​of the feature points to be processed can be sorted in descending order, and the feature point with the highest stability value can be selected as the feature point to be retained. Further, a circular region centered on the feature point to be retained is defined with a certain distance as its radius; this region can be used as the processing region. Alternatively, the processing region can be a square centered on the feature point to be retained. Further still, after determining the processing region, which contains some feature points to be processed and one feature point to be retained, all feature points in the processing region except the center point are deleted, and the stability value of the feature point to be retained is also removed, resulting in the target feature point. At this point, a new feature point with the highest stability value will emerge from the remaining feature points. This new feature point will be selected as the new feature point to be retained, and a processing region centered on this new feature point will be defined. All other feature points in this region will be deleted, and the stability value of the new feature point to be retained will be removed, making it the target feature point. Next, the feature point with the highest updated stability value will be selected. This process is repeated to obtain target feature points from which stability values ​​are removed, until no target feature points with stability values ​​are found in the current frame. At this point, downsampling is complete. Based on this method, the target feature point corresponding to the current frame is determined.

[0062] For example, the current frame contains feature points q, w, e, r, t, y, u, and i to be processed, with corresponding stability values ​​of 2, 5, 6, 8, 7, 4, 3, and 9. Here, for the sake of illustrating stability values, simple numerical values ​​are used to represent them; the actual stability values ​​are determined based on the specific circumstances of the image frame. After determining the stability values ​​for each feature point to be processed, the feature points are sorted in descending order of stability value, in the order i>r>t>e>w>y>u>q. It can be seen that feature point i has the highest stability value. Feature point i is selected as the feature point to be retained. Using i as the center and a certain distance as the radius, a region can be selected; this selected region is the region to be processed. The selected region contains feature points q, w, e, and i. Feature points q, w, and e, except for the feature point i to be retained, can be deleted. Only feature point i remains in the current region. Therefore, the stability value of feature point i is removed, and this point i is selected as the target feature point. Furthermore, the feature point with the highest stability value in the current frame is r, with a stability value of 8. Therefore, feature point r is selected as the feature point to be retained, and the processing region is further determined. This processing region contains points r and t. Point t is deleted, point r is retained, and its stability value is removed, resulting in the target feature point. Further, the point with the highest stability value among the remaining feature points is y. The processing region centered on y contains feature points y and u. Point u is deleted, point y is retained, and its stability value is removed, resulting in the target feature point. At this point, the current frame does not contain target feature points with stability values. Feature points i, r, and y are then selected as the target feature points determined after downsampling.

[0063] S240. After removing the rotational parallax between the current frame and the historical keyframes, determine the displacement parallax between the current frame and the historical keyframes.

[0064] Specifically, some universal rotational parallax removal algorithms can be used to remove the rotational parallax between historical keyframes and the current frame. After removing the rotational parallax, the displacement parallax between the current frame and historical keyframes can be calculated.

[0065] S250. If the number of target feature points reaches the first preset number threshold and the displacement parallax is greater than the first preset displacement parallax threshold, then the current frame is determined to be a key frame.

[0066] The technical solution of this embodiment obtains the current frame (not the first frame); determines the target feature points of the current frame and the displacement disparity between the current frame and historical keyframes; if the number of target feature points reaches a first preset threshold and the displacement disparity is greater than the first preset displacement disparity threshold, then the current frame is determined to be a keyframe. When obtaining the next frame, the next frame is used as the current frame, and the keyframe is used as a historical keyframe to determine whether the next frame is a keyframe based on the historical keyframes. This embodiment comprehensively filters keyframes from the perspectives of disparity, co-view, and the distribution of feature points in the current frame. This solves the problem in the prior art where the selection criteria for keyframes are singular and lack judgment on observation point coverage, leading to risks in the robustness of SLAM. It achieves the technical effect of effective and accurate keyframe filtering.

[0067] Example 3

[0068] Figure 3 This is a flowchart illustrating a method for determining keyframes according to Embodiment 3 of this disclosure. Based on the aforementioned embodiments, this embodiment further determines keyframes by combining second and third keyframe adjustment strategies when the number of target feature points does not reach a first preset threshold and / or the displacement disparity is less than a preset displacement disparity threshold. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0069] like Figure 3 As shown, the method specifically includes the following steps:

[0070] S310. Obtain the current frame; where the current frame is not the first frame.

[0071] S320. Determine the target feature points in the current frame and the displacement parallax between the current frame and historical keyframes;

[0072] S330. If the number of target feature points does not reach the first preset number threshold and / or the displacement disparity is less than the preset displacement disparity threshold, then the current frame is processed based on the second keyframe adjustment strategy and / or the third keyframe adjustment strategy, so that when the processing result is consistent with the corresponding preset result, the current frame is determined to be a keyframe.

[0073] The second and third keyframe adjustment strategies both include methods for determining keyframes, which are applicable to situations where the number of target feature points in the current frame does not meet the first preset number threshold and / or the displacement disparity is less than the preset displacement disparity threshold.

[0074] Specifically, if the number of target feature points in the current frame does not meet the first preset number threshold and / or the displacement disparity is less than the preset displacement disparity threshold, then the key frame determination method of S130 cannot be used to determine whether the current frame can be used as a key frame. In this case, the second key frame adjustment strategy and the third key frame adjustment strategy are used to determine whether the current frame can be used as a key frame. The current frame is processed according to the corresponding adjustment strategy to obtain a processing result. If the processing result is consistent with the preset result of the strategy, then the current frame can be used as a key frame.

[0075] Optionally, the current frame is processed based on the second keyframe adjustment strategy to determine the current frame as a keyframe when the processing result is consistent with the corresponding preset result. This includes: determining the common feature points of the current frame and historical keyframes, and performing downsampling processing in the current frame based on the common feature points to determine the target feature points; and determining the displacement deviation between the current frame and historical keyframes. If the number of target feature points is less than the number of feature points to be processed in the current frame, and the displacement deviation is less than the second preset displacement deviation, then the current frame is determined as a keyframe.

[0076] Commonly seen feature points refer to some feature points shared by the current frame and historical keyframes. For example, if the feature points of the current frame include a, b, c, f, and g, and the feature points of the historical keyframes include a, d, k, f, and m, then feature points a and f are the commonly seen feature points of the current frame and the historical keyframes.

[0077] The second preset displacement deviation can be a parallax distance threshold set by the developer to determine whether the parallax distance generated by the displacement between the current frame and the historical keyframes meets the requirements.

[0078] Specifically, a similarity algorithm can be used to calculate the similarity value between feature points in historical keyframes and feature points in the current frame. If the similarity value between two feature points reaches a certain threshold, then the two points are considered highly similar, and this point in the current frame can be considered a co-visible feature point with the historical keyframe. Based on this method, all co-visible feature points in the current frame are determined. Then, the co-visible points in the current frame are downsampled. The downsampling method is the same as in the above embodiment; the current frame can be downsampled according to the stability value of the co-visible feature points. After downsampling, the remaining co-visible feature points can be used as target feature points. Furthermore, the disparity between the current frame and historical keyframes is calculated. After removing the rotational disparity, the displacement disparity between the historical keyframes and the current frame is determined, and the value corresponding to this displacement disparity is used as the displacement deviation. Furthermore, it is determined whether the number of target feature points is less than the number of feature points to be processed contained in the current frame, and whether the displacement deviation is less than the time difference distance threshold preset by the developer. If the number of target feature points in the current frame is less than the number of feature points to be processed in the current frame, and the displacement deviation between the current frame and the historical key frame is less than the second preset displacement deviation, it means that the current frame is not so similar to the historical key frame, that is, the current frame contains more information, and the current frame can be determined as a key frame.

[0079] For example, the second preset displacement deviation can be 5, the number of feature points to be processed in the current frame can be 100, and the displacement deviation is 3. After sampling, the number of common viewpoints between the current frame and historical keyframes is 60, which is less than 70% of the number of feature points to be processed in the current frame, and the displacement deviation is less than the second preset displacement deviation. Therefore, the current frame can be determined as a keyframe.

[0080] In this embodiment, the third keyframe adjustment strategy can be: processing the current frame based on the third keyframe adjustment strategy, so that when the processing result is consistent with the third preset result, the current frame is determined to be a keyframe, including: triangulating the feature points to be processed in the current frame to obtain the point cloud data to be processed; downsampling the point cloud data to be processed to obtain the target feature points; determining the displacement deviation between the current frame and the historical keyframes; if the number of target feature points is less than or equal to the number of co-view feature points, and the displacement deviation is less than the third preset displacement deviation, then the current frame is determined to be a keyframe; wherein, the co-view feature points are the co-view points of the current frame and the historical keyframes.

[0081] The point cloud data to be processed refers to the cloud data points obtained by connecting the feature points to be processed in the current frame with triangles to form a triangular mesh, and establishing the topological connection relationship between each data point and its neighboring points. The third preset displacement parallax threshold is a parallax threshold set by the developers in advance to determine whether the displacement parallax between the current frame and the historical keyframes meets the requirements.

[0082] Specifically, the feature points to be processed in the current frame can be triangulated using a triangulation algorithm to obtain point cloud data, i.e., the point cloud data to be processed. This point cloud data is then projected onto the current frame to determine which points are projected onto it. These points are then downsampled. The specific downsampling process is the same as the downsampling method described in the previous embodiment and will not be repeated here. After downsampling, the target feature points of the current frame can be obtained. If the number of target feature points is less than or equal to the number of consensus points between the current frame and historical keyframes, and the displacement deviation between the current frame and historical keyframes is less than a third preset displacement deviation, then the current frame can be used as a keyframe.

[0083] For example, the third preset displacement parallax is 6, the number of co-view points between the current frame and historical keyframes is 60, and the displacement parallax is 4. The number of remaining feature points after downsampling of the triangulated points in the current frame is 20, which is less than 50% of the number of co-view points in the current frame, i.e., less than 30. Furthermore, the displacement parallax of 4 is less than the third preset displacement parallax, so the current frame can be used as a keyframe.

[0084] It should be noted that the three strategies for determining keyframes described above can be processed in parallel or sequentially. These strategies can be used simultaneously or sequentially to determine whether the current frame is a keyframe. A keyframe determined based on any of the above strategies can be considered a keyframe.

[0085] The technical solution of this embodiment acquires the current frame; determines the target feature points of the current frame; and determines the displacement disparity between the current frame and historical keyframes. If the number of target feature points reaches a first preset threshold and the displacement disparity is greater than the first preset displacement disparity threshold, then the current frame is determined to be a keyframe. When acquiring the next frame, the next frame is used as the current frame, and the keyframe is used as a historical keyframe to determine whether the next frame is a keyframe based on the historical keyframe. The technical solution of this embodiment comprehensively filters keyframes from the perspectives of disparity, co-view, and the distribution of feature points in the current frame. This solves the problem in the prior art where the selection criteria for keyframes are singular and lacks judgment on the coverage of observation points, leading to risks in the robustness of SLAM. It achieves effective filtering of keyframes, improving the accuracy and robustness of SLAM while ensuring real-time performance.

[0086] Example 4

[0087] Figure 4 This is a structural block diagram of an apparatus for determining keyframes provided in Embodiment 4 of this disclosure. It can execute the method for determining keyframes provided in any embodiment of this disclosure, and possesses the corresponding functional modules and beneficial effects for executing the method. For example... Figure 4As shown, the device specifically includes: a current frame acquisition module 410, a feature point and disparity determination module 420, and a key frame determination module 430.

[0088] The current frame acquisition module 410 is used to acquire the current frame; wherein, the current frame is not the first frame;

[0089] The feature point and disparity determination module 420 is used to determine the target feature points of the current frame and the displacement disparity between the current frame and the historical key frame.

[0090] The keyframe determination module 430 is used to determine the current frame as a keyframe if the number of target feature points reaches a first preset number threshold and the displacement parallax is greater than the first preset displacement parallax threshold.

[0091] Based on the above technical solutions, the keyframe determination module 430 includes:

[0092] The new keyframe determination module is used to, when obtaining the next frame of the current frame, take the next frame as the current frame and take the key frame as the historical key frame, so as to determine whether the next frame is a key frame based on the historical key frame.

[0093] Based on the above technical solutions, the device also includes:

[0094] The first frame determination module is used to determine the current frame as a key frame if the current frame is the first frame, and to use the key frame as the historical key frame of the next frame.

[0095] Based on the above technical solutions, the feature point and disparity determination module 420 includes:

[0096] The feature point determination module is used to determine the feature points to be processed in the current frame, as well as the stability value of each feature point to be processed.

[0097] The feature point downsampling module is used to downsample the feature points to be processed based on the stability value to obtain the target feature points of the current frame.

[0098] Based on the above technical solutions, the feature point downsampling module includes:

[0099] The stability value elimination module is used to determine the region to be processed centered on the feature point to be retained with the highest current stability value, and to delete the feature points to be processed in the region to obtain the target feature point from which the stability value of the feature point to be retained is eliminated.

[0100] The target feature point acquisition module is used to repeatedly determine the region to be processed centered on the feature point to be retained with the highest current stability value, and delete the feature points to be processed in the region to be processed, so as to obtain the target feature points to be removed from the stability value of the feature points to be retained, until the current frame does not contain any target feature points with stability values.

[0101] Based on the above technical solutions, the feature point and disparity determination module 420 also includes:

[0102] The displacement parallax determination module is used to determine the displacement parallax between the current frame and the historical keyframe after removing the rotational parallax between the current frame and the historical keyframe.

[0103] Based on the above technical solutions, the device for determining keyframes also includes:

[0104] The keyframe determination unit is used to process the current frame based on the second keyframe adjustment strategy and / or the third keyframe adjustment strategy if the number of target feature points does not reach the first preset number threshold and / or the displacement disparity is less than the preset displacement disparity threshold, so as to determine the current frame as a keyframe when the processing result is consistent with the corresponding preset result.

[0105] Based on the above technical solutions, the keyframe determination unit further includes:

[0106] A common-view feature point and displacement deviation determination unit is used to determine the common-view feature points of the current frame and the historical key frame, and to determine the target feature point based on the common-view feature points in the current frame by downsampling; and to determine the displacement deviation between the current frame and the historical key frame.

[0107] The second displacement disparity determination unit is used to determine the current frame as a key frame if the number of target feature points is less than the number of feature points to be processed in the current frame and the displacement deviation is less than a second preset displacement deviation.

[0108] Based on the above technical solutions, the keyframe determination unit further includes:

[0109] The triangulation unit is used to triangulate the feature points to be processed in the current frame to obtain the point cloud data to be processed.

[0110] The downsampling unit is used to downsample the point cloud data to be processed to obtain target feature points;

[0111] The third displacement deviation determination unit is used to determine the displacement deviation between the current frame and the historical key frame;

[0112] A keyframe determination subunit is used to determine the current frame as a keyframe if the number of target feature points is less than or equal to the number of co-view feature points and the displacement deviation is less than a third preset displacement deviation; wherein, the co-view feature points are the co-view points of the current frame and the historical keyframes.

[0113] The technical solution of this embodiment obtains the current frame (not the first frame); determines the target feature points of the current frame and the displacement disparity between the current frame and historical keyframes; if the number of target feature points reaches a first preset threshold and the displacement disparity is greater than the first preset displacement disparity threshold, then the current frame is determined to be a keyframe. When obtaining the next frame, the next frame is used as the current frame, and the keyframe is used as a historical keyframe to determine whether the next frame is a keyframe based on the historical keyframes. This embodiment comprehensively filters keyframes from the perspectives of disparity, common view, and the distribution of feature points in the current frame. This solves the problem in the prior art where the selection criteria for keyframes are singular and lack judgment on observation point coverage, leading to risks in the robustness of SLAM. It achieves effective keyframe filtering, improving the accuracy and robustness of SLAM while ensuring real-time performance.

[0114] The apparatus for determining keyframes provided in this disclosure can execute the method for determining keyframes provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method.

[0115] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0116] Example 5

[0117] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of this disclosure. Refer to the following... Figure 5 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 5 The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0118] like Figure 5 As shown, electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0119] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0120] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0121] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0122] The electronic device provided in this embodiment and the method for determining keyframes provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0123] Example 6

[0124] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the method for determining keyframes provided in the above embodiments.

[0125] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0126] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0127] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0128] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0129] Obtain the current frame; wherein the current frame is not the first frame;

[0130] Determine the target feature points of the current frame, and determine the displacement parallax between the current frame and historical keyframes;

[0131] If the number of target feature points reaches a first preset number threshold, and the displacement disparity is greater than the first preset displacement disparity threshold, then the current frame is determined to be a key frame. When obtaining the next frame of the current frame, the next frame is used as the current frame and the key frame is used as the historical key frame, so as to determine whether the next frame is a key frame based on the historical key frame.

[0132] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0134] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0135] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0136] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] According to one or more embodiments of this disclosure, [Example 1] provides a method for determining keyframes, the method comprising:

[0138] Obtain the current frame; wherein the current frame is not the first frame;

[0139] Determine the target feature points of the current frame, and determine the displacement parallax between the current frame and historical keyframes;

[0140] If the number of target feature points reaches a first preset number threshold, and the displacement parallax is greater than the first preset displacement parallax threshold, then the current frame is determined to be a key frame.

[0141] According to one or more embodiments of this disclosure, [Example 2] provides a method for determining keyframes, further comprising:

[0142] Optionally, after determining that the current frame is a keyframe, the method further includes:

[0143] When obtaining the next frame of the current frame, the next frame is used as the current frame and the key frame is used as the historical key frame, so as to determine whether the next frame is a key frame based on the historical key frame.

[0144] According to one or more embodiments of this disclosure, [Example 3] provides a method for determining keyframes, further comprising:

[0145] If the current frame is the first frame, then the current frame is determined to be a key frame, and the key frame is used as the historical key frame for the next frame.

[0146] According to one or more embodiments of this disclosure, [Example 4] provides a method for determining keyframes, further comprising:

[0147] Optionally, determining the target feature points of the current frame includes:

[0148] Determine the feature points to be processed in the current frame, and the stability value of each feature point to be processed;

[0149] The target feature points of the current frame are obtained by downsampling the feature points to be processed based on the stability value.

[0150] According to one or more embodiments of this disclosure, [Example 5] provides a method for determining keyframes, further comprising:

[0151] Optionally, the step of downsampling the feature points to be processed based on the stability value to obtain the target feature points of the current frame includes:

[0152] Determine the region to be processed centered on the feature point with the highest current stability value, and delete the feature points to be processed in the region to obtain the target feature point from which the stability value of the feature point to be retained is removed;

[0153] The process repeats, identifying the region to be processed centered on the feature point with the highest current stability value, and deleting the feature points in the region to be processed, thus obtaining the target feature point from which the stability value of the feature point to be retained is removed, until the current frame does not contain any target feature points with stability values.

[0154] According to one or more embodiments of this disclosure, [Example Six] provides a method for determining keyframes, further comprising:

[0155] Optionally, determining the displacement parallax between the current frame and historical keyframes includes:

[0156] After removing the rotational parallax between the current frame and the historical keyframe, the displacement parallax between the current frame and the historical keyframe is determined.

[0157] According to one or more embodiments of this disclosure, [Example Seven] provides a method for determining keyframes, further comprising:

[0158] If the number of target feature points does not reach the first preset number threshold and / or the displacement disparity is less than the preset displacement disparity threshold, the current frame is processed based on the second keyframe adjustment strategy and / or the third keyframe adjustment strategy, so that when the processing result is consistent with the corresponding preset result, the current frame is determined to be a keyframe.

[0159] According to one or more embodiments of this disclosure, [Example Eight] provides a method for determining keyframes, further comprising:

[0160] Optionally, the current frame is processed based on a second keyframe adjustment strategy to determine it as a keyframe when the processing result matches a corresponding preset result, including:

[0161] Determine the co-view feature points of the current frame and the historical keyframe, and perform downsampling processing in the current frame based on the co-view feature points to determine the target feature points; and determine the displacement deviation between the current frame and the historical keyframe.

[0162] If the number of target feature points is less than the number of feature points to be processed in the current frame, and the displacement deviation is less than the second preset displacement deviation, then the current frame is determined to be a key frame.

[0163] According to one or more embodiments of this disclosure, [Example Nine] provides a method for determining keyframes, further comprising:

[0164] Optionally, the current frame is processed based on a third keyframe adjustment strategy to determine it as a keyframe when the processing result matches a third preset result, including:

[0165] The feature points to be processed in the current frame are triangulated to obtain the point cloud data to be processed.

[0166] The point cloud data to be processed is downsampled to obtain the target feature points;

[0167] Determine the displacement deviation between the current frame and the historical keyframe;

[0168] If the number of target feature points is less than or equal to the number of co-visible feature points, and the displacement deviation is less than a third preset displacement deviation, then the current frame is determined to be a key frame.

[0169] The co-view feature point is the co-view point of the current frame and the historical key frame.

[0170] According to one or more embodiments of this disclosure, [Example 10] provides an apparatus for determining keyframes, comprising:

[0171] The current frame acquisition module is used to acquire the current frame; wherein, the current frame is not the first frame;

[0172] The feature point and disparity determination module is used to determine the target feature points of the current frame and the displacement disparity between the current frame and historical key frames.

[0173] The keyframe determination module determines the current frame as a keyframe if the number of target feature points reaches a first preset number threshold and the displacement parallax is greater than the first preset displacement parallax threshold.

[0174] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0175] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0176] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for determining keyframes, characterized in that, include: Obtain the current frame; wherein the current frame is not the first frame; Determine the target feature points of the current frame, and determine the displacement parallax between the current frame and historical keyframes; If the number of target feature points reaches a first preset number threshold, and the displacement disparity is greater than the first preset displacement disparity threshold, then the current frame is determined to be a key frame. Determining the target feature points of the current frame includes: Determine the feature points to be processed in the current frame, and the stability value of each feature point to be processed; The feature points to be processed are downsampled based on the stability value to obtain the target feature points of the current frame; The stability value is used to characterize the stability of the feature point to be processed, and the stability is determined based on the frequency of occurrence of the feature point to be processed in multiple consecutive video frames. The step of downsampling the feature points to be processed based on the stability value to obtain the target feature points of the current frame includes: Determine the processing region centered on the feature point with the highest current stability value, delete the feature points in the processing region, and remove the stability values ​​of the feature points to be retained to obtain the target feature points from which the stability values ​​of the feature points to be retained are removed. The process involves repeatedly determining the region to be processed, deleting the feature points to be processed in the region, removing the stability values ​​of the feature points to be retained, and obtaining target feature points from which the stability values ​​of the feature points to be retained are removed, until the current frame does not contain any target feature points with stability values.

2. The method according to claim 1, characterized in that, After determining that the current frame is a keyframe, the process also includes: When obtaining the next frame of the current frame, the next frame is used as the current frame and the key frame is used as the historical key frame, so as to determine whether the next frame is a key frame based on the historical key frame.

3. The method according to claim 1, characterized in that, Also includes: If the current frame is the first frame, then the current frame is determined to be a key frame, and the key frame is used as the historical key frame for the next frame.

4. The method according to claim 1, characterized in that, Determining the displacement disparity between the current frame and historical keyframes includes: After removing the rotational parallax between the current frame and the historical keyframe, the displacement parallax between the current frame and the historical keyframe is determined.

5. The method according to claim 1, characterized in that, Also includes: If the number of target feature points does not reach the first preset number threshold and / or the displacement disparity is less than the preset displacement disparity threshold, the current frame is processed based on the second keyframe adjustment strategy and / or the third keyframe adjustment strategy, so that when the processing result is consistent with the corresponding preset result, the current frame is determined to be a keyframe.

6. The method according to claim 5, characterized in that, The current frame is processed based on a second keyframe adjustment strategy, and when the processing result matches the corresponding preset result, the current frame is determined to be a keyframe, including: Determine the co-view feature points of the current frame and the historical keyframe, and perform downsampling processing in the current frame based on the co-view feature points to determine the target feature points; and determine the displacement deviation between the current frame and the historical keyframe. If the number of target feature points is less than the number of feature points to be processed in the current frame, and the displacement deviation is less than the second preset displacement deviation, then the current frame is determined to be a key frame.

7. The method according to claim 5, characterized in that, The current frame is processed based on a third keyframe adjustment strategy, and when the processing result is consistent with a third preset result, the current frame is determined to be a keyframe, including: The feature points to be processed in the current frame are triangulated to obtain the point cloud data to be processed. The point cloud data to be processed is downsampled to obtain the target feature points; Determine the displacement deviation between the current frame and the historical keyframe; If the number of target feature points is less than or equal to the number of co-visible feature points, and the displacement deviation is less than a third preset displacement deviation, then the current frame is determined to be a key frame. The co-view feature point is the co-view point of the current frame and the historical key frame.

8. An apparatus for determining keyframes, characterized in that, include: The current frame acquisition module is used to acquire the current frame; wherein, the current frame is not the first frame; The feature point and disparity determination module is used to determine the target feature points of the current frame and the displacement disparity between the current frame and historical key frames. The keyframe determination module determines the current frame as a keyframe if the number of target feature points reaches a first preset number threshold and the displacement disparity is greater than the first preset displacement disparity threshold. Feature point and disparity determination module, including: The feature point determination module is used to determine the feature points to be processed in the current frame, as well as the stability value of each feature point to be processed. The feature point downsampling module is used to downsample the feature points to be processed based on the stability value to obtain the target feature points of the current frame. The stability value is used to characterize the stability of the feature point to be processed, and the stability is determined based on the frequency of occurrence of the feature point to be processed in multiple consecutive video frames. The feature point downsampling module includes: The stability value elimination module is used to determine the region to be processed centered on the feature point to be retained with the highest current stability value, delete the feature points to be processed in the region to be processed, eliminate the stability value of the feature points to be retained, and obtain the target feature point from which the stability value of the feature points to be retained is eliminated. The target feature point acquisition module is used to repeatedly determine the region to be processed, delete the feature points to be processed in the region to be processed, remove the stability values ​​of the feature points to be retained, and obtain the target feature points with the stability values ​​of the feature points to be retained removed, until the current frame does not contain any target feature points with stability values.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining keyframes as described in any one of claims 1-7.

10. A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the method for determining keyframes as described in any one of claims 1-7.

Images