Methods, apparatuses, devices, media, and program products for determining co-sighting keyframes

Abstract

The method, device, equipment, medium and program product for determining co-view key frames provided by the present disclosure determine a plurality of voxels of a physical space corresponding to a current frame image, determine visible voxels from the plurality of voxels, and the visible voxels include a plurality of map points. Then, the visible map points of the current frame image are determined according to the physical coordinates of the map points, and the key frame image in which the visible map points of the current frame image are located in the stored key frame set is determined as the co-view key frame image of the current frame image. The co-view key frame does not need to be determined first, and then all the key frames are traversed to find the co-view key frame, which guarantees the real-time performance of the visual SLAM system, and also reduces the pose drift and improves the positioning accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a method, apparatus, device, medium, and program product for determining co-view keyframes. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a computer vision technique that can build a model of the surrounding environment while moving, even when the surrounding environment is unknown, and simultaneously estimate its own motion. SLAM technology can be applied to real-time devices such as drones, robots, and autonomous vehicles to achieve navigation functions.

[0003] The positioning accuracy of SLAM technology typically depends on the ability of the current frame to establish connections with all keyframes in the map. Current technologies usually require identifying keyframes across all frames and then traversing all keyframes to find common-view keyframes. The frame containing a visible map point is called a common-view keyframe. The more visible map points in a common-view keyframe, the smaller the camera pose drift and the higher the positioning accuracy.

[0004] Traversing all keyframes to find common keyframes causes the time consumed by the visual SLAM system to increase linearly with the increase of running time and running range, which cannot meet the real-time requirements of the visual SLAM system. Summary of the Invention

[0005] This disclosure provides a method, apparatus, device, medium, and program product for determining common-view keyframes, which solves the problem in the prior art that traversing all keyframes to find common-view keyframes causes the time consumption of the visual SLAM system to increase linearly with the increase of running time and running range, thus failing to meet the real-time requirements of the visual SLAM system.

[0006] In a first aspect, this disclosure provides a method for determining common-view keyframes, including:

[0007] Obtain multiple voxels in the physical space corresponding to the current frame image;

[0008] A visible voxel is determined from the plurality of voxels, wherein the visible voxel contains a plurality of map points;

[0009] The visible map points of the current frame image are determined based on the physical coordinates of the map points in the visible voxels. The visible map points of the current frame image include map points in the visible voxels that can be projected onto the current frame image.

[0010] Determine the keyframe image containing the visible map points of the current frame image from the stored keyframe set, and define the keyframe image containing the visible map points of the current frame image as the co-view keyframe image of the current frame image.

[0011] Secondly, this disclosure provides an apparatus for determining common-view keyframes, comprising:

[0012] The acquisition module is used to acquire multiple voxels in the physical space corresponding to the current frame image;

[0013] A first determining module is used to determine a visible voxel from the plurality of voxels, wherein the visible voxel contains a plurality of map points;

[0014] The second determining module is used to determine the visible map points of the current frame image based on the physical coordinates of the map points in the visible voxels. The visible map points of the current frame image include map points in the visible voxels that can be projected onto the current frame image.

[0015] The third determining module is used to determine the keyframe image where the visible map point of the current frame image is located in the stored keyframe set, and to determine the keyframe image where the visible map point of the current frame image is located as the common keyframe image of the current frame image.

[0016] Thirdly, this disclosure provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0017] The memory stores computer-executed instructions;

[0018] The processor executes computer execution instructions stored in the memory to implement the method for determining co-view keyframes as described in the first aspect.

[0019] Fourthly, this disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method for determining co-view keyframes as described in the first aspect.

[0020] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method for determining co-view keyframes as described in the first aspect.

[0021] This disclosure provides a method, apparatus, device, medium, and program product for determining common-view keyframes. It acquires multiple voxels in the physical space corresponding to the current frame image, then determines visible voxels from these voxels, each including multiple map points. Based on the physical coordinates of these map points, it determines the visible map points of the current frame image. Finally, it identifies the keyframe image containing the visible map points of the current frame image from a stored set of keyframes as the common-view keyframe image. This eliminates the need to first determine keyframes and then traverse all keyframes to find common-view keyframes, ensuring the real-time performance of the visual SLAM system while reducing pose drift and improving positioning accuracy. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0023] Figure 1 A flowchart illustrating a method for determining common-view keyframes provided in Embodiment 1 of this disclosure;

[0024] Figure 2 A schematic diagram of a visual voxel as an example;

[0025] Figure 3 A schematic diagram illustrating voxel division as an example;

[0026] Figure 4 This is a structural block diagram of an apparatus for determining common-view keyframes provided in Embodiment 2 of this disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 2 of this disclosure.

[0028] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0030] The positioning accuracy of SLAM technology usually depends on the ability of the current frame to establish a connection with all keyframes in the map. In existing technologies, it is usually necessary to determine keyframes in all frames and then traverse all keyframes to find common keyframes.

[0031] In this context, the co-view keyframe is the frame containing visible map points. A map point is a point in 3D space whose pose information (position and orientation, such as rotation angle and orientation) is captured or observed by a vision sensor (such as a camera). Visible map points can be understood as the map points that can be observed in the current frame image. The more visible map points in the co-view keyframe, the smaller the camera pose drift and the higher the positioning accuracy.

[0032] The aforementioned shared-view keyframes are keyframes that correspond to map points (i.e., visible map points) that can be observed by the camera, contained in voxels (pixels in three-dimensional space) where the camera's view cone (the cone formed by the extensions of the lines connecting the optical center and the four corners of the camera plane) intersects in space. It can be understood that shared-view keyframes are a subset of keyframes.

[0033] Traversing all keyframes to find common keyframes causes the time consumption of the visual SLAM system to increase linearly with the increase of running time and running range, which cannot meet the real-time requirements of the system. Alternatively, in order to ensure the real-time performance of the system, only a portion of the keyframes within a certain time range or a certain spatial range can be traversed, but this approach will lose some visible map points, resulting in greater camera pose drift.

[0034] Considering that the number of voxels in physical space is fixed and will not change with the number of keyframes, pose drift can be reduced while meeting the system's real-time requirements.

[0035] Therefore, this application proposes a method, apparatus, device, medium, and program product for determining common-view keyframes. By finding the visible voxels in the physical space where the current frame image is located, and then determining whether the map points in the visible voxels are visible map points, that is, whether these map points can be projected into the current frame image, the keyframe where the visible map points are located can be determined as the common-view keyframe. It is not necessary to first determine the keyframe and then traverse all the keyframes to find the common-view keyframe, which ensures the real-time performance of the visual SLAM system. At the same time, it also reduces pose drift and improves positioning accuracy.

[0036] The application scenarios of this disclosure can include intelligent devices that integrate cameras and visual SLAM systems. These devices can be virtual reality headsets, drones, robots, autonomous vehicles, and other devices that use SLAM technology for navigation. It is understood that the fast method for finding common-view keyframes provided in this disclosure includes, but is not limited to, the above application scenarios.

[0037] The technical solutions of this disclosure and how they solve the aforementioned technical problems will be described in detail below with specific embodiments. These specific embodiments may exist independently or in combination with each other. The same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0038] refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for determining common-view keyframes according to Embodiment 1 of this disclosure. The method can be executed by a device for determining common-view keyframes, which is equipped with a camera and is specifically executed by a visual SLAM system integrated in the device (hereinafter referred to as the system). The method includes the following steps.

[0039] S101. Obtain multiple voxels in the physical space corresponding to the current frame image.

[0040] To achieve navigation, a visual SLAM system needs to find as many visible map points as possible for the current frame image to ensure accurate positioning. The system acquires the current frame image captured by the camera.

[0041] After the system acquires the current frame image captured by the camera, it can determine the visible voxel from all voxels in the physical space corresponding to the current frame image. This visible voxel includes multiple map points, as specifically implemented below:

[0042] Before determining the visible voxels, the system can divide the physical space into multiple voxels based on specific map points in the keyframe images of the keyframe set, using a preset size. For example, the preset size can be 1m × 1m × 1m (meters). Figure 2 This diagram illustrates the physical space divided into sections of a preset size. The number and location of map points and voxels in the diagram are merely examples, and the map points are specific map points of the image. Optionally, these specific map points may include the center point and four vertices of the image.

[0043] After obtaining multiple voxels, the physical coordinates of each vertex of each voxel are determined, that is, the physical coordinates of the eight vertices. These physical coordinates can be understood as three-dimensional coordinates in space. The keyframe set is determined based on each frame image preceding the current frame image. For a detailed description of the keyframe set, please refer to step S104.

[0044] S102. Determine the visible voxel from multiple voxels, where the visible voxel contains multiple map points.

[0045] Once the system has identified the visible voxel, the following steps can be used to determine the visible voxel from multiple voxels:

[0046] The system uses the camera's optical center as the origin and connects this origin to the four vertices of the current frame image, forming four rays. These four rays have deterministic functional expressions. The system then substitutes the physical coordinates of each voxel's vertex into the functional expressions of the four rays to determine whether a map point passes through a ray. In other words, if the physical coordinates of a vertex satisfy the functional expression of a ray, then that vertex passes through that ray. Therefore, the system can determine vertices that lie on at least one ray; that is, a vertex can lie on multiple rays simultaneously.

[0047] After determining the vertices of at least one ray, the system can determine the maximum and minimum values ​​of each coordinate (i.e., x, y, and z) from the physical coordinates of these vertices. For example, the maximum value is determined as x1 and the minimum value as x2 from the x-coordinates of these vertices; the maximum value is determined as y1 and the minimum value as y2 from the y-coordinates of these vertices; and the maximum value is determined as z1 and the minimum value as z2 from the z-coordinates of these vertices.

[0048] Then, based on the maximum and minimum values ​​of the physical coordinates at the vertex of at least one ray, the voxels within the range of the maximum and minimum values ​​of each coordinate value are determined. For example, voxels within the range of minimum value x2 to maximum value x1, minimum value y2 to maximum value y1, and minimum value z2 to maximum value z1 can be determined.

[0049] The system can then identify visible voxels as those within the range of maximum and minimum coordinate values, as well as voxels containing vertices that pass through at least one ray. An exemplary diagram of visible voxels can be found here. Figure 3 .

[0050] S103. Determine the visible map points of the current frame image based on the physical coordinates of the map points in the visible voxels.

[0051] After the system determines the visible voxels, it can determine the map points in each visible voxel that can be projected into the current frame image based on the coordinates of the map points within the visible voxels. These map points are referred to as visible map points below.

[0052] Specifically, the system can convert the physical coordinates of each map point in each visible voxel into camera coordinates according to formula (1).

[0053]

[0054] Where W represents the physical coordinate system and C represents the camera coordinate system. Let i be the physical coordinates of point i on the map. These are the matrix parameters for transforming the physical coordinate system to the camera coordinate system.

[0055] After determining the camera coordinates of the map points in the visible voxels, the system can use formula (2) to determine the visible map points in the visible voxels based on the camera coordinates and the pixel size of the current frame.

[0056]

[0057] in, Let K be the camera coordinates of map point i, and K be the camera intrinsic parameters. Let i be the pixel coordinates of map point i. Based on formula (2), we determine whether the map point is a visible map point according to formula (3).

[0058]

[0059] Where w and h are the pixel sizes of the current frame image.

[0060] Once the visible map points in each visible voxel are determined, the visible map points of the current frame image can be obtained. These visible map points are map points that can be projected onto the current frame image.

[0061] S104. Determine the keyframe image where the visible map points of the current frame image are located in the stored keyframe set, and determine the keyframe image where the visible map points of the current frame image are located as the common keyframe image of the current frame image.

[0062] After determining the visible map points of the current frame image, the system selects the keyframe image containing the visible map points of the current frame image from the stored keyframe set as the common keyframe image of the current frame image.

[0063] Optionally, the system can determine whether the current frame is a keyframe based on the number of visible map points or shared keyframe images. Specifically, when the number of visible map points is less than a first preset threshold, or the number of shared keyframes is less than a second preset threshold, the current frame is designated as a keyframe. In other words, the current frame has relatively few visible map points, meaning a significant number of other map points are not visible, thus the current frame can be used as a keyframe.

[0064] The system can then add the current frame image to the keyframe set for storage, so that it can be used by subsequent frame images when determining common keyframes.

[0065] If the current frame is a keyframe, the system can also use this keyframe to generate new map points, thereby increasing the number of map points and the number of shared keyframes in subsequent frames, thus improving the system's positioning accuracy.

[0066] Specifically, the system extracts the point corresponding to the random pixel coordinates (u, v) from the current frame image as the new map point. The system can determine the initial depth z of the new map point based on the physical coordinates of the visible map points in the current frame, such as using the average depth of the visible map points near the pixel coordinates of the new map point. Then, based on the depth z of the new map point and the pixel coordinates (u, v), the physical coordinates of the new map point are obtained by the above formulas (1) and (2).

[0067] After obtaining new map points, the system can update the voxels of the physical space corresponding to the current frame image based on the new map points.

[0068] Specifically, the system determines the voxels to which the new map points belong based on their physical coordinates, and then inserts the new map points into their respective voxels. When a voxel contains only new map points, it indicates that the voxel is a new voxel relative to the voxels in the physical space corresponding to the current frame image in step S102. These new voxels are used in subsequent frame images when determining the co-view frame image, as one of all the voxels determined in step S102 for the visible voxels. In other words, the number of voxels in the physical space corresponding to the current frame image is increased.

[0069] In this embodiment, the system uses multiple voxels in the physical space corresponding to the current frame image, and then determines the visible voxel from these voxels. This visible voxel includes multiple map points. Then, based on the physical coordinates of these map points, the system determines the visible map points of the current frame image. The system then selects the keyframe image containing the visible map points of the current frame image from the stored keyframe set as the co-view keyframe image of the current frame image. This eliminates the need to first determine the keyframes and then traverse all keyframes to find the co-view keyframes, ensuring the real-time performance of the visual SLAM system while also reducing pose drift and improving positioning accuracy.

[0070] Corresponding to the method for determining common-view keyframes in the above embodiments, Figure 4 This is a structural block diagram of an apparatus for determining common-view keyframes according to Embodiment 2 of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown.

[0071] Reference Figure 4 The device 40 includes: an acquisition module 401, a first determination module 402, a second determination module 403, and a third determination module 404.

[0072] The acquisition module 401 is used to acquire multiple voxels in the physical space corresponding to the current frame image.

[0073] The first determining module 402 is used to determine a visible voxel from multiple voxels, wherein the visible voxel contains multiple map points.

[0074] The second determining module 403 is used to determine the visible map points of the current frame image based on the physical coordinates of the map points in the visible voxels. The visible map points of the current frame image include all map points in the visible voxels that can be projected into the current frame image.

[0075] The third determining module 404 is used to determine the keyframe image where the visible map points of the current frame image are located in the stored keyframe set, and to determine the keyframe image where the visible map points of the current frame image are located as the common keyframe image of the current frame image.

[0076] Optionally, the first determining module 402 is specifically used for:

[0077] With the camera's optical center as the origin, four rays are formed by connecting the origin to the four vertices of the current frame image.

[0078] Determine the vertex on at least one ray from the vertices of multiple voxels based on their physical coordinates.

[0079] Determine the voxels within the range of the maximum and minimum values ​​of the physical coordinates of the vertices on at least one ray.

[0080] The voxels within the range of maximum and minimum coordinate values, as well as the voxels containing vertices that pass through at least one ray, are defined as visible voxels.

[0081] Optionally, the second determining module 403 is specifically used for:

[0082] Convert the physical coordinates of map points in the visible voxels to camera coordinates.

[0083] Based on the camera coordinates and the pixel size of the current frame image, determine the map points in the visible voxels that can be projected into the current frame image, and define the map points that can be projected into the current frame image as the visible map points of the current frame image.

[0084] Optionally, device 40 further includes: a generation module,

[0085] The generation module is specifically used to: after determining the keyframe image where the visible map points of the current frame image are located as the co-view keyframe image of the current frame image, determine the current frame image as a keyframe based on the number of visible map points or co-view keyframe images.

[0086] Generate new map points based on the physical coordinates of the visible map points in the current frame image.

[0087] Update the voxels of the physical space corresponding to the current frame image based on the new map points.

[0088] Optionally, the generation module is specifically used to: determine the current frame image as a keyframe when the number of visible map points is less than a first preset threshold, or when the number of shared keyframe images is less than a second preset threshold.

[0089] Optionally, the device 40 further includes a storage module for storing keyframes into a keyframe set.

[0090] Optionally, the acquisition module 401 is specifically used to: divide the physical space into multiple voxels and the physical coordinates of each vertex of the voxels according to the special map points of the keyframe images in the keyframe set. The voxels include the special map points of any keyframe in the keyframe set. The special map points are part of the map points of the image. The keyframe set is determined based on the images before the current frame image.

[0091] Optional, special map points include the center point of the image and the four vertices.

[0092] The apparatus in this embodiment can be used to execute the steps of a method for determining common keyframes in Embodiment 1. The specific implementation and technical effects are similar, and will not be described again here.

[0093] refer to Figure 5 The present disclosure provides a schematic diagram of the structure of an electronic device 50, which can be an intelligent device integrating a camera and a visual SLAM system. The intelligent device may include, but is not limited to, devices requiring navigation functions such as virtual reality headsets, drones, robots, and autonomous vehicles. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0094] like Figure 5 As shown, the electronic device 50 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 50. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0095] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 50 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 50 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0096] Embodiment 4 of the present invention provides a computer-readable storage medium storing computer-executable instructions. These instructions include program code for performing the method described in Embodiment 1. In this embodiment, the computer-executable instructions can be downloaded and installed from a network via a communication device 509, installed from a storage device 508, or installed from a ROM 502. When the computer-executable instructions are executed by a processing device 501, they perform the functions defined in the method of this disclosure embodiment. Specific implementation methods and technical effects are similar and will not be repeated here.

[0097] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0098] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0099] Embodiment 5 of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of a washing machine drum collision handling method as described in Embodiment 1 above. The specific implementation method and technical effects are similar, and will not be repeated here.

[0100] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0101] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0102] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for determining co-view keyframes, characterized in that, The method includes: Obtain multiple voxels in the physical space corresponding to the current frame image; A visible voxel is determined from the plurality of voxels, wherein the visible voxel contains a plurality of map points; Based on the coordinates of map points within the visible voxels, determine the map points in each visible voxel that are projected into the current frame image. The visible map points in the current frame image include the map points in the visible voxels that are projected into the current frame image. In the stored set of keyframes, determine the keyframe image where the visible map point of the current frame image is located, and determine the keyframe image where the visible map point of the current frame image is located as the common keyframe image of the current frame image. Determining the visible voxels from the plurality of voxels includes: With the camera's optical center as the origin, four rays are formed by connecting the origin to the four vertices of the current frame image; Substitute the physical coordinates of the vertices of the multiple voxels into the function expressions of the four rays to determine the vertex on at least one ray from the multiple vertices of the multiple voxels; Based on the maximum and minimum values ​​of the physical coordinates at the apex of at least one ray, determine the voxels within the range of the maximum and minimum values ​​of the coordinates. The voxels within the range of the maximum and minimum values ​​of the coordinates, as well as the voxels containing the vertices on at least one ray, are determined as visible voxels.

2. The method according to claim 1, characterized in that, The step of determining the map points in each visible voxel that can be projected onto the current frame image based on the coordinates of map points within the visible voxel includes: Convert the physical coordinates of map points in the visible voxels into camera coordinates; Based on the camera coordinates and the pixel size of the current frame image, map points in the visible voxels that can be projected onto the current frame image are determined, and the map points projected onto the current frame image are defined as the visible map points of the current frame image.

3. The method according to claim 1 or 2, characterized in that, After determining the keyframe image where the visible map points of the current frame image are located as the co-view keyframe image of the current frame image, the method further includes: The current frame image is determined to be a keyframe based on the number of visible map points or the number of shared keyframe images; Generate new map points based on the physical coordinates of the visible map points in the current frame image; Update the voxels of the physical space corresponding to the current frame image based on the new map points.

4. The method according to claim 3, characterized in that, The step of determining the current frame image as a keyframe based on the number of visible map points or shared keyframe images includes: When the number of visible map points is less than a first preset threshold, or when the number of shared keyframe images is less than a second preset threshold, the current frame image is determined to be a keyframe.

5. The method according to claim 4, characterized in that, The method further includes: The keyframes are stored in the keyframe set.

6. The method according to claim 4 or 5, characterized in that, Obtain multiple voxels in the physical space corresponding to the current frame image, including: Based on the special map points of the keyframe images in the keyframe set, the physical space is divided into multiple voxels and the physical coordinates of each vertex of the voxels. The voxels include the special map points of any keyframe in the keyframe set. The special map points are part of the map points of the image. The keyframe set is determined based on the images preceding the current frame image.

7. The method according to claim 6, characterized in that, The special map point includes the center point of the image and four vertices.

8. An apparatus for determining co-view keyframes, characterized in that, include: The acquisition module is used to acquire multiple voxels in the physical space corresponding to the current frame image; A first determining module is used to determine a visible voxel from the plurality of voxels, wherein the visible voxel contains a plurality of map points; The second determining module is used to determine the map points in each visible voxel that are projected into the current frame image based on the coordinates of the map points in the visible voxel. The visible map points in the current frame image include the map points in the visible voxel that are projected into the current frame image. The third determining module is used to determine the key frame image where the visible map point of the current frame image is located in the stored key frame set, and to determine the key frame image where the visible map point of the current frame image is located as the common key frame image of the current frame image. The first determining module is specifically used to connect the camera optical center with the four vertices of the current frame image to form four rays, with the camera optical center as the origin. Substitute the physical coordinates of the vertices of the multiple voxels into the function expressions of the four rays to determine the vertex on at least one ray from the multiple vertices of the multiple voxels; Based on the maximum and minimum values ​​of the physical coordinates at the apex of at least one ray, determine the voxels within the range of the maximum and minimum values ​​of the coordinates. The voxels within the range of the maximum and minimum values ​​of the coordinates, as well as the voxels containing the vertices on at least one ray, are determined as visible voxels.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method for determining common-view keyframes as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method for determining common-view keyframes as described in any one of claims 1-7.

11. A computer program product, characterized in that, The method includes a computer program that, when executed by a processor, implements the method for determining common keyframes as described in any one of claims 1-7.

Images