Information processing method and apparatus for sparse reconstruction

Abstract

This disclosure provides an information processing method and apparatus for sparse reconstruction, comprising: updating the total number of currently registered images and updating an intermediate variable representing the increment of registered images when a new registered image is added during the sparse reconstruction process; performing coordinate space transformations under different modes based on the relationship between the total number of currently registered images and a first preset value, and the relationship between the intermediate variable and a second preset value; wherein the parameters used for coordinate space transformations under different modes are different. By performing coordinate space transformations under different modes based on the number of registered images in an incremental sparse reconstruction approach, the stability of values ​​is ensured during large-scale image space calculations, thereby overcoming the problem of numerical instability in large-scale image space calculations in related technologies.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and specifically to information processing methods and apparatus for sparse reconstruction. Background Technology

[0002] Incremental sparse reconstruction performs spatial calculations on the input image to ultimately calculate the pose and sparse point cloud of the registered image. This process requires constructing a local coordinate space upon which both the image pose and the sparse point cloud are built. Typically, this local coordinate space is constructed as follows: In the initial stage of incremental sparse reconstruction, two relevant images, ImageA and ImageB, are selected using methods such as epipolar constraints for spatial calculations. If the calculation is successful, one of the images (ImageA in this case) is usually chosen, with the center point of the camera lens as the origin and the direction the camera lens is facing as the positive Z-axis, to construct the coordinate space. New images are then continuously registered based on this foundation. This default coordinate system selection is highly random, and the relative positions of ImageA and other registered images can be significant. Therefore, the coordinate system established using ImageA may cause numerical stability issues during large-scale image space calculations.

[0003] Public content

[0004] The main objective of this disclosure is to provide an information processing method and apparatus for sparse reconstruction.

[0005] To achieve the above objectives, according to a first aspect of this disclosure, an information processing method for sparse reconstruction is provided, comprising updating the total number of currently registered images and updating an intermediate variable representing the increment of registered images if a new registered image is added during the sparse reconstruction process; performing coordinate space transformation under different modes based on the relationship between the total number of currently registered images and a first preset value, and the relationship between the intermediate variable and a second preset value; wherein the parameters used for coordinate space transformation under different modes are different.

[0006] Optionally, the coordinate space transformation in different modes includes: in the first mode, determining a first bounding box based on the projection center points in all currently registered images and the sparse point cloud determined in the sparse reconstruction process; and determining first data for coordinate space transformation based on the first bounding box, wherein the first data includes the coordinates of a first reference point for translation on which the coordinate space transformation is based and a first transformation scale value.

[0007] Optionally, the coordinate space transformation in different modes includes: in the second mode, determining a second bounding box based on the projection center point in all currently registered images; and determining second data for coordinate space transformation based on the second bounding box, wherein the second data includes the coordinates of a second reference point for translation on which the coordinate space transformation is based, and a second transformation scale value.

[0008] Optionally, based on the relationship between the total number of currently registered images and a first preset value, and the relationship between the intermediate variable and a second preset value, performing coordinate space transformation under different modes includes: if the total number of currently registered images is less than the first preset value, then determining whether the current intermediate variable is not less than the second preset value; if the current intermediate variable is not less than the second preset value, then performing coordinate space transformation under the first mode.

[0009] Optionally, based on the relationship between the total number of currently registered images and the first preset value, and the relationship between the intermediate variable and the second preset value, performing coordinate space transformation under different modes includes: if the total number of currently registered images is not less than the first preset value, then determining whether the current intermediate variable is not less than the second preset value; if the current intermediate variable is not less than the second preset value, then performing coordinate space transformation under the second mode.

[0010] Optionally, the method further includes resetting the current intermediate variable to zero after performing coordinate space transformations under different modes.

[0011] According to a second aspect of this disclosure, an information processing apparatus for sparse reconstruction is provided, comprising an increment unit configured to update the total number of currently registered images and an intermediate variable representing the increment of registered images if a new registered image is added during the sparse reconstruction process; and a calculation unit configured to perform coordinate space transformations under different modes based on the relationship between the total number of currently registered images and a first preset value, and the relationship between the intermediate variable and a second preset value; wherein the parameters used for coordinate space transformations under different modes are different.

[0012] Optionally, the coordinate space transformation in different modes includes: in a first mode, determining a first bounding box based on the projection center points in all currently registered images and the sparse point cloud determined during sparse reconstruction; and determining first data for coordinate space transformation based on the first bounding box, wherein the first data includes the coordinates of a first reference point for translation and a first transformation scale value on which the coordinate space transformation is based; in a second mode, determining a second bounding box based on the projection center points in all currently registered images; and determining second data for coordinate space transformation based on the second bounding box, wherein the second data includes the coordinates of a second reference point for translation and a second transformation scale value on which the coordinate space transformation is based.

[0013] According to a third aspect of this disclosure, a computer-readable storage medium is provided storing computer instructions for causing the computer to perform the method described in any implementation of the first aspect.

[0014] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the method described in any implementation of the first aspect.

[0015] The information processing method and apparatus for sparse reconstruction in this disclosure includes: updating the total number of currently registered images and updating an intermediate variable representing the increment of registered images when a new registered image is added during the sparse reconstruction process; performing coordinate space transformations under different modes based on the relationship between the total number of currently registered images and a first preset value, and the relationship between the intermediate variable and a second preset value; wherein the parameters used for coordinate space transformations under different modes are different. By performing coordinate space transformations under different modes based on the number of registered images in the auto-incrementing sparse reconstruction method, the stability of values ​​is ensured during large-scale image space calculation, thereby overcoming the problem of numerical instability in large-scale image space calculation processes in related technologies. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1This is a flowchart of an information processing method for sparse reconstruction according to an embodiment of the present disclosure;

[0018] Figure 2 This is an application scenario diagram of the information processing method for sparse reconstruction according to the embodiments of this disclosure;

[0019] Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present disclosure, the technical solutions of the present disclosure will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present disclosure.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] It should be understood that in the various embodiments of this disclosure, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0023] It should be understood that in this disclosure, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0024] It should be understood that in this disclosure, "multiple" refers to two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B, and C", "Contains A, B, and C" means that all three A, B, and C are contained; "Contains A, B, or C" means that one of A, B, and C is contained; "Contains A, B, and / or C" means that any one, two, or three of A, B, and C are contained.

[0025] It should be understood that in this disclosure, "B corresponding to A", "B corresponding to A", "A corresponds to B", or "B corresponds to A" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Matching A and B is defined as a similarity between A and B that is greater than or equal to a preset threshold.

[0026] Depending on the context, "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection."

[0027] The technical solutions of this disclosure will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] According to embodiments of this disclosure, a method for estimating camera parameters in a 3D modeling scene is provided, such as... Figure 1 As shown, the method includes the following steps 101 to 102:

[0030] Step 101: If a new registered image is added during the sparse reconstruction process, update the total number of currently registered images and the intermediate variable used to represent the increment of registered images.

[0031] In this embodiment, an intermediate variable NewM can be pre-set to represent the number of newly registered images during the sparse reconstruction process; the intermediate variable can be reset to zero after each coordinate space transformation. N can be used to represent the total number of currently registered images during the sparse reconstruction process.

[0032] For example, in incremental sparse reconstruction, two registration images can be selected initially. After successful registration, more registration images can be added continuously, and the pose information and sparse point cloud of the registration images are calculated and optimized during this process. When adding images, the intermediate variables can be incremented by one for each successfully registered image.

[0033] Step 102: Based on the relationship between the total number of currently registered images and the first preset value, and the relationship between the intermediate variable and the second preset value, perform coordinate space transformation under different modes; wherein the parameters used for coordinate space transformation under different modes are different.

[0034] In this embodiment, the total number of currently registered images can be less than a first preset value. If it is less than the first preset value, it means that the total number of currently registered images and the number of points in the sparse point cloud are relatively small. In this case, coordinate space transformation in the first mode can be performed.

[0035] If it is not less than the first preset value, it means that the total number of currently registered images and the number of points in the sparse point cloud are relatively large. In this case, the coordinate space transformation in the second mode can be performed.

[0036] Coordinate space transformations under different modes may include different reference points used by the image projection center during coordinate transformation, and may also include different transformation ratios.

[0037] As an optional implementation of this embodiment, the coordinate space transformation in different modes includes: in the first mode, determining a first bounding box based on the projection center points in all currently registered images and the sparse point cloud determined in the sparse reconstruction process; and determining first data for coordinate space transformation based on the first bounding box, wherein the first data includes the coordinates of a first reference point for translation on which the coordinate space transformation is based and a first transformation scale value.

[0038] In this optional implementation, in the first mode, each registered image corresponds to a projection center point, therefore multiple registered images can have multiple projection center points C. i Furthermore, during incremental sparse reconstruction, the sparse point cloud of the system can be calculated; based on all the projection center points and the points in the sparse point cloud, a minimum bounding box is established, and the height h of this bounding box can then be obtained. all Simultaneously, the centroid o of all points can be calculated based on this bounding box. all Based on the centroid o all For point p in a sparse point cloud j And the projection center point C in the registered image i Translation transformations, such as: p j’＝ p j -o all Ci’＝ C i -o all .

[0039] Furthermore, based on the height h of the bounding box all and the preset height threshold HT all Determine the transformation ratio, for example: scale = HT all / h all HT all The threshold range can be set as needed, for example [10, 50]. Then: p j’＝ p j* scale; C i’＝ C i* scale.

[0040] As an optional implementation of this embodiment, in the second mode, a second bounding box is determined based on the projection center point in all currently registered images; based on the second bounding box, second data for coordinate space transformation is determined, wherein the second data includes the coordinates of a second reference point for translation on which the coordinate space transformation is based, and a second transformation scale value.

[0041] In this optional implementation, the minimum bounding box can be determined only for the projection center point of the registered image, and the height h of the bounding box can be determined based on the minimum bounding box. c And based on the bounding box, all centroids o are calculated. c Based on the centroid o c For point p in a sparse point cloud j And the projection center point C in the registered image i Translation transformation, p j’＝ p j -o c C i’＝ C i -o c Based on the height h of the bounding box c and the preset height threshold HT c Determine the transformation ratio, for example: scale = HT c / h c HT all The threshold range can be set as needed, such as [5, 40], p j’＝ p j* scale; C i’＝ C i* scale.

[0042] As an optional implementation of this embodiment, the coordinate space transformation under different modes is performed based on the relationship between the total number of currently registered images and the first preset value, and the relationship between the intermediate variable and the second preset value. This includes: if the total number of currently registered images is less than the first preset value, then determining whether the current intermediate variable is not less than the second preset value; if the current intermediate variable is not less than the second preset value, then performing the coordinate space transformation under the first mode.

[0043] In this optional implementation, if it is less than the first preset value NT, then the current intermediate variable NewM can be compared with the second preset value, such as the set threshold NewMT. C The size relationship between them, if NewM is not less than NewMT C Then, the coordinate space transformation in the first mode can be performed. By ensuring that the image increment is greater than the second preset value before performing the coordinate space transformation, sufficient data can be guaranteed as the basis for calculation, thereby ensuring the accuracy of the coordinate space transformation. For example, the first preset value can be [100, 1000]; the set threshold NewMT C It can be [10, 500].

[0044] As an optional implementation of this embodiment, the coordinate space transformation under different modes is performed based on the relationship between the total number of currently registered images and the first preset value, and the relationship between the intermediate variable and the second preset value. This includes: if the total number of currently registered images is not less than the first preset value, then determining whether the current intermediate variable is not less than the second preset value; if the current intermediate variable is not less than the second preset value, then performing the coordinate space transformation under the second mode.

[0045] In this optional implementation, if it is less than the first preset value, then the current intermediate variable NewM can be compared with the set threshold NewMT. C The size relationship between them, if NewM is not less than NewMT C Then, the coordinate space transformation in the second mode can be performed. This ensures sufficient data as a basis for computation, thereby guaranteeing the accuracy of the coordinate space transformation. For example, the first preset value can be [100, 1000]; the set threshold NewMT C It can be [10, 500].

[0046] As an optional implementation of this embodiment, the method further includes setting the current intermediate variable to zero after performing coordinate space transformations under different modes.

[0047] In this optional implementation, after each spatial transformation is completed, the intermediate variables can be reset to zero, and the coordinate space transformation can be performed again.

[0048] refer to Figure 2 , Figure 2 An exemplary schematic diagram of the application of the method of this embodiment is shown, wherein "All-Points" is the aforementioned first mode; "Center-Points" is the aforementioned second mode. As can be seen from the figure, different execution modes are determined based on the total number of currently registered images and the image increment.

[0049] This embodiment ensures numerical stability in large-scale image space solving by performing coordinate space transformations under different conditions during the self-incremental sparse reconstruction process.

[0050] According to embodiments of this disclosure, an information processing apparatus for sparse reconstruction is also provided, including an increment unit configured to update the total number of currently registered images and an intermediate variable representing the increment of registered images if a new registered image is added during the sparse reconstruction process; and a calculation unit configured to perform coordinate space transformations under different modes based on the relationship between the total number of currently registered images and a first preset value, and the relationship between the intermediate variable and a second preset value; wherein the parameters used for coordinate space transformations under different modes are different.

[0051] As an optional implementation of this embodiment, the coordinate space transformation in different modes includes: in a first mode, determining a first bounding box based on the projection center points in all currently registered images and the sparse point cloud determined during sparse reconstruction; and determining first data for coordinate space transformation based on the first bounding box, wherein the first data includes the coordinates of a first reference point for translation and a first transformation scale value on which the coordinate space transformation is based; in a second mode, determining a second bounding box based on the projection center points in all currently registered images; and determining second data for coordinate space transformation based on the second bounding box, wherein the second data includes the coordinates of a second reference point for translation and a second transformation scale value on which the coordinate space transformation is based.

[0052] As an optional implementation of this embodiment, the coordinate space transformation under different modes is performed based on the relationship between the total number of currently registered images and the first preset value, and the relationship between the intermediate variable and the second preset value. This includes: if the total number of currently registered images is less than the first preset value, then determining whether the current intermediate variable is not less than the second preset value; if the current intermediate variable is not less than the second preset value, then performing the coordinate space transformation under the first mode.

[0053] As an optional implementation of this embodiment, the coordinate space transformation under different modes is performed based on the relationship between the total number of currently registered images and the first preset value, and the relationship between the intermediate variable and the second preset value. This includes: if the total number of currently registered images is not less than the first preset value, then determining whether the current intermediate variable is not less than the second preset value; if the current intermediate variable is not less than the second preset value, then performing the coordinate space transformation under the second mode.

[0054] As an optional implementation of this embodiment, the method further includes resetting the current intermediate variable to zero after performing coordinate space transformations under different modes.

[0055] This disclosure provides an electronic device, such as... Figure 3 As shown, the electronic device includes one or more processors 31 and a memory 32. Figure 3 Take a processor 31 as an example.

[0056] The controller may also include an input device 33 and an output device 34.

[0057] The processor 31, memory 32, input device 33, and output device 34 can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0058] Processor 31 can be a Central Processing Unit (CPU). Processor 31 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0059] The memory 32, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the control method in the embodiments of this disclosure. The processor 31 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 32, thereby implementing the method of the above-described method embodiments.

[0060] The memory 32 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of the processing device operated by the server. Furthermore, the memory 32 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 32 may optionally include memory remotely located relative to the processor 31, and these remote memories can be connected to a network connection device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0061] Input device 33 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the server's processing device. Output device 34 may include display devices such as a display screen.

[0062] One or more modules are stored in memory 32, and when executed by one or more processors 31, they perform actions such as... Figure 1 The method shown.

[0063] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0064] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An information processing method for sparse reconstruction, characterized in that, include: During sparse reconstruction, if a new registered image is added, update the total number of currently registered images and the intermediate variable used to represent the increment of registered images; Based on the relationship between the total number of currently registered images and the first preset value, and the relationship between the intermediate variable and the second preset value, coordinate space transformations are performed under different modes; wherein the parameters used for coordinate space transformations are different under different modes. The coordinate space transformation under different modes includes: in the first mode, determining a first bounding box based on the projection center point in all currently registered images and the sparse point cloud determined in the sparse reconstruction process; and determining first data for coordinate space transformation based on the first bounding box, wherein the first data includes the coordinates of a first reference point for translation on which the coordinate space transformation is based, and a first transformation scale value. The coordinate space transformation under different modes includes: in the second mode, determining a second bounding box based on the projection center point in all currently registered images; and determining second data for coordinate space transformation based on the second bounding box, wherein the second data includes the coordinates of a second reference point for translation on which the coordinate space transformation is based, and a second transformation scale value. Wherein, when determining the first data for coordinate space transformation based on the first bounding box, or when determining the second data for coordinate space transformation based on the second bounding box, the method includes: The centroids of all points calculated based on the bounding box o all For points in sparse point clouds p j and the projection center point in the registered image C i Translation transformation p j’= p j - o all ; C i’= C i - o all Based on the height of the bounding box h all and preset height threshold HT all Determine the transformation ratio.

2. The information processing method for sparse reconstruction according to claim 1, characterized in that, Based on the relationship between the total number of currently registered images and the first preset value, and the relationship between the intermediate variable and the second preset value, coordinate space transformations under different modes are performed, including: If the total number of currently registered images is less than a first preset value, then determine whether the current intermediate variable is not less than a second preset value; If the current intermediate variable is not less than the second preset value, then the coordinate space transformation in the first mode is performed.

3. The information processing method for sparse reconstruction according to claim 1, characterized in that, Based on the relationship between the total number of currently registered images and the first preset value, and the relationship between the intermediate variable and the second preset value, coordinate space transformations under different modes are performed, including: If the total number of currently registered images is not less than a first preset value, then determine whether the current intermediate variable is not less than a second preset value; If the current intermediate variable is not less than the second preset value, then the coordinate space transformation in the second mode is performed.

4. The information processing method for sparse reconstruction according to claim 1, characterized in that, The method further includes resetting the current intermediate variable to zero after performing coordinate space transformations under different modes.

5. An information processing device for sparse reconstruction, characterized in that, include: The increment unit is configured to update the total number of registered images and the intermediate variable used to represent the increment of registered images if a new registered image is added during the sparse reconstruction process. The calculation unit is configured to perform coordinate space transformations under different modes based on the relationship between the total number of currently registered images and a first preset value, and the relationship between the intermediate variable and a second preset value; wherein the parameters used for coordinate space transformations are different under different modes. The coordinate space transformation under different modes includes: in the first mode, determining a first bounding box based on the projection center point in all currently registered images and the sparse point cloud determined in the sparse reconstruction process; and determining first data for coordinate space transformation based on the first bounding box, wherein the first data includes the coordinates of a first reference point for translation on which the coordinate space transformation is based, and a first transformation scale value. The coordinate space transformation under different modes includes: in the second mode, determining a second bounding box based on the projection center point in all currently registered images; and determining second data for coordinate space transformation based on the second bounding box, wherein the second data includes the coordinates of a second reference point for translation on which the coordinate space transformation is based, and a second transformation scale value. Wherein, when determining the first data for coordinate space transformation based on the first bounding box, or when determining the second data for coordinate space transformation based on the second bounding box, the method includes: The centroids of all points calculated based on the bounding box o all For points in sparse point clouds p j and the projection center point in the registered image C i Translation transformation p j’= p j - o all ; C i’= C i - o all Based on the height of the bounding box h all and preset height threshold HT all Determine the transformation ratio.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the information processing method for sparse reconstruction as described in any one of claims 1-4.

7. An electronic device, characterized in that, include: At least one processor; The at least one processor is also connected in communication with a memory, wherein the memory stores a computer program that can be executed by the at least one processor to cause the at least one processor to perform the information processing method for sparse reconstruction as described in any one of claims 1-4.

Images