Grid generation method, device, electronic device and computer readable storage medium

Abstract

The present disclosure provides a grid generation method, device, electronic device and computer-readable storage medium. The present disclosure determines the voxel corresponding to the pixel in the preset voxel space based on the image coordinates and depth value of the pixel in the current depth image; determines the incremental structured data information corresponding to the neighboring voxel based on the neighborhood voxel of the voxel and the structured data corresponding to the previous depth image; and finally generates the target grid corresponding to the current depth image based on the incremental structured data information corresponding to the neighborhood voxel. The present disclosure extracts the grid and generates the target grid based on the incremental structured data information corresponding to the voxel, rather than extracting the grid for all structured data, which can reduce the amount of calculation for grid extraction, reduce the time consumption for grid extraction, and effectively improve the efficiency of grid extraction.

Description

[0001] Technology Neighborhood

[0002] The present disclosure relates to the field of image processing technology, and in particular to a grid generation method, device, electronic device and computer-readable storage medium. Background Art

[0003] Augmented reality is an important topic in the field of 3D perception, and using dense grids in augmented reality applications can more completely describe 3D scene information. Based on known dense grids, some common 3D scene interactions or special effects can be achieved, such as object occlusion, collision, shadow mapping, etc. The traditional dense grid generation algorithm sequentially fuses depth maps of different perspectives into structured data according to the internal and external parameters of the camera, and extracts the corresponding dense grid.

[0004] Traditional dense grid generation algorithms extract grids from all structured data after fusing each depth image into structured data, which is computationally intensive, time-consuming, and inefficient. Summary of the invention

[0005] The embodiments of the present disclosure at least provide a grid generation method and device.

[0006] In a first aspect, an embodiment of the present disclosure provides a grid generation method, comprising:

[0007] Acquire a current depth image and structured data corresponding to a previous depth image used for mesh generation before the current depth image;

[0008] Determining a voxel corresponding to the pixel in a preset voxel space based on the image coordinates and the depth value of the pixel in the current depth image;

[0009] Determine, based on the structured data corresponding to the neighboring voxels of the voxel and the previous depth image, the structured data increment information corresponding to the neighboring voxels of the voxel;

[0010] Based on the structured data increment information corresponding to the neighborhood voxels of the voxel, a target grid corresponding to the current depth image is generated.

[0011] In this regard, meshes are extracted and target meshes are generated based on the incremental information of structured data corresponding to voxels, rather than extracting meshes for all structured data. This can reduce the amount of computational effort and time spent on mesh extraction, and effectively improve mesh extraction efficiency.

[0012] In a possible implementation, determining the incremental structured data information corresponding to the neighboring voxels of the voxel based on the neighboring voxels of the voxel and the structured data corresponding to the previous depth image includes:

[0013] Determining bias information corresponding to the neighborhood voxel based on the voxel coordinates of the neighborhood voxel in the preset voxel space;

[0014] Determining a key value of the neighborhood voxel based on the bias information of the neighborhood voxel;

[0015] Based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image, the structured data increment information corresponding to the neighborhood voxel of the voxel is determined.

[0016] In this implementation, using the offset information to generate the key value of the voxel can avoid hash collision when querying voxels outside the voxel space. In addition, by comparing the key value of the voxel with the structured data corresponding to the previous depth image, more accurate structured data increment information can be determined.

[0017] In a possible implementation, the structured data corresponding to the front depth image includes key values ​​of voxels corresponding to the front depth image;

[0018] The determining, based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image, structured data increment information corresponding to the neighborhood voxel of the voxel includes:

[0019] In response to a key value of the neighborhood voxel being included in the structured data, projecting the neighborhood voxel onto the current depth image;

[0020] In response to the projection of the neighborhood voxel onto the current depth image, and when the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image is within a preset threshold range, based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image, determine the incremental information of the structured data corresponding to the neighborhood voxel; or

[0021] In response to the key value of the neighborhood voxel not being included in the structured data, incremental structured data information corresponding to the neighborhood voxel is determined based on voxel depth information corresponding to the neighborhood voxel.

[0022] This implementation method can relatively accurately determine the structured data increment information by using the projection result of the neighborhood voxels and the voxel depth information and / or pixel depth information corresponding to the voxels.

[0023] In a possible implementation manner, the structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel;

[0024] The determining, based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image, the structured data increment information corresponding to the neighborhood voxel comprises:

[0025] Taking the difference between the voxel depth information corresponding to the neighborhood voxel and the pixel depth information corresponding to the neighborhood voxel as the increment value of the truncated signed distance function corresponding to the neighborhood voxel;

[0026] The state information of the neighborhood voxel is set to an updated state.

[0027] In a possible implementation manner, the structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel;

[0028] The determining, based on the voxel depth information corresponding to the neighborhood voxel, the structured data increment information corresponding to the neighborhood voxel comprises:

[0029] Using the voxel depth information as an increment value of a truncated signed distance function corresponding to the neighborhood voxel;

[0030] The state information of the neighborhood voxel is set to an added state.

[0031] In this implementation, the structured data increment information can be determined more accurately using the voxel depth information and / or pixel depth information corresponding to the voxel.

[0032] In a possible implementation, the determining, based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image, the structured data increment information corresponding to the neighborhood voxel of the voxel further includes:

[0033] In response to the neighborhood voxel not being projected onto the current depth image, or the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image being not within the preset threshold range, the state information of the neighborhood voxel is set to a deleted state.

[0034] In this implementation, by setting voxels that cannot be projected onto the current depth image or voxels whose depth information is not within a preset threshold range to a deletion state, voxels with low accuracy can be deleted, which is beneficial to improving the quality of the generated mesh.

[0035] In a possible implementation, the bias information includes first bias information and second bias information of a corresponding voxel;

[0036] The determining, based on the voxel coordinates of the neighborhood voxels in the preset voxel space, the bias information corresponding to the neighborhood voxels includes:

[0037] Filter the coordinate item with the largest absolute value from the voxel coordinates;

[0038] Determining the first offset information based on the coordinate item with the largest absolute value and the number of unit voxels included in the preset voxel space;

[0039] The second offset information is determined based on the first offset information and the number of unit voxels included in the preset voxel space.

[0040] This embodiment can generate offset information more accurately based on voxel coordinates.

[0041] In a possible implementation, the generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel includes:

[0042] Determine target structural data of the voxel based on the structural data increment information of the voxel and the structural data;

[0043] Based on the target structured data of the voxels, a target grid corresponding to the current depth image is generated.

[0044] In this implementation, the structured data increment information is added to the original structured data to generate target structured data, and a relatively accurate target grid can be generated using the target structured data.

[0045] In a possible implementation manner, the structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel;

[0046] The generating a target grid corresponding to the current depth image based on the target structured data of the voxels includes:

[0047] In response to the state information of the voxel of the neighborhood voxel being a preset state, and when it is determined that the number of times the state information of the voxel of the neighborhood voxel is the preset state is greater than a preset number, generating an incremental grid corresponding to the voxel based on the target structure data of the voxel;

[0048] Based on the incremental grid corresponding to the voxel and the grid corresponding to the previous depth image, a target grid corresponding to the current depth image is generated.

[0049] This implementation method extracts a grid for the corresponding target structured data only when the number of updates of the state information to the preset state is greater than a preset number, thereby avoiding repeated extraction of grids, reducing the amount of calculation for grid extraction, saving computing resources, and being conducive to improving the efficiency of grid extraction and reducing the noise of grid extraction.

[0050] In a possible implementation, the above-mentioned grid generation method further includes:

[0051] Obtaining optimized posture parameters corresponding to the shooting device that shoots the depth image and the posture parameters when the shooting device shoots each depth image;

[0052] For each depth image, determining a posture change value based on the posture parameters corresponding to the depth image and the optimized posture parameters;

[0053] In response to the posture change value being greater than a preset posture change threshold, the incremental structured data information of the voxel corresponding to the depth image is removed from the target structured data, and based on the optimized posture parameter, the final incremental structured data information of the voxel corresponding to the depth image is determined;

[0054] Based on the final structured data increment information of the voxels corresponding to the depth image, the structured data corresponding to the depth image is determined.

[0055] In this implementation, when the posture of the shooting device changes significantly, the structured data incremental information generated based on the unoptimized posture parameters will be removed from the target structured data, and new and more accurate structured data incremental information will be generated based on the optimized posture parameter information. Based on the structured data incremental information, more accurate structured data can be generated, which helps to reduce the phenomenon of grid layer misalignment.

[0056] In a possible implementation, the structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel;

[0057] The grid generation method further comprises:

[0058] In response to removing the incremental structured data information of the voxel corresponding to the depth image from the target structured data, when the truncated signed distance function value of the voxel corresponding to the depth image is less than zero, the state information of the voxel corresponding to the depth image is set to a deleted state.

[0059] In this implementation, voxels with truncated signed distance function values ​​less than zero are set to a deleted state, which can delete voxels with low accuracy, thereby improving the quality of the generated grid and reducing the grid layering misalignment phenomenon.

[0060] In a possible implementation manner, before generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel, the method further includes:

[0061] Filtering target voxels whose structured data increment information is empty from the voxels corresponding to the current depth image;

[0062] Projecting the target voxel onto the current depth image;

[0063] In response to the target voxel being projected onto the current depth image, and when the voxel depth information obtained by projecting the target voxel onto the current depth image is within a preset threshold range, the structured data incremental information corresponding to the target voxel is determined based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image.

[0064] This implementation method uses the unupdated structured data and depth images to determine the moving object, and updates the corresponding structured data increment information, which can improve the accuracy of mesh extraction.

[0065] In a possible implementation manner, the structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel;

[0066] The determining, based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image, the structured data increment information corresponding to the target voxel comprises:

[0067] In response to the case where a difference value obtained by subtracting the pixel depth information corresponding to the target voxel from the voxel depth information corresponding to the target voxel is less than zero, using the difference value obtained by subtracting the pixel depth information from the voxel depth information as a truncated signed distance function increment value corresponding to the target voxel;

[0068] The state information of the target voxel is set to an update state.

[0069] In this implementation, the structured data increment information of voxels whose difference between voxel depth information and corresponding pixel depth information is less than zero is updated, which essentially updates the structured data increment information of the moving object, and can improve the accuracy of mesh extraction.

[0070] In a possible implementation manner, before determining the voxel corresponding to the pixel in the preset voxel space based on the image coordinates of the pixel in the current depth image, the method further includes:

[0071] For each pixel in the current depth image, calculate a pixel depth difference between a pixel depth of the pixel and a pixel depth of a neighboring pixel of the pixel;

[0072] In response to the pixel depth difference corresponding to the pixel not being within the preset depth difference range, the pixel is set to be invalid.

[0073] This implementation method performs a neighborhood pixel depth consistency check on pixels in the current depth image, removes noise in the current depth image, and helps to improve the accuracy of grid extraction.

[0074] In a second aspect, the present disclosure also provides a grid generation device, including:

[0075] A data acquisition module, used to acquire a current depth image and structured data corresponding to a previous depth image used for grid generation before the current depth image;

[0076] A voxel determination module, configured to determine a voxel corresponding to a pixel in a preset voxel space based on the image coordinates and the depth value of the pixel in the current depth image;

[0077] An incremental information determination module, configured to determine incremental structured data information corresponding to neighboring voxels of the voxel based on the neighboring voxels of the voxel and the structured data corresponding to the previous depth image;

[0078] The grid generation module is used to generate a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel.

[0079] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0080] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0081] For a description of the effects of the above-mentioned grid generation device, electronic device, and computer-readable storage medium, please refer to the description of the above-mentioned grid generation method, which will not be repeated here.

[0082] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0084] Figure 1 A flow chart of a grid generation method provided by an embodiment of the present disclosure is shown;

[0085] Figure 2 A schematic diagram of a neighborhood voxel in an embodiment of the present disclosure is shown;

[0086] Figure 3 A schematic diagram of a preset voxel space in an embodiment of the present disclosure is shown;

[0087] Figure 4 A schematic diagram showing a grid extracted in an embodiment of the present disclosure is shown;

[0088] Figure 5 A schematic diagram of a grid generation device provided by an embodiment of the present disclosure is shown;

[0089] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by technicians in the field without making creative work belong to the scope of protection of the present disclosure.

[0091] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0092] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0093] In the prior art, when extracting a grid, after each depth image is fused into the structured data, the grid is extracted for all the structured data, which is computationally intensive, time-consuming, and inefficient. To address this defect, the embodiments of the present disclosure provide a grid generation method, device, electronic device, and computer-readable storage medium. The embodiments of the present disclosure extract the grid and generate the target grid based on the incremental information of the structured data corresponding to the voxels, rather than extracting the grid for all the structured data, which can reduce the computational complexity of the grid extraction, reduce the time consumption of the grid extraction, and effectively improve the efficiency of the grid extraction.

[0094] The grid generation method, device, electronic device and storage medium disclosed in the present disclosure are described below through specific embodiments.

[0095] like Figure 1 As shown, the embodiment of the present disclosure discloses a grid generation method, which can be applied to a device such as a server that needs to generate a grid, and is used to generate a target grid based on a depth image. Specifically, the grid generation method may include the following steps:

[0096] S110: Acquire a current depth image and structured data corresponding to a previous depth image used for mesh generation before the current depth image.

[0097] The depth image may be taken by a depth camera and transmitted to a device that executes the mesh generation method. The previous depth image may include one or more depth images, which are images that generate structured data before the current depth image and perform mesh extraction based on the extracted structured data.

[0098] This embodiment describes how to extract a grid from a current depth image. The method for extracting a grid from a previous depth image and the method for extracting a grid from a current depth image may be the same.

[0099] The "grid" in the grid extraction and generated target grid in this embodiment refers to a dense grid. The extracted or generated dense grid can be used to generate augmented reality AR images or AR special effects.

[0100] When generating or extracting a grid, the structured data corresponding to the depth image is first fused into the structured data obtained after the previous depth image fusion, and then the grid extraction operation is performed on the fused structured data to obtain a grid. Therefore, when the previous depth image generates a grid, the corresponding structured data and the fused structured data are first generated.

[0101] S120. Determine, based on the image coordinates of a pixel in the current depth image, a voxel corresponding to the pixel in a preset voxel space.

[0102] Here, if Figure 3 As shown, the preset voxel space can be flexibly set according to the actual application scenario. For example, the preset voxel space can be set to be composed of N 3 The voxel space is composed of unit voxels, N = 500. The voxel coordinate system range of the voxel space is [-N / 2, N / 2], and the actual size of the unit voxel is ζ, ζ = 0.06 meters. The actual size of the preset voxel space is N*ζ.

[0103] Figure 3 In the figure, the voxel located in the preset voxel space is voxel 301, and the voxel located outside the preset voxel space is voxel 302.

[0104] When specifically determining the voxel of a pixel, the following steps can be used:

[0105] When determining the voxel coordinates of a pixel, it is necessary to combine the depth value of the pixel. Specifically, the pixel coordinates p(i, j) of the pixel are obtained, and the depth is d(i, j). Using the internal and external parameters of the camera that captured the current depth image, the pixel is projected into the world coordinate system to obtain the three-dimensional coordinates Then the three-dimensional coordinates Projection to voxel coordinates Get the voxel coordinates (x,y,z).

[0106] To save memory, we only need to manage the transformed voxels instead of maintaining the entire preset voxel space.

[0107] S130 . Determine structured data increment information corresponding to the neighborhood voxels of the voxel based on the neighborhood voxels of the voxel and the structured data corresponding to the previous depth image.

[0108] like Figure 2 As shown, the neighborhood voxels of the above-mentioned certain voxel may include voxels 201 located on the eight vertices of the cube corresponding to the unit voxel where the voxel is located.

[0109] In a specific embodiment, the structured data incremental information may include the state information of the corresponding voxel and the truncated signed distance function tsdf incremental value of the corresponding voxel. The above-mentioned state information may include: adding state, updating state and deleting state. The truncated signed distance function incremental value may be the difference between the voxel depth information obtained by projecting a certain voxel onto the current depth image and the pixel depth information of the pixel corresponding to the voxel.

[0110] When determining the above-mentioned structured data incremental information, the key value of the neighborhood voxel can be calculated first, and then it can be determined whether the calculated key value exists in the structured data obtained by the previous depth image fusion. If not, the structured data of the neighborhood voxel is used as the structured data incremental information. If it exists, the structured incremental information is determined based on the voxel coordinates of the voxel and the corresponding pixel coordinates.

[0111] S140 . Generate a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel.

[0112] Here, the incremental structured data information of a certain voxel can be fused with the structured data corresponding to the voxel in the structured data obtained by fusion of the previous depth image to obtain the target structured data of the voxel, and then the target grid is generated based on the grid obtained by grid extraction of the target structured data.

[0113] In a specific implementation, the incremental information of the structured data may be added to the structured data corresponding to the corresponding voxel by weighted summation to obtain the target structured data.

[0114] The structured data incremental information is added to the original structured data to generate target structured data, and a more accurate target grid can be generated using the target structured data.

[0115] In some embodiments, the above-mentioned determining the incremental structured data information corresponding to the neighboring voxels of the voxel based on the neighboring voxels of the voxel and the structured data corresponding to the previous depth image can be specifically implemented by the following steps:

[0116] Step 11: Determine the offset information corresponding to the neighborhood voxel based on the voxel coordinates of the neighborhood voxel in the preset voxel space.

[0117] In a specific implementation, the above-mentioned bias information may include first bias information and second bias information of the neighborhood voxels. Specifically, it can be implemented by the following sub-steps:

[0118] Sub-step 111 : Filter the coordinate item with the largest absolute value from the voxel coordinates of the neighborhood voxels.

[0119] Here, we can use the following formula to select the coordinate item with the largest absolute value:

[0120]

[0121] Indicates selecting the coordinate item with the largest absolute value.

[0122] Sub-step 112: determining the first offset information based on the coordinate item with the largest absolute value and the number of unit voxels included in the preset voxel space.

[0123] exist When , the first bias information can be calculated using the following formula:

[0124]

[0125] In the formula, s l Indicates the first bias information.

[0126] When , the first bias information can be calculated using the following formula:

[0127]

[0128] Sub-step 113: determining the second offset information based on the first offset information and the number of unit voxels included in the preset voxel space.

[0129] Here, the second bias information can be calculated using the following formula:

[0130]

[0131] In the formula, s G represents the second bias information, and k represents the index value of the voxel.

[0132] This embodiment can generate offset information more accurately based on voxel coordinates.

[0133] Step 12: Determine the key value of the neighborhood voxel based on the bias information of the neighborhood voxel.

[0134] In specific implementation, the key value of the neighborhood voxel can be calculated using the following formula:

[0135]

[0136] in,

[0137] In the formula, The key value representing the neighborhood voxel.

[0138] The above key value calculation method can avoid hash collision when querying voxels outside the preset voxel space, and the query range is not only limited to the preset voxel space, but can be voxels converted from any depth image, which means that grid generation is no longer limited by the set generation range.

[0139] Step 13: Determine the incremental structured data information corresponding to the neighborhood voxels of the voxel based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image.

[0140] Here, the structured data includes the key value of the voxel corresponding to the previous depth image. After obtaining the key value of the neighborhood voxel corresponding to the current depth image, determine whether the key value of the neighborhood voxel is included in the structured data. If the key value of the neighborhood voxel is included in the structured data, project the neighborhood voxel onto the current depth image. Afterwards, if the neighborhood voxel is projected onto the current depth image, and the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image is within a preset threshold range, then based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image, determine the incremental structured data information corresponding to the neighborhood voxel.

[0141] Specifically, the difference between the voxel depth information corresponding to the neighborhood voxel and the pixel depth information corresponding to the neighborhood voxel is used as the increment value of the truncated signed distance function corresponding to the neighborhood voxel; and the state information of the neighborhood voxel is set to an updated state.

[0142] The above preset threshold range can be flexibly set according to the specific application scenario, for example, it can be set to [0.4m, 2.5m].

[0143] If the key value of the neighborhood voxel is not included in the structured data, the incremental structured data information corresponding to the neighborhood voxel is determined based on the voxel depth information corresponding to the neighborhood voxel.

[0144] Specifically, the voxel depth information is used as the incremental value of the truncated signed distance function corresponding to the neighborhood voxel; and the state information of the neighborhood voxel is set to an adding state.

[0145] By using voxel depth information and / or pixel depth information corresponding to the voxels, the structured data increment information can be determined more accurately.

[0146] In response to the neighborhood voxel not being projected onto the current depth image, or the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image being not within the preset threshold range, the state information of the neighborhood voxel is set to a deleted state.

[0147] By setting voxels that cannot be projected onto the current depth image or voxels whose depth information is not within a preset threshold range to a deletion state, voxels with low accuracy can be deleted, which is beneficial to improving the quality of the generated mesh.

[0148] The above embodiment utilizes the projection result of the neighborhood voxels and the voxel depth information and / or pixel depth information corresponding to the voxels to relatively accurately determine the structured data increment information.

[0149] Using the bias information to generate the key value of the voxel can avoid hash collision when querying voxels outside the voxel space. In addition, by comparing the key value of the voxel with the structured data corresponding to the previous depth image, more accurate incremental information of the structured data can be determined.

[0150] After the above steps are completed, a counter can be used to record the update status of each neighborhood voxel state information. Specifically, if the state information of a neighborhood voxel is in the adding state, the counter of the neighborhood voxel is set to 1; if the state information of a neighborhood voxel is in the updating state, the counter of the neighborhood voxel is increased by 1 based on the original value; if the state information of a neighborhood voxel is in the deleting state, the counter of the neighborhood voxel is set to 0.

[0151] Based on the value of the neighborhood voxel recorded by the counter, it is determined whether to perform grid extraction on the neighborhood voxel, which can be specifically achieved by the following steps:

[0152] Step 21, in response to the state information of the voxel of the neighborhood voxel being a preset state, such as an adding state or an updating state, and determining that the number of times the state information of the voxel of the neighborhood voxel is the preset state is greater than a preset number, that is, when the value in the counter corresponding to the neighborhood voxel is greater than the preset number, generate an incremental grid corresponding to the voxel based on the target structured data of the voxel.

[0153] The above preset number of times can be flexibly set according to the specific application scenario, for example, it can be set to 3.

[0154] When extracting the grid from the target structured data, the Marching Cubes algorithm can be used to implement it.

[0155] After extracting the grid for a certain voxel, the state information of the voxel is set to a normal state.

[0156] Step 22: Generate a target grid corresponding to the current depth image based on the incremental grid corresponding to the voxel and the grid corresponding to the previous depth image.

[0157] Specifically here, the incremental grid generated by each voxel is added to the grid generated by the previous depth image to obtain the target grid.

[0158] like Figure 4 As shown, the incremental grid 401 is added to the existing grid 402, and the area of ​​the grid becomes larger and larger.

[0159] When the number of updates of the state information to the preset state is greater than the preset number, the grid is extracted for the corresponding target structured data, which avoids repeated grid extraction, reduces the amount of calculation for grid extraction, saves computing resources, and is conducive to improving the efficiency of grid extraction and reducing the noise of grid extraction.

[0160] If there is a voxel in the neighborhood voxels of the voxel whose status information is a deleted state, the grid is not extracted for the voxel.

[0161] The pose of the camera that captures the depth image may change. If the pose of the camera is optimized and has changed significantly from the pose used for mesh extraction, the previously generated structured data needs to be deleted or updated. This can be achieved using the following steps:

[0162] Step 31: Obtain optimized posture parameters corresponding to the shooting device for shooting the depth image and the posture parameters when the shooting device shoots each depth image.

[0163] In a specific implementation, the external parameters of a shooting device, such as a camera, can be expressed as T i t The external parameters include six parameters, the first three parameters are the translation parameters of the camera, and the last three parameters are the rotation parameters of the camera. The above i represents the frame number of the depth image, and t represents the time sequence number of the depth image.

[0164] Step 32: For each depth image, determine a posture change value based on the posture parameters corresponding to the depth image and the optimized posture parameters.

[0165] After receiving the optimized pose parameters, the pose change value is calculated based on the pose parameters corresponding to each frame of the depth image. Specifically, the pose change value can be calculated using the following formula:

[0166] ΔT i t′ =||T i t′ -T i t ||2

[0167] Where, T i t′ represents the optimized pose parameters, and ||*||2 represents the L2 norm.

[0168] Step 33: In response to the posture change value being greater than a preset posture change threshold, the incremental structured data information of the voxels corresponding to the depth image is removed from the target structured data, and based on the optimized posture parameters, the final incremental structured data information of the voxels corresponding to the depth image is determined.

[0169] The preset posture change threshold can be flexibly set according to the specific application scenario, for example, it can be set to 0.01.

[0170] When removing the incremental structured data information, the pose parameters corresponding to the depth image can be used to determine the voxel coordinates of each voxel in the depth image, and then the incremental structured data information corresponding to each voxel can be calculated using the same method as in the above embodiment, and the calculated incremental structured data information can be deleted from the target structured data. Alternatively, the incremental structured data information corresponding to the stored depth image can be directly used to delete the incremental structured data information from the target structured data.

[0171] After the incremental structured data information of the voxel corresponding to the depth image is removed from the target structured data, if the truncated signed distance function value of the voxel corresponding to the depth image is less than zero, the state information of the voxel corresponding to the depth image is set to a deleted state.

[0172] Setting the voxels whose truncated signed distance function value is less than zero to a deleted state can delete the voxels with low accuracy, which is beneficial to improving the quality of the generated grid and reducing the grid layering and misalignment phenomenon.

[0173] Afterwards, the voxel coordinates of each voxel in the depth image are determined by optimizing the posture parameters, and the structured data increment information corresponding to each voxel is calculated by the same method as the above embodiment.

[0174] Step 34: Determine the structured data corresponding to the depth image based on the final structured data increment information of the voxels corresponding to the depth image.

[0175] Here, the incremental information of the structured data may be added to the structured data corresponding to the corresponding voxel by weighted summation.

[0176] In the above-mentioned embodiment, when the posture of the shooting device changes significantly, the structured data incremental information generated according to the unoptimized posture parameters will be eliminated from the target structured data, and new and more accurate structured data incremental information will be generated based on the optimized posture parameter information. Based on the structured data incremental information, more accurate structured data can be generated, which helps to reduce the phenomenon of grid layer misalignment.

[0177] The mesh generation technology in the prior art does not perform special processing on the dynamic objects in the scene, and cannot update the dynamic objects in the scene in real time, and the generated mesh is not accurate enough. The embodiment of the present disclosure can specifically use the following steps to implement the processing of dynamic objects:

[0178] Before generating the target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel, the target voxels with empty structured data increment information are screened from the voxels corresponding to the current depth image; and then the target voxels are projected onto the current depth image. When the target voxel is projected onto the current depth image, and the voxel depth information obtained by projecting the target voxel onto the current depth image is within a preset threshold range, the structured data increment information corresponding to the target voxel is determined based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image.

[0179] When projecting the target voxel onto the current depth image, this can be achieved by utilizing the posture parameters of the shooting device.

[0180] The above preset threshold range can be flexibly set according to the specific application scenario, for example, it can be set to [0.4m, 2.5m].

[0181] The above-mentioned determining the structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image may specifically be:

[0182] In response to the situation that a difference value obtained by subtracting pixel depth information corresponding to the target voxel from voxel depth information corresponding to the target voxel is less than zero, the difference value obtained by subtracting pixel depth information from voxel depth information is used as an increment value of a truncated signed distance function corresponding to the target voxel; and the state information of the target voxel is set to an updated state.

[0183] Updating the structured data increment information of voxels whose difference between voxel depth information and corresponding pixel depth information is less than zero essentially updates the structured data increment information of the moving object, which can improve the accuracy of mesh extraction.

[0184] When processing dynamic objects, it should be noted that although the state information of the voxel is updated, the value in the counter corresponding to the voxel is not modified.

[0185] The mesh generation technology in the prior art is mostly used to process high-precision depth images collected by depth sensors. For depth images with strong noise and poor quality, the generated mesh has a lot of noise. The embodiment of the present disclosure can perform noise reduction processing before processing the depth image to improve the accuracy of the generated mesh, which can be achieved by the following steps:

[0186] Step 41: Before determining the corresponding voxel of the pixel in the preset voxel space based on the image coordinates of the pixel in the current depth image, for each pixel in the current depth image, calculate the pixel depth difference between the pixel depth and the pixel depth of the neighboring pixels of the pixel.

[0187] The above-mentioned neighborhood pixels may include pixels within a certain range adjacent to the pixel. For example, if the coordinates of a certain pixel are p(i, j), the neighborhood pixels of the pixel may be pixels with coordinates within {i+Δi, j+Δj}. Where Δi∈(-R, R), Δj∈(-R, R), and the neighborhood radius may be set to 3 pixels.

[0188] Step 42: In response to the pixel depth difference corresponding to the pixel not being within a preset depth difference range, the pixel is set to be invalid.

[0189] The preset depth difference value may be set to a fixed value according to a specific scenario, or may be determined according to the depth information of the pixel. For example, the preset depth difference value may be set to 15% of the depth value corresponding to the pixel.

[0190] Performing a neighborhood pixel depth consistency check on pixels in the current depth image and removing noise from the current depth image helps improve the accuracy of mesh extraction.

[0191] The grid generation method in the above embodiment extracts the grid from the corresponding target structured data and merges it into the previous grid only when the cumulative number of updates of the status information in each depth image reaches a certain value, thereby avoiding repeated access to all structured data, and the noise of the generated grid is also greatly reduced. The above embodiment reduces the amount of structured data to be maintained, helps to extract the grid, and does not cause voxel access conflicts. It can be arbitrarily extended beyond the preset voxel space without setting the grid generation range in advance. Further, the above embodiment solves the problem of high memory usage in the traditional method, and even low-end mobile phones can run the above grid generation method in real time. Further, in the process of generating structured data, it is determined whether there are dynamic objects in the depth image. If there are, the corresponding incremental information of the structured data will be updated and eliminated, which can improve the accuracy of the generated grid. Further, before processing the depth image, the above embodiment performs a consistency check to eliminate the noise in the depth image, thereby reducing the noise of the generated grid. Furthermore, the above-mentioned embodiment reintegrates the frames with large changes in optimized posture parameters, that is, deletes the incremental information of the structured data integrated with the old posture parameters of the frame, and uses the optimized posture parameters of the frame, and reintegrates the optimized posture parameters into the incremental information of the structured data, which helps to reduce the grid stratification phenomenon.

[0192] Corresponding to the above-mentioned grid generation method, the present disclosure also discloses a grid generation device, each module in the device can implement each step in the grid generation method of each embodiment above, and can achieve the same beneficial effect, so the same parts will not be repeated here. Specifically, Figure 5 As shown, the grid generating device comprises:

[0193] The data acquisition module 510 is used to acquire a current depth image and a structured number corresponding to a previous depth image used for mesh generation before the current depth image.

[0194] The voxel determination module 520 is configured to determine the voxel corresponding to the pixel in the preset voxel space based on the image coordinates and the depth value of the pixel in the current depth image.

[0195] The incremental information determination module 530 is used to determine the incremental information of the structured data corresponding to the neighboring voxels of the voxel based on the neighboring voxels of the voxel and the structured data corresponding to the previous depth image.

[0196] The grid generation module 540 is used to generate a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel.

[0197] In some embodiments, when determining the incremental information corresponding to the neighboring voxels of the voxel based on the neighboring voxels of the voxel and the structured data corresponding to the previous depth image, the incremental information determining module 530 is configured to:

[0198] Determining bias information corresponding to the neighborhood voxel based on the voxel coordinates of the neighborhood voxel in the preset voxel space;

[0199] Determining a key value of the neighborhood voxel based on the bias information of the neighborhood voxel;

[0200] Based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image, the structured data increment information corresponding to the neighborhood voxel of the voxel is determined.

[0201] In some embodiments, the structured data includes key values ​​of voxels corresponding to the front depth image;

[0202] When determining the incremental information corresponding to the neighboring voxels of the voxel based on the key value of the neighboring voxel and the structured data corresponding to the previous depth image, the incremental information determining module 530 is used to:

[0203] In response to a key value of the neighborhood voxel being included in the structured data, projecting the neighborhood voxel onto the current depth image;

[0204] In response to the projection of the neighborhood voxel onto the current depth image, and when the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image is within a preset threshold range, based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image, determine the incremental information of the structured data corresponding to the neighborhood voxel; or

[0205] In response to the key value of the neighborhood voxel not being included in the structured data, incremental structured data information corresponding to the neighborhood voxel is determined based on voxel depth information corresponding to the neighborhood voxel.

[0206] In some embodiments, the structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel;

[0207] When determining the incremental information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image, the incremental information determining module 530 is used to:

[0208] Taking the difference between the voxel depth information corresponding to the neighborhood voxel and the pixel depth information corresponding to the neighborhood voxel as the increment value of the truncated signed distance function corresponding to the neighborhood voxel;

[0209] Setting the state information of the neighborhood voxels to an updated state;

[0210] The determining, based on the voxel depth information corresponding to the neighborhood voxel, the structured data increment information corresponding to the neighborhood voxel comprises:

[0211] Using the voxel depth information as an increment value of a truncated signed distance function corresponding to the neighborhood voxel;

[0212] The state information of the neighborhood voxel is set to an added state.

[0213] In some embodiments, when the incremental information determination module 530 determines the incremental information of the structured data corresponding to the neighboring voxels of the voxel based on the key value of the neighboring voxel and the structured data corresponding to the previous depth image, it is used to:

[0214] In response to the neighborhood voxel not being projected onto the current depth image, or the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image being not within the preset threshold range, the state information of the neighborhood voxel is set to a deleted state.

[0215] In some embodiments, the bias information includes first bias information and second bias information of the corresponding voxel;

[0216] When determining the offset information corresponding to the neighborhood voxel based on the voxel coordinates of the neighborhood voxel in the preset voxel space, the incremental information determining module 530 is used to:

[0217] Filter the coordinate item with the largest absolute value from the voxel coordinates;

[0218] Determining the first offset information based on the coordinate item with the largest absolute value and the number of unit voxels included in the preset voxel space;

[0219] The second offset information is determined based on the first offset information and the number of unit voxels included in the preset voxel space.

[0220] In some embodiments, when generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel, the grid generation module 540 is used to:

[0221] Determine target structural data of the voxel based on the structural data increment information of the voxel and the structural data;

[0222] Based on the target structured data of the voxels, a target grid corresponding to the current depth image is generated.

[0223] In some embodiments, the structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel;

[0224] When the grid generation module 540 generates a target grid corresponding to the current depth image based on the target structured data of the voxels, it is used to:

[0225] In response to the state information of the voxel of the neighborhood voxel being a preset state, and when it is determined that the number of times the state information of the voxel of the neighborhood voxel is the preset state is greater than a preset number, generating an incremental grid corresponding to the voxel based on the target structure data of the voxel;

[0226] Based on the incremental grid corresponding to the voxel and the grid corresponding to the previous depth image, a target grid corresponding to the current depth image is generated.

[0227] In some embodiments, the incremental information determination module 530 is further used to:

[0228] Obtaining optimized posture parameters corresponding to the shooting device that shoots the depth image and the posture parameters when the shooting device shoots each depth image;

[0229] For each depth image, determining a posture change value based on the posture parameters corresponding to the depth image and the optimized posture parameters;

[0230] In response to the posture change value being greater than a preset posture change threshold, the incremental structured data information of the voxel corresponding to the depth image is removed from the target structured data, and based on the optimized posture parameter, the final incremental structured data information of the voxel corresponding to the depth image is determined;

[0231] Based on the final structured data increment information of the voxels corresponding to the depth image, the structured data corresponding to the depth image is determined.

[0232] In some embodiments, the structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel;

[0233] The incremental information determination module 530 is further used for:

[0234] In response to removing the incremental structured data information of the voxel corresponding to the depth image from the target structured data, when the truncated signed distance function value of the voxel corresponding to the depth image is less than zero, the state information of the voxel corresponding to the depth image is set to a deleted state.

[0235] In some embodiments, before generating the target grid corresponding to the current depth image based on the structured data incremental information corresponding to the neighborhood voxels of the voxel, the incremental information determining module 530 is further configured to:

[0236] Filtering target voxels whose structured data increment information is empty from the voxels corresponding to the current depth image;

[0237] Projecting the target voxel onto the current depth image;

[0238] In response to the target voxel being projected onto the current depth image, and when the voxel depth information obtained by projecting the target voxel onto the current depth image is within a preset threshold range, the structured data incremental information corresponding to the target voxel is determined based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image.

[0239] In some embodiments, the structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel;

[0240] When determining the incremental information of the structured data corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image, the incremental information determining module 530 is used to:

[0241] In response to the case where a difference value obtained by subtracting the pixel depth information corresponding to the target voxel from the voxel depth information corresponding to the target voxel is less than zero, using the difference value obtained by subtracting the pixel depth information from the voxel depth information as a truncated signed distance function increment value corresponding to the target voxel;

[0242] The state information of the target voxel is set to an update state.

[0243] In some embodiments, before determining the voxel corresponding to the pixel in the preset voxel space based on the image coordinates of the pixel in the current depth image, the voxel determination module 520 is further configured to:

[0244] For each pixel in the current depth image, calculate a pixel depth difference between a pixel depth of the pixel and a pixel depth of a neighboring pixel of the pixel;

[0245] In response to the pixel depth difference corresponding to the pixel not being within the preset depth difference range, the pixel is set to be invalid.

[0246] Corresponding to the above-mentioned grid generation method, the embodiment of the present disclosure further provides an electronic device 600, such as Figure 6 FIG. 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present disclosure, including:

[0247] Processor 61, memory 62, and bus 63; memory 62 is used to store execution instructions, including internal memory 621 and external memory 622; the internal memory 621 is also called internal memory, which is used to temporarily store the operation data in the processor 61 and the data exchanged with the external memory 622 such as a hard disk. The processor 61 exchanges data with the external memory 622 through the internal memory 621. When the electronic device 600 is running, the processor 61 communicates with the memory 62 through the bus 63, so that the processor 61 executes the following instructions:

[0248] Acquire a current depth image and structured data corresponding to a previous depth image used for grid generation before the current depth image; determine the voxel corresponding to the pixel in a preset voxel space based on the image coordinates of the pixel in the current depth image; determine the structured data increment information corresponding to the neighborhood voxels of the voxel based on the neighborhood voxels of the voxel and the structured data corresponding to the previous depth image; generate a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel.

[0249] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the grid generation method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0250] The present disclosure also provides a computer program product, including a computer-readable storage medium storing program code, wherein the program code includes instructions that can be used to execute the steps of the grid generation method described in the above method embodiment, and specifically, the computer program product can be implemented in hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0251] The present disclosure relates to the field of augmented reality, which obtains image information of target objects in a real environment, and then uses various vision-related algorithms to detect or identify the relevant features, states and attributes of the target objects, so as to obtain an AR effect that combines virtuality and reality and matches the specific application. Exemplarily, the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms may involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, and position or depth detection of objects. Specific applications may not only involve interactive scenes such as guided tours, navigation, explanations, reconstructions, virtual effect overlay displays, etc. related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effects display, virtual model display and other interactive scenes.

[0252] The convolutional neural network can be used to detect or identify the relevant features, states and attributes of the target object. The above convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0253] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0254] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0255] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0256] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0257] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A grid generation method, characterized in that: include: Acquire a current depth image and structured data corresponding to a previous depth image used for mesh generation before the current depth image; Determining a voxel corresponding to the pixel in a preset voxel space based on the image coordinates and the depth value of the pixel in the current depth image; Determine, based on the structured data corresponding to the neighboring voxels of the voxel and the previous depth image, the structured data increment information corresponding to the neighboring voxels of the voxel; Generate a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel; The determining, based on the structured data corresponding to the neighboring voxels of the voxel and the previous depth image, the structured data increment information corresponding to the neighboring voxels of the voxel includes: Determining bias information corresponding to the neighborhood voxel based on the voxel coordinates of the neighborhood voxel in the preset voxel space; Determining a key value of the neighborhood voxel based on the bias information of the neighborhood voxel; Based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image, the structured data increment information corresponding to the neighborhood voxel of the voxel is determined.

2. The grid generation method according to claim 1, characterized in that: The structured data corresponding to the front depth image includes key values ​​of voxels corresponding to the front depth image; The determining, based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image, structured data increment information corresponding to the neighborhood voxel of the voxel includes: In response to a key value of the neighborhood voxel being included in the structured data, projecting the neighborhood voxel onto the current depth image; In response to the projection of the neighborhood voxel onto the current depth image, and when the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image is within a preset threshold range, based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image, determining the structured data increment information corresponding to the neighborhood voxel; or, In response to the key value of the neighborhood voxel not being included in the structured data, incremental structured data information corresponding to the neighborhood voxel is determined based on voxel depth information corresponding to the neighborhood voxel.

3. The grid generation method according to claim 2, characterized in that: The structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel; The determining, based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image, the structured data increment information corresponding to the neighborhood voxel comprises: Taking the difference between the voxel depth information corresponding to the neighborhood voxel and the pixel depth information corresponding to the neighborhood voxel as the increment value of the truncated signed distance function corresponding to the neighborhood voxel; The state information of the neighborhood voxel is set to an updated state.

4. The grid generation method according to claim 2, characterized in that: The structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel; The determining, based on the voxel depth information corresponding to the neighborhood voxel, the structured data increment information corresponding to the neighborhood voxel comprises: Using the voxel depth information as an increment value of a truncated signed distance function corresponding to the neighborhood voxel; The state information of the neighborhood voxel is set to an added state.

5. The grid generation method according to claim 3, characterized in that: The step of determining the incremental structured data information corresponding to the neighboring voxels of the voxel based on the key value of the neighboring voxel and the structured data corresponding to the previous depth image further includes: In response to the neighborhood voxel not being projected onto the current depth image, or the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image being not within the preset threshold range, the state information of the neighborhood voxel is set to a deleted state.

6. The grid generation method according to any one of claims 1 to 5, characterized in that: The bias information includes first bias information and second bias information of the corresponding voxel; The determining, based on the voxel coordinates of the neighborhood voxels in the preset voxel space, the bias information corresponding to the neighborhood voxels includes: Filter the coordinate item with the largest absolute value from the voxel coordinates; Determining the first offset information based on the coordinate item with the largest absolute value and the number of unit voxels included in the preset voxel space; The second offset information is determined based on the first offset information and the number of unit voxels included in the preset voxel space.

7. The grid generation method according to any one of claims 1 to 5, characterized in that: The step of generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel comprises: Determine target structural data of the voxel based on the structural data increment information of the voxel and the structural data; Based on the target structured data of the voxels, a target grid corresponding to the current depth image is generated.

8. The grid generation method according to claim 7, characterized in that: The structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel; The generating a target grid corresponding to the current depth image based on the target structured data of the voxels includes: In response to the state information of the voxel of the neighborhood voxel being a preset state, and when it is determined that the number of times the state information of the voxel of the neighborhood voxel is the preset state is greater than a preset number, generating an incremental grid corresponding to the voxel based on the target structure data of the voxel; Based on the incremental grid corresponding to the voxel and the grid corresponding to the previous depth image, a target grid corresponding to the current depth image is generated.

9. The grid generation method according to claim 7, characterized in that: Also includes: Obtaining optimized posture parameters corresponding to the shooting device that shoots the depth image and the posture parameters when the shooting device shoots each depth image; For each depth image, determining a posture change value based on the posture parameters corresponding to the depth image and the optimized posture parameters; In response to the posture change value being greater than a preset posture change threshold, the incremental structured data information of the voxel corresponding to the depth image is removed from the target structured data, and based on the optimized posture parameter, the final incremental structured data information of the voxel corresponding to the depth image is determined; Based on the final structured data increment information of the voxels corresponding to the depth image, the structured data corresponding to the depth image is determined.

10. The grid generation method according to claim 9, characterized in that: The structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel; The grid generation method further comprises: In response to the truncated signed distance function value of the voxel corresponding to the depth image being less than a preset value after the structured data incremental information of the voxel corresponding to the depth image is removed from the target structured data, the state information of the voxel corresponding to the depth image is set to a deleted state.

11. The grid generation method according to any one of claims 1 to 5, characterized in that: Before generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel, the method further includes: Filtering target voxels whose structured data increment information is empty from the voxels corresponding to the current depth image; Projecting the target voxel onto the current depth image; In response to the target voxel being projected onto the current depth image, and when the voxel depth information obtained by projecting the target voxel onto the current depth image is within a preset threshold range, the structured data incremental information corresponding to the target voxel is determined based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image.

12. The grid generation method according to claim 11, characterized in that: The structured data increment information includes state information of the corresponding voxel and a truncated signed distance function increment value of the corresponding voxel; The determining, based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image, the structured data increment information corresponding to the target voxel comprises: In response to the case where a difference value obtained by subtracting the pixel depth information corresponding to the target voxel from the voxel depth information corresponding to the target voxel is less than zero, using the difference value obtained by subtracting the pixel depth information from the voxel depth information as a truncated signed distance function increment value corresponding to the target voxel; The state information of the target voxel is set to an update state.

13. The grid generation method according to any one of claims 1 to 5, characterized in that: Before determining the voxel corresponding to the pixel in the preset voxel space based on the image coordinates of the pixel in the current depth image, the method further includes: For each pixel in the current depth image, calculate a pixel depth difference between a pixel depth of the pixel and a pixel depth of a neighboring pixel of the pixel; In response to the pixel depth difference corresponding to the pixel not being within the preset depth difference range, the pixel is set to be invalid.

14. A grid generation device, characterized in that: include: A data acquisition module, used to acquire a current depth image and structured data corresponding to a previous depth image used for grid generation before the current depth image; A voxel determination module, configured to determine a voxel corresponding to a pixel in a preset voxel space based on the image coordinates and the depth value of the pixel in the current depth image; An incremental information determination module, configured to determine incremental structured data information corresponding to neighboring voxels of the voxel based on the neighboring voxels of the voxel and the structured data corresponding to the previous depth image; A grid generation module, used for generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxel; The incremental information determination module is specifically used for: Determining bias information corresponding to the neighborhood voxel based on the voxel coordinates of the neighborhood voxel in the preset voxel space; Determining a key value of the neighborhood voxel based on the bias information of the neighborhood voxel; Based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image, the structured data increment information corresponding to the neighborhood voxel of the voxel is determined.

15. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the grid generation method as described in any one of claims 1 to 13 are performed.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the grid generation method according to any one of claims 1 to 13 are executed.

Images