A spatial annotation method and system based on depth completion

Abstract

The application relates to a space labeling method and system based on depth completion, wherein the method comprises the following steps: acquiring environment depth information of sparse pixels from frame pictures of a video stream; performing grid division on the frame pictures of the video stream to obtain a plurality of grid regions; calculating average depth values of the grid regions according to the environment depth information, and assigning the corresponding average depth values to all pixel points in the corresponding grid regions; and completing space labeling of the frame pictures of the video stream according to pixel coordinates and average depth values of the pixel points. Through the application, the problem of how to improve the robustness and accuracy of space labeling is solved, the completion of complete picture environment depth information based on sparse environment depth information is realized, an additional depth sensor is not needed, the AR space labeling with a 100% success rate is ensured, the user experience of AR users is greatly improved, and the equipment cost is reduced.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a spatial annotation method and system based on depth completion. Background Technology

[0002] With the continuous development and advancement of AR technology, it is gradually being applied to all aspects of daily life, such as AR annotation, AR remote assistance, and AR content placement. Efficient and accurate spatial annotation is a crucial guarantee for the successful application of AR technology. Spatial annotation refers to calculating the 3D coordinates of a point on a video stream displayed on the screen using a camera projection algorithm.

[0003] Currently, spatial annotation relies on the camera's pose information when capturing video footage, the pixel coordinates of points on the screen, and complete environmental depth information. Typically, obtaining the camera's pose requires SLAM tracking technology. However, SLAM tracking technology used in AR applications cannot provide complete environmental depth information in real time; it can only provide sparse environmental depth information. Therefore, existing spatial annotation schemes, without adding a depth sensor, struggle to obtain complete environmental depth information, easily leading to annotation errors or inaccuracies.

[0004] Currently, no effective solution has been proposed for improving the robustness and accuracy of spatial annotation in related technologies. Summary of the Invention

[0005] This application provides a spatial annotation method and system based on depth completion, which at least addresses the problem of how to improve the robustness and accuracy of spatial annotation in related technologies.

[0006] In a first aspect, embodiments of this application provide a spatial annotation method based on depth completion, the method comprising:

[0007] Environmental depth information of sparse pixels is obtained from the frames of the video stream;

[0008] The video stream frames are divided into grids to obtain several grid regions;

[0009] Based on the environmental depth information, calculate the average depth value of each grid region, and assign the corresponding average depth value to all pixels in the corresponding grid region.

[0010] Spatial annotation of the video stream frames is completed based on the pixel coordinates of the pixels and the average depth value.

[0011] In some embodiments, the video stream frames are divided into grids to obtain several grid regions, including:

[0012] Based on the distribution of the sparse pixels, the video stream frames are divided into grids using a preset grid division algorithm to obtain several grid regions. The preset grid division algorithm is used to ensure that each grid region contains at least a first preset number of sparse pixels.

[0013] In some embodiments, calculating the average depth value of each grid region based on the environmental depth information includes:

[0014] Iterate through each grid region to determine if there are sparse pixels;

[0015] If so, the average depth value of the current grid region is calculated based on the environmental depth information of the sparse pixels in the current grid region.

[0016] If not, then find the second preset number of sparse pixels closest to the current grid region, and calculate the average depth value of the current grid region based on the environmental depth information of the second preset number of sparse pixels.

[0017] In some embodiments, calculating the average depth value of the current grid region based on the environmental depth information of sparse pixels within the current grid region includes:

[0018] Determine whether the number of sparse pixels in the current grid area is greater than or equal to the first preset number;

[0019] If so, the average depth value of the current grid region is calculated based on the environmental depth information of the sparse pixels in the current grid region.

[0020] If not, then find the third preset number of sparse pixels closest to the current grid region, and calculate the average depth value of the current grid region based on the environmental depth information of the sparse pixels in the current grid region and the environmental depth information of the third preset number of sparse pixels.

[0021] In some embodiments, obtaining environmental depth information of sparse pixels from frames of a video stream includes:

[0022] The environmental depth information of sparse pixels is obtained from the frames of the video stream using a preset SLAM algorithm.

[0023] Secondly, embodiments of this application provide a spatial annotation system based on depth completion, the system including an information extraction module, a grid division module, an information completion module, and a spatial annotation module;

[0024] The information extraction module is used to obtain environmental depth information of sparse pixels from the frames of the video stream.

[0025] The grid division module is used to divide the frame of the video stream into grids to obtain several grid regions;

[0026] The information completion module is used to calculate the average depth value of each grid region based on the environmental depth information, and assign the corresponding average depth value to all pixels in the corresponding grid region.

[0027] The spatial annotation module is used to complete the spatial annotation of the video stream frames based on the pixel coordinates of the pixels and the average depth value.

[0028] In some embodiments, the grid division module is used to divide the frame of the video stream into grids according to the distribution of the sparse pixels using a preset grid division algorithm to obtain several grid regions, wherein the preset grid division algorithm is used to ensure that each grid region contains at least a first preset number of sparse pixels.

[0029] In some embodiments, the information completion module is used to traverse and determine whether there are sparse pixels in each grid area;

[0030] If so, the average depth value of the current grid region is calculated based on the environmental depth information of the sparse pixels in the current grid region.

[0031] If not, then find the second preset number of sparse pixels closest to the current grid region, and calculate the average depth value of the current grid region based on the environmental depth information of the second preset number of sparse pixels.

[0032] In some embodiments, the information completion module is used to determine whether the number of sparse pixels in the current grid area is greater than or equal to a first preset number.

[0033] If so, the average depth value of the current grid region is calculated based on the environmental depth information of the sparse pixels in the current grid region.

[0034] If not, then find the third preset number of sparse pixels closest to the current grid region, and calculate the average depth value of the current grid region based on the environmental depth information of the sparse pixels in the current grid region and the environmental depth information of the third preset number of sparse pixels.

[0035] In some embodiments, the information extraction module is used to obtain environmental depth information of sparse pixels from the frames of the video stream using a preset SLAM algorithm.

[0036] Compared to related technologies, the spatial annotation method and system based on depth completion provided in this application embodiment obtains environmental depth information of sparse pixels from the frames of a video stream; divides the frames of the video stream into grids to obtain several grid regions; calculates the average depth value of each grid region based on the environmental depth information, and assigns the corresponding average depth value to all pixels in the corresponding grid region; and completes the spatial annotation of the video stream frames based on the pixel coordinates and average depth value of the pixels. This solves the problem of how to improve the robustness and accuracy of spatial annotation, and realizes the completion of complete environmental depth information of the image based on sparse environmental depth information. It does not require an additional depth sensor, ensuring 100% success rate for AR spatial annotation, greatly improving the user experience of AR users, and reducing equipment costs. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0038] Figure 1 This is a flowchart of the steps of the spatial annotation method based on depth completion according to an embodiment of this application;

[0039] Figure 2 This is a flowchart illustrating the steps for calculating the average depth value of a grid region according to an embodiment of this application;

[0040] Figure 3 This is a structural block diagram of a depth-complete-based spatial annotation system according to an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of spatial annotation technology according to an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application.

[0043] Figure descriptions: 31. Information extraction module; 32. Grid division module; 33. Information completion module; 34. Spatial annotation module. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0045] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0046] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0047] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0048] This application provides a spatial annotation method based on depth completion. Figure 1 This is a flowchart illustrating the steps of the spatial annotation method based on depth completion according to an embodiment of this application, as follows: Figure 1 As shown, the method includes the following steps:

[0049] Step S102: Obtain the environmental depth information of sparse pixels from the frame of the video stream;

[0050] Specifically, environmental depth information of sparse pixels is obtained from frames of a video stream using a preset SLAM algorithm. Preferably, this preset SLAM algorithm is an AR application SLAM algorithm, which can obtain the camera's pose in real time, but cannot provide complete environmental depth information corresponding to the image in real time.

[0051] Step S104: Divide the frame of the video stream into a grid to obtain several grid regions;

[0052] Specifically, based on the distribution of sparse pixels, the video stream frames are divided into grid regions using a preset grid partitioning algorithm. The preset grid partitioning algorithm ensures that each grid region contains at least a first preset number of sparse pixels. Preferably, the preset grid partitioning algorithm can be the Delaunay grid partitioning algorithm, or other grid partitioning algorithms that achieve the same function; this is not limited to any particular algorithm.

[0053] It should be noted that the above-mentioned preset grid division algorithm ensures that each grid region contains at least a first preset number of sparse pixels, so that in the subsequent depth information completion step S106, each grid region only uses its own sparse pixels to calculate the average depth value, which can greatly reduce the amount of spatial annotation calculation.

[0054] Step S106: Calculate the average depth value of each grid region based on the environmental depth information, and assign the corresponding average depth value to all pixels in the corresponding grid region.

[0055] Specifically, Figure 2 This is a flowchart illustrating the steps for calculating the average depth value of a grid region according to an embodiment of this application, as follows: Figure 2 As shown, step S106 specifically includes the following steps:

[0056] Step S202: Iterate through each grid region to determine if there are sparse pixels. If yes, proceed to step S204; otherwise, proceed to step S210.

[0057] Step S204: Determine whether the number of sparse pixels in the current grid area is greater than or equal to a first preset number. If yes, proceed to step S206; otherwise, proceed to step S208.

[0058] Step S206: Calculate the average depth value of the current grid region based on the environmental depth information of sparse pixels in the current grid region;

[0059] Step S208: Find the third preset number of sparse pixels closest to the current grid area, and calculate the average depth value of the current grid area based on the environmental depth information of the sparse pixels in the current grid area and the environmental depth information of the third preset number of sparse pixels.

[0060] Step S210: Find the second preset number of sparse pixels closest to the current grid area, and calculate the average depth value of the current grid area based on the environmental depth information of the second preset number of sparse pixels.

[0061] It should be noted that through the above steps S202 to S210, it is ensured that even if the grid area divided in step S104 does not contain the first preset number of sparse pixels, the average depth value of each grid area can be calculated relatively accurately, which further improves the robustness of spatial annotation.

[0062] Step S108: Based on the pixel coordinates and average depth values ​​of the pixels, complete the spatial annotation of the video stream frames.

[0063] Specifically, the pose of the camera capturing the video stream is obtained through SLAM tracking technology, and the spatial annotation of the frames of the video stream is completed based on the pixel coordinates and average depth values ​​of the pixels.

[0064] Steps S102 to S108 in the embodiments of this application solve the problem of how to improve the robustness and accuracy of spatial annotation, realize the completion of complete environmental depth information of the image based on sparse environmental depth information, without the need for additional depth sensors, ensure 100% success rate of AR spatial annotation, greatly improve the user experience of AR users, and reduce equipment costs.

[0065] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0066] This application provides a spatial annotation system based on depth completion. Figure 3 This is a structural block diagram of a depth-complete-based spatial annotation system according to an embodiment of this application, such as... Figure 3 As shown, the system includes an information extraction module 31, a grid division module 32, an information completion module 33, and a spatial annotation module 34;

[0067] The information extraction module 31 is used to obtain environmental depth information of sparse pixels from the frame of the video stream;

[0068] The grid division module 32 is used to divide the frame of the video stream into a grid to obtain several grid regions;

[0069] The information completion module 33 is used to calculate the average depth value of each grid area based on the environmental depth information, and assign the corresponding average depth value to all pixels in the corresponding grid area.

[0070] The spatial annotation module 34 is used to complete the spatial annotation of the video stream frames based on the pixel coordinates and average depth value of the pixels.

[0071] The information extraction module 31, grid division module 32, information completion module 33, and spatial annotation module 34 in this embodiment solve the problem of how to improve the robustness and accuracy of spatial annotation. They achieve the completion of complete environmental depth information based on sparse environmental depth information without the need for additional depth sensors. This ensures 100% success rate in AR spatial annotation, greatly improves the user experience for AR users, and reduces equipment costs.

[0072] In some embodiments, the grid division module 32 is used to divide the frame of the video stream into grids according to the distribution of sparse pixels using a preset grid division algorithm to obtain several grid regions. The preset grid division algorithm is used to ensure that each grid region contains at least a first preset number of sparse pixels.

[0073] In some embodiments, the information completion module 33 is used to traverse and determine whether there are sparse pixels in each grid area;

[0074] If so, calculate the average depth value of the current grid region based on the environmental depth information of the sparse pixels in the current grid region;

[0075] If not, find the second preset number of sparse pixels closest to the current grid area, and calculate the average depth value of the current grid area based on the environmental depth information of the second preset number of sparse pixels.

[0076] In some embodiments, the information completion module 33 is used to determine whether the number of sparse pixels in the current grid area is greater than or equal to a first preset number.

[0077] If so, calculate the average depth value of the current grid region based on the environmental depth information of the sparse pixels in the current grid region;

[0078] If not, find the third preset number of sparse pixels closest to the current grid area, and calculate the average depth value of the current grid area based on the environmental depth information of the sparse pixels in the current grid area and the environmental depth information of the third preset number of sparse pixels.

[0079] In some embodiments, the information extraction module 31 is used to obtain environmental depth information of sparse pixels from the frames of the video stream using a preset SLAM algorithm.

[0080] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0081] It should be noted that, Figure 4 This is a schematic diagram of spatial annotation technology according to an embodiment of this application, such as... Figure 4 As shown, spatial annotation technology generally refers to the technique used in AR (Augmented Reality) scenarios to map the pixel location clicked by a user on a terminal device (such as a mobile phone or camera) to a location in three-dimensional physical space. Based on this technology, AR users can place AR virtual content at desired locations in real physical space, achieving a superimposed effect between the virtual and real worlds.

[0082] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0083] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0084] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0085] Furthermore, in conjunction with the depth-complete-based spatial annotation methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the depth-complete-based spatial annotation methods in the above embodiments.

[0086] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, a network interface, a display screen, and an input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements a spatial annotation method based on depth completion. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0087] In one embodiment, Figure 5 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 5 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 5 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement a spatial annotation method based on depth completion, and the database stores data.

[0088] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0090] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A spatial labeling method based on depth completion, characterized in that, The method includes: Environmental depth information of sparse pixels is obtained from the frames of the video stream; Based on the distribution of the sparse pixels, the frames of the video stream are divided into grids using a preset grid division algorithm to obtain several grid regions. The preset grid division algorithm is used to ensure that each grid region contains at least a first preset number of sparse pixels. Based on the environmental depth information, the average depth value of each grid region is calculated, and the corresponding average depth value is assigned to all pixels in the corresponding grid region; wherein, calculating the average depth value of each grid region based on the environmental depth information includes: Iterate through each grid region to determine if there are sparse pixels; if so, calculate the average depth of the current grid region based on the environmental depth information of the sparse pixels in the current grid region; if not, find the second preset number of sparse pixels closest to the current grid region, and calculate the average depth of the current grid region based on the environmental depth information of the second preset number of sparse pixels. Spatial annotation of the video stream frames is completed based on the pixel coordinates of the pixels and the average depth value.

2. The method according to claim 1, characterized in that, Calculating the average depth value of the current grid region based on the environmental depth information of sparse pixels within the current grid region includes: Determine whether the number of sparse pixels in the current grid area is greater than or equal to the first preset number; If so, the average depth value of the current grid region is calculated based on the environmental depth information of the sparse pixels in the current grid region. If not, then find the third preset number of sparse pixels closest to the current grid region, and calculate the average depth value of the current grid region based on the environmental depth information of the sparse pixels in the current grid region and the environmental depth information of the third preset number of sparse pixels.

3. The method according to claim 1, characterized in that, The environmental depth information of sparse pixels obtained from the frames of the video stream includes: The environmental depth information of sparse pixels is obtained from the frames of the video stream using a preset SLAM algorithm.

4. A spatial annotation system based on depth completion, characterized in that, The system includes an information extraction module, a grid division module, an information completion module, and a spatial annotation module; The information extraction module is used to obtain environmental depth information of sparse pixels from the frames of the video stream. The grid division module is used to divide the frame of the video stream into grids according to the distribution of the sparse pixels using a preset grid division algorithm to obtain several grid regions. The preset grid division algorithm is used to ensure that each grid region contains at least a first preset number of sparse pixels. The information completion module is used to calculate the average depth value of each grid region based on the environmental depth information, and assign the corresponding average depth value to all pixels in the corresponding grid region; wherein, calculating the average depth value of each grid region based on the environmental depth information includes: Iterate through each grid region to determine if there are sparse pixels; if so, calculate the average depth of the current grid region based on the environmental depth information of the sparse pixels in the current grid region; if not, find the second preset number of sparse pixels closest to the current grid region, and calculate the average depth of the current grid region based on the environmental depth information of the second preset number of sparse pixels. The spatial annotation module is used to complete the spatial annotation of the video stream frames based on the pixel coordinates of the pixels and the average depth value.

5. The system according to claim 4, characterized in that, The information completion module is used to determine whether the number of sparse pixels in the current grid area is greater than or equal to a first preset number. If so, the average depth value of the current grid region is calculated based on the environmental depth information of the sparse pixels in the current grid region. If not, then find the third preset number of sparse pixels closest to the current grid region, and calculate the average depth value of the current grid region based on the environmental depth information of the sparse pixels in the current grid region and the environmental depth information of the third preset number of sparse pixels.

6. The system according to claim 4, characterized in that, The information extraction module is used to obtain environmental depth information of sparse pixels from the frames of the video stream using a preset SLAM algorithm.

Images