Binocular depth map-based hole optimization method, terminal device and medium
By using NEON instructions to process depth map data in an embedded system, building masks and replacing arrays, the problem of hollowing when generating depth maps by binocular cameras is solved, and rapid hollowing optimization and improved target detection accuracy are achieved.
Patent Information
- Application Number
- CN202411686358.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-03-28
AI Technical Summary
When generating depth maps, the embedded binocular cameras cannot match due to the lack of obvious features, resulting in hollow problems. In addition, the existing algorithms are time-consuming to deploy and cannot meet the needs of real-time object detection.
Using the method based on NEON instructions, the depth map data is processed in a group of 16 bytes, a mask array and a replacement array are constructed, each element is traversed to determine whether it is a hole, and the corresponding replacement process is performed.
The speed of hollow optimization is significantly improved to reach less than 3 milliseconds, and the subsequent object detection accuracy based on depth maps is improved.
Smart Images
Figure CN119722431A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of hole optimization, and in particular to a hole optimization method based on a binocular depth map, a terminal device and a medium. Background Art
[0002] The embedded binocular camera can synthesize the depth map and the depth of the camera Figure 1 It is usually provided by the interface of the embedded chip. However, when matching the left and right views of the binoculars, the features are not obvious and the depth value cannot be calculated. Therefore, the generated depth map has holes (such as Figure 1 The place where the value is 0 is a hole).
[0003] Embedded binocular cameras can also synthesize depth maps through existing algorithms. However, since the neural network needs to be trained and the model takes a long time to deploy (greater than 50 milliseconds), the synthesis speed is very slow and basically cannot meet the subsequent real-time target detection.
[0004] NEON is an optimized instruction set under the ARM architecture, mainly used to implement SIMD (Single Instruction Multiple Data), which greatly improves processing efficiency by packing multiple operands in large registers and operating multiple operands simultaneously under one instruction. NEON technology is particularly suitable for processing some simple and highly repetitive algorithms, which can significantly improve performance. Simply put, NEON can process multiple bits of data at a time (normally 128 bits, 8 bits is a byte, which is equivalent to processing 16 bytes at a time, which has the effect of accelerating 16 times).
[0005] Hole optimization is to fill the place where the number is 0 with a number. There are many ways to fill the hole. The conventional method is to diffuse outward from the current point of the hole, such as Figure 2 The number 0 in the second row and second column is a hole. In order to fill it, the average of the non-hole values around it is taken. The current value = (8+10+9+3) / 4 = 7.5, which is of int type and is 7.
[0006] Although the above method can achieve hole optimization, considering the computing power of the embedded platform, this method is time-consuming and cannot be applied to NEON acceleration. Summary of the invention
[0007] In order to solve the above problems, the present invention proposes a hole optimization method based on a binocular depth map, a terminal device and a medium.
[0008] The specific plan is as follows:
[0009] A hole optimization method based on a binocular depth map is provided. After the depth map data is divided into groups of 16 bytes, each group of data is processed in sequence through NEON instructions. Each processing includes the following steps:
[0010] Construct a mask array consisting of 16 elements. The value of the i-th element in the mask array is the mask of the i-th byte in the processed data. The mask of the i-th byte is set according to whether the value of the i-th byte represents a hole.
[0011] Construct a replacement array consisting of 16 elements, where the value of each element in the replacement array is the value of the last byte in the data processed last time;
[0012] Traverse each element in the processed data and perform the following processing: for the i-th element, i∈[1,16], determine whether the value of the i-th element in the mask array represents a hole. If so, replace the value of the i-th element with the value of the i-th element in the replacement array; otherwise, set the value of the i-th element unchanged.
[0013] Furthermore, when the value of the i-th byte indicates a hole, the mask is 1; when the value of the i-th byte does not indicate a hole, the mask is 0.
[0014] Further, when the processed data is the first processed data, the value of each element in the replacement data is a preset fixed value.
[0015] Furthermore, when the processed data is the first processed data, it is not processed.
[0016] A hole optimization terminal device based on a binocular depth map includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above in an embodiment of the present invention are implemented.
[0017] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above in an embodiment of the present invention are implemented.
[0018] The present invention adopts the above technical solution and can be deployed in an embedded system, thereby improving the speed of hole optimization (below 3 milliseconds) and the accuracy of subsequent target detection based on the depth map. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 Shown is a schematic diagram of a hole in a depth map.
[0020] Figure 2 Shown is a schematic diagram of a conventional void filling method.
[0021] Figure 3 Shown is a schematic diagram of 16-byte data loaded in the first embodiment of the present invention.
[0022] Figure 4 Shown is a flow chart of the method in this embodiment.
[0023] Figure 5 Shown is a schematic diagram of the mask array constructed in this embodiment.
[0024] Figure 6 Shown is a schematic diagram of the replacement array constructed in this embodiment.
[0025] Figure 7 Shown is a schematic diagram of the cavity optimization result in this embodiment.
[0026] Figure 8 The figure shows the before and after comparison of the hole-optimized picture after color mapping in this embodiment. DETAILED DESCRIPTION
[0027] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.
[0028] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.
[0029] Embodiment 1:
[0030] Considering that NEON processes 16 bytes at a time, which is equivalent to loading 16 bytes at a time for batch processing, based on this, an embodiment of the present invention provides a hole optimization method based on a binocular depth map. In the method, after the depth map data is segmented into groups of 16 bytes, each group of data is processed in turn by a NEON instruction, that is, the NEON instruction loads 16 bytes of data each time, such as Figure 3 shown.
[0031] In each treatment, Figure 4 As shown, the following steps are included:
[0032] Step 1: Construct a mask array consisting of 16 elements. The value of the i-th element in the mask array is the mask of the i-th byte in the processed data. The mask of the i-th byte is set according to whether the value of the i-th byte represents a hole.
[0033] Step 2: Construct a replacement array consisting of 16 elements, where the value of each element in the replacement array is the value of the last byte in the data processed last time.
[0034] Step 3: Traverse each element in the processed data and perform the following processing: For the i-th element, i∈[1,16], determine whether the value of the i-th element in the mask array represents a hole. If so, replace the value of the i-th element with the value of the i-th element in the replacement array; otherwise, set the value of the i-th element to remain unchanged.
[0035] In the mask setting of this embodiment, when the value of the i-th byte represents a hole (the value is 0), the mask is 1; when the value of the i-th byte does not represent a hole (the value is not 0), the mask is 0. Figure 3 The mask array corresponding to the data shown is as follows Figure 5 shown.
[0036] like Figure 6 Shown Figure 3 Schematic diagram of the replacement array corresponding to the data shown, where the value of the last byte in the last processed data is 6, so the values of the 16 elements in the replacement array are all 0.
[0037] It should be noted that since the data processed for the first time does not contain the data processed last time, two processing methods are provided in this embodiment. One is not to process the data processed for the first time (i.e., keep the original data); the other is to replace it with a preset fixed value, that is, the value of each element in the replacement data is a preset fixed value.
[0038] By using the method of this embodiment, the filling data of the hole value 0 is no longer obtained by taking the average value of the surrounding area, but is obtained by continuously expanding the left value to the right, such as Figure 7 and Figure 8 shown.
[0039] In the embodiment of the present invention, the optimization of the void is combined with NEON through a simple solution to achieve the purpose of accelerated processing, which can play a good effect in the field of embedded low computing power and provide a good foundation for the subsequent target detection algorithm.
[0040] Embodiment 2:
[0041] The present invention also provides a hole optimization terminal device based on a binocular depth map, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method embodiment of the first embodiment of the present invention when executing the computer program.
[0042] Further, as an executable solution, the hole optimization terminal device based on the binocular depth map can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The hole optimization terminal device based on the binocular depth map may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the composition structure of the above-mentioned hole optimization terminal device based on the binocular depth map is only an example of a hole optimization terminal device based on the binocular depth map, and does not constitute a limitation on the hole optimization terminal device based on the binocular depth map. It may include more or less components than the above, or a combination of certain components, or different components. For example, the hole optimization terminal device based on the binocular depth map may also include input and output devices, network access devices, buses, etc., and the embodiments of the present invention do not limit this.
[0043] Further, as an executable solution, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the binocular depth map-based hole optimization terminal device, and uses various interfaces and lines to connect the various parts of the entire binocular depth map-based hole optimization terminal device.
[0044] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the hole optimization terminal device based on the binocular depth map by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the mobile phone, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0045] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.
[0046] If the module / unit integrated in the terminal device for hole optimization based on the binocular depth map is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc.
[0047] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.
Claims
1. A hole optimization method based on binocular depth map, characterized by: After the depth map data is divided into groups of 16 bytes, each group of data is processed in sequence through NEON instructions; each processing includes the following steps: Construct a mask array consisting of 16 elements. The value of the i-th element in the mask array is the mask of the i-th byte in the processed data. The mask of the i-th byte is set according to whether the value of the i-th byte represents a hole. Construct a replacement array consisting of 16 elements, where the value of each element in the replacement array is the value of the last byte in the data processed last time; Traverse each element in the processed data and perform the following processing: for the i-th element, i∈[1,16], determine whether the value of the i-th element in the mask array represents a hole. If so, replace the value of the i-th element with the value of the i-th element in the replacement array; otherwise, set the value of the i-th element unchanged.
2. The hole optimization method based on binocular depth map according to claim 1, characterized in that: When the value of the i-th byte represents a hole, the mask is 1; when the value of the i-th byte does not represent a hole, the mask is 0.
3. The hole optimization method based on binocular depth map according to claim 1, characterized in that: When the processed data is the first processed data, the value of each element in the replacement data is a preset fixed value.
4. The hole optimization method based on binocular depth map according to claim 1, characterized in that: When the processed data is the first processed data, it is not processed.
5. A hole optimization terminal device based on binocular depth map, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 4 when executing the computer program.
6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.