Cost aggregation based depth map reconstruction method, device, apparatus and storage medium

Abstract

The application provides a depth map reconstruction method and device based on cost aggregation, equipment and a storage medium. In one example of the application, the method comprises: acquiring a sequence of time-series laser line images; obtaining a plurality of spliced time-domain images from the image sequence; performing enhancement processing on the laser line region in the spliced time-domain image by using a cost aggregation method according to the fixed slope monotonicity of the laser line region in the spliced time-domain image; performing inverse splicing on the plurality of spliced time-domain images after the enhancement processing to reorganize the laser line image sequence after the enhancement processing; and performing depth map reconstruction according to the laser line image sequence after the enhancement processing. The method can improve the reliability and scene applicability of depth map reconstruction.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a method, apparatus, device and storage medium for depth map reconstruction based on cost aggregation. Background Technology

[0002] Binocular stereo vision combined with line structured light is a 3D measurement technology widely used in industrial inspection, robot navigation, and 3D reconstruction due to its high precision, high efficiency, and good real-time performance. The basic principle of this technology is as follows: a specific line structured light pattern is projected onto the surface of the object being measured using a laser. A calibrated binocular camera captures images of the laser lines that are modulated and deformed on the object's surface. Then, a stereo matching algorithm is used to calculate the disparity between corresponding points on the laser lines in the left and right images, thereby achieving 3D depth reconstruction of the object.

[0003] However, existing binocular structured light cameras still face challenges in the integrity and robustness of depth reconstruction in practical applications, especially in some special scenarios where depth data is easily missing, for example: a. For some light-absorbing or low-reflectivity objects, the surface of the object has extremely low reflectivity to actively projected laser signals. Even if the exposure gain is adjusted to a large extent, the effective laser signal received by the imaging sensor will still be very weak, resulting in a low signal-to-noise ratio of the laser line image. b. For high dynamic or high speed scenes, the system allows for very short exposure times, resulting in low contrast of laser lines in some areas.

[0004] It is evident that existing binocular structured light cameras generally suffer from a lack of depth reconstruction due to low laser line signal-to-noise ratio and insufficient contrast when facing light-absorbing objects or high-speed scenes. This severely restricts the reliability and applicability of the technology in more complex industrial environments. Summary of the Invention

[0005] In view of this, this application provides a method, apparatus, device and storage medium for depth map reconstruction based on cost aggregation.

[0006] According to a first aspect of the embodiments of this application, a depth map reconstruction method based on cost aggregation is provided, comprising: Acquire a temporal laser line image sequence; wherein, the temporal laser line image sequence includes a set of temporally continuous images, and a laser line is projected onto the scene to be reconstructed in any laser line image, and the laser line moves at a constant speed with a fixed period in the scene to be reconstructed; Multiple stitched temporal maps are obtained based on the image sequence; wherein, a stitched temporal map is obtained by stitching the same row of pixels of each laser line image in the image sequence in a temporal order; Based on the fixed slope monotonicity of the laser line region in the spliced ​​time-domain diagram, a cost-based aggregation method is used to enhance the laser line region in the spliced ​​time-domain diagram. The multiple stitched temporal images after enhancement are inversely stitched together and recombined into an enhanced laser line image sequence. Depth map reconstruction is performed based on the enhanced laser line image sequence.

[0007] According to a second aspect of the embodiments of this application, a depth map reconstruction apparatus based on cost aggregation is provided, comprising: An acquisition unit is used to acquire a time-series laser line image sequence; wherein, the time-series laser line image sequence includes a set of time-series continuous images, and a laser line is projected onto the scene to be reconstructed in any laser line image, and the laser line moves at a constant speed with a fixed period in the scene to be reconstructed; The stitching unit is used to obtain multiple stitched temporal maps based on the image sequence; wherein, a stitched temporal map is obtained by stitching the same row of pixels of each laser line image in the image sequence in a temporal order; An enhancement processing unit is used to enhance the laser line region in the spliced ​​time domain diagram by using a cost aggregation method based on the fixed slope monotonicity of the laser line region in the spliced ​​time domain diagram. The recombination unit is used to inversely stitch together multiple stitched time-domain images after enhancement processing, and reassemble them into an enhanced laser line image sequence. The reconstruction unit is used to reconstruct the depth map based on the enhanced laser line image sequence.

[0008] According to a third aspect of the present application, an electronic device is provided, including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0009] According to a fourth aspect of the embodiments of this application, a machine-readable storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0010] The depth map reconstruction method based on cost aggregation in this application acquires a temporal laser line image sequence, and obtains multiple stitched temporal maps by stitching the image sequence together using temporal image rows. Based on the fixed slope monotonicity of the laser line region in the stitched temporal map, cost aggregation is used to enhance the laser line region in the stitched temporal map. Then, the enhanced multiple stitched temporal maps are inversely stitched together to reconstruct an enhanced laser line image sequence. Depth map reconstruction is performed based on the enhanced laser line image sequence. This method can improve the contrast of laser lines in the laser line image, effectively improve the depth map reconstruction rate in scenes with dark areas or objects with very low contrast, and enhance the reliability and scene applicability of depth map reconstruction. Attached Figure Description

[0011] Figure 1 This is a schematic flowchart of a depth map reconstruction method based on cost aggregation provided in an embodiment of this application; Figure 2 This is a schematic diagram of a time-series laser line image sequence provided in an embodiment of this application; Figure 3 This is a schematic diagram of a spliced ​​time-domain diagram provided in an embodiment of this application; Figure 4A This is a schematic diagram of a laser line image under the condition of a relatively dim laser line, provided in an embodiment of this application; Figure 4B This is a schematic diagram of a depth map reconstructed when the laser line is relatively dark, provided in an embodiment of this application; Figure 5 This is a schematic diagram of laser line images before and after enhancement processing provided in an embodiment of this application; Figure 6 This is a reconstructed depth map corresponding to an enhanced laser line image provided in an embodiment of this application; Figure 7 This is a schematic diagram of a depth map reconstruction device based on cost aggregation provided in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0013] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0014] Please see Figure 1 This is a flowchart illustrating a depth map reconstruction method based on cost aggregation provided in an embodiment of this application. Figure 1 As shown, the depth map reconstruction method based on cost aggregation may include the following steps: Step S100: Obtain a temporal laser line image sequence; wherein, the temporal laser line image sequence includes a set of temporally continuous images, and a laser line is projected onto the scene to be reconstructed in any laser line image, and the laser line moves at a constant speed with a fixed period in the scene to be reconstructed.

[0015] For example, a configured binocular structured light camera system can be used to scan the scene under test (also known as the scene to be reconstructed).

[0016] The laser projects a linear laser beam at a fixed period and speed, moving uniformly across the scene surface. Left and right cameras simultaneously capture a series of images, forming a temporal sequence of laser line images.

[0017] For example, each laser projection creates multiple parallel and uniformly spaced laser lines on the scene surface. These laser lines, as a whole, move at a constant speed across the scene surface at a fixed period. Each frame of the image synchronously captured by the binocular camera contains the modulation and deformation of these multiple laser lines.

[0018] For example, a schematic diagram of a time-series laser line image sequence can be shown as follows: Figure 2 As shown, in a complete scan cycle, the pixel distance that the laser line moves in the image is the same as the number of pixels between two adjacent laser lines.

[0019] Step S110: Obtain multiple stitched time-domain images based on the image sequence; wherein, one stitched time-domain image is obtained by stitching the same row of pixels of each laser line image in the image sequence in time sequence.

[0020] In this embodiment of the application, for any row of each laser line image in the image sequence, the pixel values ​​of that row in each laser line image can be extracted to obtain multiple rows of pixel values ​​(the number of rows is the same as the number of frames of the laser line images in the image sequence), and the multiple rows of pixel values ​​are spliced ​​together in time sequence to obtain a spliced ​​time domain image.

[0021] For example, the number of frames in the stitched temporal graph is the same as the number of rows in the laser line image in the image sequence, and the width of the stitched temporal graph is the same as the width of the laser line image in the image sequence.

[0022] For example, suppose there are T frames of laser line images in the image sequence, and the size of each laser line image is H (rows) × W (columns).

[0023] For each row h (0 ≤ h < H, row number starts from 0) of the laser line image (which can be called the original image), construct an independent stitched temporal map: Each time-domain plot has a size of T×W, where: Row index (vertical axis): Represents the time sequence, from row 0 (first frame) to row T-1 (last frame), arranged from top to bottom.

[0024] Column index (horizontal axis): Represents the position of a pixel in row h, from column 0 (leftmost) to column W-1 (rightmost), arranged from left to right.

[0025] The grayscale value of the t-th row and x-th column in the time-domain plot is the inverted grayscale value of the pixel in the h-th row and x-th column in the (t+1)-th frame image.

[0026] For example, suppose there are T=3 frames of images, each frame has a size of H=2 rows and W=3 columns.

[0027] For row h=0, construct the time-domain graph: The 0th row (top row) of the time-domain plot corresponds to the image row h=0 of the 0th frame.

[0028] The first row (middle) of the time-domain plot corresponds to the image row h=0 of the first frame.

[0029] The second row (bottom row) of the time-domain plot corresponds to the image row h=0 of the second frame.

[0030] Step S120: Based on the fixed slope monotonicity of the laser line region in the spliced ​​time domain diagram, the laser line region in the spliced ​​time domain diagram is enhanced using a cost aggregation method.

[0031] In this embodiment, the laser line region exhibits a diagonal line feature with a fixed slope in the stitched time-domain image obtained as described above. Based on this characteristic, a cost aggregation algorithm can be used to enhance the laser line region. This process uses the fixed slope as a constraint condition for path selection, strengthening paths that conform to the slope pattern and suppressing noise and background interference.

[0032] In one example, obtaining multiple stitched temporal images from an image sequence as described above can include: Invert the grayscale values ​​of each laser line image in the image sequence; The same rows of each laser line image after grayscale inversion are stitched together in time sequence to obtain multiple stitched time-domain images.

[0033] For example, considering that the laser line region is bright (high grayscale value) in the original image, but the core of the cost aggregation algorithm is usually to find the minimum cost path, the image grayscale can be inverted first to make the laser line region dark (low grayscale value, i.e., low cost), so that the aggregation algorithm is attracted to the laser line region.

[0034] For example, for any frame of a laser line image in an image sequence, its grayscale inverted image I can be calculated. inverted =255-I original (Taking an 8-bit image as an example), that is, for any pixel position in the inverted image, its gray value is equal to 255 minus the gray value of that pixel position in the original image, and a spliced ​​temporal map is constructed according to the inverted image sequence in the above manner.

[0035] For example, a schematic diagram of the stitched time-domain image obtained by stitching together the same rows of each laser line image after grayscale inversion in a time sequence can be shown as follows: Figure 3 As shown.

[0036] like Figure 3 As shown, with time as the vertical axis and the image row direction as the horizontal axis, the temporal variation of a particular row of images can be determined. The black diagonal lines (representing the laser line areas in the original image) represent the areas that need enhancement. The slope of the black diagonal lines can be calculated.

[0037] In one example, the slope of the black diagonal line can be determined based on the ratio of the number of frames of the laser line image in the image sequence to the number of pixels between adjacent laser lines in the laser line image.

[0038] For example, the slope = number of frames of laser line images in the image sequence / number of pixels between adjacent laser lines in the laser line images.

[0039] S130. Perform inverse stitching on the multiple stitched time-domain images after enhancement processing to reassemble them into an enhanced laser line image sequence.

[0040] In this embodiment of the application, when each spliced ​​time-domain image is enhanced in accordance with the manner described in the above embodiments, the multiple enhanced time-domain images (each corresponding to a row of the original image) can be reverse spliced, that is, the images are remapped from the "row-time" two-dimensional space back to the standard "row-column" image space, and the enhanced laser line image sequence is recombined.

[0041] Step S140: Reconstruct the depth map based on the enhanced laser line image sequence.

[0042] For example, based on the principle of stereo vision, stereo matching and triangulation can be performed on the enhanced laser line image sequence acquired by the left and right cameras to reconstruct the depth map of the scene.

[0043] It can be seen that, in Figure 1 In the method described, a temporal laser line image sequence is acquired. Based on the image sequence, multiple stitched temporal maps are obtained by stitching together temporal images. Then, based on the fixed slope monotonicity of the laser line region in the stitched temporal map, a cost aggregation method is used to enhance the laser line region in the stitched temporal map. Next, the enhanced stitched temporal maps are inversely stitched together to reconstruct an enhanced laser line image sequence. Depth map reconstruction is then performed based on this enhanced laser line image sequence. This method can improve the contrast of laser lines in the laser line image, effectively increasing the depth map reconstruction rate for scenes with dark areas or objects with very low contrast, thus improving the reliability and scene applicability of depth map reconstruction.

[0044] In some embodiments, the enhancement processing of the laser line region in the spliced ​​time domain image based on the fixed slope monotonicity of the laser line region in the spliced ​​time domain image using a cost aggregation method may include: For each pixel location in the stitched temporal graph, the aggregation cost of that pixel location is determined based on its original cost, continuation cost, and continuity cost. The calculation of both the continuation cost and the continuity cost is constrained by the monotonicity of the fixed slope.

[0045] For example, the purpose of cost aggregation is to find and enhance the laser line path that best fits a fixed slope (which can be denoted as k) in the time-domain graph. For each pixel position (which can be denoted as p) in the time-domain graph, its aggregated cost can be determined based on the original cost, the continuation cost, and the continuity cost.

[0046] For example, the original cost can characterize the initial matching cost of the pixel location in the current state.

[0047] For example, the value of the original cost can be the initial grayscale value of the pixel location in the stitched temporal map.

[0048] In the spliced ​​time-domain image after grayscale inversion, the grayscale values ​​of the points on the laser line are low, so the original cost is also low.

[0049] For example, the continuation cost can characterize the continuity of the path, propagating from the cost of the previous pixel (e.g., the left pixel). Its calculation is constrained by a fixed slope, and it preferentially selects paths consistent with the slope direction for propagation. By considering the aggregated cost of the previous pixel position in the same or adjacent state, it ensures that the cost can be continuation and accumulation on paths that conform to the preset fixed slope, thereby enhancing the continuous laser line trajectory.

[0050] For example, the continuity cost constrains the path change through a penalty parameter mechanism, the core of which is the expected change pattern of the state variable, which is determined by a fixed slope.

[0051] In one example, continuity cost is characterized by introducing a penalty parameter mechanism to characterize the constraint strength of the path, including: Based on a fixed slope, determine the expected change pattern of the state variables; For paths where the change in state conforms to the expected change pattern, their cost is used as a candidate continuity cost. For paths where the change in state deviates from the expected change pattern by a fixed unit, the cost plus the first penalty value is used as the candidate continuity cost. For paths where the state change does not conform to the expected change pattern, the minimum cost among all states of the previous pixel plus the second penalty value is used as the candidate continuity cost. The candidate continuity cost with the lowest substitution value is determined as the final continuity cost. The second penalty value is greater than the first penalty value.

[0052] For example, for a path whose state change conforms to the expected change pattern, its candidate generation value can be the corresponding generation value of the previous pixel (the previous pixel in the aggregation direction). For a path where the state change deviates from the expected change pattern by a fixed unit, its candidate value can be the sum of the corresponding value of the previous pixel and the first penalty value. For paths where the state change does not conform to the expected change pattern, the candidate cost value can be the sum of the minimum cost value among all states of the previous pixel and the second penalty value.

[0053] In some embodiments, the above-described inverse stitching of multiple stitched temporal images after enhancement processing to reassemble them into an enhanced laser line image sequence may include: For any augmented spliced ​​temporal map, determine the maximum and minimum aggregation costs of that augmented spliced ​​temporal map; Based on the maximum and minimum aggregation costs of the enhanced spliced ​​time-domain graph, the enhanced spliced ​​time-domain graph is normalized. The normalized spliced ​​time-domain images are inversely spliced ​​and recombined into an enhanced laser line image sequence.

[0054] For example, for any enhanced stitched temporal map, the maximum and minimum values ​​of the aggregation cost can be determined based on the aggregation cost of each pixel position in the enhanced stitched temporal map, and the enhanced stitched temporal map can be normalized based on the determined maximum and minimum values.

[0055] For example, for any pixel location in the enhanced stitched temporal map, the aggregation cost of that pixel location can be normalized in the following way: V normalized =255×(V-Min) / (Max-Min) Among them, V normalized V is the normalized aggregation cost for this pixel location, V is the current aggregation cost for this pixel location, Min is the minimum aggregation cost in the stitched temporal graph after enhancement, and Max is the maximum aggregation cost in the stitched temporal graph after enhancement.

[0056] Using the above method, the aggregation cost can be mapped to a grayscale range of 0~255 (taking an 8-bit image as an example).

[0057] For example, the normalized stitched temporal maps (each corresponding to a row of the original image) can be reassembled.

[0058] For example, all pixels in the first row of each normalized stitched temporal image can be extracted and stitched together to form the corresponding row of the first frame of the enhanced image sequence; all pixels in the second row can be extracted and stitched together to form the corresponding row of the second frame, and so on. Finally, an enhanced laser line image sequence with the same number of frames as the original input sequence can be reconstructed.

[0059] It should be noted that, in the embodiments of this application, if the laser line image is subjected to grayscale inversion during the process of enhancing the laser line region in the spliced ​​time domain image using the cost aggregation method, then after obtaining the enhanced laser line image sequence in the above manner, it is also necessary to perform grayscale inversion on the obtained enhanced laser line image sequence so that the laser line appears as a high-brightness, clear stripe in the final output enhanced image sequence.

[0060] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below in conjunction with specific application scenarios.

[0061] In this embodiment, considering the presence of objects with strong light absorption or reflection in a real-world scenario, such as black velvet or rough metal surfaces, the acquired laser line image will appear as follows: Figure 4A As shown ( Figure 4A (Acquired through low-exposure simulation of dark areas), weak laser line contrast leads to severe loss of depth information. The reconstructed depth map can be as follows: Figure 4B As shown.

[0062] by Figure 4B Taking the red rectangular area as an example, depth information is missing within the red rectangular area (black represents no depth information).

[0063] To improve the depth map reconstruction rate, this embodiment provides a cost-aggregation method for reconstructing the original image sequence (temporally, by continuously moving a group of consecutive images with a smaller linewidth within a repetitive fixed period, as illustrated in the diagram). Figure 2 The enhancement scheme (shown) improves the contrast of laser lines in the image.

[0064] In this embodiment, pixels in the same row of images (laser line images) in the original image sequence can be stitched together to obtain a stitched time-domain image. A schematic diagram of the stitched time-domain image can be shown below. Figure 3 As shown.

[0065] like Figure 3 As shown, the grayscale of pixels in the laser line region exhibits a single increasing property with a fixed slope in the stitched time-domain graph. Based on this, a cost aggregation method can be constructed to enhance the original image sequence.

[0066] In this embodiment, Figure 3 The black diagonal lines in the image (representing the laser lines in the original image) indicate the areas that need enhancement. The slope of the black diagonal lines can be calculated (e.g., the number of line-shifted images / the number of pixels between the two laser lines).

[0067] For example, cost aggregation can be implemented in the following manner: M(pr) can be determined in the following way: The farther away from d, the greater the cost; the closer to d, the smaller the cost.

[0068] For example, d can represent whether the current traversal point is on a path with a fixed slope (e.g., ...). Figure 3 On the black diagonal line (in the diagram), a small cost is added near a fixed slope (e.g., the distance to the nearest point on the black diagonal line is 1), and a large cost is added when the slope is significantly different (e.g., the distance to the nearest point on the black diagonal line is more than 1).

[0069] C(p,d) is the original cost, and its value can be the initial grayscale value of the current pixel.

[0070] M(pr) includes both the continuation cost and the continuity cost.

[0071] For example, the continuation cost can characterize the continuity of the path, propagating from the cost of the previous pixel (e.g., the left pixel). Its calculation is constrained by a fixed slope, and it preferentially selects paths consistent with the slope direction for propagation. By considering the aggregated cost of the previous pixel position in the same or adjacent state, it ensures that the cost can be continuation and accumulation on paths that conform to the preset fixed slope, thereby enhancing the continuous laser line trajectory.

[0072] For example, the continuity cost constrains the path change through a penalty parameter mechanism, the core of which is the expected change pattern of the state variable, which is determined by a fixed slope.

[0073] For example, candidate paths from the previous pixel pr to the current pixel p can be determined, and the cost of each candidate path can be determined: Path 1 (strictly continuous): L(pr, d). If the path perfectly follows the laser slope, there is no penalty, and the cost is directly inherited.

[0074] Path 2 (slightly discontinuous): L(pr, d±1)+P1. If the path deviates slightly from the slope, a small penalty P1 is added.

[0075] Path 3 (Severely Discontinuous / Reboot): If the path deviates completely from the slope, find the path with the lowest cost and add a large penalty P2.

[0076] In the above formula, min L(pr, i) is a normalization term that represents the minimum cost of all points in the previous column of data, preventing numerical overflow and maintaining numerical stability.

[0077] Through the above cost aggregation process, a cost aggregation map (i.e., the stitched temporal map after the above enhancement process) is obtained. The maximum and minimum values ​​of the cost aggregation map are found, the cost aggregation image is normalized, and the normalized cost aggregation map is inversely stitched together to reassemble into an enhanced laser line image sequence. The grayscale of the enhanced laser line image sequence is inverted to obtain the final enhanced laser line image sequence.

[0078] For example, the laser line images before and after enhancement processing can be respectively as follows: Figure 4A and Figure 5 As shown (only one frame of the laser line image in the image sequence is shown in the figure).

[0079] For example, depth map reconstruction can be performed based on the enhanced laser line image sequence.

[0080] For example, the laser line center can be extracted frame by frame from the enhanced laser line image sequence acquired by the left and right cameras, and a sub-pixel precision algorithm can be used to locate the laser line center position. Thus, for each frame of the image, a set of two-dimensional coordinates of the laser line center point can be obtained.

[0081] Match the center points of the laser lines in the left and right camera images.

[0082] Based on the principle of triangulation, the three-dimensional spatial coordinates of the center point of each successfully matched laser line are calculated. Subsequently, these sparse three-dimensional point clouds are mapped onto the image grid, and through interpolation, fusion and other processing, a regular depth map covering the entire field of view is generated.

[0083] The depth information obtained from multiple frames of laser line scanning is fused, and the integrity, density and accuracy of the depth map are improved by using time-series data. The final result is then optimized by filtering and other processes to obtain a high-quality depth map.

[0084] For example, the reconstructed depth map corresponding to the enhanced laser line image can be as follows: Figure 6 As shown.

[0085] like Figure 5 and Figure 6 As shown, compared to the reconstructed depth map corresponding to the original image, the reconstruction rate of the reconstructed depth map corresponding to the enhanced laser line image is significantly increased. Figure 6 The depth information within the red rectangular area is more complete.

[0086] The method provided in this application has been described above. The apparatus provided in this application is described below: Please see Figure 7 This is a schematic diagram of a depth map reconstruction device based on cost aggregation provided in an embodiment of this application, as shown below. Figure 7 As shown, the depth map reconstruction apparatus based on cost aggregation may include: An acquisition unit is used to acquire a time-series laser line image sequence; wherein, the time-series laser line image sequence includes a set of time-series continuous images, and a laser line is projected onto the scene to be reconstructed in any laser line image, and the laser line moves at a constant speed with a fixed period in the scene to be reconstructed; The stitching unit is used to obtain multiple stitched temporal maps based on the image sequence; wherein, a stitched temporal map is obtained by stitching the same row of pixels of each laser line image in the image sequence in a temporal order; An enhancement processing unit is used to enhance the laser line region in the spliced ​​time domain diagram by using a cost aggregation method based on the fixed slope monotonicity of the laser line region in the spliced ​​time domain diagram. The recombination unit is used to inversely stitch together multiple stitched time-domain images after enhancement processing, and reassemble them into an enhanced laser line image sequence. The reconstruction unit is used to reconstruct the depth map based on the enhanced laser line image sequence.

[0087] For example, the specific implementation process of each functional unit in the cost aggregation-based depth map reconstruction device for implementing cost aggregation-based depth map reconstruction can be found in the relevant descriptions in the above embodiments.

[0088] This application provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the package measurement method described above.

[0089] Please see Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 801 and a memory 802 storing machine-executable instructions. The processor 801 and the memory 802 can communicate via a system bus 803. Furthermore, by reading and executing the machine-executable instructions corresponding to the package measurement logic in the memory 802, the processor 801 can execute the package measurement method described above.

[0090] The memory 802 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0091] In some embodiments, a machine-readable storage medium, such as Figure 8 The memory 802 in the memory, which is a machine-readable storage medium, stores machine-executable instructions that, when executed by a processor, implement the package measurement method described above. For example, the storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

Claims

1. A depth map reconstruction method based on cost aggregation, characterized in that, include: Acquire a temporal laser line image sequence; wherein, the temporal laser line image sequence includes a set of temporally continuous images, and a laser line is projected onto the scene to be reconstructed in any laser line image, and the laser line moves at a constant speed with a fixed period in the scene to be reconstructed; Multiple stitched temporal maps are obtained based on the image sequence; wherein, a stitched temporal map is obtained by stitching the same row of pixels of each laser line image in the image sequence in a temporal order; Based on the fixed slope monotonicity of the laser line region in the spliced ​​time-domain diagram, a cost-based aggregation method is used to enhance the laser line region in the spliced ​​time-domain diagram. The multiple stitched temporal images after enhancement are inversely stitched together and recombined into an enhanced laser line image sequence. Depth map reconstruction is performed based on the enhanced laser line image sequence.

2. The method according to claim 1, characterized in that, The process of obtaining multiple stitched temporal images based on the image sequence includes: Invert the grayscale values ​​of each laser line image in the image sequence; The same rows of each laser line image after grayscale inversion are stitched together in time sequence to obtain multiple stitched time-domain images.

3. The method according to claim 1, characterized in that, The enhancement process for the laser line region in the spliced ​​time-domain image, based on the fixed slope monotonicity of the laser line region, employs a cost aggregation method, including: For each pixel position in the stitched time-domain graph, the aggregation cost of the pixel position is determined based on the original cost, the continuation cost, and the continuity cost of that pixel position. The calculation of both the continuation cost and the continuity cost is constrained by the fixed slope monotonicity.

4. The method according to claim 3, characterized in that, The continuation cost represents the cost that propagates from the previous pixel position along the aggregation direction; And / or, The continuity cost is characterized by introducing a penalty parameter mechanism to represent the constraint strength of the path, including: Based on the fixed slope, the expected change pattern of the state variables is determined; For paths whose state changes conform to the expected change pattern, their cost is used as candidate continuity costs. For paths where the state change amount deviates from the expected change pattern by a fixed unit, the cost plus the first penalty value is used as the candidate continuity cost. For paths where the state change amount does not conform to the expected change pattern, the minimum cost value among all states of the previous pixel plus the second penalty value is used as the candidate continuity cost. The candidate continuity cost with the lowest substitution value is determined as the final continuity cost. The second penalty value is greater than the first penalty value.

5. The method according to claim 1, characterized in that, The fixed slope is determined based on the ratio of the number of frames of the laser line image in the image sequence to the ratio of the number of pixels between adjacent laser lines in the laser line image.

6. The method according to claim 1, characterized in that, The step of inversely stitching together multiple stitched time-domain images after enhancement processing to reconstruct an enhanced laser line image sequence includes: For any augmented spliced ​​temporal map, determine the maximum and minimum aggregation costs of that augmented spliced ​​temporal map; Based on the maximum and minimum aggregation costs of the enhanced spliced ​​time-domain graph, the enhanced spliced ​​time-domain graph is normalized. The normalized spliced ​​time-domain images are inversely spliced ​​and recombined into an enhanced laser line image sequence.

7. A depth map reconstruction device based on cost aggregation, characterized in that, include: An acquisition unit is used to acquire a time-series laser line image sequence; wherein, the time-series laser line image sequence includes a set of time-series continuous images, and a laser line is projected onto the scene to be reconstructed in any laser line image, and the laser line moves at a constant speed with a fixed period in the scene to be reconstructed; The stitching unit is used to obtain multiple stitched temporal maps based on the image sequence; wherein, a stitched temporal map is obtained by stitching the same row of pixels of each laser line image in the image sequence in a temporal order; An enhancement processing unit is used to enhance the laser line region in the spliced ​​time domain diagram by using a cost aggregation method based on the fixed slope monotonicity of the laser line region in the spliced ​​time domain diagram. The recombination unit is used to inversely stitch together multiple stitched time-domain images after enhancement processing, and reassemble them into an enhanced laser line image sequence. The reconstruction unit is used to reconstruct the depth map based on the enhanced laser line image sequence.

8. The apparatus according to claim 7, characterized in that, The stitching unit obtains multiple stitched temporal maps based on the image sequence, including: Invert the grayscale values ​​of each laser line image in the image sequence; The same rows of each laser line image after grayscale inversion are stitched together in time sequence to obtain multiple stitched time-domain images; And / or, The enhancement processing unit, based on the fixed slope monotonicity of the laser line region in the stitched time-domain image, performs enhancement processing on the laser line region in the stitched time-domain image using a cost aggregation method, including: For each pixel position in the stitched time-domain graph, the aggregation cost of the pixel position is determined based on the original cost, the continuation cost, and the continuity cost of that pixel position. The calculation of both the continuation cost and the continuity cost is constrained by the fixed slope monotonicity. Wherein, the continuation cost represents the cost that propagates from the previous pixel position along the aggregation direction; And / or, The continuity cost is characterized by introducing a penalty parameter mechanism to represent the constraint strength of the path, including: Based on the fixed slope, the expected change pattern of the state variables is determined; For paths whose state changes conform to the expected change pattern, their cost is used as candidate continuity costs. For paths where the state change amount deviates from the expected change pattern by a fixed unit, the cost plus the first penalty value is used as the candidate continuity cost. For paths where the state change amount does not conform to the expected change pattern, the minimum cost value among all states of the previous pixel plus the second penalty value is used as the candidate continuity cost. The candidate continuity cost with the lowest substitution value is determined as the final continuity cost. Wherein, the second penalty value is greater than the first penalty value; And / or, The fixed slope is determined based on the ratio of the number of frames of the laser line image in the image sequence to the number of pixels between adjacent laser lines in the laser line image; And / or, The reassembly unit performs inverse stitching on multiple stitched time-domain images after enhancement processing, reassembling them into an enhanced laser line image sequence, including: For any augmented spliced ​​temporal map, determine the maximum and minimum aggregation costs of that augmented spliced ​​temporal map; Based on the maximum and minimum aggregation costs of the enhanced spliced ​​time-domain graph, the enhanced spliced ​​time-domain graph is normalized. The normalized spliced ​​time-domain images are inversely spliced ​​and recombined into an enhanced laser line image sequence.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-6.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-6.

Images