Depth data acquisition method and apparatus, electronic device, and machine-readable storage medium

Abstract

The application provides a depth data acquisition method and device, electronic equipment and machine readable storage medium. The method comprises: acquiring multiple frames of gray scale images of a measured scene under different textures by an image acquisition device; determining a background gray scale image according to the multiple frames of gray scale images; obtaining multiple frames of texture-enhanced gray scale images according to the multiple frames of gray scale images and the background gray scale image; encoding and synthesizing the multiple frames of gray scale images to obtain a first synthesized image; and encoding and synthesizing the multiple frames of texture-enhanced gray scale images to obtain a second synthesized image; determining a first depth image according to the first synthesized image; and determining a second depth image according to the second synthesized image; and fusing the first depth image and the second depth image to obtain a final depth image. The method can obtain a final depth image that takes into account high dynamic range and edge accuracy.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and machine-readable storage medium for acquiring deep data. Background Technology

[0002] Currently, in the field of grayscale matching-based 3D detection, a camera is typically used to acquire images of the test scene illuminated by a coded pattern generator. Then, the spatial relationship of texture information acquired by the camera is used to recover the depth information of the test scene. However, during the reconstruction process, the grayscale image captured in a single shot is limited by laser power and texture contrast. The texture in the image is often sparse and has limited contrast, resulting in limited ability to recover depth data of the scene. It is difficult to obtain depth maps with high dynamic range and edge accuracy for different scenes. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus, electronic device, and machine-readable storage medium for acquiring depth data.

[0004] According to a first aspect of the embodiments of this application, a depth data acquisition method is provided, including:

[0005] Multiple grayscale images of different textures in the tested scene are acquired using an image acquisition device.

[0006] Based on the aforementioned multi-frame grayscale images, determine the background grayscale image;

[0007] Based on the multi-frame grayscale images and the background grayscale image, multi-frame texture-enhanced grayscale images are obtained;

[0008] The multi-frame grayscale images are encoded and synthesized to obtain a first composite image; and the multi-frame texture-enhanced grayscale images are encoded and synthesized to obtain a second composite image.

[0009] Based on the first composite image, a first depth map is determined; and based on the second composite image, a second depth map is determined.

[0010] The first depth map and the second depth map are fused to obtain the final depth map.

[0011] According to a second aspect of the embodiments of this application, a depth data acquisition apparatus is provided, comprising:

[0012] The acquisition unit is used to acquire multiple frames of grayscale images of different textures in the scene under test through an image acquisition device.

[0013] The first determining unit is used to determine the background grayscale image based on the multi-frame grayscale images;

[0014] The enhancement unit is used to obtain multi-frame texture-enhanced grayscale images based on the multi-frame grayscale images and the background grayscale images;

[0015] The synthesis unit is configured to encode and synthesize the multi-frame grayscale images to obtain a first synthesized image; and to encode and synthesize the multi-frame texture-enhanced grayscale images to obtain a second synthesized image.

[0016] The second determining unit is configured to determine a first depth map based on the first composite image; and to determine a second depth map based on the second composite image.

[0017] The fusion unit is used to fuse the first depth map and the second depth map to obtain the final depth map.

[0018] According to a third aspect of the present application, an electronic device is provided, including a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0019] According to a fourth aspect of the embodiments of this application, a machine-readable storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0020] The depth data acquisition method of this application embodiment acquires multiple frames of grayscale images under different textures of the tested scene, determines a background grayscale image based on the acquired multiple frames of grayscale images, and obtains multiple frames of texture-enhanced grayscale images based on the multiple frames of grayscale images and the background grayscale image to enhance the effective texture in the grayscale images. Furthermore, on the one hand, the original multiple frames of grayscale images can be encoded and synthesized to obtain a first synthesized image, and a first depth map can be determined based on the first synthesized image. On the other hand, the multiple frames of texture-enhanced grayscale images can be encoded and synthesized to obtain a second synthesized image, and a second depth map can be determined based on the second synthesized image. By encoding and synthesizing multiple frames of grayscale images, the information of multiple frames is fully utilized to obtain a depth map with higher edge precision. In addition, by enhancing the effective texture, a high dynamic range depth map is obtained. Thus, by fusing the first depth map and the second depth map, a final depth map that takes into account both high dynamic range and edge precision is obtained. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a depth data acquisition method provided in an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating a depth data acquisition method based on a binocular depth camera that balances high dynamic range and edge accuracy, as provided in an embodiment of this application.

[0023] Figure 3 This is a flowchart illustrating a depth data acquisition method provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of a depth data acquisition device provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0028] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0029] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0030] Please see Figure 1 This is a flowchart illustrating a depth data acquisition method provided in an embodiment of this application. Figure 1 As shown, the depth data acquisition method may include the following steps:

[0031] Step S100: Obtain multiple grayscale images of different textures of the scene under test using an image acquisition device.

[0032] For example, the image acquisition device is an image acquisition device with depth information acquisition capability, which may include, but is not limited to, a monocular depth camera or a multi-view depth camera.

[0033] For example, by adjusting the pattern and / or projection angle of the texture projected onto the scene under test, multiple different textures of the scene under test can be projected to obtain multiple grayscale images of the scene under different textures, ensuring that the texture information is rich enough.

[0034] For example, when the image acquisition device is a multi-view depth camera, the grayscale image under any texture of the scene under test includes grayscale images acquired by multiple cameras.

[0035] For example, taking a binocular depth camera as an example, the grayscale image of the scene under test under any texture includes the grayscale image of the left eye and the grayscale image of the right eye.

[0036] It should be noted that when acquiring multiple frames of grayscale images of different textures of the scene under test through an image acquisition device, the imaging parameters of the image acquisition device (such as exposure time, gain, etc.) must be kept consistent, and the brightness of the texture light source must be kept consistent (either the same or the brightness difference is within the preset difference range) when acquiring each frame of grayscale image.

[0037] Step S110: Determine the background grayscale image based on the acquired multi-frame grayscale images.

[0038] Step S120: Based on the multi-frame grayscale image and the background grayscale image, obtain the multi-frame texture-enhanced grayscale image.

[0039] In this embodiment of the application, in order to obtain a high dynamic depth map, a background grayscale map can be determined based on the acquired multi-frame grayscale map, and effective texture enhancement and background suppression (also known as background texture suppression) can be performed on the acquired multi-frame grayscale map based on the background grayscale map to highlight the effective texture and obtain a multi-frame texture-enhanced (i.e., effective texture-enhanced) grayscale map.

[0040] Step S130: Encode and synthesize multiple grayscale images to obtain a first composite image; and encode and synthesize multiple texture-enhanced grayscale images to obtain a second composite image.

[0041] In this embodiment of the application, in order to merge the texture information of multiple grayscale images into the same frame image, the information of multiple grayscale images can be merged into the same frame image so as to make full use of the information of multiple grayscale images and obtain a depth map with higher edge accuracy.

[0042] Accordingly, the multiple grayscale images obtained in step S100 can be encoded and synthesized to obtain the corresponding synthesized image (referred to as the first synthesized image in this paper).

[0043] Furthermore, the multi-frame texture-enhanced grayscale images obtained in step S120 are encoded and synthesized to obtain the corresponding synthesized image (referred to as the second synthesized image in this paper).

[0044] Step S140: Determine a first depth map based on the first composite image; and determine a second depth map based on the second composite image.

[0045] Step S150: Merge the first depth map and the second depth map to obtain the final depth map.

[0046] In this embodiment of the application, after obtaining the first composite image in a certain manner, the corresponding depth image (referred to as the first depth image in this document) can be determined based on the first composite image.

[0047] For example, for a binocular depth camera, the disparity of each pixel position in the left and right images can be determined based on the matching results of the pixels in the left and right images, and then the depth information of each pixel position can be determined to obtain the corresponding depth map.

[0048] For a monocular depth camera, the depth information of each pixel position can be determined based on the grayscale image and the preset calibration image to obtain the corresponding depth map.

[0049] Similarly, once the second composite map is obtained in the same way, the corresponding depth map (referred to as the second depth map in this article) can be determined based on the second composite map.

[0050] In this embodiment of the application, since the original grayscale image has both active and passive textures (i.e., effective texture and background texture), the edge details of its corresponding depth map (i.e., the first depth map mentioned above) are usually relatively clear; while the effective texture in the texture enhancement image is enhanced, and its corresponding depth map (i.e., the second depth map mentioned above) has a higher dynamic range. Therefore, by fusing the first depth map and the second depth map, a depth map that takes into account both high dynamic range and edge accuracy can be obtained (which can be called the final depth map).

[0051] It can be seen that, in Figure 1 In the illustrated method flow, multiple frames of grayscale images under different textures of the tested scene are acquired. Based on the acquired multiple frames of grayscale images, a background grayscale image is determined. Based on the multiple frames of grayscale images and the background grayscale image, multiple frames of texture-enhanced grayscale images are obtained to enhance the effective texture in the grayscale images. Then, on the one hand, the original multiple frames of grayscale images can be encoded and synthesized to obtain a first synthesized image, and a first depth map is determined based on the first synthesized image. On the other hand, the multiple frames of texture-enhanced grayscale images are encoded and synthesized to obtain a second synthesized image, and a second depth map is determined based on the second synthesized image. Through the encoding and synthesis of multiple frames of grayscale images, the information of multiple frames is fully utilized to obtain a depth map with higher edge precision. In addition, by enhancing the effective texture, a high dynamic range depth map is obtained. Thus, by fusing the first depth map and the second depth map, a final depth map that takes into account both high dynamic range and edge precision is obtained.

[0052] In some embodiments, the image acquisition device includes at least one camera for image acquisition, and the at least one camera acquires images in any texture of the scene under test.

[0053] The above method of determining the background grayscale image based on multiple grayscale images can include:

[0054] For multiple grayscale images acquired through the same camera, the minimum grayscale value of each pixel position in the multiple grayscale images is taken, and the corresponding background grayscale image is reconstructed.

[0055] For example, the background grayscale image can be determined by effectively suppressing the texture of the acquired grayscale image.

[0056] For example, in a multi-frame grayscale image with consistent texture light source brightness, the average brightness of the background will generally also remain consistent. Since effective textures increase brightness, when the grayscale value at the same pixel location is the smallest in a multi-frame image, it means that the pixel location belongs to the background in that frame.

[0057] Therefore, for multiple grayscale images acquired through the same camera, a background grayscale image that effectively suppresses texture can be reconstructed by taking the minimum grayscale value at the same pixel position in multiple frames.

[0058] For example, assuming the image acquisition device mentioned above is a binocular depth camera, when acquiring images, the left eye camera (or left camera) and the right eye camera (or right camera) will respectively acquire multiple frames of grayscale images under different textures of the scene being tested.

[0059] For a multi-frame grayscale image captured by the left eye camera, the background grayscale image corresponding to the left eye camera can be obtained by taking the minimum grayscale value of each pixel position in the multi-frame grayscale image.

[0060] For multiple frames of grayscale images captured by the right eye camera, the background grayscale image corresponding to the right eye camera can be obtained by taking the minimum grayscale value of each pixel position in the multiple frames of grayscale images.

[0061] In some embodiments, obtaining multi-frame texture-enhanced grayscale images based on multi-frame grayscale images and background grayscale images may include:

[0062] For any given grayscale image, based on the difference between the grayscale value of each pixel position in the grayscale image and the grayscale value of the same pixel position in the corresponding background grayscale image, the grayscale values ​​of each pixel position in the grayscale image are reconstructed to obtain the texture-enhanced grayscale image corresponding to the grayscale image of that frame.

[0063] For example, considering that for any pixel location, the greater the difference between the gray value of that pixel location and the gray value of that pixel location in the background grayscale image, the greater the probability that the pixel location is a valid texture; and the closer the gray value of that pixel location is to the gray value of that pixel location in the background grayscale image, the greater the probability that the pixel location is the background.

[0064] Therefore, given that the background grayscale image has been determined in the manner described above, for any frame grayscale image, the grayscale values ​​of each pixel position in the frame grayscale image can be reconstructed based on the difference between the grayscale values ​​of each pixel position in the frame grayscale image and the grayscale values ​​of the same pixel position in the corresponding background grayscale image. Pixels belonging to the background are suppressed, and pixels with effective texture are enhanced, thus obtaining a texture-enhanced grayscale image corresponding to the frame grayscale image.

[0065] In one example, the above-mentioned reconstruction of the grayscale values ​​of each pixel position in the grayscale image based on the difference between the grayscale value of each pixel position in the frame grayscale image and the grayscale value of the same pixel position in the corresponding background grayscale image may include:

[0066] For any pixel position in the grayscale image of the frame, a weighted operation is performed based on the difference between the grayscale value of the pixel position in the grayscale image of the frame and the grayscale value of the pixel position in the corresponding background grayscale image, as well as the grayscale value of the pixel position in the grayscale image of the frame, to obtain the reconstructed grayscale value of the pixel position.

[0067] For example, for any frame of grayscale image, the grayscale values ​​of each pixel in the frame of grayscale image can be reconstructed using the following formula:

[0068]

[0069] in, Let x be the grayscale value of the pixel at position (x, y) in the grayscale image of this frame. This represents the grayscale value at pixel position (x, y) in the background grayscale image. The grayscale value at the reconstructed pixel position (x, y) as well as These are weighting coefficients, all ranging from 0 to 1, and + =1.

[0070] It should be noted that, in the embodiments of this application, for any frame of grayscale image, the effective texture can be enhanced and the background texture suppressed by directly amplifying the difference between the grayscale value of each pixel position in the frame of grayscale image and the grayscale value of the same pixel position in the corresponding background grayscale image by a preset factor (the factor is greater than 1).

[0071] In some embodiments, the image acquisition device includes at least one camera for image acquisition, and the at least one camera acquires images in any texture of the scene being tested.

[0072] The above-mentioned encoding and synthesis of multiple grayscale images to obtain the first synthesized image may include:

[0073] For multiple grayscale images acquired through the same camera, interpolation is performed on these multiple grayscale images to obtain a first composite image of multiple frames with enlarged resolution;

[0074] The first composite image is obtained by image fusion of the multiple frames of the first image to be synthesized.

[0075] The above-mentioned encoding and synthesis of multi-frame texture-enhanced grayscale images to obtain a second composite image may include:

[0076] For multiple frames of grayscale images acquired through the same camera, corresponding to multiple frames of texture-enhanced grayscale images, interpolation is performed on these multiple frames of texture-enhanced grayscale images to obtain a second multi-frame image to be synthesized after resolution magnification.

[0077] The second composite image is obtained by image fusion of the multiple frames of the second image to be synthesized.

[0078] For example, since the purpose of encoding and synthesizing multiple frames of images is to merge the texture information of multiple frames into the same image, if the resolution of the synthesized image is the same as the resolution of the original grayscale image, the texture information density of the synthesized image may be too high, and the texture information may be too dense.

[0079] Accordingly, when encoding and synthesizing multiple frames of grayscale images acquired by the same camera, interpolation can be performed on each frame of grayscale image, such as bilinear interpolation, to obtain a grayscale image with increased resolution (which can be called the first image to be synthesized), and image fusion can be performed on the multi-frame grayscale images with increased resolution to obtain the first synthesized image.

[0080] For example, the resolution magnification factor of each grayscale image is the same.

[0081] Similarly, for multiple frames of grayscale images acquired by the same camera, which correspond to multiple frames of texture-enhanced grayscale images, interpolation can be performed on these multiple frames of texture-enhanced grayscale images to obtain a texture-enhanced grayscale image with increased resolution (which can be called the second image to be synthesized). Image fusion can then be performed on these multiple frames of the second image to be synthesized to obtain the second synthesized image.

[0082] In one example, the above-mentioned image fusion of the first composite image from multiple frames to be synthesized to obtain the first composite image may include:

[0083] Take the maximum grayscale value of each pixel position in the first composite image of the multiple frames to be synthesized to obtain the first composite image;

[0084] The above-mentioned image fusion of the multiple frames of the second composite image to obtain the second composite image may include:

[0085] The maximum grayscale value of each pixel position in the second composite image is taken to obtain the second composite image.

[0086] For example, image fusion can be performed on each image to be synthesized (such as the first image to be synthesized or the second image to be synthesized) by taking the maximum value of each pixel.

[0087] Taking the first image to be synthesized as an example, for multiple frames of the first image to be synthesized, the maximum gray value of each pixel position in the multiple frames of the first image to be synthesized can be taken to obtain the first synthesized image. That is, for any pixel position, the maximum gray value of that pixel position in the multiple frames of the first image to be synthesized is determined as the gray value of that pixel position in the first synthesized image.

[0088] In some embodiments, the image acquisition device is a multi-view depth camera; the number of image frames included in the first composite image and the number of image frames included in the second composite image are consistent with the number of views of the multi-view depth camera; the resolution of the first composite image and the second composite image are both greater than the resolution of the original grayscale image;

[0089] The determination of the first depth map based on the first composite image may include:

[0090] Sampling and matching of each frame in the first composite image are performed to obtain the first depth map at the original resolution.

[0091] Determining a second depth map based on the second composite map may include:

[0092] The images of each frame in the second composite image are sampled and matched to obtain the second depth map at the original resolution.

[0093] For example, taking a binocular depth camera as the image acquisition device, the first composite image obtained in the above manner may include a first left-eye composite image and a first right-eye composite image.

[0094] For example, each pixel in the first left-eye composite image and the first right-eye composite image can be matched to determine the disparity of each pixel in the left and right eye images, thereby obtaining the corresponding depth map (i.e., the first depth map mentioned above).

[0095] Similarly, the second composite image may include a second left-eye composite image and a second right-eye composite image. Each pixel in the second left-eye composite image and the second right-eye composite image can be matched to determine the disparity of each pixel in the left and right eye images, thereby obtaining the corresponding depth map (i.e., the second depth map mentioned above).

[0096] For example, in order to improve matching efficiency, the composite image (first composite image or second composite image) can be sampled and matched, that is, the composite image is sampled according to a preset sampling method and the sampled points are matched.

[0097] In one example, the above sampling and matching of each frame in the first synthetic image to obtain the first depth map at the original resolution may include:

[0098] Based on the resolution of the first composite image and the resolution of the original grayscale image, each frame of the first composite image is sampled at intervals, and the sampling points of each frame are matched to obtain the first depth map with the original resolution.

[0099] The above-mentioned sampling and matching of each frame in the second composite image to obtain the second depth map at the original resolution includes:

[0100] Based on the resolution of the second composite image and the resolution of the original grayscale image, each frame of the second composite image is sampled at intervals, and the sampling points of each frame are matched to obtain the second depth map with the original resolution.

[0101] For example, the sampling rate can be determined based on the ratio of the resolution of the composite image to that of the original grayscale image.

[0102] For example, assuming the resolution of the composite image (first composite image or second composite image) is twice the resolution of the original grayscale image, then interval sampling can be used, that is, one sampling point is obtained for every two pixels, and the sampling point corresponds to the valid point of the original grayscale image.

[0103] In one example, the above matching of sampling points for each frame of the image can include:

[0104] Based on the sampling disparity, the sampling points of each frame of the image are matched.

[0105] For example, since the composite image is upscaled relative to the original grayscale image, the disparity range of each pixel in the composite image is also upscaled proportionally.

[0106] When performing sampling matching, you can choose not to sample disparity, or sample disparity and match sampling points based on the sampled disparity, depending on efficiency requirements.

[0107] For example, to improve matching efficiency, disparity can be sampled, and the sampled points of each frame image can be matched based on the sampled disparity.

[0108] For example, assuming the resolution of the synthesized image is twice that of the original grayscale image, disparity can be sampled, and sampling points can be matched in multiples of 2 disparity.

[0109] In some embodiments, the above-described fusion of the first depth map and the second depth map to obtain the final depth map may include:

[0110] The second depth map is subjected to edge erosion processing to obtain the processed second depth map;

[0111] The first depth map and the processed second depth map are fused together to obtain the final depth map.

[0112] For example, since the second depth map is obtained based on a texture-enhanced grayscale image, although its dynamic range is higher, its edge precision will be reduced to a certain extent. In order to avoid the influence of the second depth map on the edge precision of the final depth map, edge erosion processing can be performed on the second depth map, and the first depth map and the processed second depth map can be fused to obtain the final depth map, so as to improve the edge precision of the final depth map.

[0113] In one example, fusing the first depth map and the processed second depth map to obtain the final depth map can include:

[0114] For any pixel location, if the pixel value of that pixel location in both the first depth map and the processed second depth map is greater than 0, then a weighted average is performed on the pixel value of that pixel location in the first depth map and the pixel value of that pixel location in the processed second depth map to obtain the pixel value of that pixel location in the final depth map.

[0115] If the pixel value at this pixel location is 0 in both the first depth map and the processed second depth map, then the pixel value at this pixel location in the final depth map is determined to be 0.

[0116] If either the pixel value at that pixel location in the first depth map or the processed second depth map is 0, then the pixel value at that pixel location in the final depth map is determined based on the non-zero pixel value at that pixel location.

[0117] For example, the fusion of the first depth map and the processed second depth map can be achieved using the following formula:

[0118]

[0119] in, Let be the pixel value at pixel position (x, y) in the first depth map. The pixel value at pixel position (x, y) in the processed second depth map. This represents the pixel value at the pixel position (x, y) in the final depth map. and These are weighting coefficients. + =1.

[0120] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below in conjunction with specific application scenarios.

[0121] In this embodiment, a depth data acquisition scheme that balances high dynamic range and edge accuracy is proposed. Multiple grayscale images are acquired by changing the pattern and / or projection angle of the projected image to ensure sufficiently rich texture information. Based on these multiple grayscale images, multiple grayscale images with enhanced effective texture and suppressed background texture are reconstructed. The information from the original and enhanced grayscale images is then encoded into two sets of images respectively. Next, interval matching is performed on the two sets of encoded images to obtain two depth maps with different dynamic ranges and good edge accuracy with high efficiency. Finally, the two depth maps are fused to obtain a depth map that balances high dynamic range and edge accuracy.

[0122] The following describes the implementation process of the depth data acquisition scheme provided in this application embodiment, taking the acquisition of 4 frames of texture patterns by a binocular depth camera as an example.

[0123] like Figure 2 As shown, in this embodiment, the implementation process of the depth data acquisition scheme that balances high dynamics and edge accuracy includes:

[0124] 1. Image acquisition: Each frame of image acquired by the binocular depth camera (which can be called a set of images) consists of two images captured by the left and right cameras. The two images are captured simultaneously under the same imaging parameters (at the same acquisition time or within the allowable error range of the acquisition time); the texture image is transformed or the projection angle is changed to obtain grayscale images of different textures.

[0125] For example, two or more sets of images can be acquired, and the brightness of the texture light source of these sets of images is consistent (the brightness is the same or the brightness difference is within a preset difference range).

[0126] like Figure 2 As shown, the acquired original grayscale image can include 4 sets of grayscale images: grayscale images of texture 1 (including grayscale images captured by the left camera and the right camera of texture 1, the same below), grayscale images of texture 2, grayscale images of texture 3, and grayscale images of texture 4.

[0127] Textures 1-4 are different textures obtained by transforming the texture image or changing the projection angle.

[0128] 2. Background grayscale image: Since the average brightness of the background is consistent in multiple grayscale images with the same light source brightness, and the effective texture belongs to increasing brightness; therefore, when the grayscale value of the same pixel in multiple images is the smallest, it means that the point belongs to the background in that frame. A background grayscale image that suppresses the effective texture can be reconstructed by taking the minimum grayscale value of the same pixel in multiple frames.

[0129] It should be noted that for binocular depth cameras, in the process of obtaining the background grayscale image in the above manner, the left-eye grayscale background image can be obtained based on the image captured by the left camera, and the right-eye grayscale background image can be obtained based on the image captured by the right camera.

[0130] 3. Texture Enhancement Image: Based on the obtained background grayscale image, each original grayscale image is subtracted from the background grayscale image pixel by pixel to obtain the probability that each pixel belongs to the effective texture or the background (the larger the grayscale difference, the higher the probability of being the effective texture; the smaller the grayscale difference, the higher the probability of being the background). Pixels belonging to the background are suppressed, and pixels of the effective texture are enhanced to obtain the texture enhancement image.

[0131] For any given grayscale image, the grayscale values ​​of each pixel in the image can be reconstructed using the following formula:

[0132]

[0133] It should be noted that when performing texture enhancement processing on the original grayscale image, it is also necessary to perform texture enhancement on the grayscale image captured by the left camera based on the grayscale background image of the left eye, and on the grayscale image captured by the right camera based on the grayscale background image of the right eye.

[0134] 4. Multi-frame image encoding and synthesis: In order to merge the texture information of multiple frames into the same image, bilinear interpolation can be performed on each grayscale image to obtain an image with increased resolution, thereby expanding the information carrying area; then, the multi-frame enlarged images can be merged into an image with the same resolution, thus merging the information of multiple frames into the same image.

[0135] It should be noted that when encoding and synthesizing multiple frames of images, it is also necessary to encode and synthesize the images from the left camera separately, and the images from the right camera separately.

[0136] For example, when encoding and combining images from the left and right cameras, the magnification ratio can be set according to actual needs. For instance, the magnification ratio can be 2, meaning the resolution of the encoded composite image is twice the resolution of the original grayscale image.

[0137] For example, the encoding and synthesis of the image from the left camera is done in the same way as the encoding and synthesis of the image from the right camera, for example, taking the maximum value for each pixel.

[0138] 5. Sampling Matching: Considering that the resolution of the coded composite image is greater than that of the required depth image, in order to improve the matching efficiency, the coded composite image can be sampled at intervals, and the sampled points correspond to the valid points of the original grayscale image; the disparity range of each point in the coded composite image is also magnified to the same proportion, and you can choose not to sample (not sample disparity) or sample disparity matching according to efficiency requirements.

[0139] For example, the lower the parallax sampling rate, the higher the accuracy of the depth map.

[0140] 6. Matching Window: Since the image being matched is a composite image with increased resolution and fused information from multiple frames, when a matching window of the same size as a single frame (a single frame image with the resolution of the original grayscale image) is used to match the composite image, the actual window of the depth map obtained by sampling and matching needs to be reduced proportionally. This is equivalent to matching with a smaller window at the original resolution, so its edges will be more accurate.

[0141] 7. Depth Map Fusion: After multi-frame encoding and matching of the original grayscale image and the texture-enhanced grayscale image, two images with different dynamic ranges can be obtained. Among them, the depth map obtained from the original grayscale image (i.e., the first depth map mentioned above, which can be denoted as D1) often has clearer edge details because it has both active and passive textures. The effective texture in the texture-enhanced image is more obvious, and its depth map (i.e., the second depth map mentioned above, which can be denoted as D2) has a higher dynamic range, but its edge precision may be slightly reduced. Therefore, during fusion, each pixel can be classified into edge and smooth regions, and edge erosion processing can be performed on the second depth map. Then, pixel-by-pixel fusion with the first depth map can be performed to obtain the final depth map D3.

[0142] For example, the fusion of the first depth map and the processed second depth map can be achieved using the following formula:

[0143]

[0144] like Figure 3 As shown, the depth data acquisition process in this embodiment may include the following steps:

[0145] 1. Control the camera to switch between different textures, set the same imaging parameters, and acquire multiple frames of grayscale images containing left and right grayscale images (the left grayscale image is the grayscale image acquired by the left camera, and the right grayscale image is the grayscale image acquired by the right camera).

[0146] 2. Reconstruct the left and right background grayscale images based on the multiple grayscale images of the left and right sides respectively;

[0147] 3. Based on the background grayscale image and the original multi-frame grayscale images, reconstruct multi-frame texture-enhanced grayscale images;

[0148] 4. Encode and synthesize the original left and right multi-frame grayscale images into a set of left and right images to be matched. Then, obtain the first depth map at the original resolution by sampling and matching.

[0149] 5. Encode and synthesize a set of left and right images to be matched by encoding the texture-enhanced multi-frame grayscale images. Then, obtain the second depth map at the original resolution by sampling and matching.

[0150] 6. Classify the pixels in the second depth map into edge and smooth regions, and erode the edge pixels to obtain the updated second depth map (i.e. the second depth map after the above processing).

[0151] 7. Merge the first depth map with the updated second depth map pixel by pixel to obtain the final depth map.

[0152] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0153] Please see Figure 4 This is a schematic diagram of the structure of a depth data acquisition device provided in an embodiment of this application, as shown below. Figure 4 As shown, the depth data acquisition device may include:

[0154] The acquisition unit 410 is used to acquire multiple frames of grayscale images of different textures of the scene under test through an image acquisition device.

[0155] The first determining unit 420 is used to determine the background grayscale image based on the multi-frame grayscale images;

[0156] The enhancement unit 430 is used to obtain a multi-frame texture-enhanced grayscale image based on the multi-frame grayscale image and the background grayscale image;

[0157] The synthesis unit 440 is used to encode and synthesize the multi-frame grayscale images to obtain a first synthesized image; and to encode and synthesize the multi-frame texture-enhanced grayscale images to obtain a second synthesized image.

[0158] The second determining unit 450 is configured to determine a first depth map based on the first composite image; and to determine a second depth map based on the second composite image.

[0159] The fusion unit 460 is used to fuse the first depth map and the second depth map to obtain the final depth map.

[0160] In some embodiments, the image acquisition device includes at least one camera for image acquisition, and the at least one camera acquires images in any texture of the scene under test.

[0161] The first determining unit 420 determines a background grayscale image based on the multi-frame grayscale images, including:

[0162] For multiple grayscale images acquired through the same camera, the minimum grayscale value of each pixel position in the multiple grayscale images is taken, and the corresponding background grayscale image is reconstructed.

[0163] In some embodiments, the enhancement unit 430 obtains multi-frame texture-enhanced grayscale images based on the multi-frame grayscale images and the background grayscale image, including:

[0164] For any given grayscale image, based on the difference between the grayscale value of each pixel position in the grayscale image and the grayscale value of the same pixel position in the corresponding background grayscale image, the grayscale values ​​of each pixel position in the grayscale image are reconstructed to obtain the texture-enhanced grayscale image corresponding to the grayscale image of that frame.

[0165] In some embodiments, the enhancement unit 430 reconstructs the grayscale values ​​of each pixel position in the frame grayscale image based on the difference between the grayscale values ​​of each pixel position in the frame grayscale image and the grayscale values ​​of the same pixel position in the corresponding background grayscale image, including:

[0166] For any pixel position in the grayscale image of the frame, a weighted operation is performed based on the difference between the grayscale value of the pixel position in the grayscale image of the frame and the grayscale value of the pixel position in the corresponding background grayscale image, as well as the grayscale value of the pixel position in the grayscale image of the frame, to obtain the reconstructed grayscale value of the pixel position.

[0167] In some embodiments, the image acquisition device includes at least one camera for image acquisition, and the at least one camera acquires images in any texture of the scene under test.

[0168] The synthesis unit 440 encodes and synthesizes the multi-frame grayscale images to obtain a first synthesized image, including:

[0169] For multiple grayscale images acquired through the same camera, interpolation is performed on these multiple grayscale images to obtain a first composite image of multiple frames with enlarged resolution;

[0170] The first composite image is obtained by image fusion of the multiple frames of the first image to be synthesized.

[0171] The synthesis unit 440 encodes and synthesizes the multi-frame texture-enhanced grayscale images to obtain a second synthesized image, including:

[0172] For multiple frames of grayscale images acquired through the same camera, corresponding to multiple frames of texture-enhanced grayscale images, interpolation is performed on these multiple frames of texture-enhanced grayscale images to obtain a second multi-frame image to be synthesized after resolution magnification.

[0173] The second composite image is obtained by image fusion of the multiple frames of the second image to be synthesized.

[0174] In some embodiments, the synthesis unit 440 performs image fusion on the multiple frames of the first image to be synthesized to obtain a first synthesized image, including:

[0175] Take the maximum grayscale value of each pixel position in the first composite image of the multiple frames to be synthesized to obtain the first composite image;

[0176] The synthesis unit 440 performs image fusion on the multiple frames of the second image to be synthesized to obtain a second synthesized image, including:

[0177] The maximum grayscale value of each pixel position in the second composite image is taken to obtain the second composite image.

[0178] In some embodiments, the image acquisition device is a multi-view depth camera; the number of image frames included in the first composite image and the number of image frames included in the second composite image are both consistent with the number of views of the multi-view depth camera; the resolution of the first composite image and the second composite image are both greater than the resolution of the original grayscale image;

[0179] The second determining unit 450 determines a first depth map based on the first composite image, including:

[0180] Sampling and matching of each frame in the first composite image yields a first depth map at the original resolution.

[0181] The second determining unit 450 determines a second depth map based on the second composite image, including:

[0182] The images of each frame in the second composite image are sampled and matched to obtain the second depth map at the original resolution.

[0183] In some embodiments, the second determining unit 450 samples and matches each frame image in the first synthetic map to obtain a first depth map at the original resolution, including:

[0184] Based on the resolution of the first composite image and the resolution of the original grayscale image, each frame of the first composite image is sampled at intervals, and the sampling points of each frame are matched to obtain the first depth map with the original resolution.

[0185] The second determining unit 450 samples and matches each frame of the second synthesized image to obtain a second depth map at the original resolution, including:

[0186] Based on the resolution of the second composite image and the resolution of the original grayscale image, each frame of the second composite image is sampled at intervals, and the sampling points of each frame are matched to obtain the second depth map with the original resolution.

[0187] In some embodiments, the second determining unit 450 matches the sampling points of each frame image, including:

[0188] Based on the sampling disparity, the sampling points of each frame of the image are matched.

[0189] In some embodiments, the fusion unit 460 fuses the first depth map and the second depth map to obtain a final depth map, including:

[0190] The second depth map is subjected to edge erosion processing to obtain the processed second depth map;

[0191] The first depth map and the processed second depth map are fused to obtain the final depth map.

[0192] In some embodiments, the fusion unit 460 fuses the first depth map and the processed second depth map to obtain a final depth map, including:

[0193] For any pixel location, if the pixel value of that pixel location in both the first depth map and the processed second depth map is greater than 0, then a weighted average is performed on the pixel value of that pixel location in the first depth map and the pixel value of that pixel location in the processed second depth map to obtain the pixel value of that pixel location in the final depth map.

[0194] If the pixel value at that pixel location is 0 in both the first depth map and the processed second depth map, then the pixel value at that pixel location in the final depth map is determined to be 0.

[0195] If either the pixel value at that pixel location in the first depth map or the processed second depth map is 0, then the pixel value at that pixel location in the final depth map is determined based on the non-zero pixel value at that pixel location.

[0196] This application provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the depth data acquisition method described above.

[0197] Please see Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 501 and a memory 502 storing machine-executable instructions. The processor 501 and the memory 502 can communicate via a system bus 503. Furthermore, by reading and executing the machine-executable instructions corresponding to the depth data acquisition logic in the memory 502, the processor 501 can execute the depth data acquisition method described above.

[0198] The memory 502 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0199] In some embodiments, a machine-readable storage medium, such as Figure 5 The memory 502 in the memory, which is a machine-readable storage medium, stores machine-executable instructions that, when executed by a processor, implement the depth data acquisition method described above. For example, the storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0200] It should be noted that, in this document, relational terms such as "objective" and "target" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0201] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for acquiring depth data, characterized in that, include: The image acquisition device acquires multiple frames of grayscale images of the scene under test under different textures; the image acquisition device includes at least one camera for image acquisition, and the at least one camera acquires images of the scene under test under any texture. For multiple frames of grayscale images captured by the same camera, a background grayscale image is determined based on the multiple frames of grayscale images; Based on the background grayscale image, effective texture enhancement and background texture suppression are performed on the multi-frame grayscale images to obtain multi-frame texture-enhanced grayscale images; The multi-frame grayscale images are encoded and synthesized to obtain a first composite image; and the multi-frame texture-enhanced grayscale images are encoded and synthesized to obtain a second composite image. Based on the first composite image, a first depth map is determined; And, based on the second composite map, a second depth map is determined; The first depth map and the second depth map are fused to obtain the final depth map.

2. The method according to claim 1, characterized in that, Determining the background grayscale image based on the multiple grayscale images includes: For multiple grayscale images acquired through the same camera, the minimum grayscale value of each pixel position in the multiple grayscale images is taken, and the corresponding background grayscale image is reconstructed.

3. The method according to claim 1, characterized in that, The step of obtaining multi-frame texture-enhanced grayscale images based on the multi-frame grayscale images and the background grayscale image includes: For any given grayscale image, based on the difference between the grayscale value of each pixel position in the grayscale image and the grayscale value of the same pixel position in the corresponding background grayscale image, the grayscale values ​​of each pixel position in the grayscale image are reconstructed to obtain the texture-enhanced grayscale image corresponding to the grayscale image of that frame.

4. The method according to claim 3, characterized in that, The step of reconstructing the grayscale values ​​of each pixel in the grayscale image based on the difference between the grayscale value of each pixel in the frame grayscale image and the grayscale value of the same pixel in the corresponding background grayscale image includes: For any pixel position in the grayscale image of the frame, a weighted operation is performed based on the difference between the grayscale value of the pixel position in the grayscale image of the frame and the grayscale value of the pixel position in the corresponding background grayscale image, as well as the grayscale value of the pixel position in the grayscale image of the frame, to obtain the reconstructed grayscale value of the pixel position.

5. The method according to claim 1, characterized in that, The step of encoding and synthesizing the multiple grayscale images to obtain the first synthesized image includes: For multiple grayscale images acquired through the same camera, interpolation is performed on these multiple grayscale images to obtain a first composite image of multiple frames with enlarged resolution; The first composite image is obtained by image fusion of the multiple frames of the first image to be synthesized. The process of encoding and synthesizing the multi-frame texture-enhanced grayscale images to obtain a second composite image includes: For multiple frames of grayscale images acquired through the same camera, corresponding to multiple frames of texture-enhanced grayscale images, interpolation is performed on these multiple frames of texture-enhanced grayscale images to obtain a second multi-frame image to be synthesized after resolution magnification. The second composite image is obtained by image fusion of the multiple frames of the second image to be synthesized.

6. The method according to claim 5, characterized in that, The step of image fusion of the multiple frames of the first composite image to obtain the first composite image includes: Take the maximum grayscale value of each pixel position in the first composite image of the multiple frames to be synthesized to obtain the first composite image; The step of fusing the multiple frames of the second composite image to obtain the second composite image includes: The maximum grayscale value of each pixel position in the second composite image is taken to obtain the second composite image.

7. The method according to claim 1, characterized in that, The image acquisition device is a multi-view depth camera; the number of image frames included in the first composite image and the number of image frames included in the second composite image are consistent with the number of views of the multi-view depth camera; the resolution of the first composite image and the second composite image are both greater than the resolution of the original grayscale image; Determining the first depth map based on the first composite map includes: Sampling and matching of each frame in the first composite image yields a first depth map at the original resolution. Determining the second depth map based on the second composite map includes: The images of each frame in the second composite image are sampled and matched to obtain the second depth map at the original resolution.

8. The method according to claim 7, characterized in that, The step of sampling and matching each frame of the first synthesized image to obtain the first depth map at the original resolution includes: Based on the resolution of the first composite image and the resolution of the original grayscale image, each frame of the first composite image is sampled at intervals, and the sampling points of each frame are matched to obtain the first depth map with the original resolution. The step of sampling and matching each frame of the second synthesized image to obtain the second depth map at the original resolution includes: Based on the resolution of the second composite image and the resolution of the original grayscale image, each frame of the second composite image is sampled at intervals, and the sampling points of each frame are matched to obtain the second depth map with the original resolution.

9. The method according to claim 8, characterized in that, The matching of sampling points for each frame of the image includes: Based on the sampling disparity, the sampling points of each frame of the image are matched.

10. The method according to claim 1, characterized in that, The process of fusing the first depth map and the second depth map to obtain the final depth map includes: The second depth map is subjected to edge erosion processing to obtain the processed second depth map; The first depth map and the processed second depth map are fused to obtain the final depth map.

11. The method according to claim 10, characterized in that, The process of fusing the first depth map and the processed second depth map to obtain the final depth map includes: For any pixel location, if the pixel value of that pixel location in both the first depth map and the processed second depth map is greater than 0, then a weighted average is performed on the pixel value of that pixel location in the first depth map and the pixel value of that pixel location in the processed second depth map to obtain the pixel value of that pixel location in the final depth map. If the pixel value at that pixel location is 0 in both the first depth map and the processed second depth map, then the pixel value at that pixel location in the final depth map is determined to be 0. If either the pixel value at that pixel location in the first depth map or the processed second depth map is 0, then the pixel value at that pixel location in the final depth map is determined based on the non-zero pixel value at that pixel location.

12. A depth data acquisition device, characterized in that, include: The acquisition unit is used to acquire multiple frames of grayscale images of different textures in the scene under test through an image acquisition device. The image acquisition device includes at least one camera for image acquisition. The at least one camera acquires images of the scene under test under any texture. The first determining unit, for multiple frames of grayscale images captured by the same camera, is used to determine a background grayscale image based on the multiple frames of grayscale images; An enhancement unit is used to perform effective texture enhancement and background texture suppression on the multi-frame grayscale image based on the background grayscale image to obtain a multi-frame texture-enhanced grayscale image. The synthesis unit is configured to encode and synthesize the multi-frame grayscale images to obtain a first synthesized image; and to encode and synthesize the multi-frame texture-enhanced grayscale images to obtain a second synthesized image. The second determining unit is used to determine the first depth map based on the first composite map; And, based on the second composite map, a second depth map is determined; The fusion unit is used to fuse the first depth map and the second depth map to obtain the final depth map.

13. The apparatus according to claim 12, characterized in that, The first determining unit determines the background grayscale image based on the multiple grayscale images, including: For multiple grayscale images acquired through the same camera, the minimum grayscale value of each pixel position in the multiple grayscale images is taken, and the background grayscale image corresponding to the multiple grayscale images is reconstructed. And / or, The enhancement unit obtains multi-frame texture-enhanced grayscale images based on the multi-frame grayscale images and the background grayscale image, including: For any frame of grayscale image, based on the difference between the grayscale value of each pixel position in the grayscale image of the frame and the grayscale value of the same pixel position in the corresponding background grayscale image, the grayscale value of each pixel position in the grayscale image of the frame is reconstructed to obtain the texture-enhanced grayscale image corresponding to the frame of grayscale image. The enhancement unit reconstructs the grayscale values ​​of each pixel in the frame's grayscale image based on the difference between the grayscale values ​​of each pixel in the frame's grayscale image and the grayscale values ​​of the same pixel in the corresponding background grayscale image, including: For any pixel position in the grayscale image of this frame, a weighted operation is performed based on the difference between the grayscale value of the pixel position in the grayscale image of this frame and the grayscale value of the pixel position in the corresponding background grayscale image, as well as the grayscale value of the pixel position in the grayscale image of this frame, to obtain the reconstructed grayscale value of the pixel position. And / or, The synthesis unit encodes and synthesizes the multiple grayscale images to obtain a first synthesized image, including: For multiple grayscale images acquired through the same camera, interpolation is performed on these multiple grayscale images to obtain a first composite image of multiple frames with enlarged resolution; The first composite image is obtained by image fusion of the multiple frames of the first image to be synthesized. The synthesis unit encodes and synthesizes the multi-frame texture-enhanced grayscale images to obtain a second synthesized image, including: For multiple frames of grayscale images acquired through the same camera, corresponding to multiple frames of texture-enhanced grayscale images, interpolation is performed on these multiple frames of texture-enhanced grayscale images to obtain a second multi-frame image to be synthesized after resolution magnification. The multiple frames of the second composite image are fused to obtain the second composite image; The synthesis unit performs image fusion on the multiple frames of the first image to be synthesized to obtain a first synthesized image, including: Take the maximum grayscale value of each pixel position in the first composite image of the multiple frames to be synthesized to obtain the first composite image; The synthesis unit performs image fusion on the multiple frames of the second image to be synthesized to obtain a second synthesized image, including: Take the maximum grayscale value of each pixel position in the second composite image from the multiple frames to obtain the second composite image; And / or, The image acquisition device is a multi-view depth camera; the number of image frames included in the first composite image and the number of image frames included in the second composite image are consistent with the number of views of the multi-view depth camera; the resolution of the first composite image and the second composite image are both greater than the resolution of the original grayscale image; The second determining unit determines a first depth map based on the first composite image, including: Sampling and matching of each frame in the first composite image yields a first depth map at the original resolution. The second determining unit determines a second depth map based on the second composite map, including: Sampling and matching are performed on each frame of the second composite image to obtain a second depth map with the original resolution. The second determining unit samples and matches each frame of the first synthesized image to obtain a first depth map at the original resolution, including: Based on the resolution of the first composite image and the resolution of the original grayscale image, each frame of the first composite image is sampled at intervals, and the sampling points of each frame are matched to obtain the first depth map with the original resolution. The second determining unit samples and matches each frame of the second synthesized image to obtain a second depth map at the original resolution, including: Based on the resolution of the second composite image and the resolution of the original grayscale image, each frame of the second composite image is sampled at intervals, and the sampling points of each frame are matched to obtain the second depth map with the original resolution. The second determining unit matches the sampling points of each frame image, including: Based on the sampling disparity, the sampling points of each frame image are matched; And / or, The fusion unit fuses the first depth map and the second depth map to obtain a final depth map, including: The second depth map is subjected to edge erosion processing to obtain the processed second depth map; The first depth map and the processed second depth map are fused to obtain the final depth map; The fusion unit fuses the first depth map and the processed second depth map to obtain a final depth map, including: For any pixel location, if the pixel value of that pixel location in both the first depth map and the processed second depth map is greater than 0, then a weighted average is performed on the pixel value of that pixel location in the first depth map and the pixel value of that pixel location in the processed second depth map to obtain the pixel value of that pixel location in the final depth map. If the pixel value at that pixel location is 0 in both the first depth map and the processed second depth map, then the pixel value at that pixel location in the final depth map is determined to be 0. If either the pixel value at that pixel location in the first depth map or the processed second depth map is 0, then the pixel value at that pixel location in the final depth map is determined based on the non-zero pixel value at that pixel location.

14. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-11.

15. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-11.

Images