Image processing methods, apparatus, electronic devices and readable storage media

Abstract

The image processing method, device, electronic device and readable storage medium provided by the present invention include: determining the depth information of the target pixel points corresponding to the hole area to be processed and the hole area to be processed in the target depth map; wherein each pixel point in the hole area to be processed has no depth information; the target pixel point is adjacent to the edge pixel point of the hole area to be processed; according to the depth difference value between the target pixel points, determining whether the target interpolation model corresponding to the hole area to be processed is a linear interpolation model or a nonlinear interpolation model; based on the determined target interpolation model, determining the depth information corresponding to each pixel point. This application takes into account the scene differences represented by the hole area, selects different interpolation methods according to the depth difference value, can fill the point cloud holes, repair the point cloud missing caused by occlusion problems, and complete the elevation information so that it can be flexibly applied to plane scenes and scenes with large elevation fluctuations.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] In recent years, with the tremendous development of 3D image scanning technology, the advantages of point cloud models have become increasingly obvious. Graphics, which uses point cloud as a research object, has received more and more attention. It has a wide range of applications in medicine, modern satellite remote sensing measurement, multimedia, machine vision, intelligent monitoring, 3D reconstruction, somatosensory interaction and 3D printing.

[0003] In the visual 3D dense reconstruction, due to the occlusion, weak texture and repeated texture problems of the measured object, there are point cloud holes in the point cloud data of the measured object. The current solution to the point cloud holes is to use linear interpolation to fill the point cloud holes in the vacant places. However, the linear interpolation method only has a good effect on planar scenes, and has little effect on areas with more elevation fluctuations. If the tree scene, telephone poles, etc. cannot be completed.

[0004] Therefore, providing an interpolation solution that can be flexibly applied to different scenarios to repair point cloud holes is an urgent problem to be solved. Summary of the invention

[0005] The object of the present invention is to provide an image processing method, device, electronic device and readable storage medium, which are used to achieve the effect of flexibly selecting different interpolation methods for different scenes to repair point cloud holes.

[0006] The embodiments of the present invention can be implemented as follows:

[0007] In a first aspect, the present invention provides an image processing method, comprising: determining a hole area to be processed in a target depth map and depth information of target pixel points corresponding to the hole area to be processed; wherein each pixel point in the hole area to be processed has no depth information; the target pixel point is adjacent to the edge pixel point of the hole area to be processed; according to the depth difference value between the target pixel points, determining whether the target interpolation model corresponding to the hole area to be processed is a linear interpolation model or a nonlinear interpolation model; based on the determined target interpolation model, determining the depth information corresponding to each pixel point.

[0008] In the first aspect, the present invention provides an image processing device, comprising a determination module, used to determine the depth information of a hole area to be processed in a target depth map and a target pixel point corresponding to the hole area to be processed; wherein the pixel points in the hole area to be processed have no depth information; the target pixel point is adjacent to the edge pixel point of the hole area to be processed; the determination module is also used to determine whether the target interpolation model corresponding to the hole area to be processed is a linear interpolation model or a nonlinear interpolation model based on the depth difference value between the target pixel points; the interpolation module is used to determine the depth information corresponding to each pixel point in the hole area to be processed based on the determined target interpolation model.

[0009] In a third aspect, the present invention provides an electronic device, comprising a processor and a memory storing computer-readable instructions, wherein the processor is configured to perform the image processing method as described in the first aspect when executing the computer-readable instructions.

[0010] In a fourth aspect, the present invention provides a readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the image processing method as described in the first aspect.

[0011] The image processing method, device, electronic device and readable storage medium provided by the present invention include: first determining a hole area to be processed in a target depth map, and then determining whether the hole area to be processed is suitable for a linear interpolation model or a nonlinear interpolation model according to the depth difference value between the target pixels corresponding to the hole area to be processed, and then determining the depth information of each pixel in the hole area to be processed according to the determined target interpolation model. The difference from the prior art is that the prior art only uses a linear interpolation method to complete the depth information of the hole area, ignoring the scene difference represented by the hole area, resulting in the linear interpolation method having a good interpolation effect only for plane scenes, but is not suitable for scenes with large elevation fluctuations represented in the target depth map. The present application takes into account the scene difference represented by the hole area, and then can flexibly select different interpolation methods according to the depth difference value, thereby being able to flexibly apply to image processing methods for plane scenes and scenes with large elevation fluctuations, fill point cloud holes, repair point cloud missing caused by occlusion problems, complete elevation information, and increase the completeness of ground object recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A schematic flow chart of an image processing method provided in an embodiment of the present application;

[0014] Figure 2 A scene example diagram of a target depth map is provided for an embodiment of the present application;

[0015] Figure 3 A schematic flowchart of an implementation method of step S106 provided in an embodiment of the present application;

[0016] Figure 4 A coordinate system provided in an embodiment of the present application;

[0017] Figure 5 A schematic flowchart of an implementation method of step S109 provided in an embodiment of the present application;

[0018] Figure 6 A schematic flowchart of an implementation method of step S103 provided in an embodiment of the present application;

[0019] Figure 7 A schematic flow chart of another image processing method provided in an embodiment of the present application;

[0020] Figure 8 A schematic diagram of a scene for obtaining a target depth map provided in an embodiment of the present application;

[0021] Fig. 9 A functional module diagram of an image processing device provided in an embodiment of the present application;

[0022] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0026] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, they are based on the orientation or position relationship shown in the accompanying drawings, or are the orientation or position relationship in which the product of the invention is usually placed when used. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0027] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.

[0028] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0029] Point cloud refers to a massive collection of points of target surface characteristics, which can be obtained through scanners or 3D measurement methods. In recent years, graphics with point cloud as the research object has received more and more attention. It has been widely used in medicine, modern satellite remote sensing measurement, multimedia, machine vision, autonomous driving, intelligent monitoring, 3D reconstruction, somatosensory interaction and 3D printing. In the actual measurement and data collection process, the collected 3D point cloud will have noise, or due to local occlusion of the sample, damage to the sample itself, weak texture, repeated texture and the limitation of the measurement method, the obtained point cloud data is missing, the depth information is incomplete, and there are some holes, which brings great difficulties to the later data processing, point cloud modeling, structural information extraction, etc.

[0030] Therefore, point cloud hole repair is an important part of point cloud data preprocessing. How to propose a hole repair method that can repair the surface closest to the actual point cloud surface is a hot topic in current research. Since point cloud data and depth images can be converted, in the solution to repair point cloud holes, the relevant technology uses a linear interpolation method for depth images to adjust point cloud holes, that is, by determining a linear equation to complete the depth information at the hole, so as to achieve the purpose of filling the hole. The linear interpolation method is simple and fast, but it only has a good effect on planar scenes. For hole areas with more elevation fluctuations, such as fruit tree scenes, telephone poles, etc., the filling effect is not ideal.

[0031] Therefore, the embodiments of the present invention provide an image processing method that can be flexibly applied to planar scenes and scenes with large elevation fluctuations, which can fill point cloud holes, repair point cloud missing caused by occlusion problems, complete elevation information, and increase the completeness of ground object recovery.

[0032] See also Figure 1 , Figure 1 An image processing method provided in an embodiment of the present application includes:

[0033] S103, determining the hole area to be processed in the target depth map and the depth information of the target pixel points corresponding to the hole area to be processed.

[0034] Each pixel point in the hole area to be processed has no depth information, and the target pixel point is adjacent to the edge pixel point of the hole area to be processed.

[0035] In this embodiment, the target depth map can be a pre-stored depth image, or a depth map obtained by processing an image of an object or area captured in any scenario. The above scenarios may be, but are not limited to, autonomous driving, drone aerial survey, virtual reality, etc. The target depth image can also be a randomly generated depth image, which is not limited here.

[0036] It can be understood that in order to avoid overfitting and the loss of accuracy of point cloud data, the hole areas to be processed in this embodiment are hole areas that meet the processing conditions. That is to say, there may be many hole areas in the target depth map, but only the hole areas that meet the processing conditions in this embodiment will have their depth information completed.

[0037] In one embodiment, the above-mentioned processing conditions can be determined based on the depth difference between the target pixel points corresponding to the hole area, where the target pixel points are two pixel points adjacent to the two endpoints of the hole area to be processed. In a possible implementation, when the depth difference value between the two target pixel points is greater than 3 meters, the depth information of the hole area is not completed. If the depth difference value between the two target pixel points is less than 3 meters, the depth information of the hole area is completed.

[0038] It is understandable that the 3 meters given in the above processing condition is only an example, and the user can define a threshold value according to his or her own needs to obtain the hole area to be processed.

[0039] S106: Determine, according to the depth difference value between the target pixel points, whether the target interpolation model corresponding to the hole area to be processed is a linear interpolation model or a nonlinear interpolation model.

[0040] In this embodiment, in order to flexibly adopt different methods to complete the depth information in the hole area according to different scenes, after obtaining the depth information of the target pixel point of the hole area to be processed, the depth difference value can be calculated, and the depth difference value can be used to determine whether the hole area to be processed adopts linear interpolation or nonlinear interpolation.

[0041] It can be understood that the linear interpolation method is suitable for planar scenes, and the nonlinear interpolation method is suitable for scenes with large elevation fluctuations. Therefore, in this embodiment, the above-mentioned depth difference value can be used to characterize whether the current hole area to be processed corresponds to a planar scene or a scene with large elevation fluctuations, and then one of the interpolation methods is used to complete the depth information, thereby achieving the effect of flexibly adopting different methods to complete the depth information in the hole area according to different scenes.

[0042] In one embodiment, the user can define the condition of whether the depth difference value corresponds to a flat scene or a scene with large elevation fluctuations according to experience or needs, which is not limited here.

[0043] S109: Determine depth information corresponding to each pixel based on the determined target interpolation model.

[0044] In this embodiment, if the determined target interpolation type is a linear interpolation model, the depth information of each pixel in the hole area to be processed is determined according to the linear interpolation method. If the determined target interpolation type is a nonlinear interpolation model, the depth information of each pixel in the hole area to be processed is determined according to the nonlinear interpolation method. In other words, for a target depth map, it may be possible to achieve hole repair by only using a linear interpolation model, or it may be possible to achieve hole repair by only using a nonlinear interpolation model. It may also be necessary to use both a linear interpolation model and a nonlinear interpolation model to achieve hole repair. It can be seen that this embodiment does not use a single interpolation method to complete the depth information of the hole, but can flexibly use different interpolation methods to complete the elevation information, and the applicable scenarios are more flexible.

[0045] To understand the above implementation process, please refer to Figure 2 , Figure 2 A scene example diagram of a target depth map is provided for an embodiment of the present application. Figure 2 In , each grid represents a pixel, where the pixel represented by the gray grid has depth information, and the pixel represented by the white grid has no depth information. (u, v) is used to represent the position coordinates of the pixel in the target depth map, and d is the depth information corresponding to the pixel.

[0046] Assuming that the area composed of the two pixels (u2, v2) and (u3, v3) in the dotted box is the hole area to be processed, the target pixel is the pixel (u0, v0) and the pixel (u1, v1). Assuming that the depth information of the pixel (u0, v0) is d0 and the depth information of the pixel (u1, v1) is d1, the difference between d0 and d1 can be used to determine whether the hole area to be processed adopts a linear interpolation model or a nonlinear interpolation model. In an example, assuming that the hole area to be processed corresponds to a plane scene, the target interpolation model of the hole area to be processed is a linear interpolation model. If the hole area to be processed corresponds to a scene with large elevation fluctuations, the target interpolation model of the hole area to be processed is a nonlinear interpolation model. Furthermore, the depth information of the pixels (u2, v2) and (u3, v3) can be determined according to the target interpolation model, so as to flexibly complement the effect of the depth information.

[0047] The image processing method provided by the embodiment of the present invention first determines the hole area to be processed in the target depth map, and then determines whether the hole area to be processed is suitable for a linear interpolation model or a nonlinear interpolation model according to the depth difference value between the target pixel points corresponding to the hole area to be processed, and then the depth information of each pixel point in the hole area to be processed can be determined according to the determined target interpolation model. The difference from the prior art is that the prior art only uses linear interpolation to complete the depth information of the hole area, ignoring the scene difference represented by the hole area, resulting in the linear interpolation method having a good interpolation effect only for plane scenes, but is not suitable for scenes with large elevation fluctuations represented in the target depth map. The present application takes into account the scene difference represented by the hole area, and then different interpolation methods can be flexibly selected according to the depth difference value, thereby being an image processing method that can be flexibly applied to plane scenes and scenes with large elevation fluctuations, can fill point cloud holes, repair point cloud missing caused by occlusion problems, complete elevation information, and increase the completeness of ground object recovery.

[0048] It should also be noted that, since there may be multiple hole areas to be processed in the target depth map, the image processing method in this embodiment can be executed in the following ways: the first way is: after determining a hole area to be processed, the depth information of the hole area to be processed is completed, and then the next hole area to be processed is determined, and so on, until there is no depth area to be processed in the target depth map; the second way is: all the hole areas to be processed in the target depth map can be determined first, and then the depth information of each hole area to be processed is completed.

[0049] Optionally, with respect to the determination of the target interpolation type applicable to the hole region to be processed in step S106, this embodiment provides a possible implementation method, see Figure 3 , Figure 3The present application provides a schematic flowchart of an implementation method of step S106, where step S106 may include the following sub-steps:

[0050] S106-1: If the depth difference value is within a first preset range, determine that the target interpolation model is a linear interpolation model.

[0051] S106-2: If the depth difference value is greater than an upper limit value of a second preset range, determining that the target interpolation model is a nonlinear interpolation model.

[0052] The upper limit value of the first preset range is the lower limit value of the second preset range.

[0053] In this embodiment, the above-mentioned first preset range and the second preset range can be determined according to empirical values, or randomly defined by the user. In one embodiment, the above-mentioned first preset range can be 0 meters-0.5 meters, and the second preset range is 0.5 meters-3 meters. That is to say, when the depth difference value is between 0 meters-0.5 meters, including 0 meters and 0.5 meters, the scene corresponding to the hole area to be processed can be considered as a plane scene, which is suitable for a linear interpolation model. When the depth difference value is greater than 0.5 meters and less than or equal to 3 meters, the scene corresponding to the hole area to be processed can be considered as a scene with large elevation fluctuations, which is suitable for a nonlinear interpolation model.

[0054] With respect to step S106 - 1 , after determining that the target interpolation model is a linear interpolation model, a linear interpolation model may be constructed according to the position information and depth information of the target pixel point.

[0055] In this embodiment, the linear interpolation model may be a first-order Bezier curve, and the above method of constructing the linear interpolation model may be to construct the linear interpolation model according to the position information and depth information of the target pixel point.

[0056] Continue with Figure 2 Taking the scene graph shown as an example, first, a two-dimensional Cartesian coordinate system is constructed according to the horizontal coordinate direction of the pixel point (the same applies to the vertical coordinate), and the linear interpolation model is assumed to be a first-order Bezier curve. The pixels in the hole area to be processed are distributed on the first-order Bezier curve, where the depth information is used as the Y-axis, and the horizontal or vertical coordinate of the pixel point is used as the X-axis.

[0057] Assume that the position information and depth information of the target pixel are (u0, v0), d0, and (u1, v1), d1 respectively, and the pixel point in the hole area to be processed is (u, v). According to the characteristics of the first-order Belleset curve, the relationship model between (u, v) and the depth information change rate t can be constructed as follows:

[0058]

[0059] Among them, u and d represent the horizontal coordinate and depth information of the pixel points in the hole area respectively, u0 and d0 are the horizontal coordinate and depth information of the target pixel point (u0, v0), and u1 and d1 are the horizontal coordinate and depth information of the target pixel point (u1, v1); u0, d0, u1, and d1 are known. After determining the change rate of the depth information corresponding to each pixel point, the horizontal coordinate and the change rate of the depth information of each pixel point are brought into the above relationship model to obtain the depth information d corresponding to each pixel point.

[0060] It should be noted that the above-mentioned Belset curve can represent: the path formed in the process of moving from one pixel point to another pixel point as the starting point, and the pixels in the hole area to be processed are distributed on the path, and the above-mentioned depth information change rate representation can represent the moving step corresponding to a certain pixel point moving to the hole area to be processed.

[0061] Similarly, for step S106 - 2 , after determining that the target interpolation model is a linear interpolation model, a linear interpolation model can be constructed according to the position information and depth information of the target pixel point.

[0062] In this embodiment, the nonlinear interpolation model can be a second-order Bezier curve. The above method of constructing the nonlinear interpolation model can be to determine the position information and depth information of a reference coordinate point based on the position information and depth information of one of the target pixel points; and construct a nonlinear interpolation model based on the position information and depth information of the target pixel point and the reference coordinate point.

[0063] Continue with Figure 2 Taking the scene graph shown in the figure as an example, firstly, a two-dimensional Cartesian coordinate system is constructed according to the horizontal or vertical coordinate direction of the pixel point, such as Figure 4 As shown, Figure 4 A coordinate system is provided in an embodiment of the present application, wherein depth information is used as the Y-axis and the horizontal coordinate or vertical coordinate of the pixel is used as the X-axis.

[0064] Assume that the nonlinear interpolation model is a second-order Bezier curve. The position information and depth information of the target pixel point in the above coordinate system are: the target pixel point (u0, V0) corresponds to the coordinate point P0 (u0, d0), the target pixel point (u1, v1) corresponds to the coordinate point P1 (u1, d1), and the reference coordinate point is assumed to be (u3, v3), the depth information is d3, and the corresponding coordinate point is P3 (u3, d3). Among them, each pixel point in the hole area to be processed is distributed in the second-order Bezier curve composed of P0, P1 and P3. For example, the pixel point (u2, v2) in the hole area to be processed corresponds to the coordinate point P2 (u2, d2).

[0065] According to the characteristics of the second-order Belset curve, the relationship model between (u, v) and the rate of change t of depth information can be constructed as follows:

[0066]

[0067] Among them, u and d represent the horizontal coordinate and depth information of the pixel points in the hole area respectively, u0 and d0 are the horizontal coordinate and depth information of the target pixel point (u0, v0), u1 and d1 are the horizontal coordinate and depth information of the target pixel point (u1, v1); u3 and d3 are the horizontal coordinate and depth information of the reference coordinate point; u0, d0, u1, d1, u3, and d3 are known. After determining the change rate of the depth information corresponding to each pixel point, the horizontal coordinate and the change rate of the depth information of each pixel point are brought into the above relationship model to obtain the depth information d corresponding to each pixel point.

[0068] In one embodiment, the reference coordinate point can be determined based on the required curve curvature. In order to better adapt to the recovery of objects such as telephone poles and trees, a point with a larger depth value in the target pixel is selected as a reference to determine the depth information of the reference coordinate point. For example, assuming that d0 is less than d1, u3=u0-1 is determined based on the horizontal coordinate u0 of the target pixel point, and d3=d1-1 is determined based on the depth information d1 of the target pixel point u1. In addition, (u3, d3) can also be adjusted based on empirical values.

[0069] Optionally, a nonlinear interpolation model is obtained by combining the linear interpolation model constructed above to obtain the depth information of each pixel in the hole area to be obtained. Figure 5 To perform the steps shown, see Figure 5 , Figure 5 A schematic flowchart of an implementation method of step S109 provided in an embodiment of the present application, step S109 may include:

[0070] S109-1, determining the depth information change rate corresponding to each pixel; all depth information change rates are distributed within a preset range.

[0071] In combination with the above-mentioned embodiment of constructing the model, it can be seen that the depth information change rate is t in the above-mentioned model, and t is uniformly distributed in a preset interval, that is, distributed between 0 and 1. In a possible implementation manner, the above-mentioned method of determining the depth information change rate corresponding to each pixel point can be: first determine the number of pixels in the hole area to be processed, and then divide the number between 0 and 1 proportionally according to the determined number of pixels, so that each pixel point corresponds to a value of the depth information change rate.

[0072] For example, assuming that there are 3 pixels in the hole area to be processed, the range from 0 to 1 can be divided into 3 values ​​of 0.25, 0.5, and 0.75 in equal proportion. These 3 values ​​correspond to the values ​​of t of the 3 pixels in the hole area to be processed.

[0073] S109-2, for each pixel, input the position information and depth information change rate of the pixel into the target interpolation model to obtain the depth information of each pixel.

[0074] For example, see Figure 2 , assuming that the target interpolation model is a linear model, the horizontal coordinate of the pixel point (u2, V2) in the hole area to be processed is u2 (known), and the corresponding depth information change rate t is 0.333, then u2 (known) and 0.333 are substituted into the above linear interpolation model, and the obtained d is the depth information of (u2, V2). Further, the horizontal coordinate u7 (known) of the pixel point (u7, V7) and the corresponding value of t are substituted into the linear interpolation model to obtain the depth information of the pixel point (u7, V7). If the target interpolation model is a nonlinear model, the depth information of the pixel points (u2, V2) and (u7, V7) is obtained in the same way as the above process, which will not be repeated here.

[0075] Optionally, as can be seen from the above content, there may be multiple hole areas in the target depth map that do not meet the processing conditions. Therefore, the following is also a method for screening out the hole areas to be processed in the target depth map. Please refer to Figure 6 , Figure 6 A schematic flowchart of an implementation method of step S103 provided in an embodiment of the present application, step S103 may include:

[0076] S103 - 1 , traverse the target depth map to determine the depth information of all hole areas and the target pixel points corresponding to each hole area.

[0077] S103-2, for the target hole area, if the depth difference value of the target pixel point corresponding to the target hole area is less than or equal to the upper limit value of the second preset range, the target hole area is determined as the hole area to be processed; the target hole area is any one of all the hole areas;

[0078] S103-3, traverse all the hole areas to obtain all the hole areas to be processed.

[0079] In this embodiment, S103-1 can be implemented in the following manner:

[0080] Step 1: traverse the target depth map row by row, determine the area connected by the pixels with missing depth information in each row as the hole area, and determine the adjacent pixels of the edge pixels in the hole area along the row direction as the target pixels.

[0081] Step 2, traverse the target depth map by column, determine the area connected by the pixels with missing depth information in each column as the hole area, and determine the adjacent pixels of the edge pixels of the hole area along the column direction as the target pixels.

[0082] For example, continue with Figure 2 Take the scene graph shown as an example. The target depth map has 4 rows and 5 columns. First, traverse the target depth map by row. The result of the traversal of the first row shows that there is no hole area. In the second row, the area composed of pixels (u2, V2) and (u7, V7) is a hole area. The corresponding target pixels are the pixel (u0, V0) adjacent to the pixel (u2, V2) and the pixel (u1, V1) adjacent to the pixel (u7, V7); in the third row, the pixel (u8, V8) is a hole area. The adjacent target pixels are the pixel (u5, V5) and the pixel (u6, V6) adjacent to the pixel (u8, V8). The same is true for the remaining rows. Then traverse the target depth map by column. The principle of determining the hole area and the target pixel is the same as when traversing by row, which will not be repeated here.

[0083] It should be noted that there is no order in which the above steps 1 and 2 must be executed. Step 1 can be executed first, and then step 2, or step 2 can be executed first, and then step 1.

[0084] It should also be noted that, assuming that there is a hole area that does not meet the above processing conditions during the row-by-row traversal, then during the column-by-column traversal, the hole area may meet the above processing conditions. This is because for the same hole area, the target pixel point determined during the row-by-row traversal is different from the target pixel point determined during the column-by-column traversal, so the above situation will occur. In this case, the hole area can still be determined as a hole area to be processed, and its depth information needs to be completed.

[0085] For example, continue with Figure 2 Taking the scene shown as an example, the hole area corresponds to the pixel point (u8, V8). During the row-by-row traversal, the target pixel points corresponding to the hole area are (u5, V5) and (u6, V6), and the depth information is d5 and d6 respectively. If the depth difference between d5 and d6 is greater than 3 meters, the depth information of the vacant area will not be completed. However, during the column-by-column traversal, the target pixel points corresponding to the hole area are (u0, V0) and (u9, V9), and the depth information is d0 and d9 respectively. If the depth difference between d9 and d9 is less than 3 meters, the depth information of the vacant area will still be completed.

[0086] By determining the unprocessed hole areas in the target depth map by the above-mentioned row-by-row and column-by-column traversal method, it is possible to prevent the omission of areas caused by improper threshold setting, ensure that each unprocessed hole area can be completed with depth information, and increase the completeness of ground object recovery.

[0087] Optionally, before executing the above steps S103 to S109, this embodiment further provides a method for obtaining a target depth map, see Figure 7 , Figure 7 A schematic flowchart of another image processing method provided in an embodiment of the present application, the method may further include:

[0088] S100, controlling the drone to fly according to the flight route, and performing image acquisition operations on the object to be measured at multiple waypoint positions to obtain multiple images to be processed, wherein the object overlap between each image to be processed is greater than a preset threshold.

[0089] S102, performing depth information optimization processing on a plurality of images to be processed to obtain a target depth map.

[0090] The above-mentioned drone can be but is not limited to a remote sensing drone. The above-mentioned flight route can be pre-stored in the drone's memory, or it can be transmitted to the drone by other equipment before the drone performs the flight mission, or it can be a route generated by the drone in real time according to the object being measured, which is not limited here.

[0091] The above shooting scene can be Figure 8 As shown, see Figure 8 A schematic diagram of a scene for obtaining a target depth map is shown in FIG. Figure 8 In the route shown, during the flight from point A to point D, data is collected on the object under test at each waypoint. The setting of the waypoint position can ensure that the drone can capture multiple images with a certain degree of overlap, and then use the visual positioning algorithm and binocular stereo matching algorithm to restore the depth map of each frame of the image, and then denoise the obtained depth map to obtain the final target depth map. Often these target depth maps will have holes.

[0092] Optionally, in a possible implementation manner, the interpolated target depth map is three-dimensionally reconstructed to obtain point cloud data.

[0093] It can be understood that in the embodiment of the present invention, there are no longer any hole areas in the point cloud data obtained based on the target depth map after interpolation processing, which reduces the difficulties of later data processing, point cloud modeling, and structural information extraction, thereby facilitating the wide application of point cloud data in various fields.

[0094] See also Fig. 9 , Fig. 9 This is a functional module diagram of an image processing device provided in an embodiment of the present application. It should be noted that the basic principle and technical effects of the image processing device provided in this embodiment are the same as those of the above embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference can be made to the corresponding contents in the above embodiment. The image processing device 20 includes:

[0095] A determination module 201 is used to determine a hole area to be processed in a target depth map and depth information of a target pixel point corresponding to the hole area to be processed;

[0096] The pixel points in the hole area to be processed have no depth information; the target pixel point is adjacent to the edge pixel point of the hole area to be processed;

[0097] The determination module 201 is further used to determine whether the target interpolation model corresponding to the hole area to be processed is a linear interpolation model or a nonlinear interpolation model according to the depth difference value between the target pixel points;

[0098] The interpolation module 202 is used to determine the depth information corresponding to each pixel in the hole area to be processed based on the determined target interpolation model.

[0099] Optionally, the determination module 201 is specifically used to determine that the target interpolation model is the linear interpolation model if the depth difference value is within a preset range; if the depth difference value is greater than an upper limit value of the preset range, determine that the target interpolation model is the nonlinear interpolation model.

[0100] Optionally, the interpolation module 202 is specifically used to determine the depth information change rate corresponding to each pixel point; all the depth information change rates are distributed within a preset interval; for each pixel point, the position information of the pixel point and the depth information change rate are input into the target interpolation model to obtain the depth information of each pixel point.

[0101] Optionally, the image processing device 20 further includes a construction module, which is used to construct the linear interpolation model according to the position information and depth information of the target pixel point, and is used to determine the position information and depth information of a reference coordinate point according to the position information and depth information of the target pixel point; and to construct the nonlinear interpolation model based on the position information and depth information of the target pixel point and the reference coordinate point.

[0102] Optionally, the determination module 201 is also specifically used to traverse the target depth map to determine all hole areas and the depth information of the target pixel points corresponding to each of the hole areas; for the target hole area, if the depth difference value of the target pixel points corresponding to the target hole area is less than a preset threshold, the target hole area is determined as the hole area to be processed; the target hole area is any one of all the hole areas; all the hole areas are traversed to obtain all the hole areas to be processed.

[0103] Optionally, the determination module 201 is further specifically used to traverse the target depth map row by row, determine the area formed by connecting the pixels in each row where the depth information is missing as the hole area, and determine the adjacent pixel points of the edge pixel points in the hole area along the row direction as the target pixel points; traverse the target depth map column by column, determine the area formed by connecting the pixels in each column where the depth information is missing as the hole area, and determine the adjacent pixel points of the edge pixel points in the hole area along the column direction as the target pixel points.

[0104] Optionally, the image processing device 20 also includes an acquisition module, which is used to control the drone to fly according to a flight route, and perform image acquisition operations on the object to be measured at multiple waypoint positions to obtain multiple images to be processed, wherein the object overlap between each image to be processed is greater than a preset threshold; and perform depth information optimization processing on the multiple images to be processed to obtain the target depth map.

[0105] The image processing device provided by the embodiment of the present invention includes a determination module and an interpolation module. First, the hole area to be processed in the target depth map is determined by the determination module, and then according to the depth difference value between the target pixel points corresponding to the hole area to be processed, it is determined whether the hole area to be processed is suitable for a linear interpolation model or a nonlinear interpolation model. Then, the depth information of each pixel point in the hole area to be processed can be determined by the interpolation module according to the determined target interpolation model. The difference from the prior art is that the prior art only uses a linear interpolation method to complete the depth information of the hole area, ignoring the scene difference represented by the hole area, resulting in the linear interpolation method having a good interpolation effect only for plane scenes, but is not suitable for scenes with large elevation fluctuations represented in the target depth map. The present application takes into account the scene difference represented by the hole area, and then different interpolation methods can be flexibly selected according to the depth difference value, so that an image processing method that can be flexibly applied to plane scenes and scenes with large elevation fluctuations can be used, and point cloud holes can be filled, point cloud missing caused by occlusion problems can be repaired, elevation information can be completed, and the completeness of ground object recovery can be increased.

[0106] The embodiment of the present invention further provides an electronic device, such as Fig.10 , Fig.10This is a structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device may be, but is not limited to, an aircraft, an unmanned aerial vehicle, or other electronic devices with data analysis functions.

[0107] The electronic device 80 includes a communication interface 81, a processor 82 and a memory 83. The processor 82, the memory 83 and the communication interface 81 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The memory 83 can be used to store software programs and modules, such as program instructions / modules corresponding to the image processing method provided in the embodiment of the present invention. The processor 82 executes various functional applications and data processing by executing the software programs and modules stored in the memory 83. The communication interface 81 can be used for signaling or data communication with other node devices. In the present invention, the electronic device 80 can have multiple communication interfaces 81.

[0108] Among them, the memory 83 can be but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0109] The processor 82 may be an integrated circuit chip with signal processing capability. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0110] Optionally, the above modules can be stored in the form of software or firmware. Figure 8The memory shown in the figure or solidified in the operating system (OS) of the electronic device, and can be Figure 8 Meanwhile, the data and program codes required for executing the above modules can be stored in the memory.

[0111] An embodiment of the present invention provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, an image processing method as described in any of the above embodiments is implemented. The readable storage medium can be, but is not limited to, a U disk, a mobile hard disk, a ROM, a RAM, a PROM, an EPROM, an EEPROM, a magnetic disk, or an optical disk, and other media that can store program codes.

[0112] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0114] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, including several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0115] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Determine the hole area to be processed in the target depth map and the depth information of the target pixel points corresponding to the hole area to be processed; Wherein, each pixel point in the hole area to be processed has no depth information; the target pixel point is adjacent to the edge pixel point of the hole area to be processed; According to the depth difference value between the target pixel points, determine whether the target interpolation model corresponding to the hole area to be processed is a linear interpolation model or a nonlinear interpolation model; the depth difference value is used to characterize whether the hole area to be processed corresponds to a plane scene or a scene with large elevation fluctuations; the linear interpolation model is applicable to the plane scene; the nonlinear interpolation model is applicable to the scene with large elevation fluctuations; Based on the determined target interpolation model, the depth information corresponding to each pixel is determined, including: determining the depth information change rate corresponding to each pixel; all the depth information change rates are distributed within a preset interval; for each pixel, the position information of the pixel and the depth information change rate are input into the target interpolation model to obtain the depth information of each pixel.

2. The image processing method according to claim 1, characterized in that: Determining whether a target interpolation model corresponding to the hole area to be processed is a linear interpolation model or a nonlinear interpolation model according to the depth difference value between the target pixel points includes: If the depth difference value is within a first preset range, determining that the target interpolation model is the linear interpolation model; If the depth difference value is within a second preset range, the target interpolation model is determined to be the nonlinear interpolation model, wherein the upper limit value of the first preset range is the lower limit value of the second preset range.

3. The image processing method according to claim 2, characterized in that: After determining that the target interpolation model is the linear interpolation model, the method further includes: The linear interpolation model is constructed according to the position information and depth information of the target pixel.

4. The image processing method according to claim 2, characterized in that: After determining that the target interpolation model is the nonlinear interpolation model, the method further includes: Determine the position information and depth information of a reference coordinate point according to the position information and depth information of the target pixel point; The nonlinear interpolation model is constructed based on the position information and depth information of the target pixel point and the reference coordinate point.

5. The image processing method according to claim 2, characterized in that: Determining a hole region to be processed in the target depth map includes: Traversing the target depth map to determine all hole areas and depth information of the target pixel points corresponding to each hole area; For a target hole region, if the depth difference value of the target pixel point corresponding to the target hole region is less than or equal to the upper limit value of the second preset range, the target hole region is determined as a hole region to be processed; the target hole region is any one of all the hole regions; All the hole regions are traversed to obtain all the hole regions to be processed.

6. The image processing method according to claim 5, characterized in that: Traversing the target depth map to determine all hole regions and depth information of the target pixel points corresponding to each hole region includes: Traversing the target depth map row by row, determining an area connected by pixels with missing depth information in each row as the hole area, and determining adjacent pixels of edge pixels of the hole area along the row direction as the target pixel points; The target depth map is traversed by columns, and an area connected by pixels in each column where the depth information is missing is determined as the hole area, and adjacent pixels of edge pixels in the hole area along the column direction are determined as the target pixels.

7. The image processing method according to claim 1, characterized in that: Before determining the hole area to be processed in the target depth map and the depth information of the target pixel points corresponding to the hole area to be processed, the method further includes: Controlling the drone to fly according to the flight route, and performing image acquisition operations on the object to be measured at multiple waypoint positions to obtain multiple images to be processed, wherein the object overlap between each of the images to be processed is greater than a preset threshold; Depth information optimization processing is performed on the multiple images to be processed to obtain the target depth map.

8. The image processing method according to claim 1, characterized in that: The method further comprises: The target depth map that has been interpolated is three-dimensionally reconstructed to obtain point cloud data.

9. An image processing device, characterized in that: include: A determination module, used to determine the hole area to be processed in the target depth map and the depth information of the target pixel points corresponding to the hole area to be processed; The pixel points in the hole area to be processed have no depth information; the target pixel point is adjacent to the edge pixel point of the hole area to be processed; The determination module is further used to determine whether the target interpolation model corresponding to the hole area to be processed is a linear interpolation model or a nonlinear interpolation model according to the depth difference value between the target pixel points; the depth difference value is used to characterize whether the hole area to be processed corresponds to a plane scene or a scene with large elevation fluctuations; the linear interpolation model is applicable to the plane scene; the nonlinear interpolation model is applicable to the scene with large elevation fluctuations; An interpolation module is used to determine the depth information corresponding to each pixel in the hole area to be processed based on the determined target interpolation model, including: determining the depth information change rate corresponding to each pixel; all the depth information change rates are distributed within a preset interval; for each pixel, the position information of the pixel and the depth information change rate are input into the target interpolation model to obtain the depth information of each pixel.

10. An electronic device, characterized in that: The invention comprises a processor and a memory storing computer-readable instructions, wherein the processor is configured to execute the image processing method according to any one of claims 1 to 8 when executing the computer-readable instructions.

11. A readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the image processing method according to any one of claims 1 to 8.

Images