Image processing method, device and system based on pixel reconstruction

Abstract

The present application provides an image processing method, device and system based on pixel reconstruction, the method comprising: determining the pixel to be reconstructed in the depth image corresponding to the target object; selecting a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed, and determining whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area; if not, obtaining a fused image based on the depth image containing the pixel to be reconstructed. The present application first determines whether the pixel to be reconstructed in the depth image needs to be reconstructed, determines the accuracy of the depth value of the pixel to be reconstructed, and thereby improves the accuracy of the obtained fused image.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, device and system based on pixel reconstruction. Background Art

[0002] With the development of precision manufacturing and semiconductor manufacturing, 3D imaging and high-precision measurement of complex and tiny structures are becoming increasingly important. Among them, 3D microscopic imaging systems play an important role in computer graphics generation, point cloud processing technology, and architectural structure reconstruction. 3D microscopic imaging systems are usually composed of telecentric lenses and projection optical machines. Multi-angle projection of the target object is performed to obtain at least one depth image. If there are multiple depth images, multiple depth images need to be fused to obtain a fused image.

[0003] However, in actual operation, when the target objects in the target object are set densely, the target objects will form occlusions with each other. These occlusions will cause inaccurate depth values ​​of the occluded areas on the depth image during projection imaging. The inaccuracy of the depth image will cause the final fused image to have low accuracy. Summary of the invention

[0004] One object of the present application is to provide an image processing method based on pixel reconstruction. The present application first determines whether the pixels to be reconstructed in the depth image need to be reconstructed, determines the accuracy of the depth value of the pixels to be reconstructed, and then improves the accuracy of the obtained fused image. Another object of the present application is to provide an image processing device based on pixel reconstruction. Another object of the present application is to provide an image processing system based on pixel reconstruction. Another object of the present application is to provide a computer device. Another object of the present application is to provide a readable medium.

[0005] In order to achieve the above-mentioned object, the present application discloses, on the one hand, an image processing method based on pixel reconstruction, the method comprising:

[0006] Determine a pixel point to be reconstructed in a depth image corresponding to a target object, wherein the depth image is at least one of a plurality of depth images obtained by performing multi-angle projection imaging of the target object using a projection system, the depth image includes depth values ​​of a plurality of pixel points, the target object includes a plurality of target objects, and the setting surfaces of the plurality of target objects are used as reference surfaces, and the pixel point to be reconstructed is a pixel point on the depth image that is on the reference surface;

[0007] Selecting a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed, and determining whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area;

[0008] If not, a fused image is obtained based on the depth image including the pixel to be reconstructed.

[0009] Optionally, determining whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area includes:

[0010] Dividing the determination area into a first part and a second part along the closest target object edge direction with the pixel to be reconstructed as the center;

[0011] Calculate the characteristic values ​​of the depth values ​​of the pixels in the first part and the second part respectively;

[0012] Determine whether the pixel to be reconstructed needs to be reconstructed according to the feature values ​​of the first part and the second part.

[0013] Optionally, respectively calculating the characteristic values ​​of the depth values ​​of the first part and the second part of pixels includes:

[0014] The means and variances of the depth values ​​of the pixels of the first part and the second part are calculated respectively as the corresponding feature values ​​of the first part and the second part.

[0015] Optionally, determining whether the pixel to be reconstructed needs to be reconstructed according to the characteristic values ​​of the first part and the second part includes:

[0016] It is determined whether the eigenvalues ​​of the first portion and the second portion are both inconsistent with the eigenvalues ​​corresponding to the reference surface.

[0017] Optionally, selecting a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed includes:

[0018] A rectangular determination area surrounding the pixel to be reconstructed is selected at the position of the pixel to be reconstructed, and a size of the rectangular determination area is smaller than a minimum spacing between the multiple target objects.

[0019] Optionally, the size of the determination area is smaller than a minimum spacing between the multiple target objects.

[0020] Optionally, the method further includes determining a pixel point to be reconstructed in the depth image corresponding to the target object, before:

[0021] Extracting and obtaining the reference plane based on multiple depth images;

[0022] The pixel point to be reconstructed is determined according to the depth value of the pixel point on the reference plane.

[0023] Optionally, extracting the reference plane based on multiple depth images includes:

[0024] Generating a point cloud according to pixel depth values ​​of at least one of the depth image and the fused image;

[0025] The reference plane is obtained by extracting all pixel points that meet a preset standard from the point cloud.

[0026] Optionally, determining the pixel point to be reconstructed according to the depth value of the pixel point on the reference plane includes:

[0027] Fusing the multiple depth images to obtain a preliminary fused image;

[0028] Determine all pixel points on the preliminary fused image whose depth values ​​are higher than the depth value of the reference plane as pixel points to be corrected;

[0029] If the target pixel point corresponding to the pixel point to be corrected in the depth image is on the reference plane, the target pixel point is used as the pixel point to be reconstructed.

[0030] Optionally, obtaining a fused image based on a depth image including the pixel to be reconstructed includes:

[0031] The depth value of the corresponding pixel in the fused image is obtained based on the depth values ​​of multiple pixel points at the same position in the multiple depth images, wherein the pixel to be reconstructed exists in the pixel points in the multiple depth images corresponding to the fused pixel point of each fused image. If the pixel to be reconstructed does not need to be reconstructed, the depth value of the pixel to be reconstructed is used as the depth value of the fused pixel point; if the pixel to be reconstructed needs to be reconstructed, the depth value of the fused pixel point is obtained based on the depth value of the pixel point corresponding to the depth image where the pixel to be reconstructed does not exist.

[0032] Optionally, obtaining a fused image based on a depth image including the pixel to be reconstructed includes:

[0033] Obtaining a preliminary fused image through a fusion algorithm based on the multiple depth images;

[0034] If the depth value of the fused pixel point corresponding to the to-be-reconstructed pixel point in the fused image exceeds the depth value of the reference plane by a value greater than a preset threshold, the depth value of the fused pixel point is set to the depth value of the reference plane.

[0035] The present application also discloses an image processing device based on pixel reconstruction, comprising:

[0036] A pixel point selection module is used to determine the pixel points to be reconstructed in the depth image corresponding to the target object, the depth image is at least one of a plurality of depth images obtained by multi-angle projection imaging of the target object using a projection system, the depth image includes depth values ​​of a plurality of pixel points, the target object includes a plurality of target objects, the setting surfaces of the plurality of target objects are used as reference surfaces, and the pixel points to be reconstructed are the pixel points on the depth image that are on the reference surface;

[0037] A pixel reconstruction module is used to select a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed, and determine whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area;

[0038] The image fusion module is used to obtain a fused image based on the depth image containing the pixel points to be reconstructed.

[0039] The present application also discloses a microscopic fringe projection imaging system, comprising the image processing device as described above.

[0040] The present application also discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the above method when executing the computer program.

[0041] The present application also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0042] The present application determines the pixels to be reconstructed in the depth image corresponding to the target object based on the image processing method of pixel reconstruction, selects a determination area surrounding the pixels to be reconstructed and matching the shape of the target object according to the positions of the pixels to be reconstructed, and determines whether the pixels to be reconstructed need to be reconstructed according to the depth values ​​of the pixels in the determination area. If reconstruction is not required, it means that the depth value of the pixels to be reconstructed in the depth image is accurate, and can be used to determine the depth value of the corresponding pixels in the fused image. Therefore, the present application first determines whether the pixels to be reconstructed in the depth image need to be reconstructed, determines the accuracy of the depth value of the pixels to be reconstructed, and thereby improves the accuracy of the obtained fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 It is a flowchart of a specific embodiment of the image processing method based on pixel reconstruction of the present application;

[0045] Figure 2 It is a structural schematic diagram of a three-dimensional microscopic imaging system provided by a specific embodiment of the image processing method based on pixel reconstruction of the present application;

[0046] Figure 3 is a flowchart of a specific embodiment S000 of the image processing method based on pixel reconstruction of the present application;

[0047] Figure 4 is a flowchart of a specific embodiment S010 of the image processing method based on pixel reconstruction of the present application;

[0048] Figures 5 to 8 is a schematic diagram of depth images corresponding to multiple angles provided in an embodiment of the present application;

[0049] Fig. 9 It is a schematic diagram of a depth image recording height values ​​of different positions of a target object in a specific embodiment of an image processing method based on pixel reconstruction of the present application;

[0050] Fig.10 is a flowchart of a specific embodiment S020 of the image processing method based on pixel reconstruction of the present application;

[0051] Fig.11 is a flowchart of a specific embodiment S200 of the image processing method based on pixel reconstruction of the present application;

[0052] Fig.12 is a flowchart of a specific embodiment S300 of the image processing method based on pixel reconstruction of the present application;

[0053] Fig.13 is a schematic diagram of fusion images in the prior art;

[0054] Fig.14 is a schematic diagram of a fused image obtained by using the image processing method based on pixel reconstruction of the present application;

[0055] Fig.15 It is a structural diagram of a specific embodiment of the image processing device based on pixel reconstruction of the present application;

[0056] Fig.16 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0058] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0059] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0060] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0061] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0062] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0063] In recent years, with the development of precision manufacturing and semiconductor manufacturing, 3D imaging and high-precision measurement of complex and tiny structures have become increasingly important. Among them, micro-fringe projection 3D measurement technology (MFP-3D) has become the mainstream technology for optical three-dimensional surface imaging and measurement because of its non-contact, fast measurement speed, dense point cloud data, low cost and easy implementation, and it can better balance system flexibility and measurement accuracy. It has broad application prospects and practical value in industrial production and inspection in complex environments.

[0064] The microscopic stripe projection imaging system based on MFP-3D usually includes a front projection camera or optical machine arranged above the target object to be imaged and at least one side projection optical machine or camera arranged on the side of the target object to be imaged. The traditional microscopic stripe projection imaging system usually includes a front projection camera and a side projection optical machine (monocular single projection), and directly outputs the depth image of the target object to be imaged through the monocular single projection imaging system. However, when this monocular single projection traditional imaging system projects some target objects including multiple densely arranged or target objects with obvious height differences, the multiple target objects in the target object will block each other, which will cause the imaging system projection optical machine to be unable to project to some areas, resulting in the depth image obtained by projecting the target object will lose the depth information of some areas of the target object, resulting in poor imaging effect of the target object.

[0065] On this basis, in order to improve the projection imaging quality of the target object, the prior art usually obtains depth images formed by multiple projection angles of the target object by adding a side projector or camera, and then fuses the obtained multiple depth images based on the existing fusion algorithm to obtain a fused image. Although the method of obtaining a fused image as the final projected image of the target object by fusing multiple depth images to a certain extent avoids the problem of poor imaging effect of single-angle projection, the existing multiple depth image fusion method usually uses a fusion algorithm such as a clustering algorithm to obtain a fused image. The clustering algorithm sometimes clusters multiple erroneous depth values ​​(height values) and excludes the correct depth value, resulting in the multi-angle depth image fusion method still cannot completely solve the image inaccuracy problem caused by occlusion between multiple target objects.

[0066] Based on this, the present application provides an image processing method based on pixel reconstruction, which determines whether to reconstruct the depth value of the pixel in combination with the depth values ​​of the pixels around the pixel to be reconstructed in the depth image, that is, first determine whether the depth value of the pixel to be reconstructed in the depth image is accurate, improve the accuracy of the depth value of the pixel in the depth image used to correct the fused image, and then correct the fused image to improve the accuracy of the corrected fused image.

[0067] Figure 1is a schematic diagram of an image processing method based on pixel reconstruction provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0068] S100: Determine the pixel points to be reconstructed in the depth image corresponding to the target object, the depth image being at least one of a plurality of depth images obtained by multi-angle projection imaging of the target object using a projection system, the depth image including depth values ​​of a plurality of pixel points, the target object including a plurality of target objects, the setting surfaces of the plurality of target objects serving as a reference plane, and the pixel points to be reconstructed being the pixel points on the depth image that are on the reference plane.

[0069] Optionally, a microscopic fringe projection imaging system may be used to project the target object at multiple angles to obtain multiple depth images. Of course, in other embodiments, multiple depth images of the target object projected at multiple angles may also be generated by other three-dimensional projection systems, and this application does not limit this.

[0070] In one or more embodiments, the micro-fringe projection imaging system may adopt any one of the imaging systems such as monocular dual projection, monocular quadruple projection and quadruple single projection that can obtain depth images at different angles. Among them, the monocular dual projection imaging system includes a front projection camera and two side projection light machines. The monocular quadruple projection imaging system includes a front projection camera and four side projection light machines. The quadruple single projection imaging system includes a front projection light machine and two side projection cameras. The front projection camera or light machine in the imaging system is usually set on the top of the target object, and the side projection light machine or camera is usually set on the side of the target object in different directions.

[0071] exist Figure 2 In the specific example shown, the monocular four-projection microscopic fringe projection imaging system includes a front projection camera 100 arranged directly above the target object and four side projection cameras 200 respectively arranged in four directions of the target object. The monocular four-projection imaging system can respectively capture four depth images of the target object projected from four angles.

[0072] It should be noted that the depth image of the pixel to be reconstructed determined by the present application may be one or more of the multiple depth images corresponding to the target object. The target object of the present application includes multiple target objects, and the reference planes set by these target objects may be planes or surfaces of other shapes, and the height of the position where the target object is not set is the height of the reference plane.

[0073] S200: Selecting a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed, and determining whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area.

[0074] S300: If not, obtain a fused image based on the depth image including the pixel to be reconstructed.

[0075] The present application determines the pixels to be reconstructed in the depth image corresponding to the target object based on the image processing method of pixel reconstruction, selects a determination area surrounding the pixels to be reconstructed and matching the shape of the target object according to the positions of the pixels to be reconstructed, and determines whether the pixels to be reconstructed need to be reconstructed according to the depth values ​​of the pixels in the determination area. If reconstruction is not required, it means that the depth value of the pixels to be reconstructed in the depth image is accurate, and can be used to determine the depth value of the corresponding pixels in the fused image. Therefore, the present application first determines whether the pixels to be reconstructed in the depth image need to be reconstructed, determines the accuracy of the depth value of the pixels to be reconstructed, and thereby improves the accuracy of the obtained fused image.

[0076] In an optional embodiment, if Figure 3 As shown, the method further includes S000 before determining the pixel point to be reconstructed in the depth image corresponding to the target object:

[0077] S010: Extracting and obtaining the reference plane based on multiple depth images.

[0078] S020: Determine the pixel point to be reconstructed according to the depth value of the pixel point on the reference plane.

[0079] The depth image records the depth value (height) of each point on the horizontal surface of the target object. The horizontal position and height of each target object in the target object can be obtained through the depth image. Therefore, the height information of the reference plane and the target object is recorded in the depth image.

[0080] In one feasible embodiment, a reference plane corresponding to the setting surfaces of multiple target objects in the target object can be obtained based on at least one of the multiple depth images. In other feasible embodiments, in order to utilize the depth information synthesized by multiple depth images, after obtaining depth images of the target object at different angles by multi-angle projection of the target object, the multiple depth images can be fused through a fusion algorithm to obtain a fused image. The height information of the reference plane and the target object is also recorded in the fused image, and the reference plane corresponding to the setting surfaces of multiple target objects in the target object can also be extracted from the fused image.

[0081] In an optional embodiment, if Figure 4 As shown, the S010 extracts the reference plane based on multiple depth images and includes:

[0082] S011: Generate a point cloud according to pixel depth values ​​of at least one of the depth image and the fused image.

[0083] S012: Extracting all pixel points that meet a preset standard from the point cloud to obtain the reference plane.

[0084] Specifically, the target object of the present application includes multiple target objects, and the multiple target objects are arranged in a staggered manner. Then, one target object may block other target objects in three-dimensional imaging, resulting in depth value errors at certain positions in the final fused image, resulting in adhesion between target objects. Among them, the multiple target objects may be arranged on the same surface, or on surfaces at different heights and angles, and at least one surface carrying the multiple target objects is connected to form a setting surface for the multiple target objects.

[0085] The setting surface may be the surface of a structure actually existing in the target object, or may be a virtual surface determined based on the positions of multiple target objects. The preset standard for extracting the reference surface from the setting surface may be pre-set, or may be obtained by analyzing the shape of the target object. For example, the setting surface shape of the setting surfaces of all target objects may be obtained by analyzing the shape of the target object, and the setting surface shape may be used as the preset standard for extraction to extract the setting surface and obtain the reference surface.

[0086] Further, after the reference plane is extracted, the depth value of the reference plane can be determined according to the depth value of the pixel point on the reference plane. When the depth images of one or more angles in the multiple depth images are generated, the depth value at the reference plane position may be higher than the depth information of the reference plane due to the occlusion between the target objects, and the depth images of other angles do not have the problem of occlusion and will obtain accurate depth values ​​of the reference plane. It can be seen that, among the multiple depth images obtained, as long as there is a depth value of a certain pixel point in a depth image that is the depth value of the reference plane, it proves that the point is a point on the reference plane obtained without occlusion, and it can be determined that the depth value of the point on the fused image should also be the depth value of the reference plane. However, in actual situations, the pixel points on a depth image may be affected by various factors and may be inaccurate, and the pixel points of the reference plane in the depth image may also be inaccurate. Then, only the depth value of the reference plane pixel point of a depth image is determined to determine the depth value of the corresponding fused image. There is a greater risk of inaccuracy. Therefore, the present application selects the pixel points to be reconstructed from the pixel points on the reference plane in the depth image, and obtains the fused image after the accuracy check of the reference plane pixel points, thereby improving the accuracy of the fused image.

[0087] For example, taking the target object as a PCB board, there are multiple components on the PCB board, each component is a target object, and the plane formed by connecting the bottom surfaces of these components is called the reference plane. Since the height of each component installed on the PCB board is higher than the PCB board, the plane formed by connecting the bottom surfaces of the components is the corresponding installation plane of the components on the PCB board. It can be considered that the PCB board is used as the reference plane in the depth image and the fused image, and the depth value of the PCB board is the depth value of the reference plane, then the point cloud on the PCB board is called the reference point.

[0088] The target object PCB is provided with a plurality of components as target objects. The micro-projection imaging system generates four sets of depth maps at different angles for the object to be measured, and can generate four depth maps calculated by four angle projection optical machines (up, down, left, and right), such as Figures 5 to 8 As shown, Figure 5 is the left optical machine depth map, Figure 6 is the right optical machine depth map, Figure 7 is the depth map of the glazing machine, Figure 8 is the depth map of the optical machine. Figure 7 The d in the figure shows the distance between two adjacent target objects. Figure 7 In the image, due to the occlusion between the target objects, the depth value of the box area cannot be accurately identified, and the depth value of some areas in the box area is higher than the depth value of the reference plane, forming an irregular shadow area, which reduces the imaging accuracy of the depth image. The depth image records the depth value of each point on the horizontal aspect of the target object. The horizontal position and height of each target object in the target object can be obtained through the depth image. In the embodiment of the present application, the projection optical machine projects the target object at multiple angles to obtain depth maps of the target object at each angle, and each depth map includes at least one target object. In one possible implementation, each of the depth images includes the horizontal position (x, y) and depth value / height value (z) of all point clouds, and each point cloud data in the depth image can be expressed in the form of (x, y, z).

[0089] Depth images are usually stored as a data set in matrix form, which includes the height value of the target object at each pixel position. The height value is displayed as different values ​​according to the accuracy, such as 0.1, 0.23, 0.3, etc., and the unit is mm.

[0090] In a specific example, Fig. 9 The depth image shown as the reference image stores the height values ​​of the target object at different positions: 0.7532 0.7526 0.7521 0.7517; 0.752 0.7515 0.751 0.7505; 0.7504 0.7499 0.7494 0.749 ;

[0094] 0.7486 0.7481 0.7478 0.7475.

[0095] If the distance between adjacent pixels in the depth image is 0.01, a three-dimensional coordinate system can be established. The three-dimensional coordinates of the depth image after being converted into a point cloud are:

[0096] (0, 0, 0.7532) (0, 0.01, 0.7526) (0, 0.02, 0.7521) (0, 0.03, 0.7517);

[0097] (-0.01, 0, 0.752) (-0.01, 0.01, 0.7515) (-0.01, 0.02, 0.751) (-0.01, 0.03, 0.7505);

[0098] (-0.02, 0, 0.7504) (-0.02, 0.01, 0.7499) (-0.02, 0.02, 0.7494) (-0.02, 0.03, 0.749);

[0099] (-0.03, 0, 0.7486) (-0.03, 0.01, 0.7481) (-0.03, 0.02, 0.7478) (-0.03, 0.03, 0.7475).

[0100] In other embodiments, for Fig. 9 The depth image shown can generate a discrete point cloud of the target object according to a preset discrete size in the height direction for each pixel in the depth image. For example, for the height of the pixel at the (0, 0) position is 0.7532, selecting a discrete size of 0.01 can generate multiple points of (0, 0, 0), (0, 0, 0.01) ... (0, 0, 0.7532) in the height direction. Discretization of all points can obtain a discrete point cloud corresponding to the three-dimensional size of the target object, thereby improving the accuracy of reference plane extraction. The discrete size can be set according to demand, and those skilled in the art can determine it according to actual conditions, and this application does not limit this.

[0101] In an optional embodiment, if Fig.10 As shown, S020 determines the pixel point to be reconstructed according to the depth value of the pixel point on the reference plane, including:

[0102] S021: Fusing the multiple depth images to obtain a preliminary fused image;

[0103] S022: Determine all pixel points on the preliminary fused image whose depth values ​​are higher than the depth value of the reference plane as pixel points to be corrected;

[0104] S023: If a target pixel corresponding to the pixel to be corrected in the depth image is on the reference plane, use the target pixel as the pixel to be reconstructed.

[0105] In one or more embodiments, the pixel points on one or more depth image reference planes may be determined as the pixel points to be reconstructed. In this embodiment, in order to reduce the amount of calculation, multiple depth images may be fused based on an existing fusion algorithm to obtain a fused image, and the pixel points in the fused image whose depth values ​​are higher than the depth values ​​of the reference plane and reach a preset threshold are determined as the pixel points to be corrected, and the pixel points in the depth image corresponding to the pixel points to be corrected are the pixel points on the reference plane as the pixel points to be reconstructed, and there is no need to correct all the pixel points on the reference plane on the depth image, so as to avoid too many useless calculation processes and improve the calculation efficiency.

[0106] In an optional embodiment, if Fig.11 As shown, the step S200 determines whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area, including:

[0107] S221: Divide the determination area into a first part and a second part along the direction of the closest target object edge with the pixel to be reconstructed as the center;

[0108] S222: Calculate characteristic values ​​of depth values ​​of pixels in the first part and the second part respectively;

[0109] S223: Determine whether the pixel to be reconstructed needs to be reconstructed according to the feature values ​​of the first part and the second part.

[0110] Specifically, it can be understood that if a pixel point is a pixel point on the reference plane, the area on at least one side of the pixel point is the reference plane area. Based on this principle, the area to be determined can be demarcated as the center of the pixel point to be reconstructed, and whether there is a reference plane area on both sides of the pixel point to be reconstructed is determined to determine whether the pixel point to be reconstructed is a point on the reference plane.

[0111] Optionally, the size of the determination area is smaller than the minimum spacing of the multiple target objects, so as to prevent the problem that the target object area is included in the determination area, resulting in an erroneous determination result of whether the pixel to be reconstructed is a point on the reference surface. The minimum spacing of the target objects can be preset or determined by real-time detection in a variety of ways such as design parameters of the target objects, target object recognition, and depth image recognition, and this application does not limit this.

[0112] Among them, it should be noted that the pixel point to be reconstructed is preferably located at the center of the judgment area. Of course, the pixel point to be reconstructed can also deviate from the center, so that the length and width of the divided first part and the second part are less than the minimum spacing between the target objects, thereby preventing the target object area from being included in the judgment area, resulting in an erroneous judgment result of whether the pixel point to be reconstructed is a point on the reference plane.

[0113] Similarly, the shape of the determination area matches the target object, and the determination area is divided into a first part and a second part along the direction of the closest target object edge with the pixel to be reconstructed as the center, so that the divided first part and the second part are closer to the shape of the reference plane, thereby preventing the problem of partial area of ​​the target object being included in the determination area, resulting in an erroneous determination result of whether the pixel to be reconstructed is a point on the reference plane.

[0114] In an optional implementation manner, the step S222 of respectively calculating the feature values ​​of the depth values ​​of the first part and the second part of the pixel points includes:

[0115] S2221: Calculate the mean and variance of the depth values ​​of the pixels of the first part and the second part respectively as the corresponding feature values ​​of the first part and the second part.

[0116] Specifically, the depth value analysis of the pixel points can be used to determine whether any side of the first part and the second part is a reference plane. For example, in one embodiment, the difference between the depth value of each pixel point in the first part and the second part and the depth value of the reference plane can be calculated as the corresponding eigenvalue to determine whether at least one of the first part and the second part is a point on the reference plane. In this embodiment, in order to reduce the computational complexity, the mean and variance are calculated as eigenvalues ​​to determine the reference plane area, which greatly improves the computational efficiency. Of course, in other embodiments, those skilled in the art may also adopt other eigenvalue calculation methods, which are not limited in this application. These technical solutions based on the same inventive concept should also be within the scope of protection of this application.

[0117] In an optional implementation manner, the step S223 of determining whether the pixel to be reconstructed needs to be reconstructed according to the feature values ​​of the first part and the second part includes:

[0118] S2231: Determine whether the eigenvalues ​​of the first part and the second part are inconsistent with the eigenvalues ​​corresponding to the reference surface.

[0119] In the embodiment of the present application, the mean and variance corresponding to the pixels of the first part and the second part are calculated respectively. If the difference between the mean corresponding to all the pixels of any part and the mean of the reference point in the preset depth image is less than the mean condition of the set mean threshold, and the variance satisfies the variance condition of the preset variance threshold range, it means that at least one side of the first part and the second part of the pixel to be reconstructed is the reference point, then the pixel to be reconstructed can be confirmed as the reference point. and variance σ can be calculated by formula (1) and (2) respectively:

[0120] Wherein, pi is the i-th pixel point of the first part or the second part, and n is the number of pixels of the first part or the second part.

[0121] In an optional implementation manner, the step S200 of selecting a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed includes:

[0122] S210: Selecting a rectangular determination area surrounding the pixel to be reconstructed at the position of the pixel to be reconstructed, wherein the size of the rectangular determination area is smaller than the minimum spacing between the multiple target objects.

[0123] It is understandable that in this embodiment, if the target object is in the shape of a cube or a cylinder, a rectangular determination area can be selected. If the horizontal cross-sectional shape of the target object is a parallelogram or other shape containing inclined edges, the corresponding edge of the determination area should also be an inclined edge. Of course, in actual applications, those skilled in the art can determine the shape of the determination area according to actual conditions.

[0124] In a specific example, taking the target object as a PCB board, there are multiple components on the PCB board, each component is a target object, and the shape of the component is a cube. Then, a rectangular judgment area can be set around the pixel to be reconstructed, so that the length and width of the rectangular judgment area are less than the minimum spacing between the components.

[0125] In other feasible implementations, the target object in the depth image and the fused image may be first identified, the target object may be identified and its features may be extracted to obtain the shape of the target object. The target object identification may be implemented using an existing image recognition algorithm, which will not be described in detail here. For example, in the embodiment of the present application, the target object is a component, such as Figure 5-Figure 8 As shown, the target object appears as an approximately rectangular component in the depth image and the fused image.

[0126] In an optional implementation manner, the step of obtaining a fused image based on the depth image including the pixel to be reconstructed in step S300 includes:

[0127] S310: Obtaining depth values ​​of corresponding pixels in the fused image based on depth values ​​of multiple pixels at the same position in the multiple depth images, wherein the pixel to be reconstructed exists in the pixel points in the multiple depth images corresponding to the fused pixel points of each fused image, and if the pixel to be reconstructed does not need to be reconstructed, the depth value of the pixel to be reconstructed is used as the depth value of the fused pixel point; if the pixel to be reconstructed needs to be reconstructed, the depth value of the fused pixel point is obtained based on the depth value of the pixel point corresponding to the depth image where the pixel to be reconstructed does not exist.

[0128] Specifically, it can be understood that image fusion is the integration of information from multiple source images (which may be from different sensors, different perspectives, and different time points) into a single image. This process aims to generate a comprehensive image that contains all key details and has enhanced characteristics by effectively merging complementary, redundant, or unique information in each source image. In one possible implementation, an image fusion splicing method can be used to directly superimpose two images, add or average the point cloud pixel values ​​of the two images, and merge them into a fused depth image. In another possible implementation, a deep learning image fusion splicing method can also be used, mainly including a method based on a generative adversarial network (GAN), a method based on a convolutional neural network (CNN), and the like. The image fusion splicing method based on a convolutional neural network mainly extracts features and performs fusion operations on the input image through structures such as convolutional layers, pooling layers, and fully connected layers. By training the network model, the local and global features of the image can be effectively learned, and a fused depth image that meets the requirements can be generated.

[0129] In this optional implementation, the fused image obtained by the fusion of depth images still has the problem of adhesion. When multiple depth images are fused, it is first determined whether the pixel point at the corresponding position of each depth image is a point on the reference plane that does not need to be reconstructed, that is, after being determined to be a point on the reference plane, if it exists, the depth value of the fused pixel point at the corresponding position of the fused image is directly set to the depth value of the reference plane. If there is a pixel point that needs to be reconstructed, it means that the pixel value of the pixel point is inaccurate, and the depth value of the fused pixel point is obtained based on the depth value of the pixel point corresponding to the depth image where the pixel point to be reconstructed does not exist. If there is no pixel point to be reconstructed, the depth value of the corresponding fused pixel point on the fused image is determined directly based on the depth value of the pixel point at the corresponding position of all depth images using a fusion algorithm such as clustering or averaging. The accuracy and efficiency of the fused image directly obtained based on this implementation can be guaranteed.

[0130] In an optional embodiment, if Fig.12 As shown, the step S300 of obtaining a fused image based on the depth image containing the pixel to be reconstructed includes:

[0131] S321: Obtain a preliminary fused image through a fusion algorithm based on the multiple depth images.

[0132] S322: If the depth value of the fused pixel point corresponding to the to-be-reconstructed pixel point in the fused image exceeds the depth value of the reference plane by a value greater than a preset threshold, setting the depth value of the fused pixel point to the depth value of the reference plane.

[0133] In this optional implementation, an existing fusion algorithm model may be used to obtain a preliminary fusion image based on multiple depth images, and the pixels of the non-reference surface in the preliminary fusion image may be corrected by determining the pixels to be reconstructed that do not need to be reconstructed to obtain the final fusion image, thereby ensuring the accuracy of the fusion image. That is, if the depth value of the fusion pixel corresponding to the pixel to be reconstructed in the fusion image exceeds the depth value of the reference surface by more than a preset threshold, the depth value of the fusion pixel is set to the depth value of the reference surface.

[0134] Fig.13 is a schematic diagram of fusion images in the prior art, Fig.13 The oval area in the figure is the adhesion area that cannot be eliminated after fusion. Fig.14 is a schematic diagram of a fused image obtained by using the image processing method based on pixel reconstruction of the present application. Fig.13 and Fig.14 The difference between them can be seen intuitively. In this application, the fused image obtained by the above method will Fig.13 In order to eliminate the adhesion part in the depth image, the present application determines whether the pixel points to be reconstructed in the depth image need to be reconstructed, determines the accuracy of the depth value of the pixel points to be reconstructed, and obtains a fused image using the depth image containing the verified reference plane pixel points, thereby improving the accuracy of the obtained fused image, which is helpful to solve the problem that the adhesion caused by occlusion in the fused image cannot be accurately eliminated.

[0135] Based on the same principle, the present application also discloses an image processing device based on pixel reconstruction. Fig.15 As shown, in this embodiment, the device includes a pixel point selection module 11, a pixel point reconstruction module 12 and an image fusion module 13.

[0136] The pixel point selection module 11 is used to determine the pixel points to be reconstructed in the depth image corresponding to the target object, the depth image is at least one of a plurality of depth images obtained by multi-angle projection imaging of the target object using a projection system, the depth image includes depth values ​​of a plurality of pixel points, the target object includes a plurality of target objects, the setting surfaces of the plurality of target objects are used as reference surfaces, and the pixel points to be reconstructed are pixel points on the depth image that are on the reference surface;

[0137] The pixel reconstruction module 12 is used to select a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed, and determine whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area;

[0138] The image fusion module 13 is used to obtain a fused image based on the depth image containing the pixel points to be reconstructed.

[0139] In an optional embodiment, the pixel reconstruction module 12 is used to divide the determination area into a first part and a second part along the direction of the closest target object edge with the pixel to be reconstructed as the center; calculate the characteristic values ​​of the depth values ​​of the pixels in the first part and the second part respectively; and determine whether the pixel to be reconstructed needs to be reconstructed based on the characteristic values ​​of the first part and the second part.

[0140] In an optional implementation, the pixel reconstruction module 12 is used to calculate the mean and variance of the depth values ​​of the pixels of the first part and the second part respectively as the corresponding feature values ​​of the first part and the second part.

[0141] In an optional implementation, the pixel reconstruction module 12 is used to determine whether the characteristic values ​​of the first part and the second part are both inconsistent with the characteristic values ​​corresponding to the reference surface.

[0142] In an optional implementation, the pixel reconstruction module 12 is used to select a rectangular determination area surrounding the pixel to be reconstructed at the position of the pixel to be reconstructed, and the size of the rectangular determination area is smaller than the minimum spacing between the multiple target objects.

[0143] In an optional implementation, the size of the determination area is smaller than the minimum spacing between the multiple target objects.

[0144] In an optional embodiment, the pixel selection module 11 is further used to extract the reference plane based on multiple depth images before determining the pixel points to be reconstructed in the depth image corresponding to the target object; and determine the pixel points to be reconstructed according to the depth values ​​of the pixel points on the reference plane.

[0145] In an optional embodiment, the pixel selection module 11 is used to generate a point cloud according to the pixel depth values ​​of at least one of the depth image and the fused image; and extract all pixel points that meet the preset standard from the point cloud to obtain the reference plane.

[0146] In an optional embodiment, the pixel selection module 11 is used to fuse the multiple depth images to obtain a preliminary fused image; determine all pixel points on the preliminary fused image whose depth values ​​are higher than the depth value of the reference plane as pixel points to be corrected; if the target pixel point corresponding to the pixel point to be corrected in the depth image is on the reference plane, the target pixel point is used as the pixel point to be reconstructed.

[0147] In an optional embodiment, the image fusion module 13 is used to obtain the depth value of the corresponding pixel point in the fused image based on the depth values ​​of multiple pixel points at the same position in the multiple depth images, wherein the pixel point to be reconstructed exists in the pixel points in the multiple depth images corresponding to the fused pixel point of each fused image. If the pixel point to be reconstructed does not need to be reconstructed, the depth value of the pixel point to be reconstructed is used as the depth value of the fused pixel point; if the pixel point to be reconstructed needs to be reconstructed, the depth value of the fused pixel point is obtained based on the depth value of the pixel point corresponding to the depth image in which the pixel point to be reconstructed does not exist.

[0148] In an optional embodiment, the image fusion module 13 is used to obtain a preliminary fused image through a fusion algorithm based on the multiple depth images; if the depth value of the fused pixel point corresponding to the pixel point to be reconstructed in the fused image exceeds the depth value of the reference plane by a value greater than a preset threshold, the depth value of the fused pixel point is set to the depth value of the reference plane.

[0149] Since the principle of solving the problem by the device is similar to that of the above method, the implementation of the device can refer to the implementation of the method and will not be described in detail here.

[0150] The present application also discloses a microscopic fringe projection imaging system, including an image processing device based on pixel reconstruction as described in this embodiment.

[0151] Optionally, the micro-fringe projection imaging system may be any one of a monocular dual-projection, a monocular four-projection, and a four-eye single-projection imaging system.

[0152] Since the principle of solving the problem by this system is similar to that of the above method and device, the implementation of this system can refer to the implementation of the method and device, which will not be repeated here.

[0153] An embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0154] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0155] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer programs to produce the systems, devices, modules or units described in the above embodiments, which can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer device, and specifically, the computer device can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0156] In a typical example, the computer device specifically includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method executed by the client as described above is implemented; or, when the processor executes the program, the method executed by the server as described above is implemented.

[0157] Reference below Fig.16 , which shows a schematic diagram of the structure of a computer device 600 suitable for implementing an embodiment of the present application.

[0158] like Fig.16 As shown, the computer device 600 includes a central processing unit (CPU) 601, which can perform various appropriate tasks and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer device 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0159] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal feedback device (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed as needed as the storage section 608.

[0160] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 609, and / or installed from a removable medium 611.

[0161] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0162] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0163] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0164] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0166] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0167] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0168] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0169] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0170] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method based on pixel reconstruction, characterized in that: The method comprises: Determine a pixel point to be reconstructed in a depth image corresponding to a target object, wherein the depth image is at least one of a plurality of depth images obtained by performing multi-angle projection imaging of the target object using a projection system, the depth image includes depth values ​​of a plurality of pixel points, the target object includes a plurality of target objects, and the setting surfaces of the plurality of target objects are used as reference surfaces, and the pixel point to be reconstructed is a pixel point on the depth image that is on the reference surface; Selecting a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed, determining whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area, wherein the size of the determination area is smaller than the minimum spacing of the multiple target objects; If not, obtaining a fused image based on the depth image including the pixel to be reconstructed, wherein obtaining the fused image based on the depth image including the pixel to be reconstructed includes: obtaining the depth value of the corresponding pixel in the fused image based on the depth values ​​of multiple pixels at the same position in the multiple depth images, wherein the pixel to be reconstructed exists in the pixels in the multiple depth images corresponding to the fused pixel of each fused image, and using the depth value of the pixel to be reconstructed as the depth value of the fused pixel; The step of determining whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area comprises: Dividing the determination area into a first part and a second part along the direction of the closest target object edge with the pixel to be reconstructed as the center; Calculate the characteristic values ​​of the depth values ​​of the pixels in the first part and the second part respectively; Determining whether the pixel to be reconstructed needs to be reconstructed according to the characteristic values ​​of the first part and the second part; determining whether the pixel to be reconstructed needs to be reconstructed according to the characteristic values ​​of the first part and the second part includes: It is determined whether the eigenvalues ​​of the first portion and the second portion are both inconsistent with the eigenvalues ​​corresponding to the reference surface.

2. The image processing method based on pixel reconstruction according to claim 1, characterized in that: The step of respectively calculating the characteristic values ​​of the depth values ​​of the first part and the second part of the pixel points comprises: The means and variances of the depth values ​​of the pixels of the first part and the second part are calculated respectively as the corresponding feature values ​​of the first part and the second part.

3. The image processing method based on pixel reconstruction according to claim 1, characterized in that: The step of selecting a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed comprises: A rectangular determination area surrounding the pixel to be reconstructed is selected at the position of the pixel to be reconstructed, and a size of the rectangular determination area is smaller than a minimum spacing between the multiple target objects.

4. The image processing method based on pixel reconstruction according to claim 1, characterized in that: Further comprising determining a pixel to be reconstructed in the depth image corresponding to the target object, before: Extracting and obtaining the reference plane based on multiple depth images; The pixel point to be reconstructed is determined according to the depth value of the pixel point on the reference plane.

5. The image processing method based on pixel reconstruction according to claim 4, characterized in that: The extracting and obtaining the reference plane based on multiple depth images comprises: Generating a point cloud according to pixel depth values ​​of at least one of the depth image and the fused image; The reference plane is obtained by extracting all pixel points that meet a preset standard from the point cloud.

6. The image processing method based on pixel reconstruction according to claim 4, characterized in that: The step of determining the pixel point to be reconstructed according to the depth value of the pixel point on the reference surface comprises: Fusing the multiple depth images to obtain a preliminary fused image; Determine all pixel points on the preliminary fused image whose depth values ​​are higher than the depth value of the reference plane as pixel points to be corrected; If the target pixel point corresponding to the pixel point to be corrected in the depth image is on the reference plane, the target pixel point is used as the pixel point to be reconstructed.

7. The image processing method based on pixel reconstruction according to claim 1, characterized in that: Also includes: If the pixel to be reconstructed needs to be reconstructed, the depth value of the fused pixel is obtained based on the depth value of the pixel corresponding to the depth image in which the pixel to be reconstructed does not exist.

8. The image processing method based on pixel reconstruction according to claim 1, characterized in that: The step of obtaining a fused image based on the depth image containing the pixel points to be reconstructed comprises: Obtaining a preliminary fused image through a fusion algorithm based on the multiple depth images; If the depth value of the fused pixel point corresponding to the to-be-reconstructed pixel point in the fused image exceeds the depth value of the reference plane by a value greater than a preset threshold, the depth value of the fused pixel point is set to the depth value of the reference plane.

9. An image processing device based on pixel reconstruction, characterized in that: include: A pixel point selection module is used to determine the pixel points to be reconstructed in the depth image corresponding to the target object, the depth image is at least one of a plurality of depth images obtained by multi-angle projection imaging of the target object using a projection system, the depth image includes depth values ​​of a plurality of pixel points, the target object includes a plurality of target objects, the setting surfaces of the plurality of target objects are used as reference surfaces, and the pixel points to be reconstructed are the pixel points on the depth image that are on the reference surface; A pixel reconstruction module is used to select a determination area surrounding the pixel to be reconstructed and matching the shape of the target object according to the position of the pixel to be reconstructed, and determine whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area, wherein the size of the determination area is smaller than the minimum spacing of the multiple target objects, and the determination of whether the pixel to be reconstructed needs to be reconstructed according to the depth value of the pixel in the determination area includes: Dividing the determination area into a first part and a second part along the direction of the closest target object edge with the pixel to be reconstructed as the center; Calculate the characteristic values ​​of the depth values ​​of the pixels in the first part and the second part respectively; Determining whether the pixel to be reconstructed needs to be reconstructed according to the characteristic values ​​of the first part and the second part; determining whether the pixel to be reconstructed needs to be reconstructed according to the characteristic values ​​of the first part and the second part includes: determining whether the eigenvalues ​​of the first portion and the second portion are both inconsistent with the eigenvalues ​​corresponding to the reference surface; An image fusion module is used to obtain a fused image based on a depth image containing the pixel to be reconstructed, and if not, the fused image is obtained based on the depth image containing the pixel to be reconstructed, wherein the fused image is obtained based on the depth values ​​of multiple pixels at the same position in the multiple depth images, obtaining the depth value of the corresponding pixel in the fused image, wherein the pixel to be reconstructed exists in the pixel points in the multiple depth images corresponding to the fused pixel of each fused image, and the depth value of the pixel to be reconstructed is used as the depth value of the fused pixel.

10. A microscopic fringe projection imaging system, characterized in that: The invention comprises the image processing device as claimed in claim 9.

11. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images