Fusion image correction method, device and imaging system
By extracting the reference of the target object to correct the depth value of the fusion image, the problem of loss of depth information caused by occlusion in the 3D imaging system of microstripe projection is solved, and the accuracy and effect of three-dimensional imaging is improved.
Patent Information
- Application Number
- CN202411146646.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-21
AI Technical Summary
When the existing microscopic stripe projection three-dimensional imaging system measures dense or highly different target objects, the depth information of some areas is lost due to occlusion of adjacent objects, which affects the imaging effect.
Multiple depth images obtained by multi-angle projection are fused, and the reference surface of the target object is extracted, and the fusion image is corrected based on the depth value of the reference surface to avoid adhesion caused by occlusion and improve the accuracy of three-dimensional imaging.
Effectively remove the adhesion problem in the fused image, improving the accuracy of the fused image formed by the fusion of multiple depth images and the three-dimensional imaging effect of the target object.
Smart Images

Figure CN118657698B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of optical measurement and detection technology, and in particular to a fusion image correction method, device and imaging system. Background Art
[0002] This section is intended to provide a background or context to the embodiments of the present application as recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.
[0003] In recent years, with the development of precision manufacturing and semiconductor manufacturing, the three-dimensional imaging and measurement of complex and tiny structures have become increasingly important. Micro-fringe projection three-dimensional measurement technology (MFP-3D) has become the mainstream technology for optical three-dimensional surface imaging and measurement because of its non-contact, fast measurement speed, dense point cloud data, low cost and easy implementation, and it can better balance system flexibility and measurement accuracy. It has broad application prospects and practical value in industrial production and inspection in complex environments.
[0004] The current microscopic fringe projection 3D imaging system is usually composed of a telecentric lens and a projection optical machine. The target object is projected at multiple angles to obtain at least one depth image. If there are multiple depth images, the multiple depth images need to be fused to obtain a fused image. However, when measuring samples with dense components or target objects with obvious height differences, the phase-shifted fringes cannot be projected to some areas due to the occlusion of adjacent objects, resulting in the loss of depth information in the area, affecting the final imaging effect. Summary of the invention
[0005] One object of the present application is to provide a fused image correction method, extract the reference plane of the target object in the image, correct the fused image by the depth value of the reference plane, and solve the adhesion problem in the fused image. Another object of the present application is to provide a fused image correction device. Another object of the present application is to provide a microscopic fringe projection imaging system. Another object of the present application is to provide a computer device. Another object of the present application is to provide a readable medium.
[0006] In order to achieve the above-mentioned object, the present application discloses a fusion image correction method, comprising:
[0007] A fused image is obtained by fusing multiple depth images obtained by projecting the target object at multiple angles;
[0008] Extracting a reference plane for setting a plurality of target objects included in the target object according to at least one of the depth image and the fused image;
[0009] The depth value of the fused image is corrected based on the depth value of the reference plane.
[0010] Optionally, extracting a reference plane for setting a plurality of target objects included in the target object according to at least one of the depth image and the fused image comprises:
[0011] Obtaining a discrete point cloud of the target object based on at least one of the depth image and the fused image;
[0012] Identifying the setting surface of the target object according to a preset standard;
[0013] The reference surface is obtained by extracting discrete points corresponding to the setting surface from the discrete point cloud.
[0014] Optionally, the identifying the setting surface of the target object according to a preset standard includes:
[0015] Determine a surface formed by connecting bottom surfaces of multiple target objects of the target object in the depth image as a setting surface of the target object.
[0016] Optionally, the identifying the setting surface of the target object according to a preset standard includes:
[0017] The setting surface of the target object is identified based on a preset setting surface shape.
[0018] Optionally, the target object includes a setting surface and a plurality of target objects set on the setting surface;
[0019] The extracting the discrete points corresponding to the setting surface from the discrete point cloud to obtain the reference surface comprises:
[0020] The discrete points corresponding to the horizontal plane with the largest area are extracted from the discrete point cloud to obtain the reference plane corresponding to the setting surface.
[0021] Optionally, the correcting the depth value of the fused image based on the depth value of the reference plane includes:
[0022] For the point to be corrected in the fused image, determining corresponding points at the same position in the plurality of depth images corresponding to the point to be corrected;
[0023] Using the depth value of the target discrete point corresponding to the point to be corrected in the reference plane to detect whether the point to be corrected needs to be corrected;
[0024] If so, the depth value of the point to be corrected is set to the depth value of the corresponding target discrete point in the reference plane.
[0025] Optionally, the using the depth value of the target discrete point corresponding to the point to be corrected in the reference plane to detect whether the point to be corrected needs to be corrected includes:
[0026] It is determined in sequence whether the difference between the depth value of the target discrete point and the depth value of the corresponding point in each corresponding depth image is less than a preset threshold.
[0027] The present application also discloses a fusion image correction device, comprising:
[0028] An image fusion module, used to fuse multiple depth images obtained by multi-angle projection of the target object to obtain a fused image;
[0029] An image processing module, configured to extract a reference plane for setting a plurality of target objects included in the target object according to at least one of the depth image and the fused image;
[0030] An image correction module is used to correct the depth value of the fused image based on the depth value of the reference plane.
[0031] The present application also discloses a microscopic fringe projection imaging system, comprising the fusion image correction device as described above.
[0032] The present application also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.
[0033] The present application also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.
[0034] The fused image correction method of the present application fuses multiple depth images obtained by multi-angle projection of the target object to obtain a fused image, extracts a reference plane set by multiple target objects contained in the target object based on at least one of the depth image and the fused image, and corrects the depth value of the fused image based on the depth value of the reference plane. Thus, the present application extracts the reference plane set by the target object in the fused image, corrects the depth value of the fused image by the reference plane, avoids the adhesion phenomenon that also appears on the fused image obtained after the fusion of the depth images with adhesion caused by occlusion on the target object when projected at multiple angles, intelligently removes the adhesion problem caused by occlusion on the fused image, improves the accuracy of the fused image formed by the fusion of multiple depth images, and improves the three-dimensional imaging effect of the target object. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0036] Figure 1 This is a flowchart of a specific embodiment of the fusion image correction method of the present application embodiment;
[0037] Figure 2 This is a schematic diagram of the structure of a microscopic fringe projection imaging system according to a specific embodiment of the fusion image correction method of the present application;
[0038] Figure 3 This is a flowchart of a specific embodiment S200 of the fusion image correction method according to an embodiment of the present application;
[0039] Figure 4 This is a schematic diagram of recording height values of a depth image in a specific embodiment of the fusion image correction method of an embodiment of the present application;
[0040] Figure 5 This is a flowchart of a specific embodiment S300 of the fusion image correction method according to an embodiment of the present application;
[0041] Figure 6 This is a structural schematic diagram of a PCB board in a specific example of the fusion image correction method in an embodiment of the present application;
[0042] Figure 7~Figure 10 This is a schematic diagram of a depth image obtained by projecting a PCB board at four angles in a specific example of the fusion image correction method in an embodiment of the present application;
[0043] Fig.11 and Fig.12 A schematic diagram of a fused image in a specific example of the fused image correction method according to an embodiment of the present application;
[0044] Fig.13 A schematic diagram of a discrete point cloud in a specific example of the fusion image correction method in an embodiment of the present application;
[0045] Fig.14 A schematic diagram of a reference plane in a specific example of the fusion image correction method in an embodiment of the present application;
[0046] Fig.15 A schematic diagram of a corrected fused image in a specific example of the fused image correction method in an embodiment of the present application;
[0047] Fig.16 This is a structural schematic diagram of a specific embodiment of the fusion image correction device of the present application embodiment;
[0048] Fig.17 A schematic diagram showing the structure of a computer device suitable for implementing an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0050] It should be understood that the terms "first", "second", etc. in the specification, claims and drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0051] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.
[0052] The microscopic stripe projection imaging system usually includes a front projection camera or optical machine arranged above the target object to be imaged and at least one side projection optical machine or camera arranged on the side of the target object to be imaged. The traditional microscopic stripe projection imaging system usually includes a front projection camera and a side projection optical machine (monocular single projection), and directly outputs the depth image of the target object to be imaged through the monocular single projection imaging system. However, when this monocular single projection traditional imaging system projects some target objects including multiple densely arranged or with obvious height differences, the multiple target objects in the target object will block each other, which will cause the imaging system projection optical machine to be unable to project to some areas, resulting in the depth image obtained by projecting the target object will lose the depth information of some areas of the target object, resulting in poor imaging effect of the target object.
[0053] On this basis, in order to improve the projection imaging quality of the target object, the prior art usually obtains depth images formed by multiple projections of the target object at different angles by adding a side projector or camera, and then fuses the obtained multiple depth images to obtain a fused image. Although the method of obtaining a fused image as the final projected depth image of the target object by fusing multiple depth images avoids the problem of poor imaging effect of the depth image projected at a single angle to a certain extent, the existing method of fusing multiple depth images still cannot completely solve the problem of occlusion formed by multiple targets.
[0054] Based on this, the present application provides a fused image correction method, device and system, which extracts the reference plane of the target object based on multiple depth images of the target object projected at multiple angles, and uses the depth value of the reference plane to correct the depth value of the fused image, so as to avoid the adhesion phenomenon on the fused image obtained after the fusion of depth images caused by occlusion on the target object when projected at multiple angles, intelligently remove the adhesion problem caused by occlusion on the fused image, improve the accuracy of the fused image formed by the fusion of multiple depth images, and improve the three-dimensional imaging effect of the target object.
[0055] According to one aspect of the present application, this embodiment discloses a fusion image correction method. Figure 1 As shown, in this embodiment, the method includes:
[0056] S100: fusing multiple depth images obtained by multi-angle projection of the target object to obtain a fused image.
[0057] Optionally, a microscopic fringe projection imaging system may be used to project the target object at multiple angles to obtain multiple depth images. Of course, in other embodiments, multiple depth images of the target object projected at multiple angles may also be generated by other three-dimensional imaging systems, and this application does not limit this.
[0058] Among them, the microscopic fringe projection imaging system can adopt any one of the imaging systems such as monocular dual projection, monocular quadruple projection and quadruple single projection. Among them, the monocular dual projection imaging system includes a front projection camera and two side projection light machines. The monocular quadruple projection imaging system includes a front projection camera and four side projection light machines. The quadruple single projection imaging system includes a front projection light machine and two side projection cameras.
[0059] exist Figure 2 In the specific example shown, the monocular four-projection microscopic fringe projection imaging system includes a front projection camera 100 arranged directly above the target object and four side projection cameras 200 respectively arranged in four directions of the upper, lower, left and right of the target object. The monocular four-projection imaging system can respectively capture four depth images of the target object projected from four angles of upper, lower, left and right.
[0060] S200: extracting a reference plane for setting a plurality of target objects included in the target object according to at least one of the depth image and the fused image.
[0061] Specifically, it can be understood that the depth images of the target object at different angles obtained by multi-angle projection of the target object are fused through a fusion algorithm to obtain a fused image. Among them, the depth image records the depth value (height) of each point on the horizontal side of the target object, and the position and height of each target object in the target object in the horizontal direction can be obtained through the depth image. Therefore, the height information of the reference plane and the target object is recorded in the depth image and the fused image, and the reference plane corresponding to the setting surface of multiple targets in the target object can be obtained according to at least one image of the multiple depth images and the fused image.
[0062] Among them, it should be noted that the target object of the present application includes multiple targets, and the multiple targets are arranged in a staggered manner. Then the target may block other targets in three-dimensional imaging, resulting in adhesion caused by errors in the depth values of certain positions in the final fused image. Among them, multiple targets may be set on the same surface, or on surfaces at different heights and angles, and at least one surface carrying multiple targets forms a setting surface for multiple targets. Among them, the setting surface can be the surface of a structure actually existing in the target object, or it can be a virtual surface determined based on the positions of multiple targets. The preset standard for extracting the setting surface can be pre-set, or it can be obtained by analyzing the shape of the target object. For example, the setting surface shape of the setting surface can be obtained by analyzing the depth image of the target object projection, and the setting surface shape is used as the preset standard for extraction to extract the setting surface to obtain a reference surface.
[0063] S300: Correcting the depth value of the fused image based on the depth value of the reference plane.
[0064] The present application extracts the reference plane set for the target object in the fused image, and corrects the depth value of the fused image through the reference plane, thereby avoiding the adhesion phenomenon that also appears on the fused image obtained after the fusion of depth images with adhesion caused by occlusion on the target object when projected at multiple angles. The adhesion problem caused by occlusion on the fused image is intelligently removed, the accuracy of the fused image formed by the fusion of multiple depth images is improved, and the three-dimensional imaging effect of the target object is improved.
[0065] In an optional embodiment, if Figure 3 As shown, the step S200 extracts a reference plane for setting a plurality of target objects included in the target object according to at least one of the depth image and the fused image, including:
[0066] S210: Obtaining a discrete point cloud of the target object based on at least one of the depth image and the fused image.
[0067] S220: Identify the setting surface of the target object according to a preset standard.
[0068] S230: Extracting discrete points corresponding to the setting surface from the discrete point cloud to obtain the reference surface.
[0069] Specifically, it is understandable that the depth image obtained by taking a 3D imaging device such as an imaging system usually stores image data in the form of discrete points, that is, the depth image includes the horizontal position of the discrete points within the imaging range and the depth value of each discrete point. When a fused image is obtained by fusing multiple depth images, the depth value of each discrete point is obtained by a preset fusion algorithm to obtain the final depth value. The fused image that records the final depth value of each point within the imaging range can better reflect the 3D size of the target object and improve the 3D imaging effect of the target object. Therefore, one or more images can be selected from the depth image and the fused image, wherein the fused image with better imaging quality is preferably selected as the reference image, and a discrete point cloud is generated based on the reference image.
[0070] Depth images are usually stored as a data set in matrix form, which includes the height value of the target object at each pixel position. The height value is displayed as different values according to the accuracy, such as 0.1, 0.23, 0.3, etc., and the unit is mm.
[0071] In a specific example, Figure 4 The depth image shown as the reference image stores the height values of the target object at different positions: 0.75320.75260.75210.7517 0.7520.75150.7510.7505 0.75040.74990.74940.749 0.74860.74810.74780.7475
[0076] If the distance between adjacent pixels in the depth image is 0.01, a three-dimensional coordinate system can be established. The three-dimensional coordinates of the depth image after being converted into a point cloud are:
[0077] (0, 0, 0.7532) (0, 0.01, 0.7526) (0, 0.02, 0.7521) (0, 0.03, 0.7517)
[0078] (-0.01, 0, 0.752) (-0.01, 0.01, 0.7515) (-0.01, 0.02, 0.751) (-0.01, 0.03, 0.7505)
[0079] (-0.02, 0, 0.7504) (-0.02, 0.01, 0.7499) (-0.02, 0.02, 0.7494) (-0.02, 0.03, 0.749)
[0080] (-0.03, 0, 0.7486) (-0.03, 0.01, 0.7481) (-0.03, 0.02, 0.7478) (-0.03, 0.03, 0.7475)
[0081] In other embodiments, for Figure 4 The depth image shown can generate a discrete point cloud of the target object according to a preset discrete size in the height direction for each pixel in the depth image. For example, for the height of the pixel at the (0, 0) position is 0.7532, selecting a discrete size of 0.01 can generate multiple points of (0, 0, 0), (0, 0, 0.01) ... (0, 0, 0.7532) in the height direction. Discretization of all points can obtain a discrete point cloud corresponding to the three-dimensional size of the target object, thereby improving the accuracy of reference plane extraction. The discrete size can be set according to demand, and those skilled in the art can determine it according to actual conditions, and this application does not limit this.
[0082] In an optional implementation manner, the step S220 of identifying the setting surface of the target object according to a preset standard includes:
[0083] S221: Determine a surface formed by connecting bottom surfaces of multiple target objects of the target object in the depth image as a setting surface of the target object.
[0084] Specifically, it is understandable that the target object may include a setting surface for placing multiple targets, or may only include multiple targets. When the target object only includes multiple targets arranged in a staggered manner, the shapes of the targets may be analyzed, and after the bottom surfaces of the targets are identified, the bottom surfaces of the multiple targets may be connected into a virtual surface as the setting surface of the target object, so that the depth values of the points around the targets in the fused image are corrected using the setting surface corresponding to the bottom surfaces of the targets, thereby eliminating the adhesion phenomenon between the targets in the fused image.
[0085] In addition, it should be noted that the setting surface can be a regular shape or an irregular shape. Those skilled in the art can set the generation algorithm of the virtual setting surface according to actual needs. For example, a generation algorithm that connects the bottom surfaces of each target object can be used to obtain a virtual setting surface. Of course, those skilled in the art can also set other generation algorithms, and this application does not limit this.
[0086] In an optional implementation manner, the step S220 of identifying the setting surface of the target object according to a preset standard includes:
[0087] S222: Identify the setting surface of the target object based on a preset setting surface shape.
[0088] Specifically, in order to extract the reference surface of the target object from the discrete point cloud of the target object, the reference surface can be automatically extracted from the discrete point cloud according to the preset standard in the form of a preset standard. Specifically, the preset standard can be the shape feature of the setting surface of the target object.
[0089] In an optional embodiment, the target object includes a setting surface and a plurality of target objects set on the setting surface. The step S230 extracting the discrete points corresponding to the setting surface from the discrete point cloud to obtain the reference surface includes:
[0090] S231: Extracting discrete points corresponding to the horizontal plane with the largest area from the discrete point cloud to obtain a reference plane corresponding to the setting surface.
[0091] It is understandable that when multiple target objects among the target objects are set on the same horizontal plane, the horizontal range of the horizontal plane is the largest, then the horizontal plane is the setting plane of the target object, and according to the shape characteristics of the setting plane, the point coordinates corresponding to the horizontal plane with the largest horizontal range can be extracted from the discrete point cloud to obtain the reference plane corresponding to the setting plane. Of course, in other embodiments, other identification standards of setting planes can also be preset to extract the reference plane, and this application is not limited to this.
[0092] In an optional embodiment, if Figure 5 As shown, the step S300 of correcting the depth value of the fused image based on the depth value of the reference plane includes:
[0093] S310: For the point to be corrected in the fused image, determine corresponding points at the same position in multiple depth images corresponding to the point to be corrected.
[0094] S320: Using the depth value of the target discrete point corresponding to the point to be corrected in the reference plane to detect whether the point to be corrected needs to be corrected.
[0095] S330: If yes, set the depth value of the point to be corrected to the depth value of the corresponding target discrete point in the reference plane.
[0096] Specifically, it can be understood that after extracting the reference plane, the extracted reference plane includes the depth value of the position where the target object is not set in the target object. The depth value of the reference plane is used to correct the depth value of the discrete points in the fused image formed by the fusion of the depth images. The discrete points with incorrect depth values in the fused image after correction can avoid the adhesion phenomenon on the final fused image.
[0097] Among them, it should be noted that when correcting the fused image according to the reference plane, all the points to be corrected can be determined first. The depth values of the corresponding points on the multiple depth images before fusion corresponding to these points to be corrected may be accurate or inaccurate. In order to avoid the problem of inaccurate depth values of the fused image caused by the elimination of the correct depth value during the image fusion process, the present application uses the depth values of each point on the reference plane to detect each depth image corresponding to the fused image to determine whether there is an angle of the depth image that is not blocked by the target object and obtains a depth value close to the reference plane. If it exists, it means that the depth value of the fused image here should also be close to the depth value of the reference plane. Therefore, based on the reference plane, it is possible to detect whether the points to be corrected need to be corrected and correct them to the correct depth value, so that the depth information of the target object in the fused image is more accurate.
[0098] In one or more embodiments, depth value correction can be performed on all points in the fused image based on the reference plane, and all points in the fused image are sequentially detected as points to be corrected. In other embodiments, the image range to be corrected can be confirmed first, and then depth value correction can be performed on all points in the image range to be corrected based on the reference plane, and all points in the image range to be corrected can be used as points to be corrected. The image range to be corrected can be determined by performing depth value analysis on the fused image, and the depth values of all points whose depth values are higher than the reference plane can be determined as points to be corrected based on the depth values, so as to reduce the number of points to be corrected to be detected and reduce the amount of calculation.
[0099] In an optional implementation manner, the step S320 uses the depth value of the target discrete point corresponding to the point to be corrected in the reference plane to detect whether the point to be corrected needs to be corrected, including:
[0100] S321: Determine in sequence whether the difference between the depth value of the target discrete point and the depth value of the corresponding point in each corresponding depth image is less than a preset threshold.
[0101] Specifically, whether there is a depth image with a corresponding point close to the reference plane can be determined by whether the difference between the depth value of the corresponding point in each depth image of the point to be corrected and the depth value of the reference plane is less than a preset threshold. If so, the depth value of the point to be corrected is corrected. When the difference between the depth value of the corresponding point and the depth value of the reference plane is less than the preset threshold, it is considered that the height of the corresponding point on the depth image is close to the height of the reference plane. Of course, in other embodiments, it can also be determined by other methods whether there is a depth image with a depth value close to the discrete point of the reference plane target. Similarly, the value of the preset threshold can be set according to actual needs, and the present application does not limit this.
[0102] The present application is further described below through a specific example.
[0103] like Figure 6 As shown in FIG. 1 , in this specific example, the target object is a PCB board, and a plurality of components as target objects are arranged on the PCB board. By using a monocular four-projection micro-fringe projection imaging system to project images on the PCB board, four depth images at different angles can be obtained, such as Figure 7~Figure 10 shown.
[0104] In the prior art, Figure 7~Figure 10 The fused image obtained by fusing the four depth images is as follows Fig.11 and Fig.12 As shown in the figure. Fig.11 and Fig.12 It can be seen that the depth value of the position indicated by the arrow on the fused image is wrong due to the occlusion of components, and the components at this position show adhesion image characteristics in the depth image. When the depth images are fused, the four depth images are usually fused using fusion algorithms such as cluster analysis, and among the four depth images, only the depth image corresponding to the side projector on the left is not blocked and the components are not adhered. However, if the depth values of the other three depth images here are relatively similar when the four depth images are fused using algorithms such as cluster analysis, the depth image obtained by the side projector on the left will be automatically confirmed as an erroneous depth value and excluded, which will result in the depth value at this point in the final fused image being wrong, and then the problem of component adhesion will still occur here on the fused image. Therefore, the fused image obtained by projecting the target object at multiple angles in the prior art will still be unable to avoid the adhesion problem on the fused image caused by the occlusion of the target object due to the image fusion algorithm.
[0105] This application will Figure 7~Figure 10 The fused image is obtained by projecting the target object at four angles and fusing the multiple depth images obtained. Among them, the four depth images can be fused to obtain the fused image by using an existing fusion algorithm such as a clustering algorithm, which is a conventional technology in the art and will not be described in detail here.
[0106] Generate a discrete point cloud of the target object based on the fused image, such as Fig.13 shown.
[0107] Since the target object includes a PCB board and multiple components on the PCB board, the upper surface of the PCB board can be defined as the setting surface, and the components are the target objects set on the setting surface. The setting surface is a horizontal surface and has a shape feature with the largest horizontal dimension, so the surface with the largest horizontal dimension can be extracted from the discrete point cloud as the reference surface, such as Fig.14 Optionally, the reference plane can be extracted from the discrete point cloud using the PCL library SACSegmenation function. Of course, in other implementations, the reference plane can also be extracted by coordinate analysis and other methods. This application does not limit the method of extracting the reference plane.
[0108] Points in the fused image whose depth values are higher than the reference plane are determined as points to be corrected, and points in the four depth images corresponding to the points to be corrected are determined as target discrete points.
[0109] The depth value of the reference plane is used to determine whether there is a target discrete point close to the reference plane among the target discrete points. If so, it is confirmed whether the depth value of the point to be corrected in the fused image is close to the reference plane. If not, the depth value of the point to be corrected in the fused image needs to be corrected. Specifically, it is determined whether the difference between the depth value of the target discrete point and the depth value of the reference plane is less than a preset threshold. If so, it is determined whether the difference between the depth value of the point to be corrected and the depth value of the reference plane is less than a preset threshold. If so, the depth value of the point to be corrected needs to be corrected.
[0110] If the depth value of the point to be corrected needs to be corrected, the depth value of the reference plane is used as the depth value of the point to be corrected to update the depth value of the point to be corrected in the fused image. Then, all the points to be corrected are traversed, and the depth values of all the points to be corrected are detected and corrected according to the depth image and the fused image. The corrected fused image is as follows: Fig.15 As shown. Fig.15 It can be seen that the fused image corrected by the fused image correction method of the present application eliminates Fig.11 The adhesion phenomenon indicated by the arrow on the fused image shown thereby improves the three-dimensional imaging effect and quality of the target object.
[0111] Based on the same principle, the present application also discloses a fusion image correction device. Fig.16 As shown, in this embodiment, the device includes an image fusion module 11, an image processing module 12 and an image correction module 13.
[0112] The image fusion module 11 is used to fuse multiple depth images obtained by multi-angle projection of the target object to obtain a fused image.
[0113] The image processing module 12 is used to extract a reference plane for setting a plurality of target objects included in the target object according to at least one of the depth image and the fused image.
[0114] The image correction module 13 is used to correct the depth value of the fused image based on the depth value of the reference plane.
[0115] In an optional embodiment, the image processing module 12 is used to obtain a discrete point cloud of the target object based on at least one of the depth image and the fused image; identify the setting surface of the target object according to a preset standard; and extract discrete points corresponding to the setting surface from the discrete point cloud to obtain the reference plane.
[0116] In an optional implementation, the image processing module 12 is used to determine that a surface formed by connecting bottom surfaces of multiple target objects of the target object in the depth image is a setting surface of the target object.
[0117] In an optional implementation, the image processing module 12 is used to identify the setting surface of the target object based on a preset setting surface shape.
[0118] In an optional embodiment, the target object includes a setting surface and a plurality of targets set on the setting surface. The image processing module 12 is used to extract the discrete points corresponding to the horizontal plane with the largest area from the discrete point cloud to obtain the reference plane corresponding to the setting surface.
[0119] In an optional embodiment, the image correction module 13 is used to determine, for the point to be corrected in the fused image, the corresponding point at the same position in multiple depth images corresponding to the point to be corrected; use the depth value of the target discrete point corresponding to the point to be corrected in the reference plane to detect whether the point to be corrected needs to be corrected; if so, set the depth value of the point to be corrected to the depth value of the corresponding target discrete point in the reference plane.
[0120] In an optional implementation, the image correction module 13 is used to sequentially determine whether a difference between a depth value of the target discrete point and a depth value of a corresponding point in each corresponding depth image is less than a preset threshold.
[0121] Since the principle of solving the problem by the device is similar to that of the above method, the implementation of the device can refer to the implementation of the method and will not be repeated here.
[0122] The present application also discloses a microscopic fringe projection imaging system, including the fusion image correction device as described in this embodiment.
[0123] Optionally, the micro-fringe projection imaging system may be any one of a monocular dual-projection, a monocular four-projection, and a four-eye single-projection imaging system.
[0124] Since the principle of solving the problem by this system is similar to that of the above method, the implementation of this system can refer to the implementation of the method and will not be repeated here.
[0125] An embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.
[0126] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0127] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer programs to produce the systems, devices, modules or units described in the above embodiments, which can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer device, and specifically, the computer device can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0128] In a typical example, the computer device specifically includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method executed by the client as described above is implemented; or, when the processor executes the program, the method executed by the server as described above is implemented.
[0129] Reference below Fig.17 , which shows a schematic diagram of the structure of a computer device 600 suitable for implementing an embodiment of the present application.
[0130] like Fig.17 As shown, the computer device 600 includes a central processing unit (CPU) 601, which can perform various appropriate tasks and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0131] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal feedback device (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed as needed as the storage section 608.
[0132] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 609, and / or installed from a removable medium 611.
[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0134] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0136] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0138] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0140] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0142] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A fusion image correction method, characterized in that: include: fusing a plurality of depth images obtained by multi-angle projection of a target object to obtain a fused image, wherein the target object includes a setting surface and a plurality of target objects set on the setting surface; Extracting, according to at least one of the depth image and the fused image, a reference plane corresponding to the setting planes of the plurality of target objects included in the target object in the fused image; The depth value of the fused image is corrected based on the depth value of the reference plane, including: for the point to be corrected in the fused image, determining whether there is a corresponding point at the same position in multiple depth images corresponding to the point to be corrected that is close to the reference plane; if so, correcting the depth value of the point to be corrected.
2. The fused image correction method according to claim 1, characterized in that: The step of extracting a reference plane for setting a plurality of target objects included in the target object according to at least one of the depth image and the fused image comprises: Obtaining a discrete point cloud of the target object based on at least one of the depth image and the fused image; Identifying the setting surface of the target object according to a preset standard; The reference surface is obtained by extracting discrete points corresponding to the setting surface from the discrete point cloud.
3. The fused image correction method according to claim 2, characterized in that: The step of identifying the setting surface of the target object according to a preset standard includes: Determine a surface formed by connecting bottom surfaces of multiple target objects of the target object in the depth image as a setting surface of the target object.
4. The fused image correction method according to claim 2, characterized in that: The step of identifying the setting surface of the target object according to a preset standard includes: The setting surface of the target object is identified based on a preset setting surface shape.
5. The fused image correction method according to claim 2, characterized in that: The extracting the discrete points corresponding to the setting surface from the discrete point cloud to obtain the reference surface comprises: The discrete points corresponding to the horizontal plane with the largest area are extracted from the discrete point cloud to obtain the reference plane corresponding to the setting surface.
6. The fused image correction method according to claim 1, characterized in that: The step of determining corresponding points at the same position in a plurality of depth images corresponding to the point to be corrected in the fused image, determining whether there are corresponding points close to the reference plane, and if so, correcting the depth value of the point to be corrected includes: For the point to be corrected in the fused image, determining corresponding points at the same position in the plurality of depth images corresponding to the point to be corrected; Using the depth value of the target discrete point corresponding to the point to be corrected in the reference plane to detect whether there is a corresponding point close to the reference plane among the corresponding points at the same position of the point to be corrected in multiple depth images to determine whether the point to be corrected needs to be corrected; If so, the depth value of the point to be corrected is set to the depth value of the corresponding target discrete point in the reference plane.
7. The fused image correction method according to claim 6, characterized in that: The step of using the depth value of the target discrete point corresponding to the point to be corrected in the reference plane to detect whether the point to be corrected needs to be corrected comprises: It is determined in sequence whether the difference between the depth value of the target discrete point and the depth value of the corresponding point in each corresponding depth image is less than a preset threshold.
8. A fusion image correction device, characterized in that: include: An image fusion module, used for fusing a plurality of depth images obtained by multi-angle projection of a target object to obtain a fused image, wherein the target object includes a setting surface and a plurality of target objects set on the setting surface; An image processing module, configured to extract, based on at least one of the depth image and the fused image, a reference surface corresponding to the setting surfaces of the multiple target objects included in the target object in the fused image; The image correction module is used to correct the depth value of the fused image based on the depth value of the reference plane, including: for the point to be corrected in the fused image, determining whether there is a corresponding point at the same position in multiple depth images corresponding to the point to be corrected that is close to the reference plane; if so, correcting the depth value of the point to be corrected.
9. A microscopic fringe projection imaging system, characterized in that: It comprises the fused image correction device as described in claim 8.
10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
