Multi-view multi-angle image processing method and system, electronic device and storage medium

Abstract

The application provides a multi-view multi-angle image processing method, system, electronic device and storage medium, comprising projecting and transforming the side-view collection graph by using the projection transformation matrix of the side-view camera relative to the front-view camera which is acquired in advance; searching the front-view collection graph to obtain a front-view matching contour matched with the front-view template contour on the front-view template graph of the measured object acquired in advance, and extracting the center of the front-view matching contour to obtain a first center point; searching the side-view collection graph to obtain a side-view matching contour matched with the side-view template contour on the side-view template graph of the measured object acquired in advance, and extracting the center of the side-view matching contour to obtain a second center point; calculating the offset of the second center point relative to the first center point, which is recorded as a real-time positioning offset; and offsetting the side-view collection graph by the real-time positioning offset to align and fuse the side-view collection graph and the front-view collection graph to obtain a fusion image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optical detection, and in particular to a multi-view multi-angle image processing method and system, an electronic device and a storage medium. BACKGROUND

[0002] With the development of semiconductor technology represented by integrated circuits, miniaturization and multi-functionality have become the common goal pursued by everyone, accelerating the development of IC (Integrated circuit) chip design, and the 3D detection capability of IC chips is more demanding.

[0003] In the prior art, the height of an object is detected by taking multiple-angle images of the object and fusing multiple images of different angles. However, the image fusion speed in the prior art is slow and the fusion accuracy is low. SUMMARY

[0004] The present application provides a multi-view multi-angle image processing method, system, electronic device and storage medium to solve the technical problems of slow image fusion speed and low fusion accuracy in the prior art.

[0005] In a first aspect, the present application provides a multi-view multi-angle image processing method for fusing multi-directional images collected by an image collection device, wherein the image collection device comprises a forward camera and a lateral camera, the forward camera is used to take a vertical image of an object to obtain a forward collection image, the lateral camera is used to take an inclined image of the object to obtain a lateral collection image, and the mirror plane of the lateral camera, the image plane of the lateral camera and the extension plane of the object plane of the object intersect at a line, and the multi-view multi-angle image processing method comprises:

[0006] Retrieving the required fused forward collection image and lateral collection image of the object;

[0007] Projecting and transforming the lateral collection image using a pre-acquired projection transformation matrix of the lateral camera relative to the forward camera to obtain a projected lateral image;

[0008] Searching the forward collection image to obtain a forward matching contour matching a reference feature on a pre-acquired forward template image of the object, and extracting the center of the forward matching contour to obtain a first center point;

[0009] Searching the lateral collection image to obtain a lateral matching contour matching a reference feature on a pre-acquired lateral template image of the object, and extracting the center of the lateral matching contour to obtain a second center point, wherein the reference feature on the forward template image and the reference feature on the lateral template image are the same reference feature on the same reference height plane of the object.

[0010] calculating an offset of the second center point relative to the first center point, denoted as a real-time positioning offset;

[0011] offsetting the side-view acquisition image by the real-time positioning offset to align the side-view acquisition image with the front-view acquisition image to obtain a fused image.

[0012] As an embodiment of the present application, the front-view matching contour and the side-view matching contour comprise:

[0013] photographing the measured object using an image acquisition device to obtain a front-view template image and a side-view template image;

[0014] acquiring a contour of the reference feature on the front-view template image, denoted as a front-view template contour;

[0015] acquiring a contour of the reference feature on the side-view template image, denoted as a side-view template contour.

[0016] As an embodiment of the present application, before the acquiring of the contour of the reference feature on the front-view template image, denoted as a front-view template contour, the method further comprises:

[0017] framing the reference feature on the front-view template image to obtain a first acquisition region;

[0018] The acquiring of the contour of the reference feature on the front-view template image, denoted as a front-view template contour, specifically comprises:

[0019] acquiring a contour of the reference feature in the first acquisition region, denoted as a front-view template contour;

[0020] Before the acquiring of the contour of the reference feature on the side-view template image, denoted as a side-view template contour, the method further comprises:

[0021] framing the reference feature on the side-view template image to obtain a second acquisition region;

[0022] The acquiring of the contour of the reference feature on the side-view template image, denoted as a side-view template contour, specifically comprises:

[0023] acquiring a contour of the reference feature in the second acquisition region, denoted as a side-view template contour.

[0024] As an embodiment of the present application, after the acquiring of the contour of the reference feature on the front-view template image, denoted as a front-view template contour, the method further comprises:

[0025] performing line thinning processing on the front-view template contour to obtain a thinned front-view template contour;

[0026] obtaining a profile of the reference feature on the side-view template image, denoted as a side-view template profile;

[0027] performing line thinning on the side-view template profile to obtain a thinned side-view template profile.

[0028] As an embodiment of the present application, before searching the front-view acquisition image to obtain a front-view matching profile matching the front-view template profile of the pre-acquired front-view template image of the measured object, and extracting the center of the front-view matching profile to obtain a first center point, the method further comprises:

[0029] offsetting the projected side-view acquisition image using a training offset of the pre-acquired side-view template image of the measured object relative to the front-view template image.

[0030] As an embodiment of the present application, the method for obtaining the training offset comprises:

[0031] extracting a center point of the front-view template profile, denoted as a third center point;

[0032] extracting a center point of the side-view template profile, denoted as a fourth center point;

[0033] calculating an offset of the fourth center point relative to the third center point, denoted as a training offset.

[0034] As an embodiment of the present application, before searching the front-view acquisition image to obtain a front-view matching profile matching the front-view template profile of the reference feature on the pre-acquired front-view template image of the measured object, the method further comprises:

[0035] performing framing on the reference feature of the front-view acquisition image to obtain a first search region;

[0036] The searching the front-view acquisition image to obtain a front-view matching profile matching the front-view template profile of the reference feature on the pre-acquired front-view template image of the measured object specifically comprises:

[0037] searching the first search region to obtain a front-view matching profile matching the front-view template profile of the reference feature on the pre-acquired front-view template image of the measured object;

[0038] As an embodiment of the present application, before searching the side-view acquisition image to obtain a side-view matching profile matching the side-view template profile of the reference feature on the pre-acquired side-view template image of the measured object, the method further comprises:

[0039] performing framing on the reference feature of the side-view acquisition image to obtain a second search region;

[0040] The searching of the side-view acquisition image is performed to obtain a side-view matching contour matching a side-view template contour of a reference feature on a pre-acquired side-view template image of the measured object.

[0041] The searching of the second search region is performed to obtain a side-view matching contour matching a side-view template contour of a reference feature on a pre-acquired side-view template image of the measured object.

[0042] As an embodiment of the present application, before the real-time positional offset of the side-view acquisition image, the method comprises:

[0043] It is judged whether the real-time positional offset is less than a pre-set offset threshold value, if yes, the step of the real-time positional offset of the side-view acquisition image is performed, if no, the side-view acquisition image is offset using a previous real-time positional offset of the measured object.

[0044] As an embodiment of the present application, the method further comprises:

[0045] A distance between a target feature on an orthographic acquisition image layer of the fused image and a target feature on a side-view acquisition image layer of the fused image is calculated along a parallax direction, and is recorded as a parallax distance; wherein the parallax direction is parallel to a normal projection of an optical axis of a lateral camera corresponding to the side-view acquisition image.

[0046] A pre-constructed height detection formula is calculated using the parallax and an inclination angle of the lateral camera corresponding to the side-view acquisition image, to obtain a height value of the target feature.

[0047] As an embodiment of the present application, the pre-constructed height detection formula is:

[0048] H=d*tan(θ);

[0049] wherein d is the parallax distance, H is the height value of the target feature, and θ is the inclination angle of the lateral camera corresponding to the side-view acquisition image.

[0050] As an embodiment of the present application, the calculation of the distance between the target feature on the orthographic acquisition image layer of the fused image and the target feature on the side-view acquisition image layer of the fused image, recorded as the parallax distance, comprises:

[0051] The target feature on the orthographic acquisition image layer of the fused image and the target feature on the side-view acquisition image layer are framed along a parallax direction, to obtain a detection region;

[0052] The target features in the detection region are acquired, to obtain first measurement points and second measurement points;

[0053] A distance between the first measurement points and the second measurement points is calculated, recorded as the parallax distance.

[0054] As one embodiment of the present application, the distance between the target feature on the front-view acquisition layer of the fusion image and the target feature on the side-view acquisition layer of the fusion image along the parallax direction is calculated, and is recorded as a parallax distance, which includes:

[0055] The target feature on the front-view acquisition layer of the fusion image and the target feature on the side-view acquisition layer are respectively framed along the parallax direction, and a first detection area and a second detection area are obtained respectively;

[0056] The target feature in the first detection area and the target feature in the second detection area are collected respectively, and a first measurement point and a second measurement point are obtained;

[0057] The distance between the first measurement point and the second measurement point is calculated, and is recorded as a parallax distance.

[0058] In a second aspect, the present application provides a multi-view multi-angle image processing system, which includes: an image retrieval module, configured to retrieve a required fusion front-view acquisition image and a side-view acquisition image of a measured object;

[0059] A projection transformation module is configured to perform projection transformation on the side-view acquisition image using a projection transformation matrix of a side camera relative to a front camera, and obtain a projected side view;

[0060] A first search module is configured to search the front-view acquisition image to obtain a front-view matching contour matched with a front-view template contour of a reference feature on a pre-acquired front-view template image of the measured object, and extract a center of the front-view matching contour to obtain a first center point;

[0061] A second search module is configured to search the side-view acquisition image to obtain a side-view matching contour matched with a side-view template contour of a reference feature on a pre-acquired side-view template image of the measured object, and extract a center of the side-view matching contour to obtain a second center point, wherein the reference feature on the front-view template image and the reference feature on the side-view template image are the same reference feature on the same reference height plane of the measured object;

[0062] A first calculation module is configured to calculate an offset amount of the second center point relative to the first center point, and record the offset amount as a real-time positioning offset amount;

[0063] A first offset module is configured to offset the side-view acquisition image by the real-time positioning offset amount, so that the side-view acquisition image is aligned with the front-view acquisition image to obtain a fusion image.

[0064] In a third aspect, the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method of the first aspect is implemented.

[0065] In a fourth aspect, the present application provides a storage medium having stored thereon a computer program which, when executed by a processor, implements the method of the first aspect.

[0066] The implementation of the embodiments of the present application has the following beneficial effects:

[0067] In the present application, when a series of measured objects are transported one by one under the image acquisition device or the image acquisition device is moved to pass each measured object in turn to acquire images of each measured object, due to reasons such as parallelism of the mechanism, repeated error of the traveling mechanism, and warping of the measured object, the shift of the side-view acquisition image of each measured object relative to the front-view acquisition image thereof is not the same. To realize image fusion of a series of measured objects, after the projection transformation matrix is obtained, the series of measured objects that need to be fused are selected, and after the image acquisition device is installed on the platform, a template image of a measured object is first selected for acquisition, the front-view template image and the side-view template image of the measured object are acquired, the reference height plane of the measured object is determined, then the same feature on the front-view template image and the side-view template image is selected as the reference feature, and the reference feature is the reference feature on the reference height plane of the measured object, so that the front-view template contour of the reference feature on the front-view template image is obtained, and the side-view template contour of the reference feature on the side-view template image is obtained, i.e., the front-view template contour and the side-view template contour are the contours of the reference feature on the reference height plane at different viewing angles. Thus, when the images of any measured object of the series of measured objects are fused, the front-view acquisition image and the side-view acquisition image of the measured object that need to be fused are first called, then the projection transformation matrix of the side-view camera relative to the front-view camera is used to perform projection transformation on the side-view acquisition image to correct the tangential distortion and the radial distortion existing in the side-view acquisition image, then the front-view acquisition image and the side-view acquisition image are searched respectively to obtain the front-view matching contour matched with the front-view template contour and the side-view matching contour matched with the side-view template contour, and the center point of the front-view matching contour and the center point of the side-view matching contour are extracted respectively to obtain the first center point and the second center point, the real-time positioning offset of the second center point relative to the first center point is calculated, and the side-view acquisition image is offset using the real-time positioning offset, so that the shift of the side-view acquisition image relative to the front-view acquisition image can be solved, the reference feature on the side-view acquisition image is aligned with the reference feature on the front-view acquisition image, i.e., the reference height plane of the measured object on the side-view acquisition image is aligned with the reference height plane of the measured object on the front-view acquisition image, so that the side-view acquisition image and the front-view acquisition image can be aligned and fused, the image fusion accuracy is improved, and the multi-angle images of any measured object of the same series of measured objects can be quickly and accurately fused using the method, thereby solving the technical problems of slow image fusion speed and low fusion accuracy in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the description are only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work based on the accompanying drawings are within the protection scope of the present application.

[0069] Figure 1 Structure diagram of the image acquisition device shown in an embodiment of the present application;

[0070] Figure 2 Imaging principle diagram of the side camera of the image acquisition device shown in an embodiment of the present application;

[0071] Figure 3 Diagram showing the relationship among the mirror plane, the phase plane of the side camera of the image acquisition device shown in an embodiment of the present application, and the object plane of the measured object;

[0072] Figure 4 Flow diagram of the multi-view multi-angle image processing method shown in an embodiment of the present application;

[0073] Figure 5 Structure block diagram of the multi-view multi-angle image processing system shown in an embodiment of the present application;

[0074] Figure 6 Structure block diagram of the electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the technical solutions of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the description are only some embodiments of the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work based on the accompanying drawings are within the protection scope of the present application.

[0076] The present application provides a multi-view multi-angle image processing method, which is used for fusing multi-directional images collected by an image acquisition device 100.

[0077] Referring to Figure 1 and Figure 2The image acquisition device 100 comprises a mounting bracket 10, and a front camera 20 and a side camera 30 mounted on the mounting bracket 10, the front camera 20 is used for vertical shooting of the measured object 200 to obtain a front view acquisition image, the side camera 30 is used for oblique shooting of the measured object 200 to obtain a side view acquisition image, and the mirror plane of the side camera 30, the image plane of the side camera 30 and the extension plane of the object plane of the measured object 200 intersect at a line.

[0078] In some specific embodiments, the front camera 20 and the side camera 30 both use telecentric lenses for shooting, the telecentric lenses have good parallelism, which means that all light rays are imaged along the optical axis, and the shooting accuracy is high.

[0079] Referring to Figure 3 , a schematic diagram of the relationship among the mirror plane of the side camera 30, the image plane of the side camera 30 and the object plane of the measured object 200 is shown in the figure, wherein the object plane of the measured object 200 refers to the plane on the measured object 200 where the shooting light beam first arrives; the extension planes of the object plane of the measured object 200, the image plane of the side camera 30 and the mirror plane of the side camera 30 intersect at a line, the angle between the object plane of the measured object 200 and the mirror plane of the side camera 30 is a, the angle between the image plane of the side camera 30 and the mirror plane of the side camera 30 is b, the distance between the object plane of the measured object 200 and the mirror plane of the side camera 30 is u, the distance between the image plane of the side camera 30 and the mirror plane of the side camera 30 is v, and the relationship among the object plane of the measured object 200, the image plane of the measured object 200 and the mirror plane of the side camera 30 is b = arctan(tan(a)*v / u).

[0080] It should be noted that the side camera 30 can comprise a plurality of side cameras, and the side cameras 30 can be arranged according to the tilting direction and the tilting angle.

[0081] In some specific embodiments, referring to Figure 1 , the side camera 30 comprises an X-axis side camera 31 and a Y-axis side camera 32, specifically, when the image acquisition device 100 is placed directly above the measured object 200, a coordinate system is established with the measured object 200 as the origin, then the orientation coordinates of the measured object 200 are (0, 0, 0); the orientation coordinates of the front camera 20 are (0, 0, Z1), Z1 is greater than 0, that is, the front camera 20 is located directly above the measured object 200; the orientation coordinates of the X-axis side camera 31 are (X2, 0, Z2), X2 and Z2 are both greater than 0, that is, the X-axis side camera 31 is located above the measured object 200 along the X-axis; the orientation coordinates of the Y-axis side camera 32 are (0, Y3, Z3), Y3 and Z3 are both greater than 0, that is, the Y-axis side camera 32 is located above the measured object 200 along the Y-axis.

[0082] Referring to Figure 4 The multi-view multi-angle image processing method comprises the following steps:

[0083] S1, calling the required fused front-view acquisition image and side-view acquisition image of the measured object 200; that is, the front-view acquisition image and side-view acquisition image to be fused are obtained by photographing the measured object 200 by the image acquisition device 100.

[0084] S2, using the projection transformation matrix of the side camera 30 relative to the front camera 20 obtained in advance to perform projection transformation on the side-view acquisition image to obtain the projected side-view acquisition image; before the image acquisition device 100 is used to acquire the image of the measured object 200, the image photographed by the side camera 30 has serious tangential distortion and slight radial distortion due to the oblique photographing of the side camera 30, so it is necessary to correct the image photographed by the side camera 30 to the common view field front camera plane; specifically, a calibration plane is selected, and the front camera 20 and the side camera 30 of the image acquisition device 100 photograph the calibration plane respectively to obtain a calibration front view and a calibration side view, and then the projection transformation matrix of the calibration side view relative to the calibration front view is obtained, that is, the projection transformation matrix of the side camera 30 relative to the front camera 20 is obtained, and after the image photographed by the side camera 30 is projected and transformed using the projection transformation matrix, the distortion correction of the image photographed by the side camera 30 is realized.

[0085] S3, searching the front-view acquisition image to obtain a front-view matching contour matched with a reference feature on a front-view template contour of a pre-acquired front-view template image of the measured object 200, and extracting the center of the front-view matching contour to obtain a first center point;

[0086] S4, searching the side-view acquisition image to obtain a side-view matching contour matched with a reference feature on a side-view template contour of a pre-acquired side-view template image of the measured object 200, and extracting the center of the side-view matching contour to obtain a second center point; wherein the reference feature on the front-view template image and the reference feature on the side-view template image are the same reference feature on the same reference height plane of the measured object;

[0087] S5, calculating the offset of the second center point relative to the first center point, denoted as a real-time positioning offset;

[0088] S6, offsetting the side-view acquisition image by the real-time positioning offset to align the side-view acquisition image with the front-view acquisition image to obtain a fused image.

[0089] In the present application, when the series of measured objects 200 are transported one by one to the image acquisition device 100 below, or when the image acquisition device 100 is moved to pass each measured object 200 in turn, in order to acquire images of each measured object 200, due to the parallelism of the mechanism, the repeated error of the travel mechanism, the warping of the measured object 200, etc., the lateral view acquisition image of each measured object 200 is offset from its front view acquisition image. In order to realize image fusion of a series of measured objects 200, after obtaining the projection transformation matrix, the series of measured objects 200 that need to be fused are selected, after the image acquisition device 100 is installed on the platform, a measured object 200 is first selected to acquire a template image, the front view template image and the lateral view template image of the measured object 200 are acquired, the reference height plane of the measured object is determined, then the same feature on the front view template image and the lateral view template image is selected as the reference feature, and the reference feature is the reference feature on the reference height plane of the measured object 200, so as to obtain the front view template contour of the reference feature on the front view template image, and the lateral view template contour of the reference feature on the lateral view template image, that is, the front view template contour and the lateral view template contour are the contours of the reference feature on the reference height plane under different viewing angles; when the images of any one of the series of measured objects 200 are fused, the required front view acquisition image and the lateral view acquisition image of the measured object 200 are first called, then the projection transformation matrix of the lateral camera 30 relative to the front camera 20 is used to project and transform the lateral view acquisition image, so as to correct the tangential distortion and the radial distortion existing in the lateral view acquisition image; then the front view acquisition image and the lateral view acquisition image are searched respectively, the front view matching contour matched with the front view template contour and the lateral view matching contour matched with the lateral view template contour are obtained respectively, the center point of the front view matching contour and the center point of the lateral view matching contour are extracted respectively, the first center point and the second center point are obtained, the real-time positioning offset of the second center point relative to the first center point is calculated, and the real-time positioning offset is used to offset the lateral view acquisition image, so as to solve the offset of the lateral view acquisition image relative to the front view acquisition image, that is, to align the reference feature on the lateral view acquisition image with the reference feature on the front view acquisition image, so that the reference height plane of the measured object on the lateral view acquisition image is aligned with the reference height plane of the measured object on the front view acquisition image, so that the lateral view acquisition image and the front view acquisition image can be aligned and fused, the image fusion accuracy is improved, and the multi-angle images of any one of the series of measured objects 200 can be quickly and accurately fused using the method, which solves the technical problems of slow image fusion speed and low fusion accuracy in the prior art.

[0090] It should be noted that after the fused image is obtained, the fused image includes a front view acquisition image layer and a lateral view acquisition image layer.

[0091] It should be noted that when image fusion is needed for other series of measured objects 200, only one measured object 200 needs to be selected from the other series of measured objects 200, the front view template image and the side view template image of the measured object 200 are collected, the reference height plane of the measured object 200 is selected, the reference feature on the reference height plane is selected, the contour of the reference feature on the front view template image (front view template contour) and the contour of the reference feature on the side view template image (side view template contour) are obtained, and then the image fusion of the series of measured objects 200 can be performed by using the running machine shooting, and the image fusion of any one of the series of measured objects 200 is performed, thereby improving the application range of the fusion method.

[0092] It should be noted that in the present application, the lateral camera 30 of the image acquisition device 100 is arranged such that the mirror plane of the lateral camera 30, the image plane of the lateral camera 30 and the extension plane of the measured object 200 intersect at a line, so that the lateral camera 30 can still clearly image the measured object 200 when shooting the measured object 200 at an angle, thereby improving the shooting clarity of the lateral camera 30, i.e. improving the clarity of the side view collection image, and further improving the clarity of the image fused from the front view collection image and the side view collection image, thereby facilitating the realization of 3D detection.

[0093] It should be noted that when the lateral camera 30 is multiple, the projection matrix of each lateral camera 30 relative to the front camera 20, the front view template contour on the front view template image of the front camera 20 and the side view template contour on the side view template image of each lateral camera 30 need to be obtained. For example, when the lateral camera 30 includes an X-axis lateral camera 31 and a Y-axis lateral camera 32, the projection transformation matrix of the X-axis lateral camera 31 relative to the front camera 20, the projection transformation matrix of the Y-axis lateral camera 32 relative to the front camera 20, the side view template contour of the reference feature on the side view template image of the X-axis lateral camera 31 and the side view template contour of the reference feature on the side view template image of the Y-axis lateral camera 32 need to be obtained.

[0094] The process of obtaining the projection transformation matrix will be described in detail below.

[0095] A feature point (x1, y1, 1) on the front view template image and a feature point (x2, y2, 1) on the side view template image corresponding to the feature point (x1, y1, 1) are determined, and a projection transformation relationship (perspective transformation) between the two is established:

[0096]

[0097] wherein H is a 3*3 matrix,

[0098] To obtain the parameters in H, two point pairs (x1, y1, 1) and (x2, y2, 1) are needed to calculate H using the correctly matched points; specifically, direct fitting is performed by least squares, and the error calculation formula for the (x1, y1, 1) point pair is as follows:

[0099]

[0100] The formula for minimizing the sum of squares of all point pair errors is as follows:

[0101]

[0102] Based on the over-constrained equation, the error is calculated, and the parameters are continuously updated in the reverse direction until the error difference meets the requirements, thereby determining the values of h11, h12, h13, h21, h22, h23, h31, h32, and h33, and thus obtaining the projection transformation matrix.

[0103] In an embodiment, before the required fused front view and side view acquisition images of the measured object 200 are retrieved, the method further comprises:

[0104] The image acquisition device 100 is used to acquire images of the measured object 200 to obtain front view and side view acquisition images, and the front view and side view acquisition images are saved to the image library.

[0105] Thus, the required fused front view and side view acquisition images of the measured object 200 are retrieved from the image library.

[0106] In some specific embodiments, the use of the image acquisition device 100 to acquire images of the measured object 200 to obtain front view and side view acquisition images, and saving the front view and side view acquisition images to the image library comprises:

[0107] The target measured object is illuminated with bright field lighting, and the image acquisition device 100 is used to take pictures of the target measured object 200 to obtain bright field front view and side view acquisition images, and the bright field front view and side view acquisition images are saved to the image library.

[0108] In this embodiment, bright field lighting is performed when the target measured object 200 is photographed, which can make the main outline of the target measured object 200 (such as the substrate of a chip) brighter and clearer, i.e., the overall outline of the target measured object 200 can be highlighted on the bright field front view and side view acquisition images.

[0109] In some specific embodiments, the use of the image acquisition device 100 to acquire images of the measured object 200 to obtain front view and side view acquisition images, and saving the front view and side view acquisition images to the image library further comprises:

[0110] The measured object 200 is dark field lighted, the image acquisition device 100 is used to take a photo of the measured object 200, the dark field front view collection image and the dark field side view collection image are obtained, and the dark field front view collection image and the dark field side view collection image are saved to the image library.

[0111] In the embodiment, the dark field light is used when the measured object 200 is taken, so that the local features (such as the gold wire on the substrate) of the measured object 200 are brighter and clearer, that is, the local features of the measured object 200 can be highlighted on the dark field front view collection image and the dark field side view collection image.

[0112] In some specific embodiments, before the front view collection image and the side view collection image of the measured object 200 are called, the method further comprises:

[0113] The image selection instruction is received, and the front view collection image and the side view collection image selected by the image selection instruction are taken as the front view collection image and the side view collection image required to be fused.

[0114] That is, the operator can select the images in the image library according to the imaging quality, fusion requirements, detection requirements and the like, so that the front view collection image and the side view collection image meeting the requirements of the operator can be called when the front view collection image and the side view collection image of the measured object 200 are called. Specifically, when the front view collection image is selected, the bright field front view collection image or the dark field front view collection image can be selected; when the side view collection image is selected, the bright field side view collection image taken by the X-axis lateral camera 31, the side view collection image taken by the X-axis lateral camera 31, the bright field side view collection image taken by the Y-axis lateral camera 32 or the dark field side view collection image taken by the Y-axis lateral camera 32 can be selected.

[0115] In an embodiment, before the front view collection image and the side view collection image are saved to the image library, the method further comprises:

[0116] The shooting completion instruction is sent, and the sending time interval between adjacent two shooting completion instructions is a predetermined time.

[0117] That is, after the image acquisition device 100 completes the shooting of the measured object 200, the front view collection image and the side view collection image obtained after shooting are saved to the image library after the shooting is completed. When the image acquisition device 100 is used to shoot the measured object 200, the connection wire is prone to shaking due to electromagnetic interference, which causes the image acquisition device 100 to repeatedly take pictures. Therefore, a predetermined time is set between the sending times of the shooting completion instructions of two adjacent times. Within the predetermined time, even if the image acquisition device 100 has completed image acquisition, the image will not be saved to the image library of the next measured object 200, because within the predetermined time, the displacement of the image acquisition device 100 relative to the next measured object 200 cannot be completed. Therefore, the image acquisition completed again within the predetermined time is most likely a repeated shooting of the previous measured object 200, which needs to be removed.

[0118] In some specific embodiments, the front view matching contour and the side view matching contour acquisition method comprises:

[0119] The image acquisition device 100 is used to shoot the measured object 200 to obtain a front view template image and a side view template image;

[0120] The contour of the reference feature on the front view template image is collected, which is recorded as a front view template contour;

[0121] The contour of the reference feature on the side view template image is collected, which is recorded as a side view template contour.

[0122] After a measured object 200 is selected from a series of measured objects, the front camera of the image acquisition device 100 is used to shoot the measured object 200 to obtain a front view template image, and the side camera of the image acquisition device 100 is used to shoot the measured object 200 to obtain a side view template image. It should be noted that the reference feature on the front view template image and the reference feature on the side view template image are the same reference feature on the same reference height plane of the measured object 200. That is, a feature on the reference height plane of the measured object 200 is selected as the reference feature, and then the contour of the reference feature on the front view template image is collected to obtain a front view template contour. The contour of the reference feature on the side view template image is collected to obtain a side view template contour.

[0123] In some specific embodiments, before the contour of the reference feature on the front view template image is collected and recorded as a front view template contour, the acquisition method further comprises:

[0124] The reference feature on the front view template image is framed to obtain a first collection area.

[0125] Correspondingly, before the contour of the reference feature on the front view template image is collected and recorded as a front view template contour, it is specifically:

[0126] Collecting the contour of the reference feature in the first collection area, and recording the contour as a front view template contour.

[0127] That is, before collecting the contour of the reference feature on the front view template map, the reference feature on the front view template map is framed first to obtain the first collection area, that is, the collection area of interest is framed; correspondingly, the contour of the reference feature in the first collection area is collected, and the collection speed of the front view template contour is improved by reducing the collection area.

[0128] In some specific embodiments, before the contour of the reference feature on the side view template map is collected and recorded as a side view template contour, the acquisition method further comprises:

[0129] Framing the reference feature on the side view template map to obtain a second collection area.

[0130] Correspondingly, before the contour of the reference feature on the side view template map is collected and recorded as a side view template contour, specifically:

[0131] Collecting the contour of the reference feature in the second collection area and recording the contour as a side view template contour.

[0132] That is, when the contour of the reference feature on the side view template map is collected, the reference feature on the side view template map is framed first to obtain the second collection area, that is, the collection area of interest is framed; correspondingly, the contour of the reference feature in the second collection area is collected, and the collection speed of the side view template contour is improved by reducing the collection area.

[0133] In some specific embodiments, after the contour of the reference feature on the front view template map is collected and recorded as a front view template contour, the method further comprises:

[0134] Performing line thinning processing on the front view template contour to obtain a thinned front view template contour.

[0135] In this embodiment, by performing line thinning processing on the front view template contour, the collection accuracy of the contour of the reference feature on the front view template map is improved, and the search amount for searching for a front view matching contour on the front view collection map is reduced.

[0136] In some specific embodiments, after the contour of the reference feature on the side view template map is collected and recorded as a side view template contour, the method further comprises:

[0137] Performing line thinning processing on the side view template contour to obtain a thinned side view template contour.

[0138] In this embodiment, by performing line thinning processing on the side view template contour, the collection accuracy of the contour of the reference feature on the side view template map is improved, and the search amount for searching for a side view matching contour on the side view collection map is reduced.

[0139] In some specific embodiments, the collecting of the contour of the reference feature on the front view template image can employ a Sobel operation to collect the contour of the reference feature on the front view template image.

[0140] Specifically, the X-direction gradient dx (i.e., Gx) and the Y-direction gradient dy (i.e., Gy) of each contour point are calculated, and the front view template contour is obtained by Gx, Gy and G; specifically:

[0141] The Y-direction Sobel gradient convolution kernel Gy of a contour point is:

[0142]

[0143] The X-direction Sobel gradient convolution kernel Gx of a contour point is:

[0144]

[0145] The gradient amplitude G of a contour point is:

[0146] The gradient direction of a contour point

[0147] In some specific embodiments, the collecting of the contour of the reference feature on the side view template image can employ a Sobel operation to collect the contour of the reference feature on the side view template image, and the Sobel operation of the front view template image can be referred to for details, which will not be described herein.

[0148] In some specific embodiments, the line thinning processing of the front view template contour can employ a Canny operation to thin the front view template contour.

[0149] Specifically, the 8-neighborhood non-maximum suppression of the gradient amplitude of the front view template image is performed, the gradient amplitude of a point is compared with the amplitudes in two directions along the gradient direction, if the amplitude of the point is greater than the amplitudes in the two directions, the point is retained; otherwise, the point is set to 0. In this way, the local maximum point, i.e., the maximum value, can be obtained, and the pixel point is considered as an edge point, and the thinned edge can be obtained.

[0150] Then the double threshold algorithm is used to detect and connect the edges; the hysteresis threshold needs two thresholds (high threshold and low threshold), if the threshold is set too high, the real edge points will be filtered out, if the threshold is set too low, the noise will also be considered as edge points; specifically, an edge image containing few false edges is obtained by screening through the high threshold, but due to the high high threshold, the generated edge may not be closed, so a low threshold is needed to connect the edges, when the edge points meeting the high threshold are connected into a contour, the algorithm will find a point meeting the low threshold in the 8-neighborhood points of the breakpoint of the contour, then the edge contour is connected according to the point, until the whole image edge is closed, and the refined orthographic template contour is obtained. By selecting appropriate high threshold and low threshold, the orthographic template contour closest to the real edge of the reference feature can be obtained.

[0151] In some specific embodiments, the line thinning processing of the side view template contour can also be performed by using the canny operation, and the thinning processing steps of the orthographic template contour using the canny operation can be referred to, and details are not described herein.

[0152] In an embodiment, before searching the orthographic acquisition image to obtain an orthographic matching contour matching the orthographic template contour of the pre-acquired orthographic template image of the measured object 200, and extracting the center of the orthographic matching contour to obtain the first center point, the method further comprises:

[0153] The projected side view acquisition image is offset using the training offset of the pre-acquired side view template image of the measured object 200 relative to the orthographic template image.

[0154] In the process of collecting the front view template image and the side view template image of the measured object 200, in order to make the front view template image and the side view template image clearly imaged, the focal length of the front camera 20 and the side camera 30 needs to be adjusted. Therefore, the focal length of the front camera 20 and the side camera 30 in the process of collecting the template image is changed compared with the focal length in the process of calculating the projection transformation matrix, which results in the offset of the side view template image relative to the front view template image in the two-dimensional plane. Therefore, before starting to take and fuse the series of measured objects 200, the training offset of the side view template image relative to the front view template image is calculated. Therefore, when fusing the image of any measured object 200 of the series of measured objects 200, the front view collected image and the side view collected image required to be fused of the measured object 200 are first called, then the projection transformation matrix of the side camera 30 relative to the front camera 20 is used to project and transform the side view collected image, so as to correct the tangential distortion and the radial distortion of the side view collected image. Then the training offset is used to offset the side view collected image, so as to solve the offset in the process of collecting the template image and calculating the projection transformation matrix, thereby reducing the offset of the side view collected image relative to the front view collected image, and further reducing the search amount of searching the front view matching contour on the front view collected image and the search amount of searching the side view matching contour on the side view collected image.

[0155] In some specific embodiments, the method for obtaining the training offset comprises:

[0156] extracting the center point of the front view template contour, denoted as a third center point;

[0157] extracting the center point of the side view template contour, denoted as a fourth center point;

[0158] calculating the offset of the fourth center point relative to the third center point, denoted as a training offset.

[0159] As known from the foregoing, the reference feature on the front view template image and the reference feature on the side view template image are the same feature on the measured object 200. Therefore, the center point of the front view template contour formed by the reference feature on the front view collected image is extracted, denoted as a third center point; the center point of the side view template contour formed by the reference feature on the side view collected image is extracted, denoted as a fourth center point, and the offset of the fourth center point relative to the third center point is the training offset of the side view template image relative to the front view template image.

[0160] In some specific embodiments, before the front view collected image is searched to obtain the front view matching contour matched with the front view template contour of the reference feature on the pre-obtained front view template image of the measured object, the method further comprises:

[0161] the reference feature of the front view collected image is framed to obtain a first search region.

[0162] Correspondingly, the searching the orthographic acquisition image to obtain the orthographic matching contour matching the orthographic template contour of the reference feature on the orthographic template image of the pre-acquired measured object is specifically:

[0163] searching the first search area to obtain the orthographic matching contour matching the orthographic template contour of the reference feature on the orthographic template image of the pre-acquired measured object.

[0164] That is, before searching the matching contour on the orthographic acquisition image, the reference feature on the orthographic acquisition image is framed to obtain the first search area; correspondingly, the orthographic matching contour is searched in the first search area, that is, the search area of interest is framed, and the search speed of the orthographic matching contour is improved by narrowing the search area.

[0165] In some specific embodiments, before the searching the lateral acquisition image to obtain the lateral matching contour matching the lateral template contour of the reference feature on the lateral template image of the pre-acquired measured object, the method further comprises:

[0166] framing the reference feature of the lateral acquisition image to obtain a second search area.

[0167] Correspondingly, the searching the lateral acquisition image to obtain the lateral matching contour matching the lateral template contour of the reference feature on the lateral template image of the pre-acquired measured object is specifically:

[0168] searching the second search area to obtain the lateral matching contour matching the lateral template contour of the reference feature on the lateral template image of the pre-acquired measured object.

[0169] That is, before searching the matching contour on the lateral acquisition image, the reference feature on the lateral acquisition image is framed to obtain the second search area; correspondingly, the lateral matching contour is searched in the second search area, that is, the search area of interest is framed, and the search speed of the lateral matching contour is improved by narrowing the search area.

[0170] In some specific embodiments, the orthographic acquisition image can be subjected to gradient level NCC template matching according to the orthographic template contour, so as to search out the orthographic matching contour matching the orthographic template contour.

[0171] Specifically, the gradient level NCC template matching formula is:

[0172]

[0173] T represents the orthographic template image, and S represents the orthographic acquisition image.

[0174] Xi, Yi, i = 0, 1, 2...n, represents the X direction gradient value of the i-th point of the orthographic template contour point set on the orthographic template image, i = 0, 1, 2...n;

[0175] Yi, i = 0, 1, 2...n, represents the Y direction gradient value of the i-th point of the orthographic template contour point set on the orthographic template image, i = 0, 1, 2...n;

[0176] Xi, Yi, i = 0, 1, 2...n, represents the gradient amplitude of the i-th point of the orthographic template contour point set on the orthographic template image, i = 0, 1, 2...n;

[0177] Xi, Yi, i = 0, 1, 2...n, respectively represent the X, Y direction gradient values of the point on the orthographic acquisition image after the coordinates Xi, Yi of the orthographic template contour point set are offset at u, v of the orthographic acquisition image;

[0178] Xi, Yi, i = 0, 1, 2...n, respectively represent the X, Y direction gradient values of the point on the orthographic template image after the coordinates Xi, Yi of the orthographic template contour point set are offset at u, v of the orthographic template image;

[0179] S u,v is a gradient level similarity measure value, S u,v The greater the value of S u,v reaches the score threshold, it means that the orthographic matching contour is searched.

[0180] In some specific embodiments, the orthographic acquisition image can be subjected to gradient level NCC template matching according to the side view template contour, so as to search for a side view matching contour matched with the side view template contour. The specific search operation can refer to the search operation of the orthographic matching contour described above, and will not be described here.

[0181] In some specific embodiments, before the step of offsetting the side view acquisition image by the real-time positioning offset, the method further comprises:

[0182] determining whether the real-time positioning offset is less than a pre-set offset threshold, if yes, performing the step of offsetting the side view acquisition image by the real-time positioning offset, and if no, offsetting the side view acquisition image by the real-time positioning offset of the previous measured object 200.

[0183] It is known from the foregoing that the height variation of the measured object 200 is reflected by the size of the real-time positioning offset; therefore, the allowable range of the real-time positioning offset is set according to the height difference range (degree of unevenness) of the measured object 200, and specifically, the offset threshold is set, and when the real-time positioning offset is less than the offset threshold, it is indicated that the calculated real-time positioning offset is only related to the parallelism of the current measured object 200, at this time, directly using the current real-time positioning offset to offset the side view collection graph can solve the reference error; when the real-time positioning offset is greater than the offset threshold, it is considered that the height difference reflected by the real-time positioning offset exceeds the inherent height difference range of the measured object 200, that is, it is considered that the calculated real-time positioning offset is wrong, therefore, the current calculated real-time positioning offset is discarded, and the real-time positioning offset of the previous measured object 200 is used to offset the current side view collection graph; because the image collection device 100 photographs the current measured object 200 after photographing the previous measured object 200, it can be considered that the front view collection graph of the previous measured object 200 and the front view collection graph of the current measured object 200 have continuity, and the side view collection graph of the previous measured object 200 and the side view collection graph of the current measured object 200 have continuity, therefore, the real-time positioning offset of the side view collection graph of the previous measured object 200 and the real-time positioning offset of the side view collection graph of the current measured object 200 are basically the same, even without difference, therefore, using the real-time positioning offset of the side view collection graph of the previous measured object 200 to offset the current side view collection graph makes the side view collection graph and the front view collection graph fused.

[0184] It should be noted that the size of the offset threshold can be adjusted according to the flatness of the measured object 200, for example, when the flatness of the measured object 200 is high, the offset threshold can be adjusted to be small.

[0185] In an embodiment, the method further comprises:

[0186] The distance between the target feature on the front view collection graph layer of the fusion image and the target feature on the side view collection graph layer of the fusion image is calculated along the parallax direction, denoted as parallax distance; wherein the parallax direction is parallel to the orthogonal projection of the optical axis of the lateral camera corresponding to the side view collection graph layer;

[0187] The height detection formula constructed in advance is calculated using the parallax and the tilt angle of the lateral camera 30 corresponding to the side view collection graph layer, to obtain the height of the target feature.

[0188] As known from the foregoing, after the lateral-view acquisition image is offset using the real-time positioning offset, the reference height plane of the measured object on the frontal-view acquisition image is aligned with the reference height plane of the measured object on the lateral-view acquisition image. With the reference height plane as a reference, the features on the frontal-view acquisition image and the lateral-view acquisition image that are not on the reference height plane are staggered. Therefore, the height of the features on the reference height plane in the fused image is 0, and the height of the features not on the reference height plane is not 0. That is, the features on the measured object 200 that are not on the reference height plane can be detected in height. The target feature and the reference feature that need to be measured in height are not on the same height plane. On the fused image, the target feature on the frontal-view acquisition image layer is staggered from the target feature on the lateral-view acquisition image layer by a certain parallax distance. The existence of this parallax distance is because the lateral camera 30 performs oblique shooting on the measured object 200. Therefore, the distance between the target feature on the frontal-view acquisition image layer and the target feature on the lateral-view acquisition image layer is calculated along the parallax direction to obtain the parallax distance, so as to ensure that the target features are the same feature. Then, the parallax distance and the oblique angle of the lateral camera 30 corresponding to the lateral-view acquisition image are used to calculate the height detection formula, so as to calculate the height of the target feature. Specifically, when the lateral-view acquisition image used for fusion is obtained by the X-axis lateral camera, as known from the foregoing, the normal projection of the optical axis of the X-axis lateral camera overlaps the X-axis. Therefore, the parallax direction is parallel to the X-axis direction. When the lateral-view acquisition image used for fusion is obtained by the Y-axis lateral camera, as known from the foregoing, the normal projection of the optical axis of the Y-axis lateral camera overlaps the Y-axis. Therefore, the parallax direction is parallel to the Y-axis direction.

[0189] Specifically, the pre-constructed height detection formula is:

[0190] H = d * tan(θ)

[0191] wherein d is the parallax distance; H is the height of the target feature, and θ is the oblique angle of the lateral camera corresponding to the lateral-view acquisition image layer.

[0192] In some specific embodiments, the distance between the target feature on the frontal-view acquisition image layer of the fused image and the target feature on the lateral-view acquisition image layer of the fused image is calculated along the parallax direction, and is recorded as the parallax distance, which includes:

[0193] The target feature on the frontal-view acquisition image layer of the fused image and the target feature on the lateral-view acquisition image layer are framed along the parallax direction to obtain a detection region;

[0194] The target feature in the framed region is collected to obtain a first measurement point and a second measurement point;

[0195] The distance between the first measurement point and the second measurement point is calculated, and is recorded as the parallax distance

[0196] In the embodiment, when the feature points on the fusion image are sparse, such as no other feature points are interposed between the two target feature points, a frame can be drawn along the parallax direction to frame the target feature on the front-view acquisition layer and the target feature on the side-view acquisition layer, so that a detection region containing the target feature on the front-view acquisition layer and the target feature on the side-view acquisition layer is obtained. The first measurement point and the second measurement point are obtained by one-time acquisition in the detection region, and the parallax distance is calculated by the distance between the first measurement point and the second measurement point. The target feature on the front-view acquisition layer and the target feature on the side-view acquisition layer are framed at one time, so that the step of obtaining the parallax distance is reduced.

[0197] In some specific embodiments, the distance between the target feature on the front-view acquisition layer of the fusion image and the target feature on the side-view acquisition layer of the fusion image along the parallax direction is calculated, and is recorded as a parallax distance.

[0198] The target feature on the front-view acquisition layer of the fusion image and the target feature on the side-view acquisition layer of the fusion image are respectively framed along the parallax direction, and a first detection region and a second detection region are obtained.

[0199] The target feature in the first detection region and the target feature in the second detection region are respectively acquired, and a first measurement point and a second measurement point are obtained.

[0200] The distance between the first measurement point and the second measurement point is calculated, and is recorded as a parallax distance.

[0201] In the embodiment, when the feature points on the fusion image are dense, such as other feature points are interposed between the two target feature points, the target feature on the front-view acquisition layer and the target feature on the side-view acquisition layer are respectively framed along the parallax direction, so that other feature points are avoided to be framed, and the target feature is not interfered by other feature points when acquired. The first acquisition point in the first detection region and the second acquisition point in the second detection region are accurately obtained, and the parallax distance is calculated by the distance between the first measurement point and the second measurement point, so that the accuracy of the parallax distance is improved.

[0202] The target feature is a gold wire, which is specifically described as follows.

[0203] In some specific embodiments, when the target feature is a gold wire, one end of the gold wire is welded on a substrate, and the other end of the gold wire is welded on a chip. Therefore, when the reference feature is the substrate, the welding points of the gold wire welded on the substrate coincide, and the welding points of the gold wire welded on the chip do not coincide. When the reference feature is the chip, the welding points of the gold wire welded on the substrate do not coincide, and the welding points of the gold wire welded on the chip coincide.

[0204] Due to the elongation of the gold wire, when detecting the height of a target feature point of the target gold wire, first, it is judged whether the angle between the target gold wire and the X axis is greater than the angle between the target gold wire and the Y axis; if the angle between the target gold wire and the X axis is greater than the angle between the target gold wire and the Y axis, then the side-view collection image captured by the X-axis side camera is selected to be fused with the front-view collection image, and a frame is drawn along the X axis on the fused image to frame the target feature point of the target gold wire on the front-view collection image layer and the target feature point of the target gold wire on the side-view collection image layer, so that the first measurement point and the second measurement point of the target feature point are collected, the parallax distance is obtained, and the height of the target feature point of the target gold wire is calculated; or two frames are drawn along the X axis direction to frame the target feature point of the target gold wire on the front-view collection image layer and the target feature point of the target gold wire on the side-view collection image layer, respectively; if the angle between the target gold wire and the X axis is less than the angle between the target gold wire and the Y axis, then the side-view collection image captured by the Y-axis side camera is selected to be fused with the front-view collection image, and a frame is drawn along the Y axis on the fused image to frame the target feature point of the target gold wire on the front-view collection image layer and the target feature point of the target gold wire on the side-view collection image layer; or, two frames are drawn along the Y axis direction to frame the target feature point of the target gold wire on the front-view collection image layer and the target feature point of the target gold wire on the side-view collection image layer, respectively.

[0205] Referring to Figure 5 The application further provides a multi-view multi-angle image processing system 300, comprising: an image calling module 310, a projection transformation module 320, a first search module 330, a second search module 340, a first calculation module 350, and a first offset module 360; the image calling module 310 is used to call the front-view collection image and the side-view collection image of the measured object 200 required to be fused; the projection transformation module 320 is used to perform projection transformation on the side-view collection image using the projection transformation matrix of the side camera 30 relative to the front camera 20 obtained in advance, to obtain a projected side view; the first search module 330 is used to search the front-view collection image to obtain a front-view matching contour matched with the front-view template contour of a reference feature on a front-view template image of the measured object 200 obtained in advance, and extract the center of the front-view matching contour to obtain a first center point; the second search module 340 is used to search the side-view collection image to obtain a side-view matching contour matched with the side-view template contour of a reference feature on a side-view template image of the measured object 200 obtained in advance, and extract the center of the side-view matching contour to obtain a second center point, wherein the reference feature on the front-view template image and the reference feature on the side-view template image are the same reference feature on the same reference height plane of the measured object 200; the first calculation module 350 is used to calculate the offset amount of the second center point relative to the first center point, denoted as a real-time positioning offset amount; and the first offset module 360 is used to offset the side-view collection image by the real-time positioning offset amount, so that the side-view collection image is aligned with the front-view collection image to obtain a fused image.

[0206] In an embodiment, the multi-view multi-angle image processing system 300 further comprises a second offset module, which is configured to offset the projected side-view acquisition image using the training offset of the pre-acquired side-view template image of the measured object 200 relative to the front-view template image before searching the front-view acquisition image to obtain a front-view matching contour matching the front-view template contour of the pre-acquired front-view template image of the measured object 200.

[0207] In an embodiment, the system further comprises a disparity obtaining module and a height calculating module, the disparity obtaining module is configured to calculate the distance between the target feature on the front-view acquisition image layer of the fusion image and the target feature on the side-view acquisition image layer of the fusion image, denoted as a disparity distance, and the height calculating module calculates the height value of the target feature using the disparity and the tilt angle of the side camera 30 corresponding to the side-view acquisition image, wherein the pre-constructed height detection formula is:

[0208] H=d*tan(θ);

[0209] wherein d is the disparity distance, H is the height value of the target feature, and θ is the tilt angle of the side camera corresponding to the side-view acquisition image.

[0210] Referring to Figure 6 The application further provides an electronic device, which further comprises a memory 601, a processor 602, and a computer program stored in the memory 601 and executable on the processor 602, and the processor 602 implements the multi-view multi-angle image processing method described in the foregoing embodiments when executing the computer program.

[0211] Further, the electronic device further comprises at least one input device 603 and at least one output device 604.

[0212] The memory 601, the processor 602, the input device 603, and the output device 604 are connected through a bus 605.

[0213] The input device 603 can be a camera, a touch panel, a physical button, a mouse, or the like, and the output device 604 can be a display screen.

[0214] The memory 601 can be a high-speed random access memory (RAM) or a non-volatile memory such as a disk memory. The memory 601 is configured to store a set of executable program codes, and the processor 602 is coupled to the memory 601.

[0215] Further, the embodiments of the present application further provide a storage medium, which can be arranged in the electronic device in the above embodiments, and the storage medium can be the memory 601 in the foregoing. The storage medium has a computer program stored thereon, and the program is executed by the processor 602 to implement the multi-view multi-angle image processing method described in the foregoing embodiments.

[0216] Further, the computer storage medium can also be a U disk, a mobile hard disk, a read-only memory 601 (ROM, Read-Only Memory), a RAM, a magnetic disk or an optical disk, and various media that can store program codes.

[0217] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another electronic device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0218] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, that is, they can be located in one place, or can be distributed to a plurality of network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0219] In addition, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can exist physically, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0220] The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can be stored in a storage medium. Based on this understanding, the technical scheme of the present application or the whole or part of the technical scheme that essentially contributes to the prior art can be embodied in the form of a software product.

[0221] It should be noted that, for the foregoing method embodiments, for the convenience of description, they are all described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0222] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0223] The above is the description of the multi-view multi-angle image processing method, system, electronic device and storage medium provided by the present application. For those skilled in the art, according to the idea of the embodiments of the present application, there will be changes in specific implementation and application range. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A multi-view multi-angle image processing method, characterized in that, The application relates to a multi-view image processing method for fusing multi-view images collected by an image collection device, wherein the image collection device comprises a forward camera and a lateral camera, the forward camera is used for vertically photographing a measured object to obtain a forward collection image, the lateral camera is used for obliquely photographing the measured object to obtain a lateral collection image, the mirror plane of the lateral camera, the image plane of the lateral camera and the extension plane of the object plane of the measured object intersect at a line, and the multi-view multi-angle image processing method comprises the following steps: an image calling step of calling the required fused forward collection image and lateral collection image of the measured object; a projection transformation step of performing projection transformation on the lateral collection image by using a projection transformation matrix of the lateral camera relative to the forward camera, so as to obtain a projected lateral collection image; a step of searching the forward collection image to obtain a forward matching contour matched with a forward template contour of a reference feature on a pre-acquired forward template image of the measured object, and extracting the center of the forward matching contour to obtain a first center point; a step of searching the lateral collection image to obtain a lateral matching contour matched with a lateral template contour of a reference feature on a pre-acquired lateral template image of the measured object, and extracting the center of the lateral matching contour to obtain a second center point, wherein the reference feature on the forward template image and the reference feature on the lateral template image are the same reference feature on the same reference height plane of the measured object; a step of calculating the offset of the second center point relative to the first center point, and recording the offset as a real-time positioning offset; a step of offsetting the lateral collection image by the real-time positioning offset, so as to align and fuse the lateral collection image with the forward collection image to obtain a fused image.

2. The multi-view multi-angle image processing method of claim 1, wherein, The acquisition method of the forward matching contour and the lateral matching contour comprises the following steps: a step of photographing the measured object by using the image collection device to obtain a forward template image and a lateral template image; a step of collecting the contour of a reference feature on the forward template image, and recording the contour as a forward template contour; and a step of collecting the contour of a reference feature on the lateral template image, and recording the contour as a lateral template contour.

3. The multi-view multi-angle image processing method of claim 2, wherein, Before the step of collecting the contour of a reference feature on the forward template image, and recording the contour as a forward template contour, the acquisition method further comprises the following steps: a step of framing the reference feature on the forward template image to obtain a first collection area; the step of collecting the contour of a reference feature on the forward template image, and recording the contour as a forward template contour specifically comprises the following step: a step of collecting the contour of a reference feature in the first collection area, and recording the contour as a forward template contour; Before the step of collecting the contour of a reference feature on the lateral template image, and recording the contour as a lateral template contour, the acquisition method further comprises the following steps: a step of framing the reference feature on the lateral template image to obtain a second collection area; the step of collecting the contour of a reference feature on the lateral template image, and recording the contour as a lateral template contour specifically comprises the following step: a step of collecting the contour of a reference feature in the second collection area, and recording the contour as a lateral template contour.

4. The multi-view multi-angle image processing method of claim 2, wherein, After the step of collecting the contour of a reference feature on the forward template image, and recording the contour as a forward template contour, the acquisition method further comprises the following steps: a step of performing line thinning processing on the forward template contour to obtain a thinned forward template contour; after the step of collecting the contour of a reference feature on the lateral template image, and recording the contour as a lateral template contour, the acquisition method further comprises the following steps: Line thinning is performed on the side-view template contour to obtain a thinned side-view template contour.

5. The multi-view multi-angle image processing method of claim 2, wherein, Before searching the front-view acquisition image to obtain a front-view matching contour matching the front-view template contour of the pre-acquired front-view template image of the measured object, and extracting a center of the front-view matching contour to obtain a first center point, the method further comprises: The projected side-view acquisition image is offset using a training offset of the pre-acquired side-view template image of the measured object relative to the front-view template image.

6. The multi-view multi-angle image processing method of claim 5, wherein, The method for obtaining the training offset comprises: A center point of the front-view template contour is extracted, denoted as a third center point. A center point of the side-view template contour is extracted, denoted as a fourth center point. An offset of the fourth center point relative to the third center point is calculated, denoted as a training offset.

7. The multi-view multi-angle image processing method of claim 1, wherein, Before the searching the front-view acquisition image to obtain a front-view matching contour matching the front-view template contour of the reference feature on the pre-acquired front-view template image of the measured object, the method further comprises: The reference feature of the front-view acquisition image is framed to obtain a first search region. The searching the front-view acquisition image to obtain a front-view matching contour matching the front-view template contour of the reference feature on the pre-acquired front-view template image of the measured object specifically comprises: The first search region is searched to obtain a front-view matching contour matching the front-view template contour of the reference feature on the pre-acquired front-view template image of the measured object. Before the searching the side-view acquisition image to obtain a side-view matching contour matching the side-view template contour of the reference feature on the pre-acquired side-view template image of the measured object, the method further comprises: The reference feature of the side-view acquisition image is framed to obtain a second search region. The searching the side-view acquisition image to obtain a side-view matching contour matching the side-view template contour of the reference feature on the pre-acquired side-view template image of the measured object specifically comprises: The second search region is searched to obtain a side-view matching contour matching the side-view template contour of the reference feature on the pre-acquired side-view template image of the measured object.

8. The multi-view multi-angle image processing method of claim 1, wherein, Before the offsetting the side-view acquisition image by the real-time positioning offset, the method further comprises: It is judged whether the real-time positioning offset is less than a pre-set offset threshold, if yes, the step of offsetting the side-view acquisition image by the real-time positioning offset is executed, if not, the side-view acquisition image is offset using a previous real-time positioning offset of the measured object.

9. The multi-view multi-angle image processing method of claim 1, wherein, The method further comprises: A parallax distance between a target feature on a front-view acquisition image layer of the fused image and a target feature on a side-view acquisition image layer of the fused image is calculated along a parallax direction, denoted as a parallax distance; wherein the parallax direction is parallel to a normal projection of an optical axis of a lateral camera corresponding to the side-view acquisition image; A pre-constructed height detection formula is calculated using the parallax and an inclination angle of the lateral camera corresponding to the side-view acquisition image to obtain a height value of the target feature.

10. The multi-view multi-angle image processing method of claim 9, wherein, The pre-constructed height detection formula is: H = d * tan(θ); wherein d is the parallax distance, H is the height value of the target feature, and θ is the inclination angle of the lateral camera corresponding to the side-view acquisition image.

11. The multi-view multi-angle image processing method of claim 9, wherein, The distance between the target feature on the front-view acquisition layer of the fusion image and the target feature on the side-view acquisition layer of the fusion image is calculated along the parallax direction, and is recorded as a parallax distance, which includes: The target feature on the front-view acquisition layer of the fusion image and the target feature on the side-view acquisition layer are framed along the parallax direction to obtain a detection area; The target feature in the detection area is acquired to obtain a first measurement point and a second measurement point; The distance between the first measurement point and the second measurement point is calculated, and is recorded as a parallax distance.

12. The multi-view multi-angle image processing method of claim 9, wherein, The distance between the target feature on the front-view acquisition layer of the fusion image and the target feature on the side-view acquisition layer of the fusion image is calculated along the parallax direction, and is recorded as a parallax distance, which includes: The target feature on the front-view acquisition layer of the fusion image and the target feature on the side-view acquisition layer are framed along the parallax direction to obtain a detection area; The target feature in the detection area is acquired to obtain a first measurement point and a second measurement point; The distance between the first measurement point and the second measurement point is calculated, and is recorded as a parallax distance.

13. A multi-view multi-angle image processing system, characterized in that, It includes: An image retrieval module is configured to retrieve a required front-view acquisition image and a side-view acquisition image of a measured object for fusion; A projection transformation module is configured to perform projection transformation on the side-view acquisition image using a projection transformation matrix of a side camera relative to a front camera to obtain a projected side-view acquisition image; A first search module is configured to search the front-view acquisition image to obtain a front-view matching contour matching a front-view template contour of a reference feature on a pre-acquired front-view template image of the measured object, and extract a center of the front-view matching contour to obtain a first center point; A second search module is configured to search the side-view acquisition image to obtain a side-view matching contour matching a side-view template contour of a reference feature on a pre-acquired side-view template image of the measured object, and extract a center of the side-view matching contour to obtain a second center point, wherein the reference feature on the front-view template image and the reference feature on the side-view template image are the same reference feature on the same reference height plane of the measured object; A first calculation module is configured to calculate an offset of the second center point relative to the first center point, which is recorded as a real-time positioning offset; A first offset module is configured to offset the side-view acquisition image by the real-time positioning offset to align the side-view acquisition image with the front-view acquisition image to obtain a fusion image.

14. An electronic device comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the multi-view multi-angle image processing method of any one of claims 1-12 is implemented.

15. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method of any one of claims 1-12.

Images