Color image synthesis method and industrial camera
By performing distortion correction and image fusion on industrial cameras, the problem of light shifting in different bands in color image imaging is solved, and the accuracy and accuracy of the image are improved.
Patent Information
- Application Number
- CN202411802739.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-04-11
AI Technical Summary
In the existing color image imaging technology, different bands of light will shift after passing through the lens, resulting in insufficient image accuracy in scenes that require refined imaging.
By acquiring the initial monochrome image of an industrial camera in multiple bands, distortion parameters are used to correct the distortion of pixel point coordinates, and image fusion is performed in combination with the internal and external parameters of the camera to generate a color fusion image.
It overcomes the problem of ray offset in different bands, improves the accuracy and accuracy of color images, and is suitable for application scenarios where high-precision images are required.
Smart Images
Figure CN119815189A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a color image synthesis method and an industrial camera. Background Art
[0002] The existing imaging method of color pictures is that light passes through the lens to reach the photosensitive chip. There are red, green and blue primary color sensing diodes on the photosensitive chip. The output signal value is sensed according to the different brightness, and then interpolated to obtain the three primary color values of each pixel.
[0003] In this process, due to the different wavelengths of light in different bands, light in different bands emitted from the same position will undergo different offsets after passing through the lens. In scenes that require refined imaging, such as cultural relics mapping, the accuracy may not meet the requirements. Summary of the invention
[0004] In view of the deficiencies in the above-mentioned prior art, the present application provides a color image synthesis method and an industrial camera to solve the problems existing in the prior art.
[0005] The technical solution adopted in the embodiment of the present application is as follows:
[0006] In a first aspect, an embodiment of the present application provides a color image synthesis method, comprising:
[0007] Acquire multiple initial monochrome images of a target object captured by an industrial camera in multiple bands;
[0008] According to the distortion parameters of the industrial camera in the multiple bands, respectively, coordinates of each pixel point in the multiple initial monochrome images are corrected to obtain multiple distortion-corrected images;
[0009] The multiple distortion-corrected images are fused according to the camera intrinsic parameters and camera extrinsic parameters of the industrial camera in the multiple bands to obtain a color fused image of the target object.
[0010] In one embodiment, fusing the multiple distortion-corrected images according to the camera intrinsic parameters and camera extrinsic parameters of the industrial camera in the multiple bands to obtain the color fused image of the target object includes:
[0011] According to the camera extrinsic parameters of the industrial camera in the multiple bands, a plurality of coordinate transformation relationships of the industrial camera are obtained, wherein the plurality of coordinate transformation relationships include: a coordinate transformation relationship of a plurality of second bands relative to the same first band;
[0012] According to the coordinate transformation relationships of the multiple coordinate transformation relationships, the camera internal parameters of the industrial camera in the first band, and the multiple distortion-corrected images, multiple color component parameters of each pixel point in the multiple bands are obtained;
[0013] The multiple color component parameters of each pixel are fused to obtain a color fused image of the target object.
[0014] In one embodiment, the coordinate transformation relationships of the plurality of coordinate transformation relationships, the camera intrinsic parameters under the first band, and the plurality of distortion-corrected images to the plurality of color component parameters of each pixel under the plurality of bands include:
[0015] Performing geometric calculations based on the multiple distortion-corrected images to obtain a fused image range;
[0016] Within the range of the fused image, traverse the pixels of the first distortion-corrected image under the first band to obtain the first image coordinates of each pixel in the first distortion-corrected image and the first color component parameters of each pixel under the first band;
[0017] According to the first image coordinates of each pixel point, using each coordinate transformation relationship and the camera intrinsic parameters under the first band, obtain the second image coordinates of each pixel point in the second distortion-corrected image under the corresponding second band;
[0018] According to the second image coordinates of each pixel point, a second color component parameter of each pixel point in the corresponding second band is determined.
[0019] In one embodiment, the step of obtaining, according to the first image coordinates of each pixel point, the second image coordinates of each pixel point in the second distortion-corrected image corresponding to the second band by using each coordinate transformation relationship and the camera intrinsic parameter under the first band includes:
[0020] According to the first image coordinates of each pixel point, each coordinate transformation relationship and the camera internal parameters under the first band are used to perform a reverse transformation on the first image coordinates of each pixel point, so as to transform the first image coordinates of each pixel point from the coordinate system of the first distortion-corrected image to the coordinate system of the second distortion-corrected image under the corresponding second band, so as to obtain the second image coordinates of each pixel point.
[0021] In one embodiment, determining the second color component parameter of each pixel point in the corresponding second band according to the second image coordinates of each pixel point includes:
[0022] According to the second image coordinates of each pixel point, a grayscale interpolation algorithm is used to calculate the second color component parameters of each pixel point in the corresponding second band.
[0023] In one embodiment, before fusing the multiple color component parameters of each pixel to obtain the color fused image of the target object, the method further includes:
[0024] According to the multiple distortion-corrected images, white balance calculation is performed on each pixel to obtain a color balance adjustment parameter of each pixel in the multiple bands;
[0025] The step of fusing the multiple color component parameters of each pixel point to obtain a color fused image of the target object includes:
[0026] According to the color balance adjustment parameters of each pixel point in the multiple bands, multiple color component parameters of each pixel point are fused to obtain a color fused image of the target object.
[0027] In one embodiment, performing geometric calculations according to the plurality of distortion-corrected images to obtain a fused image range includes:
[0028] Taking the camera imaging plane of the first waveband as a reference, projecting the multiple distortion-corrected images onto the camera imaging plane of the first waveband to obtain multiple projected images;
[0029] Calculating overlapping areas of the plurality of projection images;
[0030] Calculate the maximum enclosing rectangular area according to the overlapping area;
[0031] The preset image range is calculated according to the maximum enclosing rectangular area.
[0032] In one embodiment, the method further comprises:
[0033] The color fusion image is color corrected using a preset reference color card to obtain a target color image of the target object.
[0034] In one embodiment, before acquiring a plurality of initial monochrome images of the target object captured by the industrial camera in a plurality of bands, the method further comprises:
[0035] According to the photographing control parameters of the multiple bands, the industrial camera is controlled to image the target object in the multiple bands in sequence to obtain the multiple initial monochrome images.
[0036] In a second aspect, an embodiment of the present application provides an industrial camera, including: a lens, a photosensitive chip, a fill light of multiple bands, an imaging control circuit, a fill light control circuit, and a control circuit;
[0037] The lens faces the photosensitive surface of the photosensitive chip, the fill lights of multiple bands are evenly arranged around the lens, the photosensitive chip is electrically connected to the imaging control circuit, the fill lights of multiple bands are electrically connected to the fill light control circuit, the imaging control circuit and the fill light control circuit are respectively connected to the control circuit, the control circuit is used to connect to a host computer device, and the host computer device is used to execute the color image synthesis method described in the above embodiment.
[0038] The beneficial effects of the present application are as follows: the present application provides a color image synthesis method, including obtaining multiple initial monochrome images of a target object captured by an industrial camera in multiple bands; according to the distortion parameters of the industrial camera in multiple bands, respectively correcting the coordinates of each pixel point in the multiple initial monochrome images to obtain multiple distortion-corrected images; according to the camera internal parameters and camera external parameters of the industrial camera in multiple bands, fusing the multiple distortion-corrected images to obtain a color fused image of the target object.
[0039] Among them, multiple distortion-corrected images are fused according to the internal and external parameters of the industrial camera in multiple bands, which overcomes the problem that light in different bands will be offset after passing through the lens and improves the accuracy of obtaining color images of the target object. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0041] FIG1( a ) is a schematic diagram of an ideal optical path of light of different wavelengths after passing through a lens provided by the present application;
[0042] FIG1( b ) is a schematic diagram of the actual optical paths of light rays of different wavelengths after passing through a lens provided by the present application;
[0043] FIG. 2( a ) is a schematic diagram of the structure of an industrial camera provided in the present application;
[0044] FIG2( b ) is a block diagram of the industrial camera provided by the present application;
[0045] Figure 3 A schematic diagram of the calibration board provided for this application;
[0046] Figure 4 One of the flowcharts of the color image synthesis method provided in the embodiment of the present application;
[0047] Figure 5 The second flowchart of the color image synthesis method provided in the embodiment of the present application;
[0048] Figure 6 The third flowchart of the color image synthesis method provided in the embodiment of the present application;
[0049] Figure 7 A schematic diagram of the fused image range provided in an embodiment of the present application;
[0050] Figure 8 A fourth flowchart of the color image synthesis method provided in an embodiment of the present application;
[0051] Fig. 9 A fifth flowchart of the color image synthesis method provided in an embodiment of the present application;
[0052] Fig.10 A schematic diagram of the structure of a color image synthesis device provided in an embodiment of the present application;
[0053] Fig.11 A schematic diagram of the structure of the host computer device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.
[0055] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0056] In the description of the present application, it should be noted that if the terms "upper", "lower", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the application is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.
[0057] In addition, the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0058] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the terms "disposed" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection, it can be a direct connection, it can be an indirect connection through an intermediate medium, and it can be the internal communication of two elements. For those skilled in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0059] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.
[0060] FIG1(a) is a schematic diagram of the ideal optical path of light of different bands after passing through the lens provided by the present application, and FIG1(b) is a schematic diagram of the actual optical path of light of different bands after passing through the lens provided by the present application. As shown in FIG1(a) and FIG1(b), when an object is imaged, ideally, the positions of light of different bands after passing through the lens should be the same on the image. However, the actual situation is that the positions of light of different bands after passing through the lens will be slightly offset, which has an adverse effect on refined imaging such as high-precision mapping of cultural relics and visual measurement based on multi-band data.
[0061] Therefore, the embodiments of the present application provide a color image synthesis method and an industrial camera to solve the above problems.
[0062] Figure 2(a) is a schematic diagram of the structure of the industrial camera provided in the present application, and Figure 2(b) is a block diagram of the structure of the industrial camera provided in the present application. First, the industrial camera provided in the present application is described in combination with Figure 2(a) and Figure 2(b). The industrial camera includes a lens, a photosensitive chip, a fill light of multiple bands, an imaging control circuit, a fill light control circuit and a control circuit.
[0063] Among them, the photosensitive chip is electrically connected to the imaging control circuit, the fill light of multiple bands is electrically connected to the fill light control circuit, the imaging control circuit and the fill light control circuit are respectively connected to the control circuit, the control circuit is used to connect to the host computer device, and the host computer device is used to execute the color image synthesis method provided in the following embodiments.
[0064] Specifically, the lens faces the photosensitive surface of the photosensitive chip. When the industrial camera captures an image of the target object through the lens, light passes through the lens to the photosensitive chip to achieve imaging; fill lights of multiple bands are evenly arranged around the lens, and the fill light control circuit can control the on and off of the fill lights to obtain multiple initial monochrome images of the target object in multiple bands.
[0065] A camera can be virtualized in multiple bands respectively. The camera model can be expressed by the following collinearity equation. The collinearity equations in multiple bands are the same, but the parameters in the equations are different.
[0066]
[0067] Among them, (x, y) is the image plane coordinate of the image point; (x0, y0, f) is the internal orientation element of the image; (XS, YS, ZS) is the object space coordinate of the camera station; (X, Y, Z) is the object space coordinate of the object point; ai, bi, ci (i = 1, 2, 3) are the 9 direction cosines composed of the 3 exterior orientation elements phi, omega, kappa of the image.
[0068] (XS, YS, ZS, phi, omega, kappa) are the external parameters of the camera, and (x0, y0, f) are the intrinsic parameters of the camera.
[0069] In addition, during the shooting process, because of the lens distortion, the influence of lens distortion needs to be considered. The distortion model is as shown in formula (2):
[0070]
[0071] Among them, (k1, k2, k3, p1, p2) are distortion parameters. Substitute the distortion model into the above collinear equation. It should be noted that the distortion models under multiple bands are the same, but the distortion parameters of multiple virtual cameras are different.
[0072] The parameters in the above formula can be calculated by calibration method. This application adopts the preset Zhang Zhengyou method for calibration, which is as follows: the industrial camera is calibrated at multiple positions. Figure 3The calibration plate shown in the figure takes images, and three images are obtained at each position under multiple bands; multiple groups of images under multiple bands are obtained, and then the Zhang Zhengyou method is used to solve the external parameters of each image in each group, as well as the intrinsic parameters and distortion parameters of the three virtual cameras, thereby obtaining (XS, YS, ZS, phi, omega, kappa) and (x0, y0, f) and (k1, k2, k3, p1, p2) of multiple virtual cameras.
[0073] The color image synthesis method provided by the present application is specifically described below through multiple examples in conjunction with the accompanying drawings.
[0074] Figure 4 One of the flowcharts of the color image synthesis method provided in the embodiment of the present application is as follows: Figure 4 As shown, the method includes:
[0075] S101, obtaining a plurality of initial monochrome images of a target object captured by an industrial camera in a plurality of bands.
[0076] First, an industrial camera is used to image the target object to obtain multiple initial monochrome images of the target object in multiple bands, wherein the multiple bands may be bands corresponding to the three primary colors of light, including a red light band, a green light band, and a blue light band.
[0077] Specifically, a camera can be virtualized in multiple bands (for example, a camera in the red light band, a camera in the green light band, and a camera in the blue light band), and the industrial camera can be controlled to image the target object in multiple bands in sequence according to the photography control parameters of the multiple bands to obtain multiple initial monochrome images. The photography control parameters may include the illumination duration and illumination intensity of the fill light in multiple bands, which are used to control the exposure time of the industrial camera; and may also include the triggering method of the fill light (for example, physical button control, which is used to press a button to start shooting; or software automatic control, which automatically starts shooting according to the program).
[0078] S102 . According to the distortion parameters of the industrial camera in multiple bands, coordinates of each pixel point in a plurality of initial monochrome images are respectively subjected to distortion correction to obtain a plurality of distortion-corrected images.
[0079] After obtaining multiple initial monochrome images of the target object in multiple bands, the coordinates of each pixel in the multiple initial monochrome images can be corrected for distortion according to the pre-acquired distortion parameters of the industrial camera in multiple bands to obtain multiple corrected distortion images. After the distortion correction, the pixel coordinates of the initial monochrome image are the theoretically correct positions.
[0080] For example, after obtaining the initial monochrome image of the target object in the red light band, the initial monochrome image in the green light band, and the initial monochrome image in the blue light band, the coordinates of each pixel point in the initial monochrome image in the red light band are subjected to distortion correction according to the distortion parameters of the industrial camera in the red light band to obtain the distortion-corrected image in the red light band; the coordinates of each pixel point in the initial monochrome image in the green light band are subjected to distortion correction according to the distortion parameters of the industrial camera in the green light band to obtain the distortion-corrected image in the green light band; the coordinates of each pixel point in the initial monochrome image in the blue light band are subjected to distortion correction according to the distortion parameters of the industrial camera in the blue light band to obtain the distortion-corrected image in the blue light band.
[0081] S103 , according to the camera internal parameters and camera external parameters of the industrial camera in multiple bands, fuse the multiple distortion-corrected images to obtain a color fused image of the target object.
[0082] Finally, according to the pre-acquired camera intrinsic parameters and camera extrinsic parameters of the industrial camera in multiple bands, multiple distortion-corrected images are fused to obtain a color fused image of the target object.
[0083] For example, according to the intrinsic and extrinsic parameters of the industrial camera in the red light band, the intrinsic and extrinsic parameters of the industrial camera in the green light band, and the intrinsic and extrinsic parameters of the industrial camera in the blue light band, the distortion-corrected image in the red light band, the distortion-corrected image in the green light band, and the distortion-corrected image in the blue light band are fused to obtain a color fused image of the target object.
[0084] In summary, the embodiment of the present application provides a color image synthesis method, which can correct the distortion of the coordinates of each pixel point in multiple initial monochrome images to obtain multiple distortion-corrected images, and then fuse the multiple distortion-corrected images according to the internal camera parameters and external camera parameters of the industrial camera in multiple bands, thereby overcoming the problem that light in different bands will be offset after passing through the lens, and improving the accuracy of obtaining the color image of the target object.
[0085] Figure 5 The second flowchart of the color image synthesis method provided in the embodiment of the present application is as follows: Figure 5 As shown, the step S102 of fusing multiple distortion-corrected images according to the camera internal parameters and camera external parameters of the industrial camera in multiple bands to obtain a color fused image of the target object may include:
[0086] S201. Obtain multiple coordinate transformation relationships of the industrial camera according to the camera extrinsic parameters of the industrial camera in multiple bands.
[0087] Among them, multiple coordinate transformation relationships include: coordinate transformation relationships of multiple second bands relative to the same first band. For example, the red light band is taken as the first band and the other bands are taken as the second bands. Then the multiple coordinate transformation relationships may include: the coordinate transformation relationship of the green light band relative to the red light band (denoted as RTcam2-1) and the coordinate transformation relationship of the blue light band relative to the red light band (denoted as RTcam3-1).
[0088] The principle is: since the relative pose (position and attitude) of the three virtual cameras is unchanged, that is, no matter how the photos are taken, the three virtual cameras are a rigid body, then we can take the camera coordinate system under the red light band as the reference, that is, the camera extrinsic parameters under the red light band are recorded as (0, 0, 0, 0, 0, 0). At this time, the transformation relationship of the camera in the green light band relative to the camera in the red light band is set to RTcam2-1, where R is the rotation matrix obtained by the angle component, and T is the translation component; similarly, the transformation relationship of the camera in the blue light band relative to the camera in the red light band is set to RTcam3-1, and RTcam2-1 and RTcam3-1 can be calculated through the acquired camera extrinsic parameters.
[0089] S202, obtaining a plurality of color component parameters of each pixel point in a plurality of bands according to a coordinate transformation relationship of a plurality of coordinate transformation relationships, an internal camera parameter of an industrial camera in a first band, and a plurality of distortion-corrected images.
[0090] According to the coordinate transformation relationships of the multiple coordinate transformation relationships, the camera internal parameters of the industrial camera in the first band, and the multiple distortion corrected images, the multiple color component parameters of each pixel point in the multiple bands can be obtained. The multiple color component parameters include: red light color component parameter, represented by GreyRed; green light color component parameter, represented by GreyGreen; blue light color component parameter, represented by GreyBlue.
[0091] S203 , fusing multiple color component parameters of each pixel to obtain a color fused image of the target object.
[0092] After obtaining multiple color component parameters of each pixel point under multiple bands, the multiple color component parameters of each pixel point are fused to obtain a color fused image of the target object.
[0093] The three primary color components of the color fusion image at each pixel are: R = Kri*GreyRed, G = Kgi*GreyGreen, B = Kbi*GreyBlue. Among them, R, G, B represent the three primary color components of the color fusion image, and Kri, Kgi and Kbi are the color balance adjustment parameters of the pixel.
[0094] Figure 6The third flow chart of the color image synthesis method provided in the embodiment of the present application is as follows: Figure 6 As shown, the coordinate transformation relationships of the multiple coordinate transformation relationships, the camera internal parameters in the first band, and the multiple distortion-corrected images to the multiple color component parameters of each pixel in the multiple bands in S202 may specifically include:
[0095] S301, performing geometric calculations based on a plurality of distortion-corrected images to obtain a fused image range.
[0096] Multiple distortion-corrected images are in different camera coordinate systems. Therefore, it is necessary to perform geometric calculations based on the multiple distortion-corrected images to obtain Figure 7 The fused image range shown is used to represent the overlapping range of multiple distortion-corrected images in the camera coordinate system, that is, the fused image range is the area where the multiple distortion-corrected images actually need to be fused.
[0097] Figure 7 In the figure, camera 1 represents a virtual camera in the red light band, camera 2 represents a virtual camera in the green light band, and camera 3 represents a virtual camera in the blue light band. The fused image range is the maximum enclosing rectangular area of the overlapping projection area of multiple distortion-corrected images. The coordinates of the pixel point in the upper left corner of the area in the camera 1 coordinate system are expressed as (x m ,y m ), the length and width of the region are denoted as W and H respectively.
[0098] S302: within the range of the fused image, perform pixel traversal on the first distortion-corrected image in the first band to obtain first image coordinates of each pixel in the first distortion-corrected image and first color component parameters of each pixel in the first band.
[0099] After obtaining the fused image range, (x m ,y m ) as the starting pixel, with length, width W and H as the range, traverse the fused image range, and for the pixel coordinates (x i ,y i ), calculate the first image coordinates in the camera one coordinate system (because the first distortion-corrected image and camera one are in the same coordinate system, the first image coordinates can be obtained by directly taking the value), thereby obtaining the grayscale value of each pixel in the first band, that is, the first color component parameter (that is, the red light color component parameter), denoted as GreyRed.
[0100] S303 , according to the first image coordinates of each pixel point, using each coordinate conversion relationship and the camera intrinsic parameters under the first band, obtain the second image coordinates of each pixel point in the second distortion-corrected image under the corresponding second band.
[0101] Specifically, according to the first image coordinates of each pixel point, each coordinate transformation relationship and the camera internal parameters under the first band are used to perform a reverse transformation on the first image coordinates of each pixel point, so as to transform the first image coordinates of each pixel point from the coordinate system of the first distortion-corrected image to the coordinate system of the second distortion-corrected image under the corresponding second band, so as to obtain the second image coordinates of each pixel point.
[0102] For example, for the first image coordinate (x i ,y i ), using the intrinsic parameters of camera 1 (x0, y0, f) and the transformation relationship RTcam2-1 between camera 2 and camera 1, a reverse transformation is performed to project the coordinates of camera 1 coordinate system into camera 2, and (x i2 ,y i2 ), the grayscale value of the coordinate point is calculated in camera 2, and the second image coordinates of each pixel point under camera 2 are obtained.
[0103] For another example, for the first image coordinate (x i ,y i ), using the intrinsic parameters of camera 1 (x0, y0, f) and the conversion relationship RTcam3-1 between camera 3 and camera 1, a reverse transformation is performed to project the coordinates of camera 1 into camera 3, and (x i2 ,y i2 ), calculate the gray value of the coordinate point in camera three, and obtain the second image coordinates of each pixel point under camera three.
[0104] S304 . Determine a second color component parameter of each pixel in the corresponding second band according to the second image coordinates of each pixel.
[0105] According to the second image coordinates of each pixel point, a grayscale interpolation algorithm is used to calculate the second color component parameter of each pixel point in the corresponding second band.
[0106] For example, after obtaining the second image coordinates of each pixel under camera 2, a grayscale interpolation algorithm is used to calculate the color component parameters of each pixel under the green band corresponding to camera 2, that is, the green component GreyGreen.
[0107] After obtaining the second image coordinates of each pixel under camera three, a grayscale interpolation algorithm is used to calculate the color component parameters of each pixel under the blue band corresponding to camera three, that is, the blue component GreyBlue.
[0108] Figure 8 The fourth flowchart of the color image synthesis method provided in the embodiment of the present application is as follows: Figure 8As shown, before fusing the multiple color component parameters of each pixel point to obtain a color fused image of the target object in S103, the method of the present application further includes:
[0109] S401. Perform white balance calculation on each pixel according to a plurality of distortion-corrected images to obtain color balance adjustment parameters of each pixel in a plurality of bands.
[0110] When a color image is digitized, the colors may sometimes look a little abnormal when displayed. This is because different sensitivities, brightness enhancement factors, offsets, etc. in the color channels lead to different transformations of the three primary color components (R, G, B) in digitization, making the three primary colors in the image "unbalanced", causing the color of the target object to deviate from its original true color.
[0111] Therefore, this embodiment performs white balance calculation on each pixel point, and obtains color balance adjustment parameters of each pixel point in multiple bands.
[0112] Specifically, according to formula (3), white balance calculation can be performed on each pixel.
[0113] Y=0.333*R+0.333*G+0.333*B (3)
[0114] Among them, R, G, and B are the three primary color components of the pixel point, that is, according to formula (3), for each pixel point in the distortion-corrected image after shooting, the grayscale average value at each pixel point is taken as the ideal value, and the color balance adjustment parameter calculation method at the pixel point position is:
[0115] Kri = Y / R (4)
[0116] Kgi = Y / G (5)
[0117] Kbi = Y / B (6)
[0118] Then the step S103 of fusing the multiple color component parameters of each pixel to obtain a color fused image of the target object includes:
[0119] S402 , according to the color balance adjustment parameters of each pixel in multiple bands, multiple color component parameters of each pixel are fused to obtain a color fused image of the target object.
[0120] After obtaining the color balance adjustment parameters of each pixel in multiple bands, multiple color component parameters of each pixel are fused according to the color balance adjustment parameters of each pixel in multiple bands to obtain a color fused image of the target object.
[0121] This embodiment fuses multiple color component parameters of each pixel point, performs color correction on the image with color difference, and obtains a color fused image with normal color, so that the color of the color fused image is more realistic and the display effect of the image is enhanced.
[0122] Fig. 9 The fifth flow chart of the color image synthesis method provided in the embodiment of the present application is as follows: Fig. 9 As shown, the step S301 of performing geometric calculations based on the multiple distortion-corrected images to obtain a fused image range may include:
[0123] S501 . Taking a camera imaging plane of a first waveband as a reference, project a plurality of distortion-corrected images onto a camera imaging plane of the first waveband to obtain a plurality of projected images.
[0124] The virtual camera in the red light band is recorded as camera 1, the virtual camera in the green light band is recorded as camera 2, and the virtual camera in the blue light band is recorded as camera 3. Since cameras 2 and 3 have slight changes relative to camera 1, refer to Figure 7 , taking the camera imaging plane of the red light band (camera one) as the reference plane, and projecting multiple distortion-corrected images (including distortion-corrected images in the green light band and the blue light band) onto the camera imaging plane of the red light band, the projection images of the multiple distortion-corrected images on the reference plane (the plane of camera one) can be obtained.
[0125] S502: Calculate overlapping areas of multiple projection images.
[0126] The multiple projection images of the distortion-corrected images in the three bands on the reference plane overlap together, so the overlapping area of the multiple projection images can be calculated, including:
[0127] Take the projected image of camera 2 (virtual camera under the green light band), take a pixel in the order of image pixel arrangement, and determine whether the pixel point is included in the range of camera 1 after projection (the pixel coordinate is within the effective imaging length and width of camera 1). If so, the pixel point is included by camera 1 and marked as included by camera 2. If not, no mark is made.
[0128] The same is true for camera three. Take the projected image of camera three (virtual camera under the blue light band), take a pixel in the order of image pixel arrangement, and determine whether the pixel point is included in the range of camera one after projection (the pixel coordinate is within the effective imaging length and width of camera one). If so, the pixel point is included by camera one and marked as included by camera three. If not, it is not marked.
[0129] If a pixel in camera one (red light band) is marked by camera two and camera three at the same time, it means that the pixel is included by the three cameras at the same time. A mark is made for the three cameras, and all the pixels in camera one are traversed to obtain the overlapping area included by the three cameras at the same time, that is, the overlapping area of multiple projection images.
[0130] S503: Calculate the maximum enclosing rectangular area according to the overlapping area.
[0131] Then, based on the overlapping area, calculate the largest contained rectangle in the overlapping area, such as Figure 7 shown.
[0132] S504: Obtain a fused image range according to the maximum enclosing rectangular area.
[0133] The maximum enclosing rectangular area finally obtained, the length and width of the rectangular area are the length and width of the fused image range, recorded as W and H respectively. At the same time, the coordinates of the pixel point in the upper left corner of the fused image range in the camera coordinate system are recorded, expressed as (x m ,y m ).
[0134] In one embodiment, after obtaining the color fused image of the target object, a preset reference color card can be used to perform color correction on the color fused image to obtain a target color image of the target object. The target color image is more realistic than the color fused image, thereby enhancing the image display effect.
[0135] The following is a corresponding explanation of the device, equipment and storage medium for executing the color image synthesis method provided by any of the above embodiments of the present application. Its specific implementation process and the technical effects produced are the same as those of the corresponding method embodiments mentioned above. For the sake of brief description, for the parts not mentioned in the following embodiments, please refer to the corresponding contents in the method embodiments.
[0136] Alternatively, if Fig.10 As shown, the present application provides a color image synthesis device, comprising:
[0137] The acquisition module 10 is used to acquire a plurality of initial monochrome images of a target object captured by an industrial camera in a plurality of bands.
[0138] The correction module 20 is used to perform distortion correction on the coordinates of each pixel point in a plurality of initial monochrome images according to the distortion parameters of the industrial camera in a plurality of bands, so as to obtain a plurality of distortion-corrected images.
[0139] The fusion module 30 is used to fuse multiple distortion-corrected images according to the camera internal parameters and camera external parameters of the industrial camera in multiple bands to obtain a color fused image of the target object.
[0140] Optionally, the fusion module 30 is used to obtain multiple coordinate transformation relationships of the industrial camera based on the camera external parameters of the industrial camera in multiple bands, and the multiple coordinate transformation relationships include: coordinate transformation relationships of multiple second bands relative to the same first band; obtain multiple color component parameters of each pixel point in multiple bands based on the coordinate transformation relationships of the multiple coordinate transformation relationships, the camera internal parameters of the industrial camera in the first band, and multiple distortion corrected images; fuse the multiple color component parameters of each pixel point to obtain a color fused image of the target object.
[0141] Optionally, the fusion module 30 is used to perform geometric solution based on multiple distortion-corrected images to obtain a fused image range; within the fused image range, perform pixel traversal on the first distortion-corrected image under the first band to obtain the first image coordinates of each pixel in the first distortion-corrected image, and the first color component parameters of each pixel in the first band; based on the first image coordinates of each pixel, use each coordinate transformation relationship and the camera internal parameters under the first band to obtain the second image coordinates of each pixel in the second distortion-corrected image under the corresponding second band; based on the second image coordinates of each pixel, determine the second color component parameters of each pixel in the corresponding second band.
[0142] Optionally, the fusion module 30 is used to perform a reverse transformation on the first image coordinates of each pixel point according to the first image coordinates of each pixel point, using each coordinate transformation relationship and the camera internal parameters under the first band, so as to transform the first image coordinates of each pixel point from the coordinate system of the first distortion-corrected image to the coordinate system of the second distortion-corrected image under the corresponding second band, so as to obtain the second image coordinates of each pixel point.
[0143] Optionally, the fusion module 30 is used to calculate the second color component parameter of each pixel in the corresponding second band by using a grayscale interpolation algorithm according to the second image coordinates of each pixel.
[0144] Optionally, the color image synthesis device further includes a correction module, which is used to perform white balance calculation on each pixel point according to a plurality of distortion-corrected images, so as to obtain color balance adjustment parameters of each pixel point in a plurality of bands.
[0145] The fusion module 30 is used to fuse multiple color component parameters of each pixel point according to the color balance adjustment parameters of each pixel point in multiple bands to obtain a color fused image of the target object.
[0146] Optionally, the fusion module 30 is used to project multiple distortion-corrected images onto the camera imaging plane of the first band based on the camera imaging plane of the first band to obtain multiple projection images; calculate the overlapping area of the multiple projection images; calculate the maximum enclosing rectangular area based on the overlapping area; and calculate the preset image range based on the maximum enclosing rectangular area.
[0147] Optionally, the correction module is used to use a preset reference color card to perform color correction on the color fusion image to obtain a target color image of the target object.
[0148] Optionally, the acquisition module 10 is used to control the industrial camera to image the target object in multiple bands in sequence according to the photographing control parameters of multiple bands, so as to obtain multiple initial monochrome images.
[0149] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.
[0150] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
[0151] Alternatively, if Fig.11 As shown, the present application also provides a host computer device, including a processor 100, a storage medium 200 and a bus 300, wherein the storage medium stores program instructions executable by the processor. When the host computer device is running, the processor and the storage medium communicate through the bus, and the processor executes the program instructions to implement the color image synthesis method of any of the above embodiments.
[0152] Optionally, the present application further provides a readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the color image synthesis method of any of the above embodiments is implemented.
[0153] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0156] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.
[0157] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A color image synthesis method, characterized in that: include: Acquire multiple initial monochrome images of a target object captured by an industrial camera in multiple bands; According to the distortion parameters of the industrial camera in the multiple bands, respectively, coordinates of each pixel point in the multiple initial monochrome images are corrected to obtain multiple distortion-corrected images; The multiple distortion-corrected images are fused according to the camera intrinsic parameters and camera extrinsic parameters of the industrial camera in the multiple bands to obtain a color fused image of the target object.
2. The method according to claim 1, characterized in that The step of fusing the multiple distortion-corrected images according to the camera internal parameters and camera external parameters of the industrial camera in the multiple bands to obtain a color fused image of the target object includes: According to the camera extrinsic parameters of the industrial camera in the multiple bands, a plurality of coordinate transformation relationships of the industrial camera are obtained, wherein the plurality of coordinate transformation relationships include: a coordinate transformation relationship of a plurality of second bands relative to the same first band; According to the coordinate transformation relationships of the multiple coordinate transformation relationships, the camera internal parameters of the industrial camera in the first band, and the multiple distortion-corrected images, multiple color component parameters of each pixel point in the multiple bands are obtained; The multiple color component parameters of each pixel are fused to obtain a color fused image of the target object.
3. The method according to claim 2, characterized in that The coordinate transformation relationships according to the plurality of coordinate transformation relationships, the camera internal parameters under the first band, and the plurality of distortion-corrected images to the plurality of color component parameters of each pixel point under the plurality of bands include: Performing geometric calculations based on the multiple distortion-corrected images to obtain a fused image range; Within the range of the fused image, traverse the pixels of the first distortion-corrected image under the first band to obtain the first image coordinates of each pixel in the first distortion-corrected image and the first color component parameters of each pixel under the first band; According to the first image coordinates of each pixel point, using each coordinate transformation relationship and the camera intrinsic parameters under the first band, obtain the second image coordinates of each pixel point in the second distortion-corrected image under the corresponding second band; According to the second image coordinates of each pixel point, a second color component parameter of each pixel point in the corresponding second band is determined.
4. The method according to claim 3, characterized in that The step of obtaining, according to the first image coordinates of each pixel point, second image coordinates of each pixel point in a second distortion-corrected image corresponding to the second band by using each coordinate transformation relationship and the camera intrinsic parameters under the first band includes: According to the first image coordinates of each pixel point, each coordinate transformation relationship and the camera internal parameters under the first band are used to perform a reverse transformation on the first image coordinates of each pixel point, so as to transform the first image coordinates of each pixel point from the coordinate system of the first distortion-corrected image to the coordinate system of the second distortion-corrected image under the corresponding second band, so as to obtain the second image coordinates of each pixel point.
5. The method according to claim 3, characterized in that: The determining, according to the second image coordinates of each pixel point, the second color component parameter of each pixel point in the corresponding second band includes: According to the second image coordinates of each pixel point, a grayscale interpolation algorithm is used to calculate the second color component parameters of each pixel point in the corresponding second band.
6. The method according to claim 1, characterized in that Before fusing the multiple color component parameters of the pixels to obtain the color fused image of the target object, the method further includes: According to the multiple distortion-corrected images, white balance calculation is performed on each pixel to obtain a color balance adjustment parameter of each pixel in the multiple bands; The step of fusing the multiple color component parameters of each pixel point to obtain a color fused image of the target object includes: According to the color balance adjustment parameters of each pixel point in the multiple bands, multiple color component parameters of each pixel point are fused to obtain a color fused image of the target object.
7. The method according to claim 3, characterized in that The step of performing geometric calculations according to the plurality of distortion-corrected images to obtain a fused image range includes: Taking the camera imaging plane of the first waveband as a reference, projecting the multiple distortion-corrected images onto the camera imaging plane of the first waveband to obtain multiple projected images; calculating an overlapping area of the plurality of projection images; Calculate the maximum enclosing rectangular area according to the overlapping area; The fused image range is obtained according to the maximum enclosing rectangular area.
8. The method according to claim 1, characterized in that The method further comprises: The color fusion image is color corrected using a preset reference color card to obtain a target color image of the target object.
9. The method according to claim 1, characterized in that: The method of acquiring a plurality of initial monochrome images of a target object captured by an industrial camera in a plurality of bands includes: According to the photographing control parameters of the multiple bands, the industrial camera is controlled to image the target object in the multiple bands in sequence to obtain the multiple initial monochrome images.
10. An industrial camera, characterized in that: include: Lens, photosensitive chip, multi-band fill light, imaging control circuit, fill light control circuit and control circuit; The lens faces the photosensitive surface of the photosensitive chip, the fill lights of multiple bands are evenly arranged around the lens, the photosensitive chip is electrically connected to the imaging control circuit, the fill lights of multiple bands are electrically connected to the fill light control circuit, the imaging control circuit and the fill light control circuit are respectively connected to the control circuit, the control circuit is used to connect to a host computer device, and the host computer device is used to execute the color image synthesis method described in any one of claims 1 to 9.