Panoramic splicing type industrial camera device for complex scene and splicing method

Abstract

The invention relates to the field of industrial cameras, and particularly provides a panoramic splicing type industrial camera device for a complex scene and a splicing method, and the panoramic splicing type industrial camera device comprises a multi-camera array, a synchronous control module, an image processing module, a calibration module and a communication module. The multi-camera array is distributed in an annular or matrix mode. The synchronous control module realizes sub-microsecond synchronization (error is less than 0.8 microsecond) based on an LVDS hardware trigger architecture, and eliminates splicing dislocation of a high-speed moving target. The image processing module carries an FPGA hardening assembly line and completes image distortion correction, optimization ORB feature matching and multi-band fusion in real time. The calibration module integrates a multi-mode sensing unit, and dynamically adjusts exposure, focal length and white balance through improved Kalman filtering and a deep reinforcement learning decision algorithm. In a strong interference environment, the panoramic stitching precision reaches 0.01 mm, the defect detection sensitivity is 30 [mu] m, the power consumption is only 12 W, and the system is suitable for various industrial scenes.

Description

Technical Field

[0001] The present invention relates to the field of industrial cameras, and more particularly, to a panoramic stitching industrial camera device and a stitching method for complex scenarios. Background Art

[0002] An industrial camera is an image acquisition device designed specifically for industrial environments and is used in scenarios such as machine vision, automated inspection, and monitoring. It usually features high resolution, high frame rate, strong durability, and high anti-interference ability, and can work stably in complex environments. Common applications include production line quality inspection, robot navigation, and logistics sorting.

[0003] Traditional industrial panoramic imaging systems have significant technical bottlenecks in aspects such as multi-camera synchronization, environmental adaptability, and real-time processing: when using software triggering or discrete hardware triggering, the synchronization error between cameras generally exceeds 100 microseconds, resulting in misalignment artifacts of ≥0.1 mm in stitched images in high-speed motion scenarios (such as a 1 m / s conveyor belt), unable to meet the requirements of precision inspection; sudden changes in light (such as the instantaneous light intensity in arc welding > 10^5 lux) and mechanical vibrations (the vibration energy ratio in the 5 - 500 Hz frequency band > 30%) in complex industrial environments can cause local overexposure, out-of-focus blur, and cumulative geometric distortion. Existing automatic exposure and anti-shake technologies have a response delay > 200 ms and are not coordinated with the stitching algorithm, resulting in a stitching failure rate of dynamic scenes as high as over 15%; when the image processing link relies on the CPU / GPU architecture, the SIFT feature matching and multi-band fusion algorithms take more than 500 ms to process 4K resolution images, and the power consumption > 50 W, making it difficult to meet the real-time and mobile deployment requirements of industrial on-line inspection. In addition, traditional uniformly distributed camera arrays are prone to field-of-view blind spots when detecting curved surfaces or complex structures, and insufficient resolution in the central area leads to an increase in the missed detection rate of micron-level defects. Summary of the Invention

[0004] The main object of the present invention is to provide a panoramic stitching industrial camera device and a stitching method for complex scenarios to solve the problems in the related art.

[0005] To achieve the above object, according to one aspect of the present invention, there is provided a panoramic stitching industrial camera device and a stitching method for complex scenarios, including a multi-camera array including at least two industrial cameras, and the viewing angles of each of the industrial cameras form an overlapping area to cover a target scene; a synchronization control module connected to the multi-camera array for generating a high-precision hardware trigger signal to synchronously trigger all the industrial cameras in the multi-camera array to simultaneously acquire images with a synchronization error of less than 1 microsecond; an image processing module connected to the multi-camera array, and the image processing module includes an FPGA processor, and the FPGA processor is used for: Receive multiple images synchronously acquired by the multi-camera array in real time; Execute image preprocessing, image alignment based on feature point matching, and image fusion algorithms inside the device in real time to generate a panoramic image of the target scene; A calibration module, connected to the multi-camera array and / or the image processing module, is used to monitor at least one environmental condition in real time, and dynamically adjust the imaging parameters of at least one industrial camera in the multi-camera array according to the monitored environmental conditions to maintain image quality and stitching accuracy; A communication module, connected to the image processing module, is configured to transmit the generated panoramic image to an external device.

[0006] Further, the synchronization control module is further configured to receive an external trigger signal input, and synchronously trigger the multi-camera array based on the external trigger signal.

[0007] Further, the environmental conditions include monitoring changes in light intensity and / or device vibration.

[0008] Further, the image alignment algorithm based on feature point matching executed by the FPGA processor is the Scale-Invariant Feature Transform (SIFT) algorithm or its optimized variant; the image fusion algorithm is a multi-band fusion algorithm for eliminating stitching artifacts.

[0009] Further, the industrial cameras in the multi-camera array are distributed in a matrix, the multi-camera array includes a plurality of industrial cameras, the industrial cameras are installed on the support backplane of the support frame, and are arranged in at least a two-dimensional matrix or grid structure, and the camera spacing in the central area of the matrix or grid structure is smaller than the camera spacing in the edge area.

[0010] Further, the industrial cameras in the multi-camera array are distributed in a matrix, the multi-camera array includes a plurality of industrial cameras, the industrial cameras are installed on the support backplane of the support frame, and are arranged in at least a two-dimensional matrix or grid structure, and the camera spacing in the central area of the matrix or grid structure is smaller than the camera spacing in the edge area.

[0011] A method for generating a panoramic image using the above device, comprising the following steps: Use the synchronization control module to generate a high-precision hardware trigger signal, and synchronously trigger all industrial cameras in the multi-camera array to simultaneously acquire images of their respective fields of view with a synchronization error of less than 1 microsecond; Transmit the synchronously acquired multiple images to the image processing module in real time; Inside the device, the FPGA processor in the image processing module performs the following operations in real time: Perform image preprocessing on the received multiple images; Align adjacent images based on the feature point matching algorithm; Use the image fusion algorithm to fuse the aligned images into a panoramic image; During the image acquisition and processing, use the calibration module to monitor at least one environmental condition in real time, and dynamically adjust the imaging parameters of at least one industrial camera in the multi-camera array according to the monitored environmental conditions; Transmit the generated panoramic image to an external device through the communication module.

[0012] Further, the synchronous triggering step further includes: receiving an external trigger signal and performing synchronous triggering based on the external trigger signal.

[0013] Further, the step of dynamically adjusting the imaging parameters includes: monitoring the change in light intensity and / or the vibration of the device in real time, and correspondingly dynamically adjusting the exposure time and / or the autofocus parameters of the industrial camera.

[0014] Further, the alignment step based on the feature point matching algorithm uses the scale-invariant feature transform algorithm or its optimized variant; the image fusion step uses the multi-band fusion algorithm to eliminate stitching traces.

[0015] Compared with the prior art, the present invention has the following beneficial effects: Through the sub-microsecond-level hardware synchronous triggering architecture (synchronous error < 0.8 μs) and the optimized layout design of the ring / matrix camera array (horizontal overlap rate ≥ 30%, vertical overlap rate ≥ 15%), the present invention reduces the stitching misalignment error of the high-speed motion scene to less than 0.01 mm and eliminates 99.2% of the ghost artifacts; combined with multi-modal sensing fusion (light / vibration / ranging) and the deep reinforcement learning decision algorithm, the present invention achieves a light mutation response time ≤ 18 ms and a vibration compensation accuracy ≤ 0.02 g RMS, and maintains a stitching success rate of 98.7% in a strong interference environment; by adopting the FPGA hardened pipeline processing technology (feature matching delay ≤ 35 ms @ 4K) and the non-uniform matrix distribution (the resolution density in the central area is increased by 3.2 times), while reducing the system power consumption to 12W, the defect detection sensitivity reaches the 30-μm level; through the adaptive noise Kalman filter and the centripetal gazing optical layout, the present invention reduces the field of view blind area by 82% in the curved surface target detection, ensuring the integrity and measurement accuracy of panoramic imaging in complex industrial scenarios. Description of the Drawings

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0017] The structures, ratios, sizes, etc. shown in the drawings of this specification are only used to cooperate with the content disclosed in the specification for those who are familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Therefore, they do not have substantial technical significance. Any modification of the structure, change of the proportional relationship, or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope that can be covered by the technical content disclosed in the present invention.

[0018] Figure 1 It is a system block diagram of a preferred embodiment of the present invention; Figure 2 It is a schematic diagram of a calibration module of a preferred embodiment of the present invention; Figure 3 It is a schematic diagram of the arrangement of a multi-camera array of a preferred embodiment of the present invention; Figure 4 It is a partial view of a multi-camera array of a preferred embodiment of the present invention; Figure 5 It is a schematic diagram of the arrangement of a multi-camera array of another preferred embodiment of the present invention.

[0019] Reference numerals: 1, multi-camera array; 2, image processing module; 3, synchronization control module; 4, calibration module; 401, multi-modal sensing unit; 402, data fusion processing unit; 403, decision-making unit; 404, dynamic execution unit; 5, communication module. Detailed implementation manners

[0020] To make the invention purposes, features, and advantages of the present invention more obvious and understandable, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0021] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by terms such as "upper", "lower", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the present invention. It should be noted that when a component is considered to be "connected" to another component, it can be directly connected to the other component or there may be intermediate components present.

[0022] The technical solution of the present invention will be further described below in conjunction with the drawings and through specific embodiments.

[0023] A panoramic stitching industrial camera device for complex scenarios, as Figure 1 shown, includes: A multi-camera array 1, including at least two industrial cameras, and the viewing angles of each industrial camera form an overlapping area to cover the target scenario; A synchronization control module 3, connected to the multi-camera array 1, for generating a high-precision hardware trigger signal to synchronously trigger all the industrial cameras in the multi-camera array 1 to collect images simultaneously with a synchronization error less than 1 microsecond; The industrial cameras in the multi-camera array 1 are distributed in a ring shape, or the industrial cameras are distributed in a matrix.

[0024] When the industrial cameras are distributed in a ring shape: At least two groups of industrial cameras are included, and the industrial cameras are installed along the circumferential direction of the support frame; Among them, the optical centers of the first group of industrial cameras are located in the same plane, and their optical axes have a first preset inclination angle with respect to the normal direction of the plane. The first preset inclination angle is not zero, thereby defining a first vertical field-of-view deviation; The optical centers of the second group of industrial cameras are located in the same plane or another plane parallel thereto, and their optical axes have a second preset inclination angle with respect to the normal direction of the plane. The second preset inclination angle is different from the first preset inclination angle and has the opposite direction, defining a second vertical field-of-view deviation different from the first vertical field-of-view deviation; The arrangement of the industrial cameras ensures that between adjacent cameras along the circumferential direction, and between the first group of cameras and the second group of cameras in the vertical direction, their respective fields of view (FOVs) form an overlapping area and the overlapping area of the fields of view of adjacent cameras is not less than 20%, so as to support subsequent image stitching.

[0025] The first group of cameras and the second group of cameras are arranged in a staggered manner along the circumferential direction of the support frame.

[0026] Specifically, in this embodiment, taking scenario one (such as Figure 3 、Figure 4 As shown in the figure: continuous and dead - angle - free visual inspection is carried out on the inner wall of a large vertical cylindrical storage tank (diameter 10 meters, height 15 meters) to detect defects such as corrosion, cracks, and attachments. It is made of high - strength aluminum alloy or carbon fiber and precision - machined into a circular frame with a diameter of 1.5 meters. This frame is placed in the central area inside the storage tank through a central support rod.

[0027] Eight industrial cameras (N = 8) are selected. Each camera is equipped with: Sensor: Sony IMX264 (5 million pixels). Lens: Focal length 6mm, providing a horizontal field - of - view (HFOV) of about 60° and a vertical field - of - view (VFOV) of 45°.

[0028] The eight cameras are evenly distributed along the circular frame, and one camera of the same group is installed every 45°. For cameras 1, 3, 5, 7: the optical axis is tilted upward by α = 12° relative to the frame plane (horizontal plane). These cameras mainly cover the upper - middle area of the inner wall of the storage tank and part of the top - cover area. For cameras 2, 4, 6, 8: the optical axis is tilted downward by β=-12° relative to the frame plane (horizontal plane). These cameras mainly cover the lower - middle area of the inner wall of the storage tank and part of the bottom - plate area. According to the height - diameter ratio of the storage tank and the key detection areas, the tilt angles α and β can be adjusted in the range of 8° to 20°. A larger tilt angle can cover a wider vertical range, but may increase the splicing difficulty and edge distortion.

[0029] Horizontal direction: between adjacent cameras (such as 1 and 2, 2 and 3), ensure a 30% - 35% field - of - view overlap. Calculation basis: Overlap angle = HFOV - 360° / N = 60° - 45° = 15°. Overlap rate=(15° / 60°)*100% = 25%. To ensure robustness, the actual installation can slightly increase the overlap to more than 30%.

[0030] Vertical direction: The lower edge of the field - of - view of the upward - tilted group (1, 3, 5, 7) and the upper edge of the field - of - view of the downward - tilted group (2, 4, 6, 8) form at least a 10% - 15% vertical field - of - view overlap band at a certain distance in front of the cameras (such as the tank - wall position) to ensure seamless splicing in the vertical direction. This overlap width is related to the tilt angle, the camera VFOV, and the working distance.

[0031] On the outer side of the circular frame, at positions of 0°, 120°, and 240°, one set of multi - modal sensing units (TSL2561, AS7262, MPU6050) is installed each to form a ring - shaped sensor network. The sensors are firmly installed to reduce self - vibration interference.

[0032] In another embodiment, when the industrial cameras are distributed in a matrix: It includes a plurality of industrial cameras. The industrial cameras are installed on a support backplane and arranged in at least a two - dimensional matrix or grid structure; Among them, the spatial distribution density of the industrial cameras on the support backplane is non-uniform. Specifically, the distance between the cameras located in the central region of the matrix or grid structure is smaller than that between the cameras located in the edge region. Among them, the optical axes of a plurality of industrial cameras generally point to a common distant or target area, but are not strictly parallel. Specifically, for each or at least a part of the industrial cameras, there is a small preset inclination angle inwardly pointing to the central normal direction with respect to a central normal direction of the array, forming a centripetal convergence pointing characteristic. Moreover, the arrangement of the industrial cameras ensures that in the matrix or grid structure, the field of view (FOV) between each camera and its adjacent cameras forms an overlapping area to support subsequent image stitching.

[0033] Other embodiments are in Scenario 2 (as Figure 5 shown): High-precision and high-efficiency surface defect detection (such as scratches, sand holes, machining residues) is performed on the top surface of an automotive engine cylinder block installed on a production line. The detection area is approximately 80 cm x 60 cm, and key central areas (such as cylinder hole edges, sealing joints) need to be carefully inspected.

[0034] Nine industrial cameras (3x3 matrix) are selected, and each camera is equipped with: Sensor: OnSemi Python12K (12.5 million pixels), providing higher resolution. Lens: Focal length 12 mm, providing a horizontal field of view (HFOV) of approximately 30° and a vertical field of view (VFOV) of 22°. A longer focal length is used to obtain higher details.

[0035] An aluminum alloy or steel precision-machined backplane with sufficient thickness (such as 15 mm) is used, with dimensions of approximately 50 cm x 40 cm, ensuring high rigidity and thermal stability. The backplane is installed at the end of the robotic arm or on a fixed bracket at the inspection station.

[0036] Camera arrangement and inclination: Central camera (C): Located at the center of the backplane.

[0037] Surrounding cameras (M1 - M4): Surround the central camera, with a horizontal / vertical spacing d_center = 12 cm from the central camera.

[0038] Corner cameras (E1 - E4): Located at the four corners of the matrix, with a horizontal / vertical spacing d_edge = 18 cm from the adjacent surrounding cameras. (That is, the cameras in the central region are denser).

[0039] Centripetal gazing setting (taking a distance of 1 meter from the detection target as an example): Central camera (C): The optical axis is perpendicular to the backplane, γ = 0°.

[0040] Surround cameras (M1 - M4): The optical axes are tilted inward, pointing to the center of the array field of view, with a tilt angle γ = 1.5°.

[0041] Corner cameras (E1 - E4): The optical axes are tilted inward, pointing to the center of the array field of view, with a tilt angle γ = 2.5°.

[0042] Consideration of the tilt angle range: The range of the tilt angle γ is usually between 0.5° and 5°. The specific value depends on the working distance, the target depth / curvature, and the desired improvement in the edge viewing angle. The closer the working distance and the greater the target curvature, the larger the tilt angle may be required.

[0043] Overlap rate: Central region (between C and M1 - M4, between M and adjacent M): Ensure a high overlap rate of 40% - 50% to provide data for high-precision stitching and possible super-resolution reconstruction.

[0044] Edge region (between M and E, between E and adjacent E): Ensure a standard overlap rate of 25% - 35% to ensure complete coverage and stable stitching.

[0045] Sensor deployment (calibration module 401): Install 3 - 4 groups of multi-modal sensing units near the four corner points of the backplane (outside the E1 - E4 area) and near the midpoints of the upper and lower long sides to form a sensing network around the array.

[0046] Image processing module 2, connected to the multi-camera array 1. The image processing module 2 includes an FPGA processor, and the FPGA processor is used for: Receiving in real time multiple images synchronously acquired by the multi-camera array 1; Performing in real time image preprocessing, image alignment based on feature point matching, and image fusion algorithms inside the device to generate a panoramic image of the target scene; Specifically, configure independent physical reception interfaces for each industrial camera in the multi-camera array 1 on the FPGA. According to the specific output of the industrial camera (such as MIPI CSI-2, LVDS, Camera Link, GigE Vision, etc.), implement the corresponding physical layer (PHY) and link layer / protocol layer controllers.

[0047] The specific implementation steps are as follows: Input: The original image data stream from the input buffers of each camera.

[0048] Processing unit (in sequential pipelining): a. Bad pixel correction: Use the bad pixel map pre-stored in the FPGA internal memory (such as BRAM) to perform interpolation replacement on the detected bad pixels (dead pixels, bright pixels).

[0049] b. Black level correction: Subtract the black level reference value of the corresponding channel from the pixel value. This reference value can be fixed or dynamically fine-tuned by the calibration module 4 and then updated to the FPGA register.

[0050] c. Lens shading correction: Use the LSC look-up table (LUT) or gain map stored in the FPGA BRAM to compensate for the radial and tangential brightness attenuation of each pixel. This LUT should be generated during camera calibration. The dynamic adjustment of the calibration module 4 mainly affects the parameters of the camera itself, and the LSC maintains the calibrated value.

[0051] d. Demosaicing: If the input is in Bayer format, perform the demosaicing algorithm to convert the monochrome sensor data into a full-color (such as RGB) image.

[0052] e. White balance gain: Although the calibration module 4 adjusts the white balance of the camera itself through the dynamic execution unit 404, a digital gain interface can be reserved at the FPGA level. The FPGA receives the digital gain calculated by the calibration module 4 at this stage, which requires post-processing fine-tuning. It mainly relies on the camera's own adjustment, and the FPGA applies a fixed or basic gain.

[0053] f. Color correction matrix: Apply a 3x3 color correction matrix to convert the image color space to a standard color space (such as sRGB), or optimize the color according to the requirements of specific industrial scenarios. The CCM parameters are from calibration.

[0054] g. Gamma correction: Apply a Gamma curve (implemented through a LUT) to adjust the non-linear brightness response of the image to conform to the human eye perception or display device requirements.

[0055] h. Lens distortion correction: This is a crucial step before stitching. Use the distortion model parameters (such as radial and tangential distortion coefficients) stored in the FPGA memory. The FPGA calculates the sub-pixel coordinates of each output pixel corresponding to the input image and obtains the pixel value through bilinear or bicubic interpolation to generate an undistorted image.

[0056] Image alignment based on feature point matching: Input: Multiple pre-processed images stored in the frame buffer.

[0057] Parallel processing (for adjacent overlapping image pairs): a. Feature point detection: In the predefined overlapping regions of each image, an efficient feature point detection algorithm is executed in parallel. Considering FPGA resources and real-time requirements, FAST, Harris, or an optimized version of ORB (Oriented FAST and Rotated BRIEF) corner detectors can be selected. FPGA implementation involves parallel operations such as window sliding, pixel comparison, and threshold judgment.

[0058] b. Feature point descriptor generation: Compute descriptors for each detected feature point. For FPGA, binary descriptors are particularly suitable due to their high computational and matching efficiency. FPGA achieves this by parallelly reading pixel blocks, performing comparisons, and generating bit strings.

[0059] c. Feature point matching: Match between the sets of feature point descriptors in the overlapping regions of adjacent images. FPGA uses a parallel brute-force matcher based on Hamming distance (very efficient for binary descriptors) or approximate nearest neighbor search methods.

[0060] d. Outlier rejection and transformation matrix estimation (RANSAC): Implement the RANSAC algorithm to reject outlier pairs and estimate the geometric transformation relationship (homography matrix H) between images. FPGA can accelerate the RANSAC process by parallelizing hypothesis generation, inlier counting, and model parameter calculation (such as optimized implementations of SVD decomposition or direct linear transformation).

[0061] Output: The exact homography transformation matrix H calculated for each pair of overlapping images. These matrices will be used for the next step of image fusion.

[0062] Output: Generate a multi-channel preprocessed, undistorted, color-corrected image data stream. These data are stored in the frame buffer inside the FPGA (utilizing external high-speed DRAM such as DDR4 / LPDDR4, accessed through the FPGA's memory controller) for use in subsequent stitching steps.

[0063] Image fusion to generate a panorama: Input: Multi-channel preprocessed image data (stored in the frame buffer) and the calculated homography matrix H.

[0064] a. Panorama canvas definition and coordinate mapping: Determine the coordinate system and size of the final panoramic image based on the layout and calibration information of the camera array. The FPGA needs to calculate the coordinates on the panorama canvas corresponding to the source images (after transformation by the H matrix).

[0065] b. Image reprojection: Using the calculated homography matrix H, "project" each source image onto the panoramic canvas. The FPGA implements real-time reprojection through efficient address generation logic and interpolation units (such as bilinear interpolation). Efficient access to the source image data stored in the external DRAM is required.

[0066] c. Exposure / gain compensation: Although the calibration module 4 tries to unify the exposures of the cameras, residual differences may exist. The FPGA can analyze the brightness differences in the overlapping areas and calculate the gain compensation factors at the pixel level to make the transition smoother.

[0067] d. Image stitching and sewing: In the overlapping areas, fuse the pixels from different source images.

[0068] Output: The final panoramic image frame stored in the external DRAM controlled by the FPGA.

[0069] The calibration module 4, connected to the multi-camera array 1 and / or the image processing module 2, is used to monitor at least one environmental condition in real time and dynamically adjust the imaging parameters of at least one industrial camera in the multi-camera array 1 according to the monitored environmental conditions to maintain image quality and stitching accuracy; Although the synchronization control module 3 ensures a synchronization trigger of <1µs, the data transmission path may introduce minor delay differences. The internal logic of the FPGA should be able to receive the accurate trigger timestamp (or the synchronization frame start signal) from the synchronization control module 3 and align each input data stream based on this. And set up an independent input FIFO (First In First Out) buffer for each camera data stream. This can absorb the minor jitters in data transmission and decouple the speed differences of the subsequent processing modules to ensure data integrity. The buffer size needs to be determined according to the camera resolution, frame rate, and the throughput capacity of the FPGA processing pipeline. The received raw data (such as RAW Bayer format) needs to be preliminarily sorted out. The internal logic of the FPGA converts it into a unified internal processing format (such as pixel stream or a specific block / Tile format) for easy pipelining processing.

[0070] Specifically, as Figure 2 shown, the calibration module 4 includes a multi-modal sensing unit 401 for real-time acquisition of light intensity, color temperature, and vibration data; Among them, for light intensity: use a TSL2561 digital light sensor (range 0.1 - 40,000 lux, sampling rate 100Hz); for color temperature: an AMS AS7262 multi-spectral sensor (6-channel visible light detection); for vibration: an MPU6050 six-axis sensor (±16g acceleration, ±2000° / s angular velocity); Deployment strategy: form a ring-shaped sensor network around the camera array, and arrange a group of sensing nodes every 120 degrees.

[0071] And mark all sensor data with the global clock signal (accuracy ±10 ns) of the FPGA, establish a three-dimensional coordinate system conversion model, and map the sensor data to the camera optical center coordinate system.

[0072] The data fusion processing unit 402 realizes multi-source information fusion based on the improved Kalman filter; First, perform state space modeling: ; Among them, is the environmental light intensity at time k (unit: lux); is the environmental color temperature at time k (unit: Kelvin); is the three-axis acceleration at time k (unit: m / s²); is the three-axis angular velocity at time k (unit: rad / s).

[0073] State transition equation: ; Among them, is the state transition matrix (diagonal matrix, elements are attenuation factors); is the control input matrix; is the external control input; is the process noise.

[0074] Establish a sensor observation equation: ; Among them, is the sensor observation vector; is the observation matrix; is the observation noise.

[0075] The observation noise covariance (initial value) is: ; , the light sensor noise variance (lux²); , the color temperature sensor noise variance (K²); , the accelerometer noise variance ((m / s²)²); , the gyroscope noise variance ((rad / s)²).

[0076] And use adaptive noise covariance for updating; Process noise adaptation:

[0077] Among them, , representing the degree of historical dependence of process noise; ; is the state covariance matrix at the previous moment.

[0078] Observation noise adaptation:

[0079] , representing the temperature compensation coefficient; is the sensor temperature (°C).

[0080] Correlate light intensity and color temperature:

[0081] Among them, is the correlation decay constant, (spectral distribution difference, is the 6-channel spectral data of AS7262); , representing the spectral difference tolerance.

[0082] The correlation weight matrix is ; Finally, enter the main loop of the improved Kalman filter; Prediction: ;

[0083] Kalman gain:

[0084] represents the Hadamard product; State update: ;

[0085] Outlier suppression: .

[0086] The data fusion processing unit 402 can make the absolute error of light intensity measurement ≤ 3% and the fusion error of vibration parameters ≤ 0.02 g (RMS) by dynamically adjusting the noise parameters and data correlation weights through the above method, meeting the industrial-grade accuracy requirements.

[0087] The decision-making unit 403 generates a camera parameter adjustment strategy; specifically as follows: Taking the environmental state fused by the data fusion processing unit 402 as the input, calculate the real-time change characteristics of light intensity, vibration, and color temperature, and generate a parameter adjustment amount according to the preset rules: Light intensity change rate: , (unit: lux / s); Acceleration vibration energy: , (unit: m / s2); Color temperature deviation from the reference value: , ; Exposure time adjustment rule:

[0088] represents the light sensitivity coefficient. In this embodiment ; represents the maximum range, , is the exposure time (ms) of the current camera, is the new exposure time after adjustment.

[0089] Constraint condition: The change in exposure time does not exceed ±30%.

[0090] Next, perform focus compensation calculation:

[0091] Perform Fourier transform on the velocity data to calculate the power spectral density , and then integrate by frequency band: The compensation amplitude is large in the low frequency band (5 - 200 Hz) and small in the high frequency band (201 - 500 Hz); The white balance adjustment strategy is:

[0092] Constraint condition: The red / blue channel gain ratio is restricted between 0.7 and 2.0.

[0093] Each part in the adjustment strategy is controlled by dynamic weight distribution:

[0094] According to the type of environmental change, dynamically allocate the priority weights for parameter adjustment. When the light changes violently (ΔL k > 1000): Adjust the exposure first (weight 60%), then focus (30%), and finally white balance (10%); When there is strong vibration (E_vib > 2g): Give priority to focus compensation (weight 50%).

[0095] The dynamic execution unit 404 realizes the closed-loop control of the exposure, focal length, and white balance parameters. Convert the parameter adjustment amount generated by the decision unit 403 strategy into the actual camera control signal and achieve stable adjustment.

[0096] Exposure time control: Adopt a PID controller:

[0097] Among them, (light error), according to the IMX sensor optimization coefficient: , , When the environment changes suddenly, it automatically switches to the Bang-Bang control mode to accelerate convergence.

[0098] The focus position compensation adopts a stepped adjustment algorithm:

[0099] The maximum adjustment per frame is 50 steps to avoid out-of-step, and the vibration compensation direction is opposite to the acceleration direction.

[0100] White balance closed-loop calibration: 1. Obtain the white point coordinates (u, v) of the current image; 2. Calculate the color difference: ; 3. Iterative adjustment:

[0101] The control timing is as follows: 0 - 2 ms: Read the fused data ; 2 - 5 ms: Execute the policy generation calculation; 5 - 10 ms: Send control instructions to the camera; 10 - 16 ms: Wait for the execution to complete and collect the feedback image.

[0102] It can complete a complete closed-loop from environmental perception to parameter adjustment within 50 ms, ensuring sub-pixel-level stitching accuracy even under drastic change conditions (such as 1000 lux / s light mutation and 5 g vibration).

[0103] The communication module 5 is connected to the image processing module 2 and is configured to transmit the generated panoramic image to an external device.

[0104] The synchronization control module 3 is also used to receive an external trigger signal input and synchronously trigger the multi-camera array 1 based on the external trigger signal.

[0105] The environmental conditions include monitoring the change of light intensity and / or the vibration of the device.

[0106] The image alignment algorithm based on feature point matching executed by the FPGA processor is the Scale-Invariant Feature Transform algorithm or its optimized variant; the image fusion algorithm is the multi-band fusion algorithm for eliminating stitching traces.

[0107] It should be noted that in this embodiment, an optimized variant of the Scale-Invariant Feature Transform (SIFT) algorithm can accelerate feature detection through integral images and Hessian matrix approximation, maintaining scale invariance while significantly reducing computational complexity; A method for generating a panoramic image, comprising the following steps: Use the synchronization control module 3 to generate a high-precision hardware trigger signal to synchronously trigger all industrial cameras in the multi-camera array 1 with a synchronization error of less than 1 microsecond to simultaneously capture images of their respective fields of view; Transmit the multiple synchronously captured images to the image processing module 2 in real time; Inside the device, the FPGA processor in the image processing module 2 performs the following operations in real time: Perform image preprocessing on the received multiple images; Align adjacent images based on the feature point matching algorithm; Use the image fusion algorithm to fuse the aligned images into a single panoramic image; During the image acquisition and processing, use the calibration module 4 to continuously monitor at least one environmental condition and dynamically adjust the imaging parameters of at least one industrial camera in the multi-camera array according to the monitored environmental conditions; Transmit the generated panoramic image to an external device through the communication module 5.

[0108] The synchronous triggering step further includes: receiving an external trigger signal and performing synchronous triggering based on the external trigger signal.

[0109] The step of dynamically adjusting the imaging parameters includes: continuously monitoring changes in light intensity and / or device vibration, and accordingly dynamically adjusting the exposure time and / or autofocus parameters of the industrial camera.

[0110] The alignment step based on the feature point matching algorithm uses the Scale-Invariant Feature Transform (SIFT) algorithm or its optimized variant; the image fusion step uses the multi-band fusion algorithm to eliminate stitching artifacts.

[0111] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0112] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0113] The above are only the preferred embodiments of the present invention, and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A panoramic stitching industrial camera device for complex scenarios, characterized in that, Comprising: A multi-camera array (1), including at least two industrial cameras, where the viewing angles of each of the industrial cameras form an overlapping area to cover a target scene; A synchronization control module (3), connected to the multi-camera array (1), for generating a high-precision hardware trigger signal to synchronously trigger all the industrial cameras in the multi-camera array (1) to simultaneously capture images with a synchronization error less than 1 microsecond; An image processing module (2), connected to the multi-camera array (1), the image processing module (2) includes an FPGA processor, and the FPGA processor is used for: Receiving in real time multiple images synchronously captured by the multi-camera array (1); Performing image preprocessing, image alignment based on feature point matching, and an image fusion algorithm inside the device in real time to generate a panoramic image of the target scene; A calibration module (4), connected to the multi-camera array (1) and / or the image processing module (2), for monitoring at least one environmental condition in real time and dynamically adjusting the imaging parameters of at least one industrial camera in the multi-camera array (1) according to the monitored environmental condition; A communication module (5), connected to the image processing module (2), configured to transmit the generated panoramic image to an external device.

2. The device according to claim 1, characterized in that, The synchronization control module (3) is further used for receiving an external trigger signal input and synchronously triggering the multi-camera array (1) based on the external trigger signal.

3. The device according to claim 1, wherein The environmental condition includes monitoring the change in light intensity and / or the vibration of the device.

4. The device according to claim 1, characterized in that, The image alignment algorithm based on feature point matching executed by the FPGA processor is the scale-invariant feature transform algorithm or its optimized variant; the image fusion algorithm is the multi-band fusion algorithm for eliminating stitching artifacts.

5. The device according to claim 1, characterized in that, The industrial cameras in the multi-camera array (1) are distributed in a ring shape, the multi-camera array (1) includes at least two groups of industrial cameras, and the industrial cameras are installed along the circumferential direction of the support frame; Among them, the optical centers of the first group of industrial cameras are located in the same plane, and their optical axes have a first preset inclination angle with respect to the normal direction of the plane, and the first preset inclination angle is not zero; The optical centers of the second group of industrial cameras are located in the same plane or another plane parallel thereto, and their optical axes have a second preset inclination angle with respect to the normal direction of the plane, the second preset inclination angle is different from the first preset inclination angle and has the opposite direction.

6. The device according to claim 1, wherein The industrial cameras in the multi-camera array (1) are distributed in a matrix, the multi-camera array (1) includes a plurality of industrial cameras, the industrial cameras are installed on the support backplane of the support frame and are arranged in at least a two-dimensional matrix or grid structure, and the camera spacing in the central area of the matrix or grid structure is smaller than the camera spacing in the edge area.

7. A method for generating a panoramic image using the device according to any one of claims 1 to 6, characterized in that, Including the following steps: Using the synchronization control module (3) to generate a high-precision hardware trigger signal to synchronously trigger all the industrial cameras in the multi-camera array (1) to simultaneously capture images of their respective fields of view with a synchronization error less than 1 microsecond; Transmitting the synchronously captured multiple images to the image processing module (2) in real time; Inside the device, the following operations are performed in real time by the FPGA processor in the image processing module (2): Perform image preprocessing on the received multiple images; Align adjacent images based on the feature point matching algorithm; Use the image fusion algorithm to fuse the aligned images into a panoramic image; During the image acquisition and processing, use the calibration module (4) to monitor at least one environmental condition in real time, and dynamically adjust the imaging parameters of at least one industrial camera in the multi-camera array according to the monitored environmental conditions; Transmit the generated panoramic image to an external device through the communication module (5).

8. The method according to claim 7, wherein The synchronous triggering step further includes: receiving an external trigger signal and performing synchronous triggering based on the external trigger signal.

9. The method according to claim 7, wherein The step of dynamically adjusting the imaging parameters includes: monitoring the change in light intensity and / or the vibration of the device in real time, and dynamically adjusting the exposure time and / or the autofocus parameters of the industrial camera accordingly.

10. The method according to claim 7, wherein The alignment step based on the feature point matching algorithm uses the scale-invariant feature transform algorithm or its optimized variant; the image fusion step uses the multi-band fusion algorithm to eliminate stitching artifacts.

Images