Target detection system and method based on image stitching technology

Abstract

The invention discloses a target detection system and method based on an image splicing technology, and relates to the field of optics, image processing and machine vision. Comprising a software system module, an edge computing terminal module, a zoom lens and camera combination module, a lifting and rotating combination module, a light source module and an object placement module, the zoom lens module and the camera module are controlled by the edge computing terminal module to shoot an object in real time, and a corresponding shooting scheme is formulated according to a detected industrial object. A detected object is comprehensively shot according to the shooting angle, the focal length, the shooting position and the distance, incomplete collection of industrial object data is prevented, then the edge computing terminal carries out panoramic image splicing on images shot by a camera in real time through a panoramic image splicing module in a control software system module, and the panoramic image splicing module carries out panoramic image splicing. And finally, a target detection module in the software system module performs target detection on the panoramic image, namely, a target position and a target category are detected, so that the target detection efficiency is remarkably improved, and the capability of obtaining the full view of the object is improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of optics, image processing and machine vision, and in particular to an object detection system and method based on image stitching technology. Background Art

[0002] Existing object detection systems and methods based on machine vision mainly detect a part of a certain device, and there are problems such as a small detection field of view and low detection efficiency, and large-field object detection cannot be achieved. For example, when existing object detection methods based on machine vision detect a large field of view, they are usually limited by the camera shooting range, and use corresponding object detection algorithms to identify the collected images. However, this method cannot achieve large-field object detection and lacks the ability to recognize the entire field of view; that is, currently, the method of using a single image to detect the target area is limited by the camera shooting height and camera focal length, and a single image cannot cover the entire target detection area, resulting in a decrease in detection efficiency. Secondly, when existing object detection systems detect large-scale devices, they often take pictures of the target area by fixing the camera, which leads to the need to design a dedicated appearance object detection system for specific devices when replacing other devices; for example, a dedicated defect detection system for lithium battery electrodes and a dedicated object detection method for beverage liquids and bottle caps. These object detection systems can only detect specific types of specific devices, resulting in limited detection capabilities. Summary of the Invention

[0003] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide an object detection system and method based on image stitching technology to achieve the ability of large-field detection, especially for object detection systems for large-scale devices.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions: An object detection system based on image stitching technology, comprising: a software system module, an edge computing terminal module, a zoom lens and camera combination module, a lifting and rotating combination module, a light source module, and an object placement module; The software system module includes: an object image stitching module and an object appearance detection module. Among them, the image stitching module is used to stitch panoramic images of multiple images to be taken to obtain the entire field-of-view appearance of industrial products. Subsequently, the object appearance detection module detects the panoramic image, that is, detects the target position and target category; The edge computing terminal module is used to deploy the software system module to better apply to industrial scenarios; The described zoom lens and camera combination module adjusts the lens focal length for real-time shooting of industrial appearance images at different magnifications. The images captured by this module are transmitted to the image stitching system module for panoramic image stitching; The described lifting and rotating module is used to adjust the vertical position and in-plane rotation angle of the zoom lens and camera combination module to better adapt to the appearance detection of different industrial devices; The described light source module is used to provide good lighting conditions to obtain high-resolution images with good brightness and uniformity; The described object placement module is used to place different types of objects to achieve the purpose of detecting different devices.

[0005] Furthermore, the described object detection system based on image stitching technology further includes a display.

[0006] Furthermore, the described software system module contains a pre-trained object detection model; For the described object detection model, different object detection algorithms can be selected according to different detection devices. Taking the improved YOLOv8 algorithm as an example, specifically, by adding a channel attention mechanism module (SE, Squeeze-and-Excitation Networks) to the Neck module in the original algorithm to obtain the importance of each image in the feature map, and then using this importance to assign a weight value to each feature, so that the neural network focuses on certain feature channels, enhances the channels useful for the current task, and suppresses the channels that are not very useful for the current task. That is, before adding the attention mechanism, the network structure considers that the importance of each channel is the same. After adding SENet, the importance of each channel becomes different, which makes the model pay more attention to the channels with large weights, thereby enhancing the feature extraction ability of the model.

[0007] An object detection system and method based on the above-mentioned image stitching technology includes: a method for stitching images, a software system integration method, and a software system edge deployment method.

[0008] The described image stitching method adopts three methods according to the image type. The first method is the traditional image stitching method, including: image input, image stitching preprocessing, feature point extraction, feature descriptor generation, feature matching, homography matrix calculation, and stitching. The second method is the image stitching method based on deep learning, including: image input, rough image alignment, ablation constraint, convolution, image downsampling, and image stitching. The third method is a combination of the image stitching method based on deep learning and image optimization, where image optimization includes: edge noise suppression, image edge rectification, and gap stitching.

[0009] The described software system integration method includes: initializing the system, configuring system parameters, selecting an input source, data processing, object appearance detection algorithm, image stitching algorithm, image stitching target detection and display, interactive interface, end interface, and outputting detection results.

[0010] The described edge computing module includes: a core control board, a computing power accelerator, a radiator, a core control board housing, a memory card, an HDMI cable, a USB cable, a power cord, and a display.

[0011] Furthermore, the core control board, computing power calculator, radiator, core control board housing, memory card, HDMI cable, USB cable, and power cord together form a unit for loading pre-trained image stitching algorithm models and object detection algorithm models.

[0012] Furthermore, the display is used to display the target detection results in real time.

[0013] The beneficial effects of the present invention are: providing a target detection system based on image stitching technology, which can achieve a relatively high target detection field of view and have good resolution, especially for target detection systems of different devices. Description of the Drawings

[0014] Figure 1 It is a schematic structural diagram of the target detection system based on image stitching technology of the present invention.

[0015] Figure 2 It is a schematic diagram of the target detection algorithm provided in the present invention.

[0016] Figure 3 It is a schematic diagram of the image stitching algorithm provided by the present invention.

[0017] Figure 4 It is a schematic diagram of the software system module provided in an embodiment of the present invention.

[0018] Figure 5 It is a schematic diagram of the target detection result of image stitching of a car photographed by a drone.

[0019] Figure 6 It is a schematic diagram of the target detection result of image stitching of a football game.

[0020] Figure 7 It is a schematic diagram of the target detection result of image stitching of biomedical CT images provided in the embodiment.

[0021] In the figure: 1. Edge computing terminal module, 2. Software system module, 3. Zoom lens and camera combination module, 4. Six-axis robotic arm module, 5. Light source module, 6. Object placement module. Detailed Embodiments

[0022] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of this specification. It should be noted that the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work belong to the scope of protection of the present invention.

[0023] Please refer to Figures 1-7 , as Figure 1 shown in the structural schematic diagram of the target detection system based on the image stitching technology in the present invention. The system includes: an edge computing terminal module 1, a software system module 2, a zoom lens and camera combination module 3, a six-axis robotic arm module 4, a light source module 5, and an object placement module 6. Specifically: The edge computing terminal module 1 takes pictures of the appearance graphics of the target object to be detected through the control software system module 2, and the edge computing terminal module 1 performs panoramic image stitching on the captured images through the control software system module 2; Further, the control software system module 2 realizes real-time shooting of the target object to be detected by controlling the zoom lens and camera combination module 3; Furthermore, the zoom lens can adjust its focal length through the control software system module 2 to adapt to the shooting of different objects; Further, the control software system module 2 realizes the appearance shooting of different target objects by controlling the lifting and rotation of the lifting and rotation combination module 4.

[0024] Further, the control software system module 2 controls the light source module 5 to adjust the light source brightness and uniformity, so as to obtain an image of the target object to be detected with better brightness and uniformity.

[0025] Further, the control software system module 2 controls the object placement module 6 to adjust the relative angle between the object and the module 3, so as to shoot the angles of objects in different orientations.

[0026] As Figure 2 shown in the schematic diagram of the target detection algorithm provided in the present invention, this algorithm adopts the YOLOv8 algorithm with an added attention mechanism. The added attention mechanism algorithm is only one way of the embodiments of the present invention. For the appearance detection of different objects, the algorithm can be improved, such as the YOLO-Word algorithm; Further, the YOLOv8 network architecture algorithm with an added attention mechanism includes: a feature extraction module (Backbone), a feature fusion module (Neck), a detection head module (Head), an attention mechanism module (SeAttention), and other optimization modules (Anchor optimization, Soft-NMS module) Furthermore, the feature extraction module is responsible for extracting features from the input image, converting the input image into multi-layer feature maps for subsequent object detection tasks.

[0027] Furthermore, the feature fusion module is responsible for multi-scale fusion of the feature maps and passing these features to the prediction layer.

[0028] Furthermore, the detection head module is responsible for final regression prediction and classification. According to different feature maps, detection heads of different sizes are used to predict the object bounding boxes.

[0029] Furthermore, the attention mechanism module is responsible for enhancing the multi-scale information extraction ability of the network.

[0030] Furthermore, other optimization modules use Soft-NMS to improve the problem of densely overlapping cells.

[0031] Furthermore, the attention mechanism module adds a channel attention mechanism module (SE, Squeeze-and-Excitation Networks) to the Neck module in the original algorithm to obtain the importance of each image in the feature map, and then assigns a weight value to each feature with this importance, so that the neural network focuses on certain feature channels. Channels useful for the current task are enhanced, and channels less useful for the current task are suppressed. That is, before adding the attention mechanism, the network structure considers that the importance of each channel is the same. After adding SENet, the importance of each channel becomes different, so that the model pays more attention to the channels with large weights, thus improving the feature extraction ability of the model.

[0032] As Figure 3 shown is a schematic diagram of the image stitching algorithm provided by the present invention. The algorithm architecture includes: image stitching and image optimization. Among them, image stitching consists of a warping transformation and a combination module. The warping transformation part includes two or several images with overlapping regions. The to-be-stitched images I1 and I2 with overlapping regions are input to the combination module through warping transformation, and a highly robust warped image is formed through a residual network, global transformation, and local transformation. Subsequently, the warped image (I W1 ,I W2 ) is input to the second module to predict the synthesis masks (M C1 ,M C2 ). The finally stitched image can be expressed in the following form as, (1) After stitching two images using the existing image stitching method, the stitched image has a non-rectangular edge problem (i.e., the stitched image is not rectangular), which leads to errors in object detection. To solve this problem, a motion diffusion model and image diffusion are introduced to optimize the stitched image. Through image optimization, the irregular boundaries after stitching can be corrected into rectangles.

[0033] Furthermore, the motion diffusion model generates a motion field, converting the stitched image with irregular edges and white boundaries into a seamless rectangular form, eliminating these boundaries. Since the motion diffusion model introduces noise and morphological transformations due to the jitter of the motion field and the complexity of the remapping operation, a content diffusion model is introduced again to improve artifacts and refine the image.

[0034] Furthermore, the sequential image input module is responsible for processing the images collected in real time, converting the video into a frame-by-frame image sequence; Furthermore, the image conversion module is responsible for processing the frame-by-frame image sequence and converting it into a unified image format; Even further, to achieve real-time image acquisition, the present invention acquires image data in real time to facilitate processing by the network model; Even further, to achieve real-time image stitching and detection, the present invention deploys the object detection model and the image stitching model to the edge computing terminal; Even further, the edge computing terminal includes: Raspberry Pi 5 (8G), Raspberry Pi 5 CLB AI DP HAT+ (26TOPS) computing power, 256G solid-state drive, AI module radiator, copper pillar bracket, power supply, Mirco to HDMI cable, 128G TF card, network cable, fixing screws, and housing; As Figure 4 Schematic diagram of the software system module provided in an embodiment. The module includes: initializing system parameters, configuring system parameters, selecting an input source, four upload methods (uploading industrial pictures, uploading folders, uploading videos, starting real-time camera shooting), data processing, YOLO algorithm prediction, panoramic image real-time stitching, target real-time display, table real-time display, interactive interface, end interface, result export, end.

[0035] Furthermore, the described initialization system is used for hardware detection, loading system firmware, and setting the system environment; Furthermore, the described configuration of system parameters is used to configure the parameters necessary for the software to operate; Furthermore, the input source and the four upload methods are for real-time object detection of pictures; Furthermore, the data processing module is used to perform image enhancement and filtering on the acquired data to eliminate the influence of environmental factors on the image; Furthermore, the panoramic image stitching module is used to stitch the input images to obtain a larger panoramic image; Furthermore, the YOLO prediction module is used to load the trained module to perform target prediction on the panoramic image; Furthermore, the target real-time display module is used to display the target detection results of the panoramic image in real time; Furthermore, the table display module is used to display the target positions and target categories in the panoramic image; Furthermore, the interaction interface is used to facilitate user operations; Furthermore, the end interface is used to export and save the target detection position and category information.

[0036] As shown in Figures 5 - 7, the target detection results of the panoramic image provided in the embodiment are shown. It can be seen that the stitched panoramic image not only expands the field of view, but also improves the overall performance of target detection. Using the panoramic image, global positioning of the target can be achieved under a broader field of view, enabling efficient detection and helping to identify the overall morphology of the target. For example, the drone panoramic image shown in Figure 5 can accurately detect cars within a large field of view, significantly expanding the detection area. In contrast, Figure 7 shows the detection of brain tumors based on the panoramic image, which can detect 4 slices simultaneously in the same view, while only frame-by-frame detection can be performed without image stitching, resulting in relatively low detection efficiency.

[0037] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0038] The parts not detailed in the present invention are prior art.

Claims

1. An object detection system based on image stitching technology, comprising: Software system module, edge computing terminal module, zoom lens and camera combination module, lifting and rotating combination module, light source module and object placement module; characterized in that: The software system module includes: an object image stitching module and an object appearance detection module. Among them, the image stitching module is used to stitch multiple images to be taken into a panoramic image to obtain the overall visual appearance of the industrial product. Subsequently, the object appearance detection module detects the panoramic image, that is, detects the target position and target category; The edge computing terminal module is used to deploy the software system module for better application in industrial scenarios; The zoom lens and camera combination module is used to take industrial appearance images of different magnifications in real time by adjusting the lens focal length. The images taken by this module are transmitted to the image stitching system module to stitch panoramic images of images at the same magnification; The lifting and rotating module is used to adjust the up and down position and the rotation angle in the plane of the zoom lens and camera combination module to better adapt to the appearance detection of different industrial devices; The light source module is used to provide good lighting conditions to obtain a resolution image with better brightness and uniformity; The object placement module is used to place different types of objects to achieve the purpose of detecting different devices.

2. The object detection system based on image stitching technology according to claim 1, wherein: The software system module contains a pre-trained object detection model; The object detection model selects different object detection algorithms for different detection devices. Taking the improved YOLOv8 algorithm as an example, specifically, by adding a channel attention mechanism module (SE, Squeeze-and-Excitation Networks) to the Neck module in the original algorithm, the importance of each image in the feature map is obtained, and then weight values are assigned to each feature according to the importance, so that the neural network focuses on certain feature channels, enhances the channels useful for the current task, and suppresses the channels that are not very useful for the current task. That is, before adding the attention mechanism, the network structure considers that the importance of each channel is the same. After adding SENet, the importance of each channel becomes different, so that the model pays more attention to the channels with large weights, thereby enhancing the feature extraction ability of the model.

3. A method for an object detection system based on image stitching technology according to any one of claims 1 to 2, including the following parts: 1): Image stitching method, which adopts three methods according to the image type; Method one is the traditional image stitching method, including: Image input, image stitching preprocessing, feature point extraction, feature descriptor generation, feature matching, homography matrix calculation and stitching; Method two is the image stitching method based on deep learning, including: image input, rough image alignment, ablation constraint, convolution, image downsampling and image stitching; Method three is a combination of the image stitching algorithm based on deep learning and image optimization. Among them, image optimization includes: edge noise suppression, image edge rectification and gap stitching; The described image stitching algorithm selects different stitching methods according to the required image stitching speed and image stitching quality. Taking the proposed image stitching based on deep learning and its optimization as an example, specifically, the algorithm for image stitching based on optimization mainly consists of two parts. One part is the image stitching algorithm based on the unsupervised deep learning framework, and the other part is to process the stitched image based on the diffusion model; 2): Software system integration method, including: initializing the system, configuring system parameters, selecting input sources, data processing, object appearance detection algorithm, image stitching algorithm, image stitching target detection and display, interactive interface, end interface, and outputting detection results; 3): Software system edge deployment method, forming an edge computing module, including: a core control board, a computing power accelerator, a radiator, a core control board housing, a memory card, an HDMI cable, a USB cable, a power cord, and a display.

4. The method of an object detection system based on image stitching technology according to claim 3, characterized in that: The described core control board, computing power calculator, radiator, core control board housing, memory card, HDMI cable, USB cable, and power cord together form a unit for loading pre-trained image stitching algorithm models and object detection algorithm models.

5. The method of an object detection system based on image stitching technology according to claim 3, characterized in that: The described display is used to display the target detection results in real time.

Images