Industrial visual inspection system based on multi-modal fusion

Abstract

The invention discloses an industrial visual detection system based on multi-modal fusion, which belongs to the technical field of industrial data acquisition, processing and processing, and comprises a multi-modal data acquisition and calibration module, a scene image capture module, a visual information synthesis module, an algorithm optimization and training module and a multi-modal image output module, the multiple modules are integrated into the industrial visual detection system for multi-modal data acquisition, processing and fusion, high-quality original data are provided for subsequent image analysis, images of industrial scenes can be automatically captured under different illumination conditions, an original image database containing various illumination and scene conditions is constructed, and the visual detection system has the advantages of being simple in structure, convenient to operate and high in practicability. The system can operate stably, high-precision analysis and processing of the image data set in the industrial scene are realized, the corresponding fusion image is generated, the industrial visual inspection report is generated based on the fusion image, and the adaptability and reliability of the industrial visual inspection system are improved.

Description

Technical Field

[0001] The present invention relates to an industrial vision inspection system, and in particular to an industrial vision inspection system based on multimodal fusion, and belongs to the technical field of industrial data collection, processing and machining. Background Art

[0002] With the advancement of industrialization, the demand for precision manufacturing and automated inspection is growing. Industrial visual inspection technology has become an indispensable part of the manufacturing industry. It automatically analyzes images to identify product defects, monitor production processes, and even predict equipment failures. However, traditional single-modal visual inspection systems mainly rely on a single perspective or a single type of sensor (such as only using visible light imaging), which is easily affected by light interference, occlusion, or surface reflection in a changing industrial environment, limiting its application effect.

[0003] Some industrial visual inspection systems in current existing technologies attempt to improve inspection performance by integrating multiple sensor data, and aim to capture comprehensive information about products from different angles by combining visible light, infrared, X-ray and other imaging technologies. However, the integrated processing of these multimodal data sources faces many challenges, including data synchronization, development of fusion algorithms for heterogeneous data, computational efficiency, and real-time response capabilities. In addition, how to ensure the stability and accuracy of the fusion results, especially in dynamic and complex industrial environments, is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to provide an industrial vision inspection system based on multimodal fusion, which can efficiently process and fuse data from multiple sensors, improve the accuracy and response speed of detection, and solve the shortcomings of the prior art.

[0005] The present invention provides the following scheme:

[0006] An industrial visual inspection system based on multimodal fusion includes a multimodal data acquisition and calibration module, a scene image capture module, a visual information synthesis module, an algorithm optimization and training module, and a multimodal image output module:

[0007] The multimodal data acquisition and calibration module is used to install and calibrate image acquisition equipment for multimodal industrial visual analysis;

[0008] The scene image capture module is used to automatically capture images of industrial scenes under different lighting conditions using an image acquisition device to construct an original image database;

[0009] The visual information synthesis module is used to build an image quality improvement algorithm, a feature extraction algorithm and a deep learning image analysis algorithm into the visual information synthesis module to form a multimodal synthesis model;

[0010] The algorithm optimization and training module is used to train and optimize the multimodal synthesis model by defining a cost function and adjusting weights;

[0011] The multimodal image output module is used to obtain an image data set from the original image database, input the image data set processed by the algorithm optimization and training module into the multimodal synthesis model, execute the image fusion instruction and generate a corresponding fused image, and generate an industrial visual inspection report based on the fused image.

[0012] Furthermore, the multimodal data acquisition and calibration module further includes:

[0013] An automatic white balance adjustment module automatically adjusts the white balance setting of the image acquisition device according to different lighting conditions at the scene where the image acquisition device is located;

[0014] The adaptive resolution adjustment module dynamically adjusts the resolution of the captured image according to the environmental information of the scene where the image acquisition device is located.

[0015] Furthermore, the visual information synthesis module further includes:

[0016] A real-time performance optimization module is used to optimize the execution efficiency of the algorithm and perform real-time image processing on the image data set obtained from the original image database through the optimization algorithm;

[0017] The multimodal data alignment module is used to set the image dataset obtained from the original image database calculated by the optimization algorithm to be consistent in time and space.

[0018] Furthermore, in the visual information synthesis module, the real-time performance optimization module realizes real-time image processing based on the optimized image processing algorithm, dynamically allocates the computing resources of the CPU and GPU according to the complexity of the current real-time image processing and the system load, monitors the execution time of the algorithm in real time, and optimizes the execution order of tasks by giving priority to high-priority tasks and processing low-priority tasks in parallel or batches;

[0019] The multimodal data alignment module uses the global positioning system GPS time or the network time protocol NTP service to stamp a unified time stamp for the image data set that has undergone real-time image processing, uses the image calibration algorithm to determine the geometric relationship and spatial position of each image data set that has undergone real-time image processing, and finally corrects the image data set that has undergone real-time image processing through the transformation matrix;

[0020] Feature matching is performed between the multimodal data alignment module and the real-time performance optimization module, and feature point detection and description extraction are performed on different image data sets that have undergone real-time image processing. The spatial consistency is ensured by feature matching. The description extraction specifically involves adding descriptors to feature points during feature point detection.

[0021] Furthermore, the algorithm optimization and training module further includes:

[0022] An online learning module for continuously updating model parameters in the multimodal synthesis model while the system is running;

[0023] The feedback loop module is used to collect feedback of the system output and use it for further training and optimization of the multimodal synthesis model.

[0024] Furthermore, the multimodal image output module further comprises:

[0025] A dynamic range imaging unit for synthesizing an image dataset with a dynamic range;

[0026] The color correction unit is used to adjust the color of the fused image to match a specific application scenario.

[0027] Furthermore, in the visual information synthesis module, the process of synthesizing visual information based on the image quality improvement algorithm built into the visual information synthesis module is specifically as follows:

[0028] Based on the image quality improvement algorithm, the local features of the image data set are processed to remove noise, and the noise detection is performed by analyzing the local area of ​​the image to identify the noise pattern;

[0029] Identify the edge and texture features of the image in the image data set, perform feature recognition to distinguish noise from useful information, and apply adaptive filters to the detected noise;

[0030] Perform edge enhancement processing on the image data set to highlight the edges in the image data set, improve the image contour clarity, and use edge detection operators to identify the edges in the image data set;

[0031] Edge mapping is performed based on the detected edges of the image data set, the contrast of the image is adjusted to enhance the edge display, and the contour is sharpened by enhancing the difference in pixel values ​​near the edge to improve the contour clarity of the overall image.

[0032] Furthermore, the process of forming a multimodal synthesis model based on the built-in feature extraction algorithm and deep learning image analysis algorithm in the visual information synthesis module is specifically as follows:

[0033] The convolutional neural network is used to extract features and learn images from image datasets, and the recurrent neural network is used to process image sequences and extract temporal dynamic information from image datasets.

[0034] Furthermore, the process of forming the multimodal synthesis model also includes image fusion instructions, specifically:

[0035] According to preset image fusion objectives and system performance requirements, an image fusion rule component is defined, and according to the image fusion rule component, the defined image fusion rule is stored in an industrial vision system database;

[0036] Obtain image data characteristics of the image data set, retrieve corresponding fusion rules from the industrial vision system database, and apply the retrieved fusion rules to the image fusion process;

[0037] The image dataset is evaluated based on the peak signal-to-noise ratio (PSNR) or structural similarity (SSIM) indicator, visual feature values ​​are extracted from the image dataset, and fusion quality feedback information is generated based on the visual feature values.

[0038] The image data set is adjusted and optimized through the fusion quality feedback information to generate a fused image.

[0039] Furthermore, the process of forming the multimodal synthesis model also includes:

[0040] Preprocessing, preprocessing the multimodal image data in the image data set involved in the fusion, wherein the preprocessing means include denoising, enhancement, and standardization;

[0041] An alignment operation is used to ensure the consistency of the multimodal image data set in space and time, and the alignment operation includes feature point matching and geometric transformation;

[0042] A fusion operation is performed on the preprocessed and aligned image data sets according to the fusion rules to generate a preliminary fused image;

[0043] Post-processing: performing post-processing operations on the preliminary fused image, wherein the post-processing operations include sharpening, color correction, and contrast adjustment.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] The present invention provides an industrial vision inspection system that integrates multimodal data acquisition, processing and fusion. Through a highly automated and intelligent method, high-precision analysis and processing of image data in industrial scenes is achieved. By installing and calibrating a variety of image acquisition devices, the accuracy and consistency of image data acquired from different visual modalities are ensured, and high-quality original data is provided for subsequent image analysis. Based on the high-quality original data, a scene image capture module can be performed, which can adapt to different lighting conditions, automatically capture images of industrial scenes, and build a database of original images under a variety of lighting and scene conditions, so that the system can operate stably in a changeable industrial environment, thereby improving the adaptability and reliability of the system.

[0046] The present invention has built-in multiple image processing algorithms, including image quality improvement algorithms, feature extraction algorithms and deep learning image analysis algorithms, to form a multimodal synthesis model with powerful computing power. By comprehensively processing and analyzing multi-source image data, the comprehensive utilization efficiency of image information and the accuracy of analysis results are improved. By defining appropriate cost functions and adjusting algorithm parameters, the multimodal synthesis model is systematically trained and optimized to ensure that the system can continue to learn and improve, thereby improving the accuracy and efficiency of the system. Finally, the processed image data set is input into the synthesis model, image fusion instructions are executed, and high-quality fused images are generated, thereby realizing efficient collection, processing and fusion of industrial scene image data, significantly improving the accuracy, efficiency and adaptability of industrial visual inspection, and providing strong technical support for industrial automation and intelligentization. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is the architecture diagram of the industrial vision inspection system based on multimodal fusion.

[0049] Figure 2 It is a structural diagram of the multimodal data acquisition and calibration module.

[0050] Figure 3 It is a structural diagram of the visual information synthesis module.

[0051] Figure 3A It is a real-time image processing flow chart based on the visual information synthesis module.

[0052] Figure 4It is a structural diagram of the algorithm optimization and training module.

[0053] Figure 5 It is a structural diagram of the multimodal image output module.

[0054] Figure 6 It is a flow chart of the process of visual information synthesis based on the image quality improvement algorithm.

[0055] Figure 7 It is a flow chart of the process of forming a multimodal synthesis model. DETAILED DESCRIPTION

[0056] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] like Figure 1 The industrial visual inspection system based on multimodal fusion shown in FIG. 1 includes a multimodal data acquisition and calibration module, a scene image capture module, a visual information synthesis module, an algorithm optimization and training module, and a multimodal image output module:

[0058] The multimodal data acquisition and calibration module is used to install and calibrate image acquisition equipment for multimodal industrial visual analysis;

[0059] The scene image capture module is used to automatically capture images of industrial scenes under different lighting conditions using an image acquisition device to construct an original image database;

[0060] The visual information synthesis module is used to build an image quality improvement algorithm, a feature extraction algorithm and a deep learning image analysis algorithm into the visual information synthesis module to form a multimodal synthesis model;

[0061] The algorithm optimization and training module is used to train and optimize the multimodal synthesis model by defining a cost function and adjusting weights;

[0062] The multimodal image output module is used to obtain an image data set from the original image database, input the image data set processed by the algorithm optimization and training module into the multimodal synthesis model, execute the image fusion instruction and generate a corresponding fused image, and generate an industrial visual inspection report based on the fused image.

[0063] The industrial visual inspection system based on multimodal fusion provided by the present invention can comprehensively collect and process various types of image data, and optimize and fuse them through multiple algorithms, thereby realizing high-precision and high-efficiency industrial visual analysis, among which:

[0064] The multimodal data acquisition and calibration module ensures the accuracy and consistency of image data acquired from different visual modalities, provides high-quality raw data for subsequent image analysis, and ensures the reliability of the system analysis results; the scene image capture module enables the industrial vision inspection system to operate stably in a changing industrial environment by constructing a database of raw images under various lighting and scene conditions, thereby improving the adaptability and robustness of the system; the visual information synthesis module has built-in multiple image processing algorithms to form a multimodal synthesis model, which improves the comprehensive utilization efficiency of image information and the accuracy of analysis results by comprehensive processing and analysis of multi-source image data; algorithm optimization and training module: by defining appropriate cost functions and adjusting algorithm parameters, the multimodal synthesis model is systematically trained and optimized to ensure that the system can continue to learn and improve, improve the accuracy and efficiency of the system, and finally input the processed image data set into the synthesis model, execute image fusion instructions, generate high-quality fused images, and be able to generate industrial vision inspection reports based on the fused images.

[0065] It can be seen that the industrial visual inspection system based on multimodal fusion of this embodiment implements the multimodal industrial visual inspection method through multiple steps:

[0066] Multimodal data acquisition and calibration steps: Install and calibrate the image acquisition equipment required for multimodal industrial visual analysis to ensure that the images of industrial scenes captured are accurate;

[0067] Scene image capture step: used to automatically capture images of industrial scenes under different lighting conditions and build an original image database;

[0068] Visual information synthesis step: Through the built-in image quality improvement algorithm, feature extraction algorithm and deep learning image analysis algorithm, a multimodal synthesis model is formed to create a comprehensive image information processing flow;

[0069] Algorithm optimization and training steps: By defining cost functions and adjusting weights, the multimodal synthesis model is trained and optimized to improve the accuracy and efficiency of the multimodal industrial visual inspection system;

[0070] Multimodal image output step: obtaining an image data set from the original image database, inputting the image data set processed by the algorithm optimization and training module into the multimodal synthesis model, executing the image fusion instruction and generating a corresponding fused image, and generating an industrial visual inspection report based on the fused image.

[0071] Through the above steps, the industrial visual inspection method based on multimodal fusion can comprehensively collect and process various types of image data sets, and optimize and fuse them through specific algorithms to achieve high-precision and high-efficiency industrial visual analysis.

[0072] like Figure 2 As shown, the multimodal data acquisition and calibration module further includes:

[0073] An automatic white balance adjustment module automatically adjusts the white balance setting of the image acquisition device according to different lighting conditions at the scene where the image acquisition device is located;

[0074] The adaptive resolution adjustment module dynamically adjusts the resolution of the captured image according to the environmental information of the scene where the image acquisition device is located.

[0075] The multimodal data acquisition and calibration module further improves the adaptability and flexibility of the multimodal industrial visual inspection system through intelligent image acquisition device management. It automatically adjusts the white balance and dynamically adjusts the image resolution to ensure that high-quality image data can be obtained under different lighting and environmental conditions, thereby improving the performance and reliability of the entire system.

[0076] The automatic white balance adjustment module can automatically adjust the white balance settings of the image acquisition device according to the on-site lighting conditions of the image acquisition device, ensuring that the color of the images collected in different lighting environments (such as sunlight, cloudy days or artificial light sources) remains true and consistent, avoiding color deviation caused by changes in lighting conditions, and enhancing the adaptability of the multimodal industrial vision inspection system under different ambient lighting conditions.

[0077] The adaptive resolution adjustment module dynamically adjusts the resolution of the captured image according to the on-site environmental information of the image acquisition device, so that the multimodal industrial vision inspection system can optimize the quality and transmission efficiency of image data according to actual needs and scene complexity.

[0078] For example, when a multimodal industrial vision device captures images with rich details or requires high-precision analysis, the image resolution can be automatically increased to capture more details. In scenarios where the image content is relatively simple or has high real-time requirements, the multimodal industrial vision device can appropriately reduce the resolution to increase processing speed and reduce data volume. The adaptive adjustment mechanism improves the flexibility and efficiency of the multimodal industrial vision system, ensuring that the optimal image data can be obtained in different application scenarios.

[0079] like Figure 3 As shown, the visual information synthesis module further includes:

[0080] A real-time performance optimization module is used to optimize the execution efficiency of the algorithm and perform real-time image processing on the image data set obtained from the original image database through the optimization algorithm;

[0081] The multimodal data alignment module is used to set the image dataset obtained from the original image database calculated by the optimization algorithm to be consistent in time and space.

[0082] The visual information synthesis module improves the processing efficiency of the multimodal industrial vision inspection system and ensures data consistency. The real-time performance optimization module improves the algorithm execution speed, ensures that the system can process large amounts of data and meet real-time requirements, and ensures the consistency of image data from different data sources in time and space, making the fusion results more accurate and reliable.

[0083] The real-time performance optimization module is used to realize real-time image processing. By dynamically adjusting resource allocation, optimizing algorithm flow and implementing parallel processing strategies, it significantly improves the image processing speed and reduces processing delays. This enables the system to quickly respond to image capture actions at industrial sites without sacrificing processing quality, meeting the needs of rapid detection and instant feedback.

[0084] The multimodal data alignment module is responsible for ensuring the temporal and spatial consistency of data from different sources (such as cameras in different locations, different types of sensors). By precisely controlling time delays and utilizing image registration algorithms, it can eliminate time delays and spatial deviations between different data sources, ensuring that the image data before fusion has a high degree of alignment accuracy, which is crucial to improving the quality of subsequent image analysis and fusion, especially in complex industrial environments, and can provide more accurate and reliable visual information.

[0085] like Figure 3A As shown, exemplarily, in the visual information synthesis module, the real-time performance optimization module realizes real-time image processing based on the optimized image processing algorithm, and the specific method steps are as follows:

[0086] Step S1: dynamically allocate CPU and GPU computing resources according to the complexity of current real-time image processing and system load, monitor the execution time of the algorithm in real time, and optimize the task execution order by giving priority to high-priority tasks and processing low-priority tasks in parallel or batches;

[0087] Step S2: The multimodal data alignment module uses the global positioning system GPS time or the network time protocol NTP service to stamp a unified time stamp for the image data set that has undergone real-time image processing, uses an image calibration algorithm to determine the geometric relationship and spatial position of each image data set that has undergone real-time image processing, and finally corrects the image data set that has undergone real-time image processing through a transformation matrix;

[0088] Step S3: Feature matching is performed between the multimodal data alignment module and the real-time performance optimization module, feature point detection and description extraction are performed on different image data sets that have undergone real-time image processing, and spatial consistency is ensured through feature matching.

[0089] The description extraction is specifically to add descriptors to feature points during feature point detection, and establish a measure (vector) of image attributes in the neighborhood around the feature points. The descriptor remains invariant to image features such as rotation, scaling, and brightness changes, and has high robustness. Even after the feature points have undergone multiple transformations, the same feature points can still be matched by the descriptor, that is, the descriptor contains information around the feature points, such as gradient direction, color intensity, or other pixel intensity statistics, and the above information can be encoded into a fixed-length feature vector.

[0090] The real-time image processing method based on the visual information synthesis module achieves efficient image processing and accurate data fusion through the collaborative work of the real-time performance optimization module and the multimodal data alignment module. The real-time performance optimization module improves the execution efficiency of the algorithm through dynamic resource management and intelligent task scheduling to ensure that the needs of real-time image processing are met. The multimodal data alignment module ensures the consistency of data from different sources in time and space through time synchronization and spatial correction. The interactive cooperation of these two modules further improves the accuracy and real-time performance of multimodal image data fusion, and realizes dynamic resource management and task scheduling, time synchronization and spatial correction, and feature matching and spatial consistency.

[0091] The real-time performance optimization module dynamically allocates CPU, GPU and other computing resources according to the complexity of the current processing task and the system load, and optimizes the task execution order by real-time monitoring of the algorithm execution time, so that the system can flexibly respond to different workloads and maintain efficient operation. Even under high load conditions, it can ensure real-time processing performance. The multimodal data alignment module uses GPS or NTP services to stamp a unified timestamp for the image data set that has undergone real-time image processing, and determines the geometric relationship and spatial position of the data source through the image calibration algorithm. The image data set that has undergone real-time image processing is corrected using the transformation matrix, ensuring the synchronization of image data sets from different data sources in time series and alignment in spatial position, providing high-quality input data for subsequent fusion processing. The multimodal data alignment module and the real-time performance optimization module jointly perform feature matching, and ensure the spatial consistency of image data from different sources through feature point detection and description extraction, eliminating the errors caused by different perspectives and sensor characteristics, making the fused image more accurate and reliable.

[0092] like Figure 4The algorithm optimization and training module shown, the algorithm optimization and training module further includes:

[0093] An online learning module for continuously updating model parameters in the multimodal synthesis model while the system is running;

[0094] The feedback loop module is used to collect feedback of the system output and use it for further training and optimization of the multimodal synthesis model.

[0095] The algorithm optimization and training module realizes the continuous updating and optimization of model parameters through online learning and feedback loop mechanism, so that the industrial vision inspection system can continuously learn and adapt to new data in actual operation, thereby improving the accuracy and adaptability of the model. The online learning module can update the model parameters in real time during runtime. By analyzing the current data and processing results, the model can be adjusted instantly to adapt to new patterns and changes, which is equivalent to providing a real-time update mechanism, significantly improving the system's response speed and processing capabilities to new situations, and ensuring that the model maintains optimal performance in a changing industrial environment.

[0096] The feedback loop module is responsible for collecting feedback from the system output and using the feedback information to further train and optimize the model, identify and correct the deficiencies of the current model, thereby improving performance in each round of iteration and enabling the model to have the ability of self-learning and self-improvement, which can continuously improve the accuracy and efficiency of the model.

[0097] like Figure 5 As shown, the multimodal image output module further includes:

[0098] A dynamic range imaging unit for synthesizing an image dataset with a dynamic range;

[0099] The color correction unit is used to adjust the color of the fused image to match a specific application scenario.

[0100] The multimodal image output module can generate fused images with high dynamic range and color accuracy. Through the collaborative work of the dynamic range imaging unit and the color correction unit, high-quality images that meet specific application requirements in terms of brightness and color can be synthesized, thereby improving the application value and visual effects of the image, including:

[0101] The dynamic range imaging unit is responsible for synthesizing images with a high dynamic range. The dynamic range imaging unit overcomes the limitations of a single image in capturing brightness. It can extract and merge image information of different exposure levels from multimodal data to generate images that contain both bright details and dark details. The final fused image can more realistically reflect the visual effects of the real world, especially in scenes with strong contrast between light and dark.

[0102] The multimodal image output module can generate fused images with high dynamic range and color accuracy. Through the collaborative work of the dynamic range imaging unit and the color correction unit, it can synthesize high-quality images that meet specific application requirements in terms of brightness and color, thereby enhancing the application value and visual effect of the image.

[0103] The dynamic range imaging unit is responsible for synthesizing images with a high dynamic range. The dynamic range imaging unit overcomes the limitations of a single image in capturing brightness. It can extract and merge image information of different exposure levels from multimodal data to generate images that contain both bright details and dark details. The final fused image can more realistically reflect the visual effects of the real world, especially in scenes with strong contrast between light and dark.

[0104] The color correction unit is used to adjust the color of the fused image to ensure the color accuracy and consistency of the image. By analyzing the color characteristics and requirements of the application scenario, the color correction unit can perform color compensation and correction on the image to eliminate color deviations caused by factors such as light source changes and sensor differences. This makes the fused image have better visual effects and application adaptability in specific industrial vision application environments.

[0105] The multimodal image output module improves the quality and applicability of the fused image through the coordinated cooperation of the dynamic range imaging unit and the color correction unit. It not only enhances the visual expressiveness of the image, but also ensures the authenticity and accuracy of the image information, providing a high-quality data foundation for subsequent image analysis, processing and application.

[0106] like Figure 6 The flowchart of the process of visual information synthesis based on the image quality improvement algorithm shown in FIG. 1 is a flowchart of the process of visual information synthesis based on the image quality improvement algorithm built into the visual information synthesis module. In the visual information synthesis module, the process of visual information synthesis based on the image quality improvement algorithm built into the visual information synthesis module is specifically as follows:

[0107] Step T1, removing noise from local features of the image data set based on an image quality improvement algorithm, and performing noise detection by analyzing local areas of the image to identify noise patterns;

[0108] Step T2, identifying edge and texture features of the image in the image data set, performing feature recognition to distinguish noise from useful information, and applying an adaptive filter to the detected noise;

[0109] Step T3, performing edge enhancement processing on the image data set to highlight the edges in the image data, improve the image contour clarity, and use an edge detection operator to identify the edges in the image set;

[0110] Step T4, edge mapping is performed according to the detected edges of the image data set, the contrast of the image is adjusted to enhance the edge display, and the contour is sharpened by enhancing the difference in pixel values ​​near the edge to improve the contour clarity of the overall image.

[0111] The technical solutions provided in steps T1 to T4 indicate the visual information synthesis process in the visual information synthesis module. The visual information synthesis module improves the image quality through the technical solutions provided in steps T1 to T4 for subsequent analysis and fusion, thereby improving the clarity and readability of the image and ensuring that the synthesized image information is of high quality both in vision and analysis.

[0112] The technical solution provided in steps T1 to T4 effectively detects and removes noise in the image through local feature analysis and noise pattern recognition, applies adaptive filters to process specific noise characteristics of different image areas, improves the cleanliness and visual quality of the image, identifies key features such as edges and textures in the image, helps to distinguish noise and useful information in the image, and reduces the interference of irrelevant information. By using edge detection operators (such as Sobel and Canny) to identify edges in the image, and then mapping and contrast adjustment are performed according to the edge strength, the edge display of the image is enhanced, the contour and structure of the image are clearer, which is helpful for subsequent image analysis and understanding. Finally, by enhancing the difference in pixel values ​​near the edge, the image is contour sharpened, which further improves the contour clarity of the image, making the image more visually distinct and helping to highlight important visual elements.

[0113] Preferably, based on the feature extraction algorithm and deep learning image analysis algorithm built into the visual information synthesis module, the process of forming the multimodal synthesis model is specifically as follows:

[0114] The convolutional neural network is used to extract features and learn images from image datasets, and the recurrent neural network is used to process image sequences and extract temporal dynamic information from image datasets.

[0115] In some embodiments, an optimization technical solution can also be obtained based on the feature extraction algorithm and deep learning image analysis algorithm built into the visual information synthesis module, namely: fully mining and utilizing the rich information in the image data set through a computational model, extracting and learning image features based on a convolutional neural network (CNN), and using a recurrent neural network (RNN) to process image sequences and capture temporal dynamic information, thereby achieving in-depth analysis and understanding of the image data set. Specifically:

[0116] This embodiment uses a convolutional neural network to extract features from an image data set, and can automatically learn and identify hierarchical features in an image, from edges and textures to more complex patterns, to achieve image recognition, classification, and analysis functions. In this embodiment, due to the introduction of CNN, the model can automatically learn useful representations from the original image database, reducing the need for manual feature design and improving the accuracy and efficiency of image analysis.

[0117] This embodiment also realizes the extraction of temporal dynamic information by processing the image sequence through the recurrent neural network. The temporal dynamic information in the image data set can be captured based on the temporal dynamic information extraction, and the temporal relationship between the image frames can be understood, which is of great significance for analyzing the behavior of objects, movement patterns, and event development in the video. The application of RNN enables the model to not only process single-frame images, but also understand and predict dynamic changes in image sequences, thereby enhancing the performance of the model in processing video streams or image sequences.

[0118] like Figure 7 The flowchart of the process of forming a multimodal synthesis model shown in FIG. 1 also includes the processing flow of image fusion instructions in the process of forming a multimodal synthesis model, which is specifically as follows:

[0119] Step P1, defining an image fusion rule component according to a preset image fusion target and system performance requirements, and storing the defined image fusion rule in an industrial vision system database according to the image fusion rule component;

[0120] Step P2, obtaining image data characteristics of the image data set, retrieving corresponding fusion rules from the industrial vision system database, and applying the retrieved fusion rules to the image fusion process;

[0121] Step P3, evaluating the quality of the image data set based on the peak signal-to-noise ratio (PSNR) or the structural similarity (SSIM) index, extracting visual feature values ​​from the image data set, and generating fusion quality feedback information based on the visual feature values;

[0122] Step P4: adjusting and optimizing the image data set according to the fusion quality feedback information to generate a fused image.

[0123] The technical solution provided by steps P1 to P4 reveals the process of forming a multimodal synthesis model. The technical solution involves defining fusion rules, retrieving applicable rules, applying the rules for fusion, and providing feedback and optimizing the fusion results based on quality evaluation indicators. The process ensures that the image fusion operation can be customized according to specific task requirements and data characteristics, and achieves continuous improvement of the fusion effect through quality evaluation and feedback mechanisms.

[0124] The technical solution provided in steps P1 to P4 customizes the definition of the fusion rules. By defining the image fusion rules according to the preset fusion goals and system performance requirements, it ensures that the fusion operation can meet specific application needs and improves the flexibility and adaptability of the fusion operation. By obtaining the characteristics of the current image data set, the most appropriate fusion rules are dynamically retrieved from the database, and the fusion rules are applied to the fusion process, thereby ensuring the real-time and effectiveness of the fusion operation.

[0125] The technical solution provided in steps P1 to P4 also evaluates the quality of the image dataset by using objective evaluation indicators (such as PSNR and SSIM) and extracting visual features (such as edges, textures, colors, etc.), provides in-depth analysis and quantitative evaluation of the fusion results, and generates fusion quality feedback information to identify the advantages and disadvantages of the fusion process, thereby adjusting and optimizing the image dataset, thereby achieving continuous improvement in the quality of the fused image.

[0126] In summary, the technical solution provided by steps P1 to P4 utilizes the processing flow of image fusion instructions, and through steps such as rule definition, dynamic retrieval, quality evaluation, and feedback optimization, improves the accuracy and reliability of image fusion operations, ensures that the fused image can meet specific application requirements and quality standards, improves the performance of the industrial vision inspection system, and enhances the adaptability and stability in complex and changeable industrial environments, providing high-quality industrial vision inspection solutions.

[0127] Exemplarily, the process of forming the multimodal synthesis model also includes:

[0128] Preprocessing: preprocessing the multimodal image data in the image data set involved in the fusion, wherein the preprocessing means include denoising, enhancement, standardization, etc.;

[0129] Alignment operation, used to ensure the consistency of multimodal image data in space and time, including feature point matching, geometric transformation, etc.;

[0130] A fusion operation is performed on the preprocessed and aligned image data sets according to the fusion rules to generate a preliminary fused image;

[0131] Post-processing: performing post-processing operations on the preliminary fused image, wherein the post-processing operations include sharpening, color correction, and contrast adjustment.

[0132] In the process of forming a multimodal synthesis model in this embodiment, the quality of the image data set and the consistency requirements of the image data set are ensured through a series of image processing steps, thereby improving the accuracy and usability of the fused image: the preprocessing step improves the quality and usability of the image through means such as denoising, enhancement and standardization: denoising reduces random noise in the image, enhancement improves the contrast and brightness of the image, and standardization ensures consistent processing effects between different images, laying a good foundation for subsequent processing steps.

[0133] To ensure the spatial and temporal consistency of multimodal image data, alignment operations include feature point matching and geometric transformation. Feature point matching ensures the accurate correspondence of the same physical structure between images, while geometric transformation corrects the differences in perspective, position and scale, so that images from different sources can be accurately superimposed and fused.

[0134] According to the fusion rules, the preprocessed and aligned image data sets are fused to generate a preliminary fused image. The key information of the multimodal images is integrated through intelligent algorithms to create a new image that combines the advantages of all input data, thereby improving the richness and reliability of information. On this basis, the preliminary fused image is post-processed, including sharpening, color correction, and contrast adjustment, etc.: sharpening enhances the details and edges of the image, making the image clearer; color correction ensures the authenticity and consistency of the image color; contrast adjustment optimizes the light and dark details of the image, making the fused image more visually pleasing to users and improving the user experience.

[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0136] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the embodiments claimed in the claims can be used in any combination in the embodiments of the present invention.

[0137] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0138] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0139] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any way. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features. Throughout the specification, the same reference numerals indicate the same elements.

[0140] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition they can be divided into multiple submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including corresponding claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner can be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including corresponding claims, abstracts and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An industrial visual inspection system based on multimodal fusion, characterized in that: It includes multimodal data acquisition and calibration module, scene image capture module, visual information synthesis module, algorithm optimization and training module and multimodal image output module: The multimodal data acquisition and calibration module is used to install and calibrate image acquisition equipment for multimodal industrial visual analysis; The scene image capture module is used to automatically capture images of industrial scenes under different lighting conditions using an image acquisition device to construct an original image database; The visual information synthesis module is used to build an image quality improvement algorithm, a feature extraction algorithm and a deep learning image analysis algorithm into the visual information synthesis module to form a multimodal synthesis model; The algorithm optimization and training module is used to train and optimize the multimodal synthesis model by defining a cost function and adjusting weights; The multimodal image output module is used to obtain an image data set from the original image database, input the image data set processed by the algorithm optimization and training module into the multimodal synthesis model, execute the image fusion instruction and generate a corresponding fused image, and generate an industrial visual inspection report based on the fused image.

2. The industrial visual inspection system based on multimodal fusion according to claim 1 is characterized in that: The multimodal data acquisition and calibration module further comprises: An automatic white balance adjustment module automatically adjusts the white balance setting of the image acquisition device according to different lighting conditions at the scene where the image acquisition device is located; The adaptive resolution adjustment module dynamically adjusts the resolution of the captured image according to the environmental information of the scene where the image acquisition device is located.

3. The industrial visual inspection system based on multimodal fusion according to claim 1 is characterized in that: The visual information synthesis module further comprises: A real-time performance optimization module is used to optimize the execution efficiency of the algorithm and perform real-time image processing on the image data set obtained from the original image database through the optimization algorithm; The multimodal data alignment module is used to set the image dataset obtained from the original image database calculated by the optimization algorithm to be consistent in time and space.

4. The industrial visual inspection system based on multimodal fusion according to claim 3 is characterized in that: In the visual information synthesis module, the real-time performance optimization module realizes real-time image processing based on the optimized image processing algorithm, dynamically allocates the computing resources of the CPU and GPU according to the complexity of the current real-time image processing and the system load, monitors the execution time of the algorithm in real time, and optimizes the task execution order by giving priority to high-priority tasks and processing low-priority tasks in parallel or batches; The multimodal data alignment module uses the global positioning system GPS time or the network time protocol NTP service to stamp a unified time stamp for the image data set that has undergone real-time image processing, uses the image calibration algorithm to determine the geometric relationship and spatial position of each image data set that has undergone real-time image processing, and finally corrects the image data set that has undergone real-time image processing through the transformation matrix; Feature matching is performed between the multimodal data alignment module and the real-time performance optimization module, and feature point detection and description extraction are performed on different image data sets that have undergone real-time image processing. The spatial consistency is ensured by feature matching. The description extraction specifically involves adding descriptors to feature points during feature point detection.

5. The industrial visual inspection system based on multimodal fusion according to claim 1, characterized in that: The algorithm optimization and training module further includes: An online learning module for continuously updating model parameters in the multimodal synthesis model while the system is running; The feedback loop module is used to collect feedback of the system output and use it for further training and optimization of the multimodal synthesis model.

6. The industrial visual inspection system based on multimodal fusion according to claim 1, characterized in that: The multimodal image output module further comprises: A dynamic range imaging unit for synthesizing an image dataset with a dynamic range; The color correction unit is used to adjust the color of the fused image to match a specific application scenario.

7. The industrial visual inspection system based on multimodal fusion according to claim 1, characterized in that: In the visual information synthesis module, the process of visual information synthesis based on the image quality improvement algorithm built into the visual information synthesis module is as follows: Based on the image quality improvement algorithm, the local features of the image data set are processed to remove noise, and the noise detection is performed by analyzing the local area of ​​the image to identify the noise pattern; Identify the edge and texture features of the image in the image data set, perform feature recognition to distinguish noise from useful information, and apply adaptive filters to the detected noise; Perform edge enhancement processing on the image data set to highlight the edges in the image data set, improve the image contour clarity, and use edge detection operators to identify the edges in the image data set; Edge mapping is performed based on the detected edges of the image data set, the contrast of the image is adjusted to enhance the edge display, and the contour is sharpened by enhancing the difference in pixel values ​​near the edge to improve the contour clarity of the overall image.

8. The industrial visual inspection system based on multimodal fusion according to claim 1, characterized in that: The process of forming a multimodal synthesis model based on the built-in feature extraction algorithm and deep learning image analysis algorithm in the visual information synthesis module is specifically as follows: The convolutional neural network is used to extract features and learn images from image datasets, and the recurrent neural network is used to process image sequences and extract temporal dynamic information from image datasets.

9. The industrial visual inspection system based on multimodal fusion according to claim 8, characterized in that: The process of forming a multimodal synthesis model also includes image fusion instructions, specifically: According to preset image fusion objectives and system performance requirements, an image fusion rule component is defined, and according to the image fusion rule component, the defined image fusion rule is stored in an industrial vision system database; Obtain image data characteristics of the image data set, retrieve corresponding fusion rules from the industrial vision system database, and apply the retrieved fusion rules to the image fusion process; The image dataset is evaluated based on the peak signal-to-noise ratio (PSNR) or structural similarity (SSIM) indicator, visual feature values ​​are extracted from the image dataset, and fusion quality feedback information is generated based on the visual feature values. The image data set is adjusted and optimized through the fusion quality feedback information to generate a fused image.

10. The industrial visual inspection system based on multimodal fusion according to claim 9, characterized in that: The process of forming a multimodal synthesis model also includes: Preprocessing, preprocessing the multimodal image data in the image data set involved in the fusion, wherein the preprocessing means include denoising, enhancement, and standardization; An alignment operation is used to ensure the consistency of the multimodal image data set in space and time, and the alignment operation includes feature point matching and geometric transformation; A fusion operation is performed on the preprocessed and aligned image data sets according to the fusion rules to generate a preliminary fused image; Post-processing: performing post-processing operations on the preliminary fused image, wherein the post-processing operations include sharpening, color correction, and contrast adjustment.

Images