A visual identification detection method for multi-workpiece mixed flow production

Abstract

The application relates to the technical field of visual identification, in particular to a visual identification detection method for multi-workpiece mixed flow production. The specific implementation steps comprise the following steps: acquiring multi-modal workpiece image data by using a multispectral imaging device, combining workpiece structure data and material process parameters to construct a multi-modal digital twin, expanding the mode of the digital twin, and constructing a multi-modal sample data set. The multi-modal sample data set is input into a multi-layer meta-learning architecture to generate a mode joint identification representation. The multi-layer meta-learning architecture is used to extract the feature representation of the to-be-identified mixed flow workpiece image, and the feature representation is compared with the mode joint identification representation of the multi-modal sample data set in terms of similarity to generate a workpiece category identifier, a defect type identifier and an abnormality degree index. Through the combination of multispectral imaging, digital twin construction and a multi-layer meta-learning architecture, the application realizes accurate identification of the category, defect type and abnormality degree of mixed flow workpieces.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual recognition, in particular to a visual recognition detection method for mixed flow production of multiple workpieces. BACKGROUND

[0002] In order to ensure product quality and improve production efficiency, each flow line in a modern intelligent factory is required to achieve rapid and accurate visual recognition and detection for different types of different workpieces.

[0003] As a key function in intelligent manufacturing systems, the performance of visual recognition directly determines the stability of the entire production and the consistency of product quality. In mixed flow production mode, workpieces differ significantly in type, structure, spatial pose, and appearance features, resulting in multiple targets, severe occlusion, and complex backgrounds in images, which pose great challenges to visual systems. For large-scale customized factories with mixed flow characteristics, due to the same processing technology but significant differences in workpieces, different workpieces may appear mixed in the same image on the same production line. Complex environmental conditions often have a great impact on the target objects captured. Therefore, how to enable the equipment to effectively locate, segment, classify, and semantically understand multiple workpiece objects in a relatively complex scene has become a key research issue.

[0004] Most existing visual recognition methods rely on single-modal image input, usually lack multi-source visual information fusion processing, and cannot fully extract target-related semantic clues from multiple visual dimensions, resulting in insufficient recognition robustness in complex scenes such as strong interference, occlusion, and light changes. At the same time, current methods generally fail to construct spatial structural relationships between workpieces, lack deep modeling of contextual information in images, and limit the accuracy and generalization ability of recognition.

[0005] The lack of stable and accurate recognition ability for multiple workpiece images in complex background environments has become a core technical bottleneck for current visual recognition systems in mixed flow production applications, which needs to be addressed urgently.

[0006] Therefore, the present application provides a visual recognition detection method for mixed flow production of multiple workpieces to solve the above problems. SUMMARY

[0007] The present application aims to provide a visual recognition detection method for mixed flow production of multiple workpieces to solve the problems raised in the background.

[0008] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0009] A visual recognition detection method for mixed flow production of multiple workpieces, the method comprising:

[0010] acquire multi-modal workpiece image data including visible, infrared and ultraviolet wavelength bands using a multi-spectral imaging device; acquire workpiece structure data and material process parameters to construct a basic digital twin; bind and map the multi-modal workpiece image data to the basic digital twin to generate a multi-modal digital twin of the mixed-flow workpiece;

[0011] expand the mode of the multi-modal digital twin to construct a multi-modal sample data set;

[0012] input the multi-modal sample data set to a multi-layer meta-learning architecture, wherein a bottom layer feature recognition module is used to extract structure primitive feature data and defect primitive feature data, and a high layer mode adaptation module is used to generate a mode joint recognition representation according to the primitive feature data;

[0013] extract the feature representation of the workpiece to be identified using the multi-layer meta-learning architecture, compare the feature representation with the feature representation of the multi-modal sample data set, and generate a workpiece category identifier, a defect type identifier and an abnormality index.

[0014] Preferably, the process of acquiring multi-modal workpiece image data using a multi-spectral imaging device includes:

[0015] collect visual data including first workpiece image data in the visible light wavelength band, second workpiece image data in the infrared light wavelength band and third workpiece image data in the ultraviolet light wavelength band using a multi-spectral imaging device; integrate the first workpiece image data, the second workpiece image data and the third workpiece image data for joint denoising to generate multi-modal workpiece image data.

[0016] Preferably, the construction process of the multi-modal digital twin includes:

[0017] acquire workpiece structure data and material process parameters, wherein the workpiece structure data includes three-dimensional CAD model data, and the material process parameters include density, hardness, elastic modulus, surface roughness and thermal conductivity; construct a basic digital twin based on the workpiece structure data and the material process parameters; acquire different data from three different spectral images of the multi-modal workpiece image data; extract the geometric profile features, surface texture features and color features of the workpiece from the visible light image; extract the fluorescence response features and ultraviolet absorption features of the workpiece from the ultraviolet light image; extract the thermal radiation features and material transmission features of the workpiece from the infrared light image; bind and map the multi-modal workpiece image data to the basic digital twin to generate a multi-modal digital twin.

[0018] Preferably, the construction process of the multi-modal sample data set includes:

[0019] The workpiece structure primitive mode and the visual defect primitive mode are selected from the primitive mode parameter model library for instantiation; adjustable parameter values of the instantiated primitive mode are obtained; the multi-modal digital twin is expanded in mode, and the multi-modal digital twin is transformed in structure form and local material attribute according to the adjustable parameter values, to generate an expanded digital twin; based on the expanded digital twin, a multi-modal view is rendered using a virtual visual sensor, to generate a multi-modal expanded digital twin and accurate labeling data, including primitive mode type, shape, size, attribute change amount and belonging workpiece category identifier; the multi-modal expanded digital twin is processed in domain distribution disturbance and simulation-reality hybrid mapping, to construct a multi-modal sample data set.

[0020] Specifically, the domain distribution disturbance processing and simulation-reality hybrid mapping specific implementation process includes:

[0021] Random domain disturbance parameters are generated by a random sampling method, including but not limited to image geometric transformation parameters, illumination parameters, color parameters, noise parameters and imaging quality parameters; based on the random domain disturbance parameters, the multi-modal expanded digital twin is superimposed and fused and nonlinearly transformed, to generate a disturbed multi-modal digital twin; mode feature extraction is performed on the disturbed multi-modal digital twin and real workpiece mode data, and mode distribution difference is calculated; according to the mode distribution difference, the disturbed multi-modal digital twin and the real workpiece mode data are feature-aligned, to generate an aligned multi-modal digital twin; the aligned multi-modal digital twin and the real workpiece mode data are merged, and a multi-modal sample data set is generated in combination with the accurate labeling data.

[0022] Preferably, the generation process of the mode joint recognition representation is specifically:

[0023] According to the meta-learning task, the multi-modal sample data set is input to a multi-layer meta-learning architecture; a bottom feature recognition module extracts structure primitive feature data from the multi-modal sample data set through a convolution layer and a spatial pooling operation, and then extracts defect primitive feature data from the structure primitive feature data through an attention mechanism; a high-level mode adaptation module uses a multi-head attention mechanism to associate and combine the structure primitive feature data and the defect primitive feature data, to generate a mode joint recognition representation.

[0024] Specifically, the primitive feature meta-learning task and the mode adaptation meta-learning task are specifically:

[0025] The primitive recognition subtask is used for matching the primitive mode to obtain local mode features, and the primitive feature space is generated by combining the local mode features of all sample subtasks.

[0026] Preferably, the multi-layer meta-learning architecture is used to extract the feature representation of the mixed-flow workpiece image to be identified, and the similarity of the pattern joint recognition representation of the multi-modal sample data set is compared to generate the workpiece category identification, defect type identification and abnormality index, and the specific process includes:

[0027] The multi-modal image data of the mixed-flow workpiece to be identified is obtained, and the multi-layer meta-learning architecture is used to extract the feature representation of the mixed-flow workpiece image to be identified; the similarity of the feature representation and the pattern joint recognition representation of the multi-modal sample data set is analyzed, and the similar candidate workpiece set is generated through the similarity analysis result; the workpiece category identification, defect type identification and abnormality index are generated according to the similar candidate workpiece set.

[0028] Compared with the prior art, the beneficial effects of the present application are:

[0029] 1. The multi-spectral imaging equipment is used to obtain multi-modal workpiece image data, and the multi-modal digital twin of the mixed-flow workpiece is generated by combining the workpiece structure data and material process parameters. The characteristics of the workpiece can be more comprehensively reflected, the accurate mapping between the physical world and the digital world is realized, a high-fidelity virtual platform is provided for subsequent defect injection and pattern expansion, and the difficulty and high cost of directly obtaining rare samples from the real production line are avoided.

[0030] 2. The pattern of the multi-modal digital twin is expanded, the multi-modal digital twin is injected with structure and defects according to the workpiece structure primitive pattern and visual defect primitive pattern, and the multi-modal sample data set is constructed through domain distribution disturbance processing and simulation-realistic hybrid mapping. A large number of diversified and accurately labeled synthetic image data are systematically generated, the problems of real defect sample scarcity and high labeling cost are solved. The domain distribution disturbance processing increases the diversity and robustness of the synthetic data, the simulation-realistic hybrid mapping strategy effectively reduces the domain difference between the synthetic data and the real data, significantly improves the generalization ability of the model trained on simulation data to real scenes, avoids model overfitting and negative transfer, reduces the dependence on a large amount of real labeled data, and thus reduces the training cost.

[0031] 3.The application constructs a multi-layer meta-learning architecture containing a bottom layer feature recognition module and a high layer pattern adaptation module, extracts features from mixed flow workpieces, and generates pattern joint recognition representations. The generated pattern joint recognition representation has strong pattern representation capability and can effectively distinguish different workpieces and recognize various defects, including unknown defects, thereby improving the accuracy and versatility of pattern recognition. The meta-learning task is divided into a base element recognition meta-task and a pattern matching meta-task, so that the model not only can recognize specific workpieces and defects, but also can learn how to quickly adapt and recognize from new and small samples. By designing a hierarchical meta-learning architecture, the model can learn the combination rules and adaptation patterns between different base element features, so that it can quickly generalize when facing new workpiece types or defect combinations, thereby improving the robustness and adaptability of the model. The traditional method avoids the problem of frequent system adjustment and low efficiency caused by workpiece switching in mixed flow production, greatly improving the automation and intelligence level of the production line. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A flowchart of a visual recognition detection method for multi-workpiece mixed flow production according to an embodiment of the application;

[0033] Figure 2 A digital twin pattern expansion flowchart according to an embodiment of the application;

[0034] Figure 3 A structure diagram of a multi-layer meta-learning architecture according to an embodiment of the application;

[0035] Figure 4 A visual recognition detection flowchart for multi-workpiece mixed flow production according to an embodiment of the application. DETAILED DESCRIPTION

[0036] Next, the technical solutions of the application will be explained in detail and clearly in combination with specific examples of the application and related drawings. It should be noted that the examples described below only represent a part of the application, not the whole. Based on these examples, those skilled in the art can deduce other possible implementations of the application without creative effort, and these implementations are also protected by the application.

[0037] A visual recognition detection method for multi-workpiece mixed flow production according to an embodiment of the application, the specific implementation steps are as follows:

[0038] Obtain multi-modal workpiece image data using a multi-spectral imaging device; obtain workpiece structure data and material process parameters, and combine the multi-modal workpiece image data to construct a multi-modal digital twin of mixed flow workpieces;

[0039] Expand the pattern of the multi-modal digital twin to construct a multi-modal sample data set;

[0040] inputting the multi-modal sample dataset into a multi-layer meta-learning architecture, wherein a bottom layer feature recognition module is used to extract structure primitive feature data and defect primitive feature data, and a high layer mode adaptation module is used to generate a mode joint recognition representation according to the primitive feature data;

[0041] extracting feature representation of the mixed-flow workpiece image to be recognized by using the multi-layer meta-learning architecture, and performing similarity comparison on the mode joint recognition representation of the multi-modal sample dataset, to generate the workpiece category identification, the defect type identification and the abnormality index.

[0042] The technical solutions of the present application will be further described in detail below in combination with specific embodiments.

[0043] Embodiment one

[0044] This embodiment takes an engine blade production line as an example to elaborate the specific steps of the method. A production line simultaneously produces nickel-based alloy high-pressure turbine blades (blade A) of GE90 engines and single-crystal high-temperature alloy fan blades (blade B) of CFM56 engines.

[0045] Referring to Figure 1 , the specific implementation steps of the method proposed by the present application include:

[0046] S1, acquiring multi-modal workpiece image data by using a multi-spectral imaging device; acquiring workpiece structure data and material process parameters, and combining the multi-modal workpiece image data to construct a multi-modal digital twin of mixed-flow workpieces;

[0047] S2, expanding modes of the multi-modal digital twin to construct a multi-modal sample dataset;

[0048] S3, inputting the multi-modal sample dataset into a multi-layer meta-learning architecture, wherein a bottom layer feature recognition module is used to extract structure primitive feature data and defect primitive feature data, and a high layer mode adaptation module is used to generate a mode joint recognition representation according to the primitive feature data;

[0049] S4, extracting feature representation of the mixed-flow workpiece image to be recognized by using the multi-layer meta-learning architecture, and performing similarity comparison on the mode joint recognition representation of the multi-modal sample dataset, to generate the workpiece category identification, the defect type identification and the abnormality index.

[0050] Further, multi-modal workpiece image data is acquired by using a multi-spectral imaging device; workpiece structure data and material process parameters are acquired, and the multi-modal workpiece image data is combined to construct a multi-modal digital twin of mixed-flow workpieces, which corresponds to the above S1 step, and the specific process is as follows:

[0051] On this production line, a multispectral imaging detection equipment integrating visible light camera, infrared thermal imager and ultraviolet fluorescence imaging equipment is installed.

[0052] When the blade passes through the detection area, the visible light camera captures the blade surface image in the visible light band, generates the first workpiece image data, which reflects the geometric profile, surface texture and color features of the workpiece. The infrared thermal imager performs thermal imaging on the blade, generating the second workpiece image data, which reflects the thermal radiation features and internal material transmission features of the material. The ultraviolet fluorescence imaging equipment excites the fluorescent agent at the site where the trace pollution or cracks on the blade surface may exist, captures the fluorescence image, and generates the third workpiece image data, which reflects the fluorescence response features and ultraviolet absorption features of the material.

[0053] The first, second and third workpiece image data obtained are subjected to joint denoising processing based on wavelet transform. The different modal images are registered, then two-dimensional wavelet decomposition is applied, the images are decomposed in different scales and directions, and the wavelet coefficients are subjected to threshold processing to suppress noise. Finally, the images are reconstructed by inverse wavelet transform. In this way, random noise, background interference existing in single modal images can be effectively removed, and the signal-to-noise ratio of the images is enhanced, generating multi-modal workpiece image data.

[0054] Through visible light, infrared, ultraviolet multispectral data fusion, more comprehensive visual information (geometry, texture, color, thermal radiation, fluorescence response) of the workpiece is captured, which can more effectively identify defects (such as internal bubbles, oil stains) that are difficult to find by traditional methods, and improve the information richness and signal-to-noise ratio of the original data.

[0055] The three-dimensional CAD model data of blade A and blade B are obtained, which contains the geometric shape, dimensional tolerance and internal structure of the blade; the detailed material process parameters of blade A and blade B are obtained, including density, hardness, elastic modulus, surface roughness and thermal conductivity, to construct a basic digital twin for each blade type. The multi-modal perception features extracted from the multi-modal workpiece image data are bound and mapped with the basic digital twin, which associates the real-time multi-modal perception data of the real workpiece in the physical world with the digital model in the virtual world, forming a dynamic and real-time updated multi-modal digital twin.

[0056] Combined with CAD structure and material process parameters, a digital twin with real physical properties and reflecting multispectral characteristics is constructed. The accurate mapping between the physical world and the digital world is realized, providing a high-fidelity virtual platform for subsequent defect injection and mode expansion, avoiding the difficulty and high cost of directly obtaining rare samples from the real production line.

[0057] Further, the multi-modal digital twin is subjected to mode expansion, and a multi-modal sample data set is constructed, corresponding to the above S2 step, and the specific process is as follows:

[0058] Referring to Figure 2 , a library of parametric primitive pattern models is established, and the workpiece structure primitives include blade airfoil curves, blade root structures, and blade tip structures, which can be parameterized as airfoil thickness, chord length, bending angle, and their adjustable ranges. The visual defect primitives include surface scratches, cracks, bubbles, coating peeling, and incomplete welds, which can be parameterized as corresponding length, width, depth, direction, position, number, and their adjustable ranges.

[0059] From the library of parametric primitive pattern models, workpiece structure primitives and visual defect primitives are selected for instantiation, and the parameter adjustable ranges of the corresponding primitives are obtained.

[0060] According to the parameter adjustable ranges, structural morphology transformation and local material property transformation are performed on the multi-modal digital twin, generating a large number of extended digital twins with different micro-deformations or local defects. Based on these extended digital twins, virtual visual sensors are used to render multi-modal views. This can simulate the image acquisition process under various lighting conditions and camera angles in the real detection environment. Through rendering, multi-modal extended digital twin images are generated that are highly similar in appearance and physical properties to real multi-modal workpiece image data. At the same time, accurate labeling data corresponding to each image can be accurately obtained. Domain distribution perturbation processing and simulation-reality hybrid mapping are performed on the multi-modal extended digital twin to construct a multi-modal sample dataset.

[0061] Through parameterized workpiece structure primitives and visual defect primitives injection, and multi-modal view rendering, a large number of diverse, accurately labeled synthetic image data are systematically generated. The problem of real defect sample scarcity and high labeling cost is solved, and the model's ability to recognize different workpiece structures and defect combinations is increased through structural morphology transformation.

[0062] Specifically, the domain distribution perturbation processing and simulation-reality hybrid mapping implementation process includes:

[0063] To bridge the domain gap between simulation data and real data and improve the model's generalization ability, domain distribution perturbation processing is performed on the multi-modal extended digital twin images.

[0064] Random domain perturbation parameters are generated through random sampling methods, including image geometric transformation parameters (e.g., random rotation angle -5° to +5°, scaling factor 0.95 to 1.05), lighting parameters (e.g., random adjustment of brightness, contrast, exposure intensity), color parameters (e.g., random color jitter), noise parameters (e.g., superimposed Gaussian noise, salt and pepper noise), and imaging quality parameters (e.g., blur kernel size, JPEG compression quality).

[0065] Based on these random domain disturbance parameters, the multi-modal extended digital twin image is superimposed and fused and nonlinearly transformed to generate a multi-modal digital twin image after disturbance.

[0066] The simulation-reality hybrid mapping is performed. The mode feature extraction is performed on the multi-modal digital twin image after disturbance and the real workpiece mode data by using a feature extraction network based on ResNet. Based on the Wasserstein distance, the mode distribution difference degree between them is calculated. According to the mode distribution difference degree, the feature alignment is performed on the multi-modal digital twin image after disturbance and the real workpiece mode data by using a metric learning method, so that the feature distribution of the simulation data is as close as possible to the feature distribution of the real data, and a multi-modal digital twin image after alignment is generated.

[0067] Specifically, the metric learning method is specifically:

[0068] Combined with the sample data in the simulation domain and the real domain, a triple is constructed, including an anchor sample, a sample of the same category but different domains and a sample of different categories, and optimization learning is performed by using a triple loss function, so that the samples from the simulation domain and the real domain but belonging to the same category are closer to each other, and the samples of different categories are far away from each other.

[0069] The recognition and detection precision and generalization ability of the model for multiple workpieces and various defects in the complex and changeable real production environment are significantly improved, and the dependence on expensive and scarce real labeled data is greatly reduced.

[0070] The multi-modal digital twin images after alignment are merged with a small amount of real workpiece mode data collected from the real production line, and combined with accurate labeled data, and finally a multi-modal sample data set is constructed.

[0071] Through the domain distribution disturbance processing, the diversity and robustness of the synthetic data are increased, so that the complex changes in the real world are simulated. The simulation-reality hybrid mapping strategy effectively reduces the domain difference between the synthetic data and the real data through feature alignment, significantly improves the generalization ability of the model to the real scene after training on the simulation data, and avoids model overfitting and negative transfer.

[0072] Further, the multi-modal sample data set is input into a multi-layer meta-learning architecture, wherein a bottom feature recognition module is used to extract structure primitive feature data and defect primitive feature data, and a high-level mode adaptation module is used to generate a mode joint recognition representation according to the primitive feature data. According to the above S3 step, the specific process is:

[0073] Reference Figure 3The constructed multi-modal sample dataset is input to a customized multi-layer meta-learning architecture for training, wherein the multi-layer meta-learning architecture comprises a bottom layer feature recognition module and a high layer mode adaptation module.

[0074] The bottom layer feature recognition module comprises:

[0075] The module uses a deep convolutional neural network based on EfficientNet as a feature extractor, combines multi-level convolutional layers and spatial pooling for feature extraction in the multi-modal sample dataset, first obtains structural primitive features from the input image, and obtains shared bottom layer feature representations containing semantic information, and then further extracts defect feature from the defect primitive features in the input image through the attention mechanism.

[0076] The high layer mode adaptation module comprises:

[0077] The module uses a multi-head attention mechanism to associate and combine the structural primitive feature data and the defect primitive feature data extracted by the bottom layer module. Through the multi-head attention mechanism, the complex dependency relationship between different primitive features can be captured to generate a mode joint recognition representation.

[0078] Feature representations of the same workpiece type and the same defect type will be clustered together, while feature representations of different types or abnormalities will be far apart from each other. For example, normal samples of blade A form a cluster in space, samples of blade A with cracks form another cluster, and normal samples and defect samples of blade B also form different clusters.

[0079] Through parallel extraction of workpiece structural primitives and defect primitives at the bottom layer, and multi-head attention association and combination of the two at the high layer, a mode joint recognition representation is generated. The generated mode joint recognition representation has strong pattern representation ability and can effectively distinguish different workpieces and recognize multiple defects, including unknown defects, thereby improving the accuracy and versatility of pattern recognition.

[0080] The meta-learning task is divided into primitive recognition meta-tasks and pattern matching meta-tasks, so that the model can not only recognize specific workpieces and defects, but also learn how to quickly adapt and recognize from new and small samples, i.e. achieve the ability of "learning to learn". By designing a hierarchical meta-learning architecture, the model can learn the combination rules and adaptation patterns between different primitive features, so that it can quickly generalize when facing new workpiece types or defect combinations, thereby improving the robustness and adaptability of the model.

[0081] Further, using the multi-layer meta-learning architecture, the feature representation of the to-be-recognized mixed flow workpiece image is compared with the mode joint recognition representation of the multi-modal sample dataset for similarity, to generate a workpiece category identifier, a defect type identifier, and an abnormality index. Corresponding to the above S4 step, the specific process is as follows:

[0082] On the production line, when the mixed-flow workpiece to be identified enters the detection area, the multispectral imaging device is used to acquire multi-modal image data of the blade to be identified and perform joint denoising. The multi-modal image data is input into the trained multi-layer meta-learning architecture to extract the feature representation of the mixed-flow workpiece image to be identified. The feature representation of the mixed-flow workpiece image to be identified is compared with the pattern joint recognition representation for similarity, and the similar candidate workpiece set is generated through the similarity analysis result. For example, if the feature representation of the blade to be detected is very close to the feature cluster of blade A normal sample, it is considered to be blade A. If it is close to the feature cluster of the crack defect of blade A, it may be a blade A with a crack.

[0083] The final detection result is generated based on the similarity ranking. The workpiece class identifier and defect type identifier of the workpiece to be detected are determined according to the number of workpiece types and defect types in the similar candidate workpiece set. According to the feature representation of the workpiece in the similar candidate workpiece set, a multi-dimensional abnormality index is extracted, including structural integrity abnormality, surface quality abnormality, material attribute abnormality, and functionality abnormality. The average multi-dimensional abnormality index of the similar candidate set is calculated to generate the abnormality index of the workpiece to be detected.

[0084] The multi-dimensional abnormality vector can decompose the overall abnormality of the workpiece into specific dimensions, thereby performing more accurate fault diagnosis and greatly shortening the problem troubleshooting time, and providing a more interpretable result.

[0085] Online and real-time mixed-flow workpiece recognition and defect detection are realized. By mapping the features of the workpiece to be identified to the constructed feature space for similarity comparison, the workpiece class identifier, defect type identifier and abnormality index are quickly generated, providing accurate quality control information for the production line and improving the production automation level and product quality.

[0086] By constructing the multi-modal digital twin of the mixed-flow workpiece and the multi-layer meta-learning architecture, the problem of frequent system adjustment and low efficiency caused by workpiece switching in traditional methods in mixed-flow production is avoided, greatly improving the automation and intelligence level of the production line. Combined with the multi-modal data (vision, infrared, ultraviolet) obtained by the multispectral imaging device, and combined with the workpiece structure data and material process parameters, the multi-modal digital twin constructed can more comprehensively represent the characteristics of the workpiece, thereby realizing the recognition of defects (such as internal defects and material abnormalities) that are difficult to be found by traditional single vision, significantly improving the detection accuracy and coverage. The multi-modal sample data set is constructed by using the pattern expansion technology, and the model can learn general and transferable features from limited samples through the learning of the multi-layer meta-learning architecture, thereby having stronger recognition ability for new types of workpieces or unknown defects, reducing the dependence on a large amount of real annotation data, and reducing the training cost.

[0087] Example Two

[0088] This example takes the busbar in the new energy vehicle battery module as an example. The busbar is a key conductive component, and there may be defects such as scratches, oxidation, foreign matter, and size deviation on the surface.

[0089] In the automatic detection station of the new energy vehicle battery module production line, a visual detection system integrating multiple spectral imaging capabilities is deployed. The system includes:

[0090] A visible light imaging module is equipped with a 5 million pixel color industrial camera and configured with a ring-shaped LED white light source.

[0091] An infrared imaging module is equipped with a short-wave infrared camera with a detector resolution of 640x512 and a spectral response range of 0.9μm to 1.7μm, and configured with an infrared LED light source of 940nm wavelength.

[0092] An ultraviolet imaging module is equipped with an ultraviolet camera with a resolution of 1624x1224 and a spectral response range of 200nm to 400nm, and configured with an ultraviolet LED light source of 365nm wavelength.

[0093] The PLC controller synchronously triggers each camera to perform image acquisition, and transmits the image data to the image processing workstation in real time.

[0094] The visible light imaging module acquires first workpiece image data of the busbar in the visible light band, which reflects the surface color, macro texture and geometric contour of the busbar; the infrared imaging module acquires second workpiece image data of the busbar in the infrared light band, which is in grayscale format and reflects the absorption characteristics of the busbar material to short-wave infrared, revealing internal structural defects and the response of the surface oxidation layer to infrared light; the ultraviolet imaging module acquires third workpiece image data of the busbar in the ultraviolet light band, which is in grayscale format and reflects the fluorescence response characteristics and ultraviolet absorption characteristics of the busbar surface under ultraviolet light.

[0095] After obtaining the first workpiece image data, the second workpiece image data and the third workpiece image data, they are input into the image processing unit for joint denoising processing. This example adopts a joint denoising algorithm based on non-local mean filtering combined with three-dimensional wavelet transform.

[0096] Obtain the three-dimensional CAD model data of the busbar model to be constructed and the detailed physical properties and manufacturing process parameters of the copper alloy material used for the busbar, and construct the basic digital twin of the busbar.

[0097] The features extracted from the multi-modal workpiece image data are bound and mapped to the constructed base digital twin, generating a multi-modal digital twin of the busbar. This process transforms the digital twin from a static model to a dynamic copy that can reflect the state of the physical world in real time.

[0098] Embodiment Three

[0099] This embodiment takes the mixed production line of two different specifications of busbars (Type-A busbar and Type-B busbar) in new energy vehicle battery modules as an example. Type-A busbar is a rectangular copper bar, which may have micron-level scratches and oxidation spots on the surface; Type-B busbar is an L-shaped copper bar, which may have bending deformation and welding splash defects.

[0100] A parameterized model library containing common workpiece structure primitives and visual defect primitives of Type-A and Type-B busbars is constructed.

[0101] Workpiece structure primitive mode:

[0102] For the rectangular copper bar body mode (applicable to Type-A copper bar), its main geometric features are quantified by three parameters: represents the length, whose value range is between 149.5mm and 150.5mm; represents the width, whose value range is between 19.8mm and 20.2mm; represents the thickness, whose value range is between 2.9mm and 3.1mm.

[0103] For the L-shaped copper bar body mode (applicable to Type-B copper bar), it is defined by four parameters: represents the length of the long side; represents the length of the wide side; represents the thickness; represents the bending angle of the copper bar.

[0104] For the standard mounting hole mode common to all types of copper bars, we use three parameters to describe it: represents the diameter of the hole, whose value range is between 4.9mm and 5.1mm; and represents the coordinates of the hole center in the two-dimensional plane.

[0105] Visual defect primitive mode:

[0106] For the micron-level scratch mode (universal), it is described by four parameters: represents the length of the scratch, ranging from 0.1mm to 5mm; represents the width of the scratch, ranging from 0.01mm to 0.2mm; represents the depth of the scratch, ranging from 0.001 mm to 0.1 mm; and represents the variation of surface roughness in the scratch area.

[0107] For the oxidation spot pattern (general), we use three parameters to quantify: represents the area of the oxidation spot, ranging from 0.5 mm 2 to 10 mm 2 ; represents the color deviation of the oxidation spot, i.e., the RGB value offset relative to the standard copper color; represents the oxidation intensity.

[0108] For the L-shaped copper bar-specific defect pattern, we define the following two:

[0109] The bending deformation pattern consists of two parameters: represents the additional bending angle, ranging from 1° to 5°; represents the location where the bending occurs.

[0110] The welding spatter pattern is also defined by three parameters: represents the number of spatters, ranging from 1 to 5; represents the size of the spatters, ranging from 0.1 mm to 0.3 mm; represents the location where the spatters appear.

[0111] Based on a random strategy, samples with specific structural primitives and random defect primitives are generated for each type of busbar model, and are instantiated. Adjustable parameter values for each instantiated primitive pattern are obtained through random sampling methods. For example, for a bending deformation on a Type-B busbar, the additional bending angle parameter value is randomly set to 2.5°, and its specific location at the L-shaped bending is randomly assigned.

[0112] Based on the adjustable parameter values obtained above, the multi-modal digital twin is expanded in pattern, including structural morphology transformation and local material property transformation, to generate an expanded digital twin.

[0113] Based on the extended digital twin, multi-modal view rendering is performed using virtual visual sensors. In the simulation environment, virtual visible light cameras, infrared cameras, ultraviolet cameras similar to the actual production line are configured, and sensor inherent noise is simulated. Simulate the physical interaction of light on the surface of the extended digital twin, generate multi-modal synthetic image data with identifiable workpiece structure primitives and visual defect primitives. For example, generate Type-A busbar visible light images that clearly show minor deviations in body size, Type-B busbar depth images with slight bending deformation, and ultraviolet images with surface oxidation spots. The rendering system automatically records the precise location, geometry, type, size, attribute change amount, and workpiece category identifier of each primitive mode while generating images, generating accurate annotation data.

[0114] By random sampling method, a unique random domain perturbation parameter is generated for each rendered multi-modal extended digital twin image, including image geometric transformation parameters, lighting parameters, color parameters, noise parameters, and imaging quality parameters.

[0115] Based on the random domain perturbation parameters, the pixel values of the multi-modal extended digital twin are superimposed and fused and nonlinearly transformed to generate a perturbed multi-modal digital twin.

[0116] Using an EfficientNet-based feature extraction network, the mode feature vectors of the perturbed multi-modal digital twin and the real workpiece mode data are extracted. In the feature space, the mode distribution difference between the perturbed multi-modal digital twin (synthetic features) and the real workpiece mode data (real features) is calculated. This embodiment uses maximum mean difference as the difference measure index. According to the mode distribution difference, the perturbed multi-modal digital twin and the real workpiece mode data are aligned using a metric learning method, and combined with the accurate annotation data to form a multi-modal sample data set.

[0117] Embodiment Four

[0118] This embodiment takes the mixed production line of Type-A and Type-B busbars in new energy vehicle battery modules as an example.

[0119] The multi-modal sample data set is input to a multi-layer meta-learning architecture for training. This embodiment uses 12,000 images with accurate annotations containing Type-A and Type-B busbars, and uses a meta-learning architecture based on MAML. The training process is iteratively trained for 100 epochs using the Adam optimizer.

[0120] According to the meta-learning task, the bottom feature recognition module of the multi-layer meta-learning architecture extracts structural primitive feature data and defect primitive feature data from the multi-modal sample data set. The busbar image data in the multi-modal sample data set is input into the bottom feature recognition module, wherein the bottom feature recognition module is constructed based on EfficientNet and includes multiple convolution layers and spatial pooling layers. The bottom feature recognition module includes a special branch that extracts workpiece structural primitive feature data from busbar image data through global average pooling and fully connected layers. For example, this branch learns to identify the rectangular body features of Type-A busbars, including the aspect ratio, the shape of the four corners, and the L-shaped body features and bending angles of Type-B busbars. Another branch of the bottom feature recognition module extracts defect primitive feature data from the feature map based on the workpiece structural primitive feature data through an attention mechanism. For example, this branch learns to identify the line discontinuity of micron-level scratches, the color / texture abnormal pattern of oxidation spots, and the geometric distortion features of bending deformation. Finally, the extracted structural primitive feature data and defect primitive feature data are output in parallel.

[0121] According to the pattern adaptation meta-learning task, the high-level pattern adaptation module associates and combines the structural primitive feature data and the defect primitive feature data to generate a pattern joint recognition representation. The structural primitive feature data and the defect primitive feature data are input into the high-level pattern adaptation module, and a multi-head attention mechanism is used to associate and combine the structural primitive feature data and the defect primitive feature data. Each attention head learns the association relationship from different perspectives in parallel, for example, one head may focus on the association between scratches and the surface of the body, and another head focuses on the association between bending deformation and L-shaped structure, to generate a pattern joint recognition representation.

[0122] Embodiment Five

[0123] Reference Figure 4 This embodiment aims to use the pattern joint recognition representation that has been constructed to make real-time judgments on the busbars to be identified on the production line, and to demonstrate the visual recognition inspection process of mixed flow production of multiple workpieces.

[0124] On the production line, when a new busbar enters the detection station, its multi-modal image data is synchronously acquired by a multi-spectral imaging device. The multi-modal image data is input into the trained multi-layer meta-learning architecture. The bottom feature recognition module and the high-level pattern adaptation module of the framework perform forward propagation processing on it to generate the pattern joint recognition representation of the busbar to be identified.

[0125] The feature representation of the to-be-identified manifold is subjected to similarity analysis with the mode joint identification representation of the multi-modal sample data set. According to the similarity analysis result, a similar candidate workpiece set is generated. In the embodiment, the similarity threshold is set to 0.85, that is, all sample modes with a similarity higher than 0.85 to the to-be-identified manifold feature are included in the similar candidate workpiece set. The workpiece category and defect type with the most in the similar candidate set are selected to generate the workpiece category identifier and the defect type identifier. According to the feature representation of the workpiece in the similar candidate workpiece set, multi-dimensional abnormality indexes are extracted, including structural integrity abnormality, surface quality abnormality, material attribute abnormality and functionality abnormality. The average multi-dimensional abnormality index of the similar candidate set is calculated to generate the abnormality index of the to-be-measured workpiece.

[0126] Although specific examples of the present application have been shown, those of ordinary skill in the art should understand that various adjustments, modifications, replacements and improvements to these examples are possible without departing from the core principles and spirit of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A visual recognition detection method for mixed production of multiple workpieces, characterized in that: The following steps are involved: Use multispectral imaging equipment to obtain multimodal workpiece image data, including workpiece image data in the visible light, infrared light, and ultraviolet light bands; Acquire workpiece structural data and material process parameters to construct a basic digital twin; bind and map multimodal workpiece image data to the basic digital twin to generate a multimodal digital twin of the mixed-flow workpiece; Performing pattern expansion on the multimodal digital twin to construct a multimodal sample data set; The construction process of the multimodal sample data set includes: selecting a workpiece structure primitive pattern and a visual defect primitive pattern from a primitive pattern parameter model library for instantiation; obtaining adjustable parameter values ​​of the instantiated primitive patterns; performing pattern expansion on the multimodal digital twin, performing structural morphological transformation and local material property transformation on the multimodal digital twin according to the adjustable parameter values, and generating an extended digital twin; based on the extended digital twin, using a virtual visual sensor to perform multimodal view rendering, generating a multimodal extended digital twin and precise annotation data, including primitive pattern type, shape, size, attribute change and workpiece category identification; performing domain distribution perturbation processing and simulation-reality hybrid mapping on the multimodal extended digital twin to construct a multimodal sample data set; Inputting the multimodal sample dataset into a multi-layer meta-learning architecture, wherein the bottom-level feature recognition module is used to extract structural primitive feature data and defect primitive feature data, and the high-level pattern adaptation module is used to generate a pattern joint recognition representation based on the primitive feature data; Using a multi-layer meta-learning architecture, the feature representation of the mixed-flow workpiece image to be identified is extracted, and the similarity is compared with the feature representation of the multimodal sample dataset to generate the workpiece category identification, defect type identification and abnormality index.

2. The visual recognition detection method for mixed-flow production of multiple workpieces according to claim 1, characterized in that: The method of obtaining multimodal workpiece image data using a multispectral imaging device specifically includes: collecting visual data using the multispectral imaging device, including first workpiece image data in a visible light band, second workpiece image data in an infrared light band, and third workpiece image data in an ultraviolet light band; and integrating the first workpiece image data, the second workpiece image data, and the third workpiece image data for joint denoising to generate multimodal workpiece image data.

3. The visual recognition detection method for mixed-flow production of multiple workpieces according to claim 1, characterized in that: The construction process of the multimodal digital twin includes: obtaining workpiece structural data and material process parameters, wherein the workpiece structural data includes three-dimensional CAD model data, and the material process parameters include density, hardness, elastic modulus, surface roughness and thermal conductivity; constructing a basic digital twin based on the workpiece structural data and material process parameters; obtaining different data according to three different spectral images; extracting the geometric contour features, surface texture features and color features of the workpiece from the visible light image part; extracting the fluorescence response features and ultraviolet absorption features of the workpiece from the ultraviolet light image part; extracting the thermal radiation features and material transmission features of the workpiece from the infrared light image part; binding and mapping the multimodal workpiece image data with the basic digital twin to generate a multimodal digital twin.

4. The visual recognition detection method for mixed-flow production of multiple workpieces according to claim 1, characterized in that: The specific implementation process of the domain distribution perturbation processing and simulation-reality hybrid mapping includes: generating random domain perturbation parameters through a random sampling method, including image geometric transformation parameters, illumination parameters, color parameters, noise parameters and imaging quality parameters; based on the random domain perturbation parameters, superimposing, fusing and nonlinearly transforming the multimodal extended digital twin to generate a perturbed multimodal digital twin; extracting pattern features of the perturbed multimodal digital twin and the real workpiece pattern data, and calculating the pattern distribution difference; according to the pattern distribution difference, aligning the features of the perturbed multimodal digital twin and the real workpiece pattern data to generate an aligned multimodal digital twin; merging the aligned multimodal digital twin and the real workpiece pattern data, and combining the precisely labeled data to generate a multimodal sample data set.

5. The visual recognition detection method for mixed-flow production of multiple workpieces according to claim 1, characterized in that: The process of forming the joint recognition representation of the modalities is as follows: according to the meta-learning task, the multimodal sample dataset is input into a multi-layer meta-learning architecture; The underlying feature recognition module extracts structural primitive feature data from the multimodal sample data set through convolutional layers and spatial pooling operations, and then extracts defect primitive feature data through the attention mechanism based on the structural primitive feature data; the high-level pattern adaptation module uses a multi-head attention mechanism to associate and combine the structural primitive feature data with the defect primitive feature data to generate a joint pattern recognition representation.

6. The visual recognition detection method for mixed-flow production of multiple workpieces according to claim 5, characterized in that: The meta-learning tasks are specifically as follows: the meta-learning tasks include primitive recognition meta-tasks and pattern matching meta-tasks; the primitive recognition meta-task is composed of primitive recognition sub-tasks, each primitive recognition sub-task randomly selects different workpiece types, performs primitive pattern matching training, and obtains local pattern features; combines the local pattern features of all primitive recognition sub-tasks to generate a primitive feature space; the pattern matching meta-task is composed of pattern combination sub-tasks, each pattern combination sub-task randomly extracts different primitive feature combination units, performs primitive combination association, and generates a local primitive feature representation; combines the local pattern features of all pattern combination sub-tasks to generate a pattern joint recognition representation.

7. The visual recognition detection method for mixed-flow production of multiple workpieces according to claim 1, characterized in that: The multi-layer meta-learning architecture is used to extract the feature representation of the mixed-flow workpiece image to be identified, and the similarity is compared with the feature representation of the multimodal sample dataset to generate the workpiece category identification, defect type identification and abnormality index. The specific process includes: Acquire multimodal image data of the mixed-flow workpiece to be identified, and use a multi-layer meta-learning architecture to extract the feature representation of the mixed-flow workpiece image to be identified; perform similarity analysis with the multimodal sample data set, and generate a set of similar candidate workpieces based on the similarity analysis results; based on the set of similar candidate workpieces, generate workpiece category identification, defect type identification and abnormality index.

Images