Character defect image recognition method for scribing type marking machine

Abstract

The invention discloses a character defect image recognition method for a scribing type marking machine, and relates to the technical field of deep learning and image processing. The problem that the traditional detection technology is low in similar character defect recognition accuracy is solved. The method comprises the following steps: constructing a defect detection model, wherein the defect detection model comprises a backbone network, a feature focusing diffusion calibration network and a task interaction detector; the backbone network is used for carrying out feature extraction and outputting three feature maps P0 to P2 with different scales; the feature focusing diffusion calibration network is used for performing feature fusion, extraction and calibration on the feature maps P0 to P2, and then outputting three calibrated feature maps P3 to P5 with different scales; the task interaction detector is used for performing defect position and category prediction on the feature maps P3 to P5; and training and verifying the defect detection model by using the constructed defect data set, and performing defect identification on the character image by using the trained defect detection model. The method is mainly used for identifying the character defect image of the marking machine.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning and image processing. Background Art

[0002] Marking product information on the surface of a product by scratching is a common method in the field of industrial production. The character types generally include English letters and numbers. With the improvement of production efficiency, the quality requirements for marking characters on metal parts are constantly increasing. During the production process, the marking needle, a key component of the pneumatic marking machine, will gradually wear out over time; the air pump pressure of the pneumatic marking machine will also decay over time, resulting in changes in the depth of character engraving, and then problems such as missing pens and broken pens. Therefore, the automatic detection and automatic identification of marking character defects are of great practical significance to the actual production of industrial products.

[0003] Machine vision technology is a common method to solve this problem. Through the coordination of reasonable industrial cameras, lenses, and light sources, clear pictures of the marked characters are taken, and then image processing or deep learning algorithms are used to automatically identify character defects. Since there are 36 categories of English letters and Arabic numerals, the number of categories is large, and there are similar characters (such as 2 and Z, 0 and O, 1 and I, etc.) that lead to many similar features of the target, the traditional image processing algorithm is not ideal when facing the character defect recognition of the scratch marking machine, and the defect recognition accuracy is low.

[0004] With the development and widespread application of deep learning algorithms, some researchers have used various deep learning algorithms to detect character defects and achieved certain results. However, due to the large number of different scale features and similar features of different targets in the characters of the engraved marking machine, as well as the information loss of the traditional detection model and the lack of interaction between the independent branches of the detector, the recognition accuracy still cannot meet the actual needs.

[0005] To solve the character recognition problem, researchers have proposed several solutions:

[0006] 1. Chinese patent application No. 202110931270.4 provides a vehicle VIN code character recognition and character engraving depth detection system and detection method. The method places the workpiece to be measured at the initial scanning position of the measuring device; takes a photo of the VIN code for two-dimensional character recognition; and screens for obvious defects; scans by moving the workpiece to be measured or the sensor so that the laser bar of the sensor scans the VIN code character area, and takes photos continuously to obtain an image; solves three-dimensional information; reconstructs the character based on the three-dimensional result and compares it with the two-dimensional recognition result; if the three-dimensional recognition result is consistent with the two-dimensional recognition result, continues to output the detection result.

[0007] 2. Chinese Patent No. 202310351717.X provides a method for recognizing steel-stamped characters on paper medicine boxes based on an improved YOLOV5 model, the method comprising: using an image acquisition device to acquire a steel-stamped character image of a medicine box; performing image enhancement and other preprocessing on the image and labeling the recognition area; inputting the image into the improved YOLOV5 model for training to obtain a trained YOLOV5 model; the improved YOLOV5 model comprises: adding an efficient position attention mechanism CA (Coordinate Attention) to the backbone network of YOLOV5 so that the model network can pay attention to a wide range of position information without bringing too much computational effort, which helps to improve model performance and better locate and identify targets; replacing the SPPF fast pooling layer with SimSPPF to improve training speed; for dense small targets, an additional smaller prior box is used in the model and a correspondingly smaller detection head is added; the data set to be recognized is input for processing to obtain the final prediction result.

[0008] 3. Chinese Patent No. 202210367390.0 provides a steel billet spray-marking character recognition system based on deep learning, which includes an image acquisition device for acquiring raw image data; an inference server for receiving raw image data and forwarding the raw image data to a recognition module built into the inference server; the inference server is electrically connected to the image acquisition device; and the recognition module is used to perform spray-marking character recognition operations on the raw image data to obtain the slab number. The present invention does not need to reduce the speed of the steel billet to take pictures, thereby improving the efficiency of the production line; adapting to the environment of the collection and recognition location, greatly increasing the applicable setting locations; and recognizing rusted and worn characters, greatly improving the recognition rate.

[0009] The method described in the above patent mainly focuses on the research of character recognition technology, but lacks targeted research on the problem of similar character defect detection under multi-category conditions. The existing method ignores the impact of similar character defects on traditional detection technology and models, resulting in a decrease in recognition accuracy. The above problems need to be solved urgently. Summary of the invention

[0010] The purpose of the present invention is to solve the problem of low accuracy in identifying similar character defects by conventional detection technology; the present invention provides a method for identifying character defect images of a scratch-type marking machine.

[0011] A character defect image recognition method for a scratch-type marking machine, the method comprising the following steps:

[0012] Mark the defect locations and defect categories in all original character images and establish a defect dataset;

[0013] Building a defect detection model:

[0014] The defect detection model includes a backbone network, a feature focused diffusion calibration network and a task interaction detector;

[0015] The backbone network is used to extract features and output three feature maps P0 to P2 of different scales; the feature focus diffusion calibration network is used to fuse, extract and calibrate the feature maps P0 to P2, and then output three calibrated feature maps P3 to P5 of different scales; the task interaction detector is used to predict the defect location and category of the feature maps P3 to P5;

[0016] Using defect data sets to train the constructed defect detection model;

[0017] The trained defect detection model is used to perform defect recognition on character images.

[0018] Preferably, the backbone network includes 5 RFCBAMConv modules, 4 C2f-RFCBAM modules and the SPPF module in the YOLO algorithm;

[0019] The five RFCBAMConv modules are defined as RFCBAMConv modules No. 1 to No. 5, and the four C2f-RFCBAM modules are defined as C2f-RFCBAM modules No. 1 to No. 4;

[0020] The No. 1 RFCBAMConv module, the No. 2 RFCBAMConv module, the No. 1 C2f-RFCBAM module, the No. 3 RFCBAMConv module, the No. 2 C2f-RFCBAM module, the No. 4 RFCBAMConv module, the No. 3 C2f-RFCBAM module, the No. 5 RFCBAMConv module, the No. 4 C2f-RFCBAM module and the SPPF module are cascaded in sequence;

[0021] RFCBAMConv module No. 1 is used to extract channel and spatial region attention mechanism features from the received original character image and output a feature map with multiple channels, wherein the original character image is used as the original feature map;

[0022] RFCBAMConv modules No. 2 to No. 5 are used to extract channel and spatial region attention mechanism features from the received multi-channel feature maps and output feature maps with multiple channels;

[0023] C2f-RFCBAM modules No. 1 to No. 4 are used to fuse the feature maps of multiple channels and output the fused feature maps; and the output ends of C2f-RFCBAM modules No. 2 and No. 3 are respectively used as the first and second output ends of the backbone network, and correspondingly output the first feature map P0 and the second feature map P1 with multiple channels;

[0024] The SPPF module is used to sequentially pool and aggregate the received feature maps, and the output end of the SPPF module serves as the third output end of the backbone network to output a third feature map P2 with multiple channels;

[0025] The feature maps P0, P1 and P2 have different scales.

[0026] Preferably, the C2f-RFCBAM module is implemented by replacing all Bottleneck modules in the C2f module with the RFCBAMneck module; and the RFCBAMneck module is formed by replacing the second convolution layer in the Bottleneck module with the RFCBAConv module.

[0027] Preferably, the feature focused diffusion calibration network includes 4 splicing modules, 2 RFCBAMConv modules, 2 feature aggregation modules, 2 downsampling modules, 4 C2f modules, 1 contextual feature calibration module and 1 spatial feature calibration module;

[0028] A concatenation module is used to concatenate the received feature maps and output the concatenated feature maps;

[0029] RFCBAMConv module, which is used to extract channel and spatial region attention mechanism features from the received feature map and output a feature map with multiple channels;

[0030] A feature aggregation module is used to extract and superimpose multi-scale features on multiple received feature maps in sequence;

[0031] A downsampling module, used for performing downsampling operations;

[0032] C2f module, used for single-scale feature extraction;

[0033] A context feature calibration module, used for context calibration;

[0034] A spatial feature calibration module, used for calibrating spatial features;

[0035] The 4 splicing modules are defined as splicing modules No. 1 to No. 4, respectively. The 2 RFCBAMConv modules are defined as RFCBAMConv modules No. 6 and No. 7, respectively. The 2 feature aggregation modules are defined as feature aggregation modules No. 1 and No. 2, respectively. The 2 downsampling modules are defined as downsampling modules No. 1 and No. 2, respectively. The 4 C2f modules are defined as C2f modules No. 1 to No. 4, respectively.

[0036] The first input end of the stitching module No. 1 is used to receive the feature map P0; the output end of the stitching module No. 1 is connected to the input end of the C2f module No. 1, and the output end of the C2f module No. 1 is connected to the first input end of the stitching module No. 2 and the first input end of the feature aggregation module No. 2; the output end of the stitching module No. 2 is connected to the input end of the C2f module No. 2, and the output end of the C2f module No. 2 is connected to the input end of the context feature calibration module, and the output end of the context feature calibration module is connected to the first input end of the spatial feature calibration module, and the context feature calibration module outputs the calibrated feature map P5;

[0037] The first to third input ends of feature aggregation module No. 1 are used to receive feature maps P0 to P2 respectively; the output end of feature aggregation module No. 1 is connected to the input end of RFCBAMConv module No. 6, the input end of downsampling module No. 1 and the second input end of feature aggregation module No. 2, the output end of RFCBAMConv module No. 6 is connected to the second input end of splicing module No. 1 and the second input end of splicing module No. 2 at the same time, and the output end of downsampling module No. 1 is connected to the second input end of splicing module No. 3 and the second input end of splicing module No. 3;

[0038] The first input end of the splicing module No. 3 is used to receive the feature map P2, the output end of the splicing module No. 3 is connected to the input end of the C2f module No. 3, the output end of the C2f module No. 3 is connected to the third input end of the feature aggregation module No. 2 and the first input end of the splicing module No. 4, the third input end of the splicing module No. 4 is connected to the output end of the down-sampling module No. 2, the output end of the splicing module No. 4 is connected to the input end of the C2f module No. 4, the output end of the C2f module No. 4 is connected to the first input end of the spatial feature calibration module, and the output end of the spatial feature calibration module outputs the feature map P3;

[0039] The output end of the feature aggregation module No. 2 is connected to the input end of the RFCBAMConv module No. 7 and the input end of the downsampling module No. 2 at the same time, and the output end of the feature aggregation module No. 2 outputs the feature map P4.

[0040] Preferably, the specific process of the feature aggregation module sequentially performing multi-scale feature extraction and feature superposition on the received multiple feature maps is as follows:

[0041] The feature aggregation module performs Adonw downsampling on the feature map received at its first input end, and performs convolution operation on the feature maps received at its second and third input ends to capture feature information, and performs Concat splicing operation on the three captured feature information to obtain a local feature map L;

[0042] The context information of the local feature map L is captured through four parallel deep convolutions of different sizes, and the captured four context information are superimposed and then convolved, and the convolved feature map is superimposed with the local feature map L and output.

[0043] Preferably, the RFCBAMConv module includes the channel attention mechanism of the SE framework and the spatial attention mechanism based on convolutional features;

[0044] The channel attention mechanism of the SE framework and the spatial attention mechanism based on convolutional features are both used to receive the same input feature map, wherein the channel attention mechanism of the SE framework extracts the channel attention mechanism features of the input feature map and then sends them to the spatial attention mechanism based on convolutional features;

[0045] The spatial attention mechanism based on convolutional features weights the input feature map and the feature map extracted by the channel attention mechanism of the SE framework in different spatial dimensions, and then outputs the features after feature fusion.

[0046] Preferably, the task interaction detector implements defect location and category prediction for the calibrated feature maps P3 to P5 of three different scales including:

[0047] The feature maps P3 to P5 are convolved to extract information through two cascaded parameter-sharing convolutional layers, and the extracted information is spliced ​​through Contact to generate multi-channel task interaction features. Figure X inter , the task interaction features Figure X inter It is sent to the target classification module and the detection and positioning module at the same time;

[0048] Detection and localization module for task interaction features Figure X inter Decompose the tasks separately and analyze the interaction characteristics of the tasks Figure X inter After calculating the spatial offset and selectively extracting the position information based on the spatial offset, the task is decomposed into features X task The feature values ​​at the same position as the extracted position information are directly added to perform feature fusion to obtain the defect position prediction result;

[0049] Object classification module pairs of task interaction features Figure X inter Decompose the tasks separately and analyze the interaction characteristics of the tasks Figure X inter After selectively extracting the target category information in a local weighted sum manner, the task is decomposed into features X task The feature value at the same position as the extracted target category information is directly multiplied to obtain the defect category prediction result.

[0050] Preferably, the task interaction feature Figure X inter The implementation method of task decomposition is as follows:

[0051] X task =wX inter ;

[0052] Among them, w is the weight coefficient, X task After decomposition of the task.

[0053] Preferably, w = σ(fc2(δ(fc1(x inter ))));

[0054] x inter For X inte r is obtained by average pooling, fc1(·) and fc2(·) are the first and second fully connected operations respectively, and δ(·) is the sigmoid function.

[0055] Preferably, the defect categories include extra strokes, broken strokes, twists, and missing strokes.

[0056] The beneficial effects brought by the present invention are

[0057] The present invention constructs a defect detection model, which consists of three parts, including a backbone network, a feature focused diffusion calibration network and a task interaction detector; the backbone network is used to perform feature extraction and output feature maps P0 to P2 of three different scales; the feature focused diffusion calibration network is used to perform feature fusion, extraction and calibration on the feature maps P0 to P2, and output three calibrated feature maps P3 to P5 of three different scales; the task interaction detector is used to predict the defect position and category of the feature maps P3 to P5; different feature types for positioning and classification are optimized, so as to enhance the model's ability to extract features for similar character defects, thereby achieving more accurate positioning and classification.

[0058] Furthermore, the present invention also improves the backbone network, uses RFCBAMConv to replace the convolution module in the backbone network, and transforms the C2f module in the backbone network to enhance the model's ability to extract features.

[0059] The feature fusion network is also improved, the traditional feature fusion method is changed, and an innovative feature aggregation module is proposed to be added to the network to enhance the ability and effect of network feature fusion.

[0060] The specific structure of the task interaction detector is also designed to enhance the task interaction between the target classification and localization branches to achieve more accurate positioning and classification.

[0061] The defect detection model of the present invention can greatly improve the problems of missed detection and false detection that occur in the model, so that the model can achieve higher detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of the principle of the character defect image recognition method of the scribing marking machine described in the present invention;

[0063] Figure 2 This is a schematic diagram of the structure of the original detection model of YOLOv8;

[0064] Figure 3 It is a structural schematic diagram of the defect detection model of the present invention;

[0065] Figure 4 It is a schematic diagram of the structure of the traditional CBAM module and the RFCBAMConv module of the present invention; wherein, Figure 4 a is the structural diagram of the traditional CBAM module. Figure 4 b is a schematic diagram of the structure of the RFCBAMConv module of the present invention;

[0066] Figure 5 It is a Bottleneck module in a traditional C2f module and an RFCBAMneck module in a C2f-RFCBAM module of the present invention; wherein, Figure 5 a is the original Bottleneck module, Figure 5 b is the RFCBAMneck module of the present invention;

[0067] Figure 6 It is a schematic diagram of the structure of the traditional C2f module and the C2f-RFCBAM module; Figure 6 a is a schematic diagram of the structure of a traditional C2f module. Figure 6 b is a schematic diagram of the structure of the C2f-RFCBAM module;

[0068] Figure 7 It is a schematic diagram of the principle of the feature aggregation module;

[0069] Figure 8 This is a schematic diagram of the principle of the task interaction detector;

[0070] Fig. 9 is a schematic diagram of character images of various defect types; among them, Fig. 9 a is a schematic diagram of multi-stroke character defects. Fig. 9 b is a schematic diagram of a broken character defect. Fig. 9 c is a schematic diagram of distorted character defects. Fig. 9 d is a schematic diagram of the defect of a character with fewer strokes.

[0071] Fig.10A schematic diagram of the structure of a scratch-type marking mechanism capable of realizing automated defect detection. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0074] In order to solve the problem of low accuracy of traditional detection technology in identifying similar character defects, the present invention is improved based on the YOLOv8 deep learning algorithm to perform visual detection of various types of character defects with similarities. The original network model of the YOLOv8 deep learning algorithm is shown in the attached figure. Figure 2 The overall structure of the improved defect detection model is shown in the attached Figure 3 As shown, a character defect image recognition method for a scratch-type marking machine is formed based on the improved defect detection model, as follows:

[0075] Specific implementation method 1. Figure 1 and Figure 3 The present embodiment describes a character defect image recognition method for a scratch-type marking machine, the method comprising the following steps:

[0076] Mark the defect locations and defect categories in all original character images and establish a defect dataset; as shown in the attached Fig. 9 As shown, each image may contain four types of defects, namely, extra strokes, broken strokes, distortion, and missing strokes;

[0077] Building a defect detection model:

[0078] The defect detection model includes a backbone network, a feature focused diffusion calibration network and a task interaction detector;

[0079] The backbone network is used to extract features and output three feature maps P0 to P2 of different scales; the feature focus diffusion calibration network is used to fuse, extract and calibrate the feature maps P0 to P2, and then output three calibrated feature maps P3 to P5 of different scales; the task interaction detector is used to predict the defect location and category of the feature maps P3 to P5;

[0080] Using defect data sets to train the constructed defect detection model;

[0081] The trained defect detection model is used to perform defect recognition on character images.

[0082] In this embodiment, the present invention constructs a defect detection model, which consists of three parts, including a backbone network, a feature focused diffusion calibration network and a task interaction detector; the backbone network is used to perform feature extraction and output feature maps P0 to P2 of three different scales; the feature focused diffusion calibration network is used to perform feature fusion, extraction and calibration on the feature maps P0 to P2, and output three calibrated feature maps P3 to P5 of three different scales; the task interaction detector is used to predict the defect position and category of the feature maps P3 to P5; the partition targeted processing method enhances the model's ability to extract features for similar character defects, thereby achieving more accurate positioning and classification.

[0083] In specific applications, the specific process of realizing online recognition of the marked character image is as follows: Fig.10 The trained defect detection model is embedded in the visual controller 2, and then the character information to be marked on the workpiece is input through the virtual keyboard in the PLC integrated controller 1. Then the PLC integrated controller 1 controls the scratch marking mechanism 3 to mark on the workpiece. After the marking is completed, the PLC integrated controller sends a command to the visual controller 2, and the visual controller 2 turns on the ring light source 4 and shoots the image of the marked characters through the shooting module 5. The visual controller 2 uses the trained defect detection model to perform defect recognition detection on the character image and feeds the results back to the PLC integrated controller, so that the staff can perform subsequent processing.

[0084] The present invention Figure 2 The improved YOLOv8 deep learning algorithm is used as the basis for the improvement. The specific structure of the backbone network in the improved defect detection model can be found in Figure 3 ,Specifically: the backbone network includes 5 RFCBAMConv modules, 4 C2f-RFCBAM modules and the SPPF module in the YOLO algorithm;

[0085] The five RFCBAMConv modules are defined as RFCBAMConv modules No. 1 to No. 5, and the four C2f-RFCBAM modules are defined as C2f-RFCBAM modules No. 1 to No. 4;

[0086] The No. 1 RFCBAMConv module, the No. 2 RFCBAMConv module, the No. 1 C2f-RFCBAM module, the No. 3 RFCBAMConv module, the No. 2 C2f-RFCBAM module, the No. 4 RFCBAMConv module, the No. 3 C2f-RFCBAM module, the No. 5 RFCBAMConv module, the No. 4 C2f-RFCBAM module and the SPPF module are cascaded in sequence;

[0087] RFCBAMConv module No. 1 is used to extract channel and spatial region attention mechanism features from the received original character image and output a feature map with multiple channels, wherein the original character image is used as the original feature map;

[0088] RFCBAMConv modules No. 2 to No. 5 are used to extract channel and spatial region attention mechanism features from the received multi-channel feature maps and output feature maps with multiple channels;

[0089] C2f-RFCBAM modules No. 1 to No. 4 are used to fuse the feature maps of multiple channels and output the fused feature maps; and the output ends of C2f-RFCBAM modules No. 2 and No. 3 are respectively used as the first and second output ends of the backbone network, and correspondingly output the first feature map P0 and the second feature map P1 with multiple channels;

[0090] The SPPF module is used to sequentially pool and aggregate the received feature maps, and the output end of the SPPF module serves as the third output end of the backbone network to output a third feature map P2 with multiple channels;

[0091] The feature maps P0, P1 and P2 have different scales.

[0092] In this preferred embodiment, the backbone network is constructed by using the RFCBAMConv module, the C2f-RFCBAM module and the SPPF module to enhance the ability of the backbone network to extract features.

[0093] See also Figure 4 b, RFCBAMConv module includes the channel attention mechanism of SE framework and the spatial attention mechanism based on convolutional features;

[0094] The channel attention mechanism of the SE framework and the spatial attention mechanism based on convolutional features are both used to receive the same input feature map, wherein the channel attention mechanism of the SE framework extracts the channel attention mechanism features of the input feature map and then sends them to the spatial attention mechanism based on convolutional features;

[0095] The spatial attention mechanism based on convolutional features weights the input feature map and the feature map extracted by the channel attention mechanism of the SE framework in different spatial dimensions, and then outputs the features after feature fusion.

[0096] Attached Figure 4b. The RFCBAMConv module is an improvement on the original convolutional block attention module CBAM (Convolutional Block Attention Module) to replace the Conv convolution module in the original YOLO algorithm. It can dynamically consider the changes in the receptive field while paying attention to the important area information in the feature map, thereby improving the model's sensitivity to different scales and details.

[0097] See also Figure 4 Compared with the original convolutional block attention module CBAM (Convolutional Block Attention Module), the RFCBAMConv module uses the Squeeze and Excitation (SE) channel attention mechanism instead of the Channel Attention Mechanism (CAM) channel attention mechanism to further enhance the module's ability to focus on information-rich channels. In addition, in the RFCBAMConv module, channel and spatial attention are not performed in separate steps. Instead, they are weighted simultaneously to allow different attention maps to be obtained for each channel.

[0098] Figure 5 a is the original Bottleneck module, Figure 5 b is the RFCBAMneck module of the present invention;

[0099] See also Figure 6 , and further gives the specific structure of the C2f-RFCBAM module. The C2f-RFCBAM module is implemented by replacing all Bottleneck modules in the C2f module with the RFCBAMneck module; and the RFCBAMneck module is formed after the second layer of convolution in the Bottleneck module is replaced by the RFCBAConv module.

[0100] The C2f-RFCBAM module fuses features across layers and captures information at different scales, ensuring that the model adaptively focuses on key information areas at different scales and levels, thereby improving its performance in complex scenes and better extracting feature information. The C2f-RFCBAM module can dynamically consider changes in the receptive field while focusing on important regional information in the feature map, thereby improving the model's sensitivity to different scales and details.

[0101] See also Figure 3 ,The feature focused diffusion calibration network includes 4 splicing modules, 2 RFCBAMConv modules, 2 feature aggregation modules, 2 downsampling modules, 4 C2f modules, 1 contextual feature calibration module, and 1 spatial feature calibration module;

[0102] A concatenation module is used to concatenate the received feature maps and output the concatenated feature maps;

[0103] RFCBAMConv module, which is used to extract channel and spatial region attention mechanism features from the received feature map and output a feature map with multiple channels;

[0104] A feature aggregation module is used to extract and superimpose multi-scale features on multiple received feature maps in sequence;

[0105] A downsampling module, used for performing downsampling operations;

[0106] C2f module, used for single-scale feature extraction;

[0107] A context feature calibration module, used for context calibration;

[0108] A spatial feature calibration module, used for calibrating spatial features;

[0109] The 4 splicing modules are defined as splicing modules No. 1 to No. 4, respectively. The 2 RFCBAMConv modules are defined as RFCBAMConv modules No. 6 and No. 7, respectively. The 2 feature aggregation modules are defined as feature aggregation modules No. 1 and No. 2, respectively. The 2 downsampling modules are defined as downsampling modules No. 1 and No. 2, respectively. The 4 C2f modules are defined as C2f modules No. 1 to No. 4, respectively.

[0110] The first input end of the stitching module No. 1 is used to receive the feature map P0; the output end of the stitching module No. 1 is connected to the input end of the C2f module No. 1, and the output end of the C2f module No. 1 is connected to the first input end of the stitching module No. 2 and the first input end of the feature aggregation module No. 2; the output end of the stitching module No. 2 is connected to the input end of the C2f module No. 2, and the output end of the C2f module No. 2 is connected to the input end of the context feature calibration module, and the output end of the context feature calibration module is connected to the first input end of the spatial feature calibration module, and the context feature calibration module outputs the calibrated feature map P5;

[0111] The first to third input ends of feature aggregation module No. 1 are used to receive feature maps P0 to P2 respectively; the output end of feature aggregation module No. 1 is connected to the input end of RFCBAMConv module No. 6, the input end of downsampling module No. 1 and the second input end of feature aggregation module No. 2, the output end of RFCBAMConv module No. 6 is connected to the second input end of splicing module No. 1 and the second input end of splicing module No. 2 at the same time, and the output end of downsampling module No. 1 is connected to the second input end of splicing module No. 3 and the second input end of splicing module No. 3;

[0112] The first input end of the splicing module No. 3 is used to receive the feature map P2, the output end of the splicing module No. 3 is connected to the input end of the C2f module No. 3, the output end of the C2f module No. 3 is connected to the third input end of the feature aggregation module No. 2 and the first input end of the splicing module No. 4, the third input end of the splicing module No. 4 is connected to the output end of the down-sampling module No. 2, the output end of the splicing module No. 4 is connected to the input end of the C2f module No. 4, the output end of the C2f module No. 4 is connected to the first input end of the spatial feature calibration module, and the output end of the spatial feature calibration module outputs the feature map P3;

[0113] The output end of the feature aggregation module No. 2 is connected to the input end of the RFCBAMConv module No. 7 and the input end of the downsampling module No. 2 at the same time, and the output end of the feature aggregation module No. 2 outputs the feature map P4.

[0114] In this preferred embodiment, 4 stitching modules, 2 RFCBAMConv modules, 2 feature aggregation modules, 2 downsampling modules, 4 C2f modules, 1 context feature calibration module and 1 spatial feature calibration module are used to construct a feature focused diffusion calibration network. The purpose is to fuse the different feature maps generated in the process of extracting image features by the backbone network, so that each feature map of different scales has detailed and accurate context information, thereby improving the model's multi-scale features and different targets.

[0115] See also Figure 7 , the feature aggregation module can capture rich contextual information of different scales and improve the phenomenon of information loss in the process of model feature fusion. Specifically, the specific process of the feature aggregation module performing multi-scale feature extraction and feature superposition on multiple received feature maps in turn is given as follows:

[0116] The feature aggregation module performs Adonw downsampling on the feature map received at its first input end, and performs convolution operation on the feature maps received at its second and third input ends to capture feature information, and performs Concat splicing operation on the three captured feature information to obtain a local feature map L;

[0117] The context information of the local feature map L is captured through four parallel deep convolutions of different sizes, and the captured four context information are superimposed and then convolved, and the convolved feature map is superimposed with the local feature map L and output.

[0118] In this preferred embodiment, the feature aggregation module can capture a wide range of contextual information without compromising the integrity of the texture features. Then, the features with rich contextual information are diffused to each module of the calibration network through a diffusion mechanism, and finally, the contextual and spatial feature corrections are implemented through the introduced contextual feature calibration module and spatial feature calibration module to generate more powerful and discriminative features.

[0119] The feature map processed by the feature focus diffusion calibration network is sent to the task interaction detector for prediction (see Appendix Figure 3 ). The traditional detection model uses independent classification branches and detection frame positioning branches to predict the position, size and category of the detection target respectively, which will lead to a lack of interaction between the two independent branches, resulting in a large difference in the size of the target's detection frame and the confidence value, thus affecting the detection effect of the model. The task interaction detector of the present invention enhances the information exchange between the classification branch and the detection frame positioning branch, which can greatly alleviate the prediction conflict caused by the lack of interaction between them. The specific structure is shown in the attached figure. Figure 8 shown.

[0120] See also Figure 8 , the implementation method of the task interaction detector for defect location and category prediction on the calibrated feature maps P3 to P5 of three different scales includes:

[0121] The feature maps P3 to P5 are convolved to extract information through two cascaded parameter-sharing convolutional layers, and the extracted information is spliced ​​through Contact to generate multi-channel task interaction features. Figure X inter , the task interaction features Figure X inter It is sent to the target classification module and the detection and positioning module at the same time;

[0122] Detection and positioning module for task interaction features Figure X inter Decompose the tasks separately and analyze the interaction characteristics of the tasks Figure X inter After calculating the spatial offset and selectively extracting the position information based on the spatial offset, the task is decomposed into features X task The feature values ​​at the same position as the extracted position information are directly added to perform feature fusion to obtain the defect position prediction result;

[0123] Object classification module pairs of task interaction features Figure X inter Decompose the tasks separately and analyze the interaction characteristics of the tasks Figure X inter After selectively extracting the target category information in a local weighted sum manner, the task is decomposed into features X task The feature value at the same position as the extracted target category information is directly multiplied to obtain the defect category prediction result.

[0124] In this preferred embodiment, the task interaction detector first uses parameter-sharing convolution to extract information from multiple convolutional layers, and then generates task interaction features after contact concatenation. Figure X interThis design facilitates the interaction between tasks. Then, the calculated task interaction features Figure X inter Perform target classification and detection and positioning to allow the two independent branches to better sense each other's status.

[0125] Since the classification branch and the detection frame positioning branch have different purposes and learning mechanisms, they focus on different features. However, the branch that generates task interaction features is a single branch, which inevitably introduces a certain degree of feature interference between the detection positioning and classification branches. To avoid feature interference, a task decomposition module is designed to decompose tasks according to the specific features of different tasks. The specific structure is as follows: Figure 8 As shown. Task-specific features are calculated separately for classification or detection and positioning tasks. Specifically,

[0126] Task interaction features Figure X inter The implementation method of task decomposition is as follows:

[0127] X task =w·X inter ;

[0128] Among them, w is the weight coefficient, X task After decomposition of the task.

[0129] w is calculated from the cross-layer task interaction characteristics and can capture the dependencies between layers;

[0130] w=σ(fc2(δ(fc1(x inter ))));

[0131] x inter For X inter After average pooling, fc1(·) and fc2(·) are the first and second fully connected operations, respectively, and δ(·) is the sigmoid function.

[0132] Verification test:

[0133] In order to verify the effectiveness of the defect detection model of the present invention, the defect detection model constructed by the present invention is compared with the classic one-stage and two-stage defect detection algorithms. The specific results are shown in Table 1. In Table 1, Fastr R-CNN and MaskR-CNN are second-order algorithms, and the rest are first-order algorithms. Compared with other algorithms, the defect detection model of the present invention has obvious advantages in precision and accuracy, and shows good performance.

[0134] This experiment uses precision, recall, and mean average precision (mAP) as evaluation indicators. mAP is defined as follows:

[0135]

[0136]

[0137]

[0138]

[0139] In the formula: TP represents the number of correctly predicted positive samples, FP represents the number of negative samples that are incorrectly predicted as positive samples, and FN represents the number of undetected positive samples. Map50 and Map95 represent the average precision when the IOU threshold is set to 0.5 and the average AP corresponding to all thresholds between 0.5 and 0.95 (with a step of 0.05), respectively. In the above formula, R is the abbreviation of Recall, and P(R) is the precision value corresponding to the recall;

[0140] Table 1 Comparison of different model experiments

[0141]

[0142]

[0143] It can be concluded from Table 1 that, compared with several other algorithms, the present invention maintains a leading level in the three indicators of Map50, Map95 and Precision, which shows that the defect detection model of the present invention has obvious advantages in both accuracy and precision.

[0144] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A character defect image recognition method for a scratch-type marking machine, characterized in that: The method comprises the following steps: Mark the defect locations and defect categories in all original character images and establish a defect dataset; Building a defect detection model: The defect detection model includes a backbone network, a feature focused diffusion calibration network and a task interaction detector; The backbone network is used to extract features and output three feature maps of different scales, P0 to P2; The feature focused diffusion calibration network is used to perform feature fusion, extraction and calibration on feature maps P0 to P2, and then output three calibrated feature maps P3 to P5 of different scales; the task interaction detector is used to predict the defect location and category of feature maps P3 to P5; Using defect data sets to train the constructed defect detection model; The trained defect detection model is used to perform defect recognition on character images.

2. The character defect image recognition method of a scratch marking machine according to claim 1, characterized in that: The backbone network includes 5 RFCBAMConv modules, 4 C2f-RFCBAM modules and the SPPF module in the YOLO algorithm; The five RFCBAMConv modules are defined as RFCBAMConv modules No. 1 to No. 5, and the four C2f-RFCBAM modules are defined as C2f-RFCBAM modules No. 1 to No. 4; The No. 1 RFCBAMConv module, the No. 2 RFCBAMConv module, the No. 1 C2f-RFCBAM module, the No. 3 RFCBAMConv module, the No. 2 C2f-RFCBAM module, the No. 4 RFCBAMConv module, the No. 3 C2f-RFCBAM module, the No. 5 RFCBAMConv module, the No. 4 C2f-RFCBAM module and the SPPF module are cascaded in sequence; RFCBAMConv module No. 1 is used to extract channel and spatial region attention mechanism features from the received original character image and output a feature map with multiple channels, wherein the original character image is used as the original feature map; RFCBAMConv modules No. 2 to No. 5 are used to extract channel and spatial region attention mechanism features from the received multi-channel feature maps and output feature maps with multiple channels; C2f-RFCBAM modules No. 1 to No. 4 are used to fuse the feature maps of multiple channels and output the fused feature maps; and the output ends of C2f-RFCBAM modules No. 2 and No. 3 are respectively used as the first and second output ends of the backbone network, and correspondingly output the first feature map P0 and the second feature map P1 with multiple channels; The SPPF module is used to sequentially pool and aggregate the received feature maps, and the output end of the SPPF module serves as the third output end of the backbone network to output a third feature map P2 with multiple channels; The feature maps P0, P1 and P2 have different scales.

3. The character defect image recognition method of a scratch-type marking machine according to claim 2, characterized in that: The C2f-RFCBAM module is implemented by replacing all Bottleneck modules in the C2f module with the RFCBAMneck module; and the RFCBAMneck module is formed by replacing the second convolution layer in the Bottleneck module with the RFCBAConv module.

4. The character defect image recognition method of a scratch-type marking machine according to claim 1, characterized in that: The feature focused diffusion calibration network includes 4 splicing modules, 2 RFCBAMConv modules, 2 feature aggregation modules, 2 downsampling modules, 4 C2f modules, 1 contextual feature calibration module and 1 spatial feature calibration module; A concatenation module is used to concatenate the received feature maps and output the concatenated feature maps; RFCBAMConv module, which is used to extract channel and spatial region attention mechanism features from the received feature map and output a feature map with multiple channels; A feature aggregation module is used to extract and superimpose multi-scale features on multiple received feature maps in sequence; A downsampling module, used for performing downsampling operations; C2f module, used for single-scale feature extraction; A context feature calibration module, used for context calibration; A spatial feature calibration module, used for calibrating spatial features; The 4 splicing modules are defined as splicing modules No. 1 to No. 4, respectively. The 2 RFCBAMConv modules are defined as RFCBAMConv modules No. 6 and No. 7, respectively. The 2 feature aggregation modules are defined as feature aggregation modules No. 1 and No. 2, respectively. The 2 downsampling modules are defined as downsampling modules No. 1 and No. 2, respectively. The 4 C2f modules are defined as C2f modules No. 1 to No. 4, respectively. The first input end of the stitching module No. 1 is used to receive the feature map P0; the output end of the stitching module No. 1 is connected to the input end of the C2f module No. 1, and the output end of the C2f module No. 1 is connected to the first input end of the stitching module No. 2 and the first input end of the feature aggregation module No. 2; the output end of the stitching module No. 2 is connected to the input end of the C2f module No. 2, and the output end of the C2f module No. 2 is connected to the input end of the context feature calibration module, and the output end of the context feature calibration module is connected to the first input end of the spatial feature calibration module, and the context feature calibration module outputs the calibrated feature map P5; The first to third input terminals of the feature aggregation module No. 1 are used to receive feature maps P0 to P2 respectively; The output end of feature aggregation module No. 1 is connected to the input end of RFCBAMConv module No. 6, the input end of downsampling module No. 1 and the second input end of feature aggregation module No. 2, the output end of RFCBAMConv module No. 6 is connected to the second input end of splicing module No. 1 and the second input end of splicing module No. 2 at the same time, and the output end of downsampling module No. 1 is connected to the second input end of splicing module No. 3 and the second input end of splicing module No. 3; The first input end of the splicing module No. 3 is used to receive the feature map P2, the output end of the splicing module No. 3 is connected to the input end of the C2f module No. 3, the output end of the C2f module No. 3 is connected to the third input end of the feature aggregation module No. 2 and the first input end of the splicing module No. 4, the third input end of the splicing module No. 4 is connected to the output end of the down-sampling module No. 2, the output end of the splicing module No. 4 is connected to the input end of the C2f module No. 4, the output end of the C2f module No. 4 is connected to the first input end of the spatial feature calibration module, and the output end of the spatial feature calibration module outputs the feature map P3; The output end of the feature aggregation module No. 2 is connected to the input end of the RFCBAMConv module No. 7 and the input end of the downsampling module No. 2 at the same time, and the output end of the feature aggregation module No. 2 outputs the feature map P4.

5. The character defect image recognition method of a scratch-type marking machine according to claim 4, characterized in that: The specific process of the feature aggregation module performing multi-scale feature extraction and feature superposition on the received multiple feature maps in turn is as follows: The feature aggregation module performs Adonw downsampling on the feature map received at its first input end, and performs convolution operation on the feature maps received at its second and third input ends to capture feature information, and performs Concat splicing operation on the three captured feature information to obtain a local feature map L; The context information of the local feature map L is captured through four parallel deep convolutions of different sizes, and the captured four context information are superimposed and then convolved, and the convolved feature map is superimposed with the local feature map L and output.

6. According to claim 2, 3 or 4 The character defect image recognition method of the scribing marking machine is characterized in that: The RFCBAMConv module includes the channel attention mechanism of the SE framework and the spatial attention mechanism based on convolutional features; The channel attention mechanism of the SE framework and the spatial attention mechanism based on convolutional features are both used to receive the same input feature map, wherein the channel attention mechanism of the SE framework extracts the channel attention mechanism features of the input feature map and then sends them to the spatial attention mechanism based on convolutional features; The spatial attention mechanism based on convolutional features weights the input feature map and the feature map extracted by the channel attention mechanism of the SE framework in different spatial dimensions, and then outputs the features after feature fusion.

7. The character defect image recognition method of a scratch-type marking machine according to claim 1, characterized in that: The implementation method of the task interaction detector for defect location and category prediction of the calibrated feature maps P3 to P5 of three different scales includes: The feature maps P3 to P5 are convolved to extract information through two cascaded parameter-sharing convolutional layers, and the extracted information is spliced ​​through Contact to generate a multi-channel task interaction feature map X inter , the task interaction feature map X inter It is sent to the target classification module and the detection and positioning module at the same time; Detection and positioning module for task interaction feature map X inter Decompose the tasks separately, and extract the interactive feature graph X from the tasks inter After calculating the spatial offset and selectively extracting the position information based on the spatial offset, the task is decomposed into features X task The feature values ​​at the same position as the extracted position information are directly added to perform feature fusion to obtain the defect position prediction result; Target classification module pair task interaction feature map X inter Decompose the tasks separately, and extract the interactive feature graph X from the tasks inter After selectively extracting the target category information in a local weighted sum manner, the task is decomposed into features X task The feature value at the same position as the extracted target category information is directly multiplied to obtain the defect category prediction result.

8. The character defect image recognition method of a scratch-type marking machine according to claim 7, characterized in that: For task interaction feature map X inter The implementation method of task decomposition is as follows: X task =w·X inter ; Among them, w is the weight coefficient, X task After task decomposition, the features are derived.

9. The character defect image recognition method of a scratch-type marking machine according to claim 8, characterized in that: w=σ(fc2(δ(fc1(x inter )))); x inter For X inter After average pooling, fc1(·) and fc2(·) are the first and second fully connected operations, respectively, and δ(·) is the sigmoid function.

10. The character defect image recognition method of a scratch marking machine according to claim 1, characterized in that: Defect categories include extra strokes, broken strokes, twisted strokes, and missing strokes.

Images