Multi-line laser image defect segmentation method based on fringe multi-view convolutional network
By applying the multi-field convolution layer of the striped multi-field convolution network in the multi-line laser image, combined with the Unet3+ network, the problem of low imaging defect segmentation accuracy in the multi-line laser image is solved, and high-precision and high-efficiency defect segmentation is achieved. Improve the accuracy and efficiency of industrial inspection.
Patent Information
- Application Number
- CN202111572330.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-21
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-12-21
AI Technical Summary
The prior art is difficult to segment the imaging defects in multiline laser images with high accuracy and efficiency, which affects the accuracy and efficiency of industrial detection.
Using the multi-line laser image defect segmentation method based on the striped multi-field convolution network, the striped multi-field convolution layer with four convolution branches is designed, including 3*3 ordinary convolution branches, 3*3 hollow convolution branches, and lateral directions. The striped convolution branch and the longitudinal striped convolution branch extract the details and global features of the multi-line laser image, and combine it with the Unet3+ network for defect segmentation.
It significantly improves the segmentation accuracy and efficiency of imaging defects in multi-line laser images, and improves the imaging defect repair effect and measurement accuracy.
Smart Images

Figure CN114240969B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image analysis, and more specifically, to a multi-line laser image defect segmentation method based on a stripe multi-view convolutional network. Background Art
[0002] In recent years, computer vision technology based on deep learning has developed rapidly and is widely used in depth estimation, image restoration, target detection and other fields. The combination of laser optical detection and computer vision technology has been widely used in the field of industrial detection due to its non-contact and high efficiency.
[0003] Multi-line lasers are often used for optical measurement in the field of industrial inspection. Due to the influence of environmental interference and reflected light from the smooth surface of the object during inspection, the collected laser light strip images have poor imaging quality, and there are defects such as light spots and light strip breakage, which seriously affect the subsequent image processing and measurement accuracy. In order to ensure the accuracy of industrial inspection based on laser vision and computer vision, it is necessary to use the advantages of deep learning in the field of image segmentation to segment the imaging defects in multi-line laser images, so as to facilitate the subsequent multi-line laser imaging defect repair and thus improve the detection accuracy.
[0004] The light strips in multi-line laser images are thin and the texture features are not obvious. The existing image segmentation model is directly used for defect segmentation of multi-line laser images, which has low accuracy and low efficiency. How to perform high-precision and high-efficiency defect segmentation on multi-line laser images has become a difficulty in the field of industrial inspection. Summary of the invention
[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multi-line laser image defect segmentation method based on a stripe multi-view convolutional network. The method comprises the following steps:
[0006] The multi-line laser light strip image of the object to be measured is obtained by longitude laser projection;
[0007] The above multi-line laser light stripe image is input into the laser light stripe image defect model based on stripe multi-view convolution, and features are extracted to segment the imaging defects in the laser image. Among them, the stripe multi-view convolution layer includes four convolution branches, namely 3*3 ordinary convolution branch, 3*3 hole convolution branch, horizontal stripe convolution branch and vertical stripe convolution branch. The 3*3 ordinary convolution branch mainly extracts the detail information of the multi-line laser image, which is particularly important for small-area imaging defects. The 3*3 hole convolution branch is exactly the cross-overlapping area of two stripe convolutions, which is used to extract the surrounding features of the center position. The horizontal stripe convolution branch is used to extract the horizontal features of the multi-line laser image, mainly mining the connection information inside a single light stripe. The vertical stripe convolution branch is used to extract the vertical features of the multi-line laser image, mainly mining the association information between adjacent light stripes. After the combination of four convolution branches, the stripe multi-view convolution layer can obtain a larger range of "cross" field of view.
[0008] Compared with the prior art, the advantage of the present invention is that, targeting the characteristics of slender light strips in multi-line laser images, the multi-line laser image defect segmentation model based on stripe multi-field convolution is highlighted, which can segment common imaging defects (such as light spots and light strip broken areas) with high precision and efficiency, significantly improving the imaging defect repair effect and the final measurement accuracy.
[0009] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0011] Figure 1 is a schematic diagram of a spot defect and a fracture defect of a multi-line laser image according to an embodiment of the present invention;
[0012] Figure 2 It is a structural diagram of a laser light stripe image defect segmentation model based on stripe multi-view convolution according to an embodiment of the invention;
[0013] Figure 3 is a graphical schematic diagram of a laser light stripe image defect segmentation model based on stripe multi-view convolution according to an embodiment of the present invention;
[0014] Figure 4 is a visual diagram of a multi-view convolution of stripes with a void ratio of 2 according to an embodiment of the present invention;
[0015] Figure 5 is a schematic diagram of multi-line laser defect segmentation according to an embodiment of the present invention;
[0016] Figure 6 Schematic diagram of a laser light stripe image defect segmentation model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0017] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.
[0018] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0019] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0020] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0021] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0022] See also Figure 1 As shown, the light stripes in the multi-line laser image are relatively narrow and long, and there are usually irregular light spots and broken light stripes. In view of the imaging characteristics of the multi-line laser image, the present invention designs a stripe convolution and further constructs a stripe multi-view convolution layer.
[0023] In one embodiment, the stripe convolution is divided into a longitudinal stripe convolution branch and a transverse stripe convolution branch according to different length-to-width ratios. For example, the longitudinal stripe convolution branch first uses a 1*3 convolution kernel with a hole rate of 2 to extract short-range transverse features, and then uses a 7*1 convolution kernel with a hole rate of 2 to extract long-range longitudinal features on this basis. After the cascade of these two one-dimensional hole convolution kernels, the longitudinal stripe convolution branch uses smaller parameters to extract features with a receptive field of 13*5. The transverse stripe convolution branch first uses a 3*1 convolution kernel with a hole rate of 2 to extract short-range longitudinal features, and then uses a 1*7 convolution kernel with a hole rate of 2 to extract long-range transverse features on this basis. After the cascade of these two one-dimensional hole convolution kernels, the transverse stripe convolution branch uses smaller parameters to extract features with a receptive field of 5*13. In summary, stripe convolution can extract transverse matrix features and longitudinal matrix features of a longer range with a smaller number of parameters, thereby capturing the structural information inside a single light strip and the association information between adjacent light strips. The striped convolution not only forms a larger strip-shaped receptive field, but also achieves better segmentation effect with fewer parameters through the cascade of one-dimensional convolution kernels and the split design of the output dimension.
[0024] Furthermore, a striped multi-view convolution layer is constructed based on the designed striped convolution. Figure 2 and Figure 3 As shown in the figure, the multi-view convolution layer contains four convolution branches, namely, the ordinary 3*3 convolution branch, the 3*3 convolution branch with a void rate of 2, the horizontal stripe convolution branch and the vertical stripe convolution branch. The horizontal stripe convolution branch and the vertical stripe convolution branch mainly extract the horizontal feature information and vertical feature information of the multi-line laser, focusing on the global feature extraction of the multi-line laser image. The ordinary 3*3 convolution branch has the smallest receptive field and focuses on extracting the detailed features of the multi-line laser image, which is very important for small area defects. The receptive field of the 3*3 convolution branch with a void rate of 2 is exactly the cross-overlapping area of the horizontal stripe convolution receptive field and the vertical stripe receptive field, which strengthens the extraction of the surrounding features near the center. After the combination of the four convolution branches, the stripe multi-view convolution layer can obtain a larger range of "cross" convolution field of view. On the basis of extracting the features of the local 3*3 receptive field, it also integrates its surrounding features and horizontal and vertical extension features, making full use of the characteristics of the light stripe image.
[0025] The striped multi-view convolution layer can set the hole rate parameter of the hole convolution branch according to the size of the object to be segmented. Figure 3 The example shows a striped multi-view convolutional layer with a dilation rate of 2.
[0026] Furthermore, the stripe multi-view convolution layer designed above can be applied to a convolutional neural network, replacing some or all of the convolutional layers therein to extract features from the input multi-line laser image, and then segment the defects in the image. Convolutional neural networks can be of various types, including but not limited to ResNet, ResNeXt, DenseNet, and Unet3+.
[0027] We selected the Unet3+ network as the benchmark network for the laser light stripe image defect segmentation model. UNet3+ uses full-scale jump connections to fuse feature maps of different scales, and learns feature expressions from multi-scale aggregated feature maps through deep supervision. The second convolution layer of each layer in the encoding and decoding stages of Unet3+ is completely replaced with the stripe multi-view convolution layer. In this invention, it is named the stripe multi-view Unet3+ model.
[0028] Specifically, the Unet3+ model based on striped multi-view convolution is divided into the encoding stage and the decoding stage. The encoding stage undergoes four downsamplings, and each downsampling is followed by a traditional 3*3 convolution layer and a striped multi-view convolution layer. The decoding stage undergoes four upsamplings, and each upsampling is combined with the feature map of the same layer in the encoding stage, the feature map of the upper layer, and the feature map of the lower layer of the decoder, and then input into the striped multi-view convolution layer for decoding feature extraction. Figure 6 Shown is a schematic diagram of the laser light stripe image defect segmentation model based on stripe multi-view convolution.
[0029] The following only focuses on the multi-view convolution layer designed by the present invention, and the overall structure of the laser light stripe image defect segmentation model is not repeated. The stripe multi-view convolution layer specifically includes the following steps:
[0030] Step S110, use a common 3*3 convolution kernel to perform a convolution operation on the input feature map, the output dimension is 1 / 4 of the input dimension, and the result feature map is temporarily retained.
[0031] Step S120, use a 3*3 hole convolution kernel to perform a convolution operation on the input feature map, the output dimension is 1 / 4 of the input dimension, and the result feature map is temporarily retained.
[0032] Step S130, use a 1*3 dilated convolution kernel to perform a convolution operation on the input feature map, and the output dimension is 1 / 4 of the input dimension. Then use a 7*1 dilated convolution kernel to continue the convolution operation, the output dimension remains unchanged, and the result feature map is temporarily retained.
[0033] Step S140, use a 3*1 hole convolution kernel to perform a convolution operation on the input feature map, and the output dimension is 1 / 4 of the input dimension. Then use a 1*7 hole convolution kernel to continue the convolution, and the output dimension remains unchanged. The result feature map is temporarily retained.
[0034] Step S150, concatenating the result feature maps of the above four branches as the output feature map of the stripe multi-view convolutional layer.
[0035] Through the above method, the output result contains the features of four receptive fields of different scales, effectively utilizing the stripe characteristics of multi-line laser images and enhancing the diversity of extracted features. Figure 4 Schematic visualization of the final field of view.
[0036] To further verify the effect of the present invention, experiments were conducted. It was verified that replacing the designed multi-view convolutional layer with the Unet3+ benchmark network achieved the best defect segmentation accuracy for multi-line laser datasets while reducing the number of model parameters. Figure 5 Schematic diagram of multi-line laser image defect segmentation shown.
[0037] It should be noted that the convolution kernel size of each branch in the stripe multi-view convolution layer provided by the present invention, the void ratio, etc., the number of branches, the number and position of the stripe multi-view convolution layer set in the convolutional neural network model, etc. can be set according to actual needs or factors such as the size of the object to be segmented, and the present invention does not limit the specific structure. In addition, in actual applications, the convolutional neural network model is first pre-trained according to the sample data set to obtain the model parameters, and then the trained model can be used to perform defect segmentation on the actually collected laser image.
[0038] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.
[0039] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.
[0040] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0041] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.
[0042] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.
[0043] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0044] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0045] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.
[0046] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.
Claims
1. A multi-line laser image defect segmentation method based on a stripe multi-view convolutional network, comprising the following steps: Acquire a multi-line laser image of the target, wherein the laser image is obtained by projecting a line laser onto the surface of the target object to obtain a multi-line laser light strip image of the target object; The multi-line laser light stripe image is input into a laser light stripe image defect segmentation model based on a stripe multi-view convolution layer to extract features to segment imaging defects in the image, wherein the stripe multi-view convolution layer includes four convolution branches, namely, a 3*3 ordinary convolution branch, a 3*3 hole convolution branch, a horizontal stripe convolution branch and a vertical stripe convolution branch, the 3*3 ordinary convolution branch extracts detail information of the multi-line laser image, the 3*3 hole convolution branch is the cross-overlapping area of two stripe convolutions, and is used to extract surrounding features at the center position, the horizontal stripe convolution branch is used to extract horizontal features of the multi-line laser image to mine the structural information inside a single light stripe, and the vertical stripe convolution branch is used to extract vertical features of the multi-line laser image to mine the correlation information between adjacent light stripes. After the combination of the four convolution branches, the stripe multi-view convolution layer can obtain a larger range of "cross" field of view; Among them, the longitudinal stripe convolution branch first uses a 1*3 convolution kernel with a dilation rate of 2 to extract short-range horizontal features, and then uses a 7*1 convolution kernel with a dilation rate of 2 to extract long-range vertical features, so that the longitudinal stripe convolution branch extracts features with a receptive field of 13*5. The transverse stripe convolution branch first uses a 3*1 convolution kernel with a dilation rate of 2 to extract short-range vertical features, and then uses a 1*7 convolution kernel with a dilation rate of 2 to extract long-range horizontal features, so that the transverse stripe convolution branch extracts features with a receptive field of 5*13.
2. The method according to claim 1, characterized in that For the stripe multi-view convolution layer, the value of the void ratio is set according to the morphological characteristics of the object to be segmented. A larger void ratio is set for large-size objects, and a smaller void ratio is set for small-area objects. Among them, the convolution field of view of an ordinary 3*3 convolution branch is 3*3; the convolution field of view of a 3*3 convolution branch with a void ratio of 2 is 5*5. Through the combination of four convolution branches, the stripe multi-view convolution layer obtains a larger range of "cross" field of view.
3. The method according to claim 1, characterized in that: For the striped multi-view convolutional layer, the output dimension of each branch is 1 / 4 of the input dimension, and the output features of the four branches are concatenated according to the dimensions to obtain features containing receptive fields of four different scales as the extracted output feature map.
4. The method according to claim 2, characterized in that: The defect segmentation model is built based on the unet3+ network.
5. The method according to claim 4, characterized in that The encoding stage of the defect segmentation model and the second convolution layer of each layer in the encoding stage are set to the stripe multi-view convolution layer.
6. The method according to claim 5, characterized in that The encoding stage of the defect segmentation model undergoes four downsamplings, and each downsampling is followed by a convolution layer and a stripe multi-view convolution layer. The decoding stage undergoes four upsamplings, and each upsampling is combined with the feature map of the same layer in the encoding stage, the upper layer feature map, and the lower layer feature map in the decoding stage, and then input into the stripe multi-view convolution layer to extract the features of the decoding segment of this layer.
7. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
8. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
High-velocity motion object pose vision measurement method based on structured light
CN103727927A
Light stripe center extraction method and apparatus based on multiple dimensions
CN105005981A
Device and method for measuring three-dimensional vibration displacement based on line camera and position sensing strip
CN109357621A
A method and a device for extract that center of a laser stripe of a rail profile unde a dynamic driving environment
CN109389639A
Three-dimensional measurement method for post-welding weld surface based on laser grating projection
CN112414304A
