Template matching method and device, computer program product and detection system
The feature-enhanced template library is generated through deep convolution and feature fusion, which solves the problem of poor template matching robustness in complex industrial scenarios, and improves the accuracy and adaptability of recognition and detection.
Patent Information
- Application Number
- CN202411123670.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-12-10
AI Technical Summary
In complex industrial scenarios, the traditional template matching method is poorly robust, difficult to adapt to scale and rotation changes, and is greatly affected by noise and light changes, resulting in poor results in industrial defect detection and positioning.
By deeply convolution of the original template, multiple feature maps of different sizes are obtained and pyramids are formed, feature fusion is performed, multiple fusion feature maps are generated, and a feature enhanced template library is formed for multi-template feature matching in complex scenarios.
It improves the accuracy, robustness and adaptability of template matching, can effectively identify objects or detect defects in complex industrial scenarios, adapt to a variety of complex scenarios and application needs, and solves the problem of poor robustness.
Smart Images
Figure CN119107475A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image recognition technology, and in particular to a template matching method, device, computer-readable storage medium, computer program product and detection system. Background Art
[0002] Template matching is an important technology in computer vision and image processing, and is widely used in target detection, image recognition, pose estimation and other fields. Traditional template matching methods usually rely on a single feature representation and matching strategy, resulting in limited performance when dealing with complex scenes and diverse targets. Template matching is a very effective technology for industrial defect detection and location. It identifies and locates defects in an image by comparing the image to be detected with a known defect template.
[0003] For industrial defect detection, template matching can be used for quality inspection of various materials and products, such as metal, plastic, glass, etc. By comparing templates with known defects with real-time images on the production line, scratches, dents, stains and other defects on the product surface can be quickly discovered, thereby achieving automatic monitoring and evaluation of product quality.
[0004] In terms of positioning, template matching can help machines or systems accurately find the location of target objects. This is particularly critical in areas such as automated production lines, robot grasping, and logistics warehousing. For example, through template matching, robots can accurately identify and grasp specific objects to achieve automated assembly and handling.
[0005] The core of template matching lies in feature extraction and matching, but a single feature (such as shape, texture, color, etc.) is often difficult to fully describe the characteristics of the target object in complex industrial scenes. In industrial production, the target object may appear at different scales and angles, and traditional template matching methods often do not work well when dealing with scale and rotation changes. In industrial environments, factors such as noise, light changes, and partial occlusion often affect the effect of template matching. For example, in scenes with large light changes, color features may fail. Summary of the invention
[0006] The main purpose of the present application is to provide a template matching method, device, computer-readable storage medium, computer program product and detection system to at least solve the problem of poor robustness of template matching in complex industrial scenes in the prior art.
[0007] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a template matching method is provided, comprising: obtaining an original template, wherein the original template is used to identify an object through template matching or to detect defects of a product through template matching; performing deep convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are arranged in sequence to form a pyramid; performing feature fusion on at least one feature map combination of the plurality of feature maps to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps; forming a template library with at least one fused feature map and the original template; determining the template in the template library as a matching template when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold; and determining the object of the image to be identified or the defect in the image to be identified according to the matching template.
[0008] Optionally, at least one feature map combination of the multiple feature maps is subjected to feature fusion to obtain at least one fused feature map, including: selecting any number of the feature maps as a feature map combination; upsampling the feature maps in the feature map combination that are located at the non-bottom layer of the pyramid to obtain at least one upsampled feature map, so that the size of the upsampled feature map is consistent with the size of the feature map in the feature map combination that is located at the bottom layer of the pyramid, and the non-bottom layer is a layer other than the bottom layer of the pyramid; adding the feature map in the feature map combination that is located at the bottom layer of the pyramid to at least one upsampled feature map to obtain a preliminary fused feature map; performing a fused convolution operation on the preliminary fused feature map to obtain the fused feature map, wherein the fused convolution operation is used to reduce the aliasing effect of feature fusion.
[0009] Optionally, before selecting any number of the feature maps as a feature map combination, the method further includes: performing a dimensionality reduction operation on the multiple feature maps; after performing a fusion convolution operation on the prepared fused feature map to obtain the fused feature map, the method further includes: performing a dimensionality increase operation on the fused feature map.
[0010] Optionally, when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold, the template in the template library is determined as a matching template, including: arranging the templates in the template library in ascending order according to the fusion number to obtain a template sequence, where the fusion number is the number of the feature maps fused to form the template; matching the image to be identified with the template sequence in sequence until the matching template is obtained.
[0011] Optionally, after forming a template library with at least one of the fused feature maps and the original template, the method further includes: when the image to be identified does not match any template in the template library, determining that the object in the image to be identified cannot be identified or there is no defect in the image to be identified.
[0012] Optionally, before performing deep convolution on the original template to obtain multiple feature maps of different sizes, the method also includes: performing a preprocessing operation on the original template, the preprocessing operation including a denoising operation and a normalization operation, and the normalization operation is used for size unification.
[0013] According to another aspect of the present application, a template matching device is provided, comprising: an acquisition unit, used to acquire an original template, wherein the original template is used to identify an object through template matching or to detect defects of a product through template matching; a calculation unit, used to perform deep convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are arranged in sequence to form a pyramid; a fusion unit, used to perform feature fusion on at least one feature map combination of the plurality of feature maps to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps; a combination unit, used to form a template library by combining at least one fused feature map and the original template; a matching unit, used to determine a template in the template library as a matching template when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold; and a determination unit, used to determine the object of the image to be identified or the defect in the image to be identified according to the matching template.
[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.
[0015] According to another aspect of the present application, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, any one of the methods described above is implemented.
[0016] According to another aspect of the present application, a detection system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the described methods.
[0017] By applying the technical solution of the present application, in the above-mentioned template matching method, the original template is deeply convolved to obtain multiple feature maps of different sizes, which are arranged to form a pyramid. Each feature map contains different feature information, for example, color, shape and texture. At least one combination of the multiple feature maps is feature fused to obtain at least one fused feature map, and a feature-enhanced template can be obtained. The more feature maps are fused, the more feature information is included. Multiple templates form a template library for multi-template feature matching. The application of feature fusion in template matching greatly improves the matching accuracy, robustness and adaptability. Even in complex industrial scenes, templates containing effective features for identifying objects to be identified or defects in the image to be identified can be found, which adapts to a variety of complex scenes and application requirements, and solves the problem of poor robustness of template matching in complex industrial scenes in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A hardware structure block diagram of a mobile terminal for executing a template matching method provided in an embodiment of the present application is shown;
[0019] Figure 2 A schematic diagram of a process flow of a template matching method provided according to an embodiment of the present application is shown;
[0020] Figure 3 A template flow chart of feature enhancement provided according to an embodiment of the present application is shown;
[0021] Figure 4 A multi-template matching flow chart provided according to an embodiment of the present application is shown;
[0022] Figure 5 A structural block diagram of a template matching device provided according to an embodiment of the present application is shown.
[0023] The above drawings include the following reference numerals:
[0024] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION
[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] As introduced in the background technology, the robustness of template matching in complex industrial scenes in the prior art is poor. To solve this technical problem, the embodiments of the present application provide a template matching method, device, computer-readable storage medium, computer program product and detection system.
[0029] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a template matching method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0032] In this embodiment, a template matching method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0033] Figure 2 is a flow chart of a template matching method according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:
[0034] Step S201, obtaining an original template, the original template being used to identify an object or detect product defects through template matching;
[0035] Step S202, performing depth convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are sequentially arranged to form a pyramid;
[0036] Step S203, combining at least one feature map of the plurality of feature maps to perform feature fusion to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps;
[0037] Step S204, forming a template library by combining at least one of the fused feature maps and the original template;
[0038] Step S205, when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold, determining the template in the template library as a matching template;
[0039] Step S206: determining the object in the image to be identified or the defect in the image to be identified according to the matching template.
[0040] In the above template matching method, the original template is deeply convolved to obtain multiple feature maps of different sizes, which are arranged to form a pyramid. Each feature map contains different feature information, for example, color, shape and texture. At least one combination of the multiple feature maps is feature fused to obtain at least one fused feature map, and a feature-enhanced template can be obtained. The more feature maps are fused, the more feature information is included. Multiple templates form a template library for multi-template feature matching. The application of feature fusion in template matching greatly improves the matching accuracy, robustness and adaptability. Even in complex industrial scenes, templates containing effective features for identifying objects to be identified or defects in the image to be identified can be found, which adapts to a variety of complex scenes and application requirements, and solves the problem of poor robustness of template matching in complex industrial scenes in the prior art.
[0041] In order to obtain a feature-enhanced template, in an optional implementation, the above step S203 includes:
[0042] Step S2031, selecting any number of the above feature graphs as a feature graph combination;
[0043] Step S2032, upsampling the feature graphs in the feature graph combination that are located at a non-bottom layer of the pyramid to obtain at least one upsampled feature graph, so that the size of the upsampled feature graph is consistent with the size of the feature graph in the feature graph combination that is located at a bottom layer of the pyramid, and the non-bottom layer is a layer other than the bottom layer of the pyramid;
[0044] Step S2033, performing an addition operation on the feature map located at the bottom layer of the pyramid in the feature map combination and at least one of the up-sampled feature maps to obtain a preliminary fused feature map;
[0045] Step S2034, performing a fusion convolution operation on the prepared fusion feature map to obtain the fusion feature map. The fusion convolution operation is used to reduce the aliasing effect of feature fusion.
[0046] In the above implementation mode, if Figure 3As shown in the figure, the input template of the network is deeply convolved. The template image is first standardized by the BN layer and activated by the ReLU layer to obtain the feature map Conv_1, and the feature maps Conv_2, Conv_3 and Conv_4 are extracted by the convolution kernel accordingly. The feature maps Conv_2, Conv_3 and Conv_4 are added to obtain the first preliminary fusion feature map. The feature maps Conv_2 and Conv_3 are added to obtain the second preliminary fusion feature map. The first preliminary fusion feature map is convolved with a 3×3 convolution kernel to reduce the aliasing effect to obtain the fused feature map P3, that is, P3=Conv2+Upsample(Conv3)+Upsample(Conv4), wherein P3 is a 2-time feature enhancement template, upsample is an upsampling operation, and the second preliminary fusion feature map is convolved with a 3×3 convolution kernel to reduce the aliasing effect to obtain the fused feature map P1, that is, P1=Conv2+Upsample(Conv3).
[0047] In order to reduce the amount of calculation, in an optional implementation,
[0048] Before selecting any number of the feature maps as a feature map combination, the method further includes:
[0049] Step S301, performing a dimensionality reduction operation on the plurality of feature maps mentioned above;
[0050] After performing a fusion convolution operation on the prepared fusion feature map to obtain the fusion feature map, the method further includes:
[0051] Step S302, performing a dimensionality increase operation on the above fused feature map.
[0052] In the above implementation mode, if Figure 3 As shown in the figure, before the feature map fusion, the feature maps Conv_2, Conv_3 and Conv_4 are first reduced in dimension through a 1×1 convolution kernel to obtain Conv2_1, Conv3_1 and Conv4_1, so as to reduce the amount of calculation for subsequent feature fusion. After the fusion is completed, the dimension can be increased.
[0053] To facilitate matching, in an optional implementation, the above step S205 includes:
[0054] Step S2051, arranging the templates in the template library in ascending order according to the fusion number to obtain a template sequence, where the fusion number is the number of the feature maps fused to form the template;
[0055] Step S2052: Match the image to be identified with the template sequence in sequence until the matching template is obtained.
[0056] In the above implementation, if Figure 4 As shown, first use the original sample template P2 for matching. If it meets the conditions, it can be matched and positioned. If it does not meet the conditions, it will enter P1; P1 is a primary feature enhancement template, which has more feature information than the original template. If it meets the conditions, it can be matched and positioned. If it does not meet the conditions, it will enter P3; P3 is a secondary feature enhancement template, which has more high-frequency feature details and deeper semantic information of the image than the P1 template. If it meets the conditions, it can be matched and positioned. If it does not meet the conditions, the process will be terminated.
[0057] In order to achieve recognition, in an optional implementation, after at least one of the fused feature maps and the original template are combined into a template library, the method further includes:
[0058] Step S401 : when the image to be identified does not match any template in the template library, it is determined that the object in the image to be identified cannot be identified or there is no defect in the image to be identified.
[0059] In the above embodiment, the image to be identified does not match the templates P1, P2 and P3, and the determination target does not match, then it is determined that the object in the image to be identified cannot be identified or there is no defect in the image to be identified.
[0060] To facilitate feature enhancement, in an optional implementation, before performing deep convolution on the original template to obtain a plurality of feature maps of different sizes, the method further includes:
[0061] Step S501 , performing a preprocessing operation on the original template. The preprocessing operation includes a denoising operation and a normalization operation. The normalization operation is used for size unification.
[0062] In the above implementation, before the template is subjected to deep convolution, image preprocessing is performed on the original template to ensure that subsequent processes proceed smoothly.
[0063] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0064] The present application embodiment also provides a template matching device. It should be noted that the template matching device of the present application embodiment can be used to execute the template matching method provided by the present application embodiment. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0065] The following is an introduction to the template matching device provided in the embodiments of the present application.
[0066] Figure 5 is a structural block diagram of a template matching device according to an embodiment of the present application. Figure 5 As shown, the device comprises:
[0067] An acquisition unit 10 is used to acquire an original template, where the original template is used to identify an object or detect product defects through template matching;
[0068] A computing unit 20 is used to perform a depth convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are sequentially arranged to form a pyramid;
[0069] A fusion unit 30 is used to combine at least one feature map of the plurality of feature maps to perform feature fusion to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps;
[0070] A combining unit 40, configured to combine at least one of the fused feature maps and the original template into a template library;
[0071] A matching unit 50, configured to determine the template in the template library as a matching template when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold;
[0072] The determination unit 60 is used to determine the object in the image to be identified or the defect in the image to be identified according to the matching template.
[0073] In the above-mentioned template matching device, the original template is deeply convolved to obtain multiple feature maps of different sizes, which are arranged to form a pyramid. Each feature map contains different feature information, for example, color, shape and texture. At least one combination of the multiple feature maps is feature fused to obtain at least one fused feature map, and a feature-enhanced template can be obtained. The more feature maps are fused, the more feature information is included. Multiple templates form a template library for multi-template feature matching. The application of feature fusion in template matching greatly improves the matching accuracy, robustness and adaptability. Even in complex industrial scenes, templates containing effective features for identifying objects to be identified or defects in the image to be identified can be found, which adapts to a variety of complex scenes and application requirements, and solves the problem of poor robustness of template matching in complex industrial scenes in the prior art.
[0074] In order to obtain a template with enhanced features, in an optional implementation, the fusion unit includes:
[0075] A selection module, used for selecting any number of the above feature maps as a combination of the above feature maps;
[0076] a sampling module, configured to upsample the feature maps in the feature map combination that are located at a non-bottom layer of the pyramid to obtain at least one upsampled feature map, so that a size of the upsampled feature map is consistent with a size of the feature map in the feature map combination that is located at a bottom layer of the pyramid, wherein the non-bottom layer is a layer other than the bottom layer of the pyramid;
[0077] A fusion module, used for performing an addition operation on the feature map located at the bottom layer of the pyramid in the feature map combination and at least one of the upsampled feature maps to obtain a preliminary fused feature map;
[0078] The convolution module is used to perform a fusion convolution operation on the prepared fusion feature map to obtain the fusion feature map. The fusion convolution operation is used to reduce the aliasing effect of feature fusion.
[0079] In the above implementation, if Figure 3 As shown in the figure, the input template of the network is deeply convolved. The template image is first standardized by the BN layer and activated by the ReLU layer to obtain the feature map Conv_1, and the feature maps Conv_2, Conv_3 and Conv_4 are extracted by the convolution kernel accordingly. The feature maps Conv_2, Conv_3 and Conv_4 are added to obtain the first preliminary fusion feature map. The feature maps Conv_2 and Conv_3 are added to obtain the second preliminary fusion feature map. The first preliminary fusion feature map is convolved with a 3×3 convolution kernel to reduce the aliasing effect to obtain the fused feature map P3, that is, P3=Conv2+Upsample(Conv3)+Upsample(Conv4), wherein P3 is a 2-time feature enhancement template, upsample is an upsampling operation, and the second preliminary fusion feature map is convolved with a 3×3 convolution kernel to reduce the aliasing effect to obtain the fused feature map P1, that is, P1=Conv2+Upsample(Conv3).
[0080] In order to reduce the amount of calculation, in an optional implementation,
[0081] The above device also includes:
[0082] A dimension reduction unit, used for performing a dimension reduction operation on a plurality of the feature maps before selecting any number of the feature maps as a combination of the feature maps;
[0083] The above device also includes:
[0084] The dimension increasing unit is used to perform a fusion convolution operation on the prepared fusion feature map to obtain the fusion feature map, and then perform a dimension increasing operation on the fusion feature map.
[0085] In the above implementation mode, if Figure 3 As shown in the figure, before the feature map fusion, the feature maps Conv_2, Conv_3 and Conv_4 are first reduced in dimension through a 1×1 convolution kernel to obtain Conv2_1, Conv3_1 and Conv4_1, so as to reduce the amount of calculation for subsequent feature fusion. After the fusion is completed, the dimension can be increased.
[0086] To facilitate matching, in an optional implementation, the matching unit includes:
[0087] An arrangement module, used for arranging the templates in the template library in ascending order according to the fusion number to obtain a template sequence, where the fusion number is the number of the feature maps fused to form the template;
[0088] The matching module is used to match the above-mentioned image to be recognized with the above-mentioned template sequence in sequence until the above-mentioned matching template is obtained.
[0089] In the above implementation mode, if Figure 4 As shown, first use the original sample template P2 for matching. If it meets the conditions, it can be matched and positioned. If it does not meet the conditions, it will enter P1; P1 is a primary feature enhancement template, which has more feature information than the original template. If it meets the conditions, it can be matched and positioned. If it does not meet the conditions, it will enter P3; P3 is a secondary feature enhancement template, which has more high-frequency feature details and deeper semantic information of the image than the P1 template. If it meets the conditions, it can be matched and positioned. If it does not meet the conditions, the process will be terminated.
[0090] In order to achieve identification, in an optional implementation manner, the above device further includes:
[0091] A determination module is used to determine that the object in the image to be identified cannot be identified or there is no defect in the image to be identified when the image to be identified does not match any template in the template library after at least one of the fused feature maps and the original template is formed into a template library.
[0092] In the above embodiment, the image to be identified does not match the templates P1, P2 and P3, and the determination target does not match, then it is determined that the object in the image to be identified cannot be identified or there is no defect in the image to be identified.
[0093] To facilitate feature enhancement, in an optional implementation, the device further includes:
[0094] The preprocessing unit is used to perform a preprocessing operation on the original template before performing deep convolution on the original template to obtain multiple feature maps of different sizes. The preprocessing operation includes a denoising operation and a normalization operation. The normalization operation is used for size unification.
[0095] In the above implementation, before the template is subjected to deep convolution, image preprocessing is performed on the original template to ensure that subsequent processes proceed smoothly.
[0096] The template matching device includes a processor and a memory. The acquisition unit, calculation unit, fusion unit, combination unit, matching unit and determination unit are all stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions. The modules are all located in the same processor; or, the modules are located in different processors in any combination.
[0097] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of poor robustness of template matching in complex industrial scenes in the prior art can be solved by adjusting kernel parameters.
[0098] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.
[0099] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the template matching method.
[0100] Specifically, the template matching method includes:
[0101] Step S201, obtaining an original template, the original template being used to identify an object or detect product defects through template matching;
[0102] Step S202, performing depth convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are sequentially arranged to form a pyramid;
[0103] Step S203, combining at least one feature map of the plurality of feature maps to perform feature fusion to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps;
[0104] Step S204, forming a template library by combining at least one of the fused feature maps and the original template;
[0105] Step S205, when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold, determining the template in the template library as a matching template;
[0106] Step S206: determining the object in the image to be identified or the defect in the image to be identified according to the matching template.
[0107] An embodiment of the present invention provides a processor, and the processor is used to run a program, wherein the template matching method is executed when the program is run.
[0108] Specifically, the template matching method includes:
[0109] Step S201, obtaining an original template, the original template being used to identify an object or detect product defects through template matching;
[0110] Step S202, performing depth convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are sequentially arranged to form a pyramid;
[0111] Step S203, combining at least one feature map of the plurality of feature maps to perform feature fusion to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps;
[0112] Step S204, forming a template library by combining at least one of the fused feature maps and the original template;
[0113] Step S205, when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold, determining the template in the template library as a matching template;
[0114] Step S206: determining the object in the image to be identified or the defect in the image to be identified according to the matching template.
[0115] An embodiment of the present invention provides a detection system, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented:
[0116] Step S201, obtaining an original template, the original template being used to identify an object or detect product defects through template matching;
[0117] Step S202, performing depth convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are sequentially arranged to form a pyramid;
[0118] Step S203, combining at least one feature map of the plurality of feature maps to perform feature fusion to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps;
[0119] Step S204, forming a template library by combining at least one of the fused feature maps and the original template;
[0120] Step S205, when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold, determining the template in the template library as a matching template;
[0121] Step S206: determining the object in the image to be identified or the defect in the image to be identified according to the matching template.
[0122] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:
[0123] Step S201, obtaining an original template, the original template being used to identify an object or detect product defects through template matching;
[0124] Step S202, performing depth convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are sequentially arranged to form a pyramid;
[0125] Step S203, combining at least one feature map of the plurality of feature maps to perform feature fusion to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps;
[0126] Step S204, forming a template library by combining at least one of the fused feature maps and the original template;
[0127] Step S205, when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold, determining the template in the template library as a matching template;
[0128] Step S206: determining the object in the image to be identified or the defect in the image to be identified according to the matching template.
[0129] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0131] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0132] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0134] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0135] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0136] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0137] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0138] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:
[0139] 1) In the template matching method of the present application, the original template is deeply convolved to obtain multiple feature maps of different sizes, which are arranged to form a pyramid. Each feature map contains different feature information, for example, color, shape and texture. At least one combination of the multiple feature maps is feature fused to obtain at least one fused feature map, and a feature-enhanced template can be obtained. The more feature maps are fused, the more feature information is included. Multiple templates form a template library for multi-template feature matching. The application of feature fusion in template matching greatly improves the matching accuracy, robustness and adaptability. Even in complex industrial scenes, templates containing effective features for identifying objects to be identified or defects in the image to be identified can be found, which adapts to a variety of complex scenes and application requirements, and solves the problem of poor robustness of template matching in complex industrial scenes in the prior art.
[0140] 2) In the template matching device of the present application, the original template is deeply convolved to obtain multiple feature maps of different sizes, which are arranged to form a pyramid. Each feature map contains different feature information, for example, color, shape and texture. At least one combination of the multiple feature maps is feature fused to obtain at least one fused feature map, and a feature-enhanced template can be obtained. The more feature maps are fused, the more feature information is included. Multiple templates form a template library for multi-template feature matching. The application of feature fusion in template matching greatly improves the matching accuracy, robustness and adaptability. Even in complex industrial scenes, templates containing effective features for identifying objects to be identified or defects in the image to be identified can be found, which adapts to a variety of complex scenes and application requirements, and solves the problem of poor robustness of template matching in complex industrial scenes in the prior art.
[0141] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A template matching method, It is characterized in that include: Acquire an original template, where the original template is used to identify an object through template matching or to detect defects of a product through template matching; Performing depth convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are sequentially arranged to form a pyramid; Perform feature fusion on at least one feature map combination of the plurality of feature maps to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps; Combining at least one of the fused feature maps and the original template into a template library; When the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold, determining the template in the template library as a matching template; The object of the image to be identified or the defect in the image to be identified is determined according to the matching template.
2. The method according to claim 1, It is characterized in that Combining at least one feature map of the plurality of feature maps to perform feature fusion to obtain at least one fused feature map, comprising: Selecting any number of the feature maps as a feature map combination; Upsampling the feature maps in the feature map combination that are located at a non-bottom layer of the pyramid to obtain at least one upsampled feature map, so that a size of the upsampled feature map is consistent with a size of the feature map in the feature map combination that is located at a bottom layer of the pyramid, wherein the non-bottom layer is a layer other than the bottom layer of the pyramid; Adding the feature map at the bottom layer of the pyramid in the feature map combination to at least one of the upsampled feature maps to obtain a preliminary fused feature map; A fusion convolution operation is performed on the prepared fusion feature map to obtain the fusion feature map, and the fusion convolution operation is used to reduce the aliasing effect of feature fusion.
3. The method according to claim 2, It is characterized in that Before selecting any number of the feature maps as a feature map combination, the method further includes: Performing a dimensionality reduction operation on the plurality of feature maps; After performing a fusion convolution operation on the prepared fusion feature map to obtain the fusion feature map, the method further includes: A dimension-up operation is performed on the fused feature map.
4. The method according to claim 1, It is characterized in that When the similarity between the template in the template library and the image to be recognized is greater than a predetermined threshold, determining the template in the template library as a matching template includes: Arrange the templates in the template library in order from small to large according to the fusion number to obtain a template sequence, where the fusion number is the number of the feature maps fused to form the template; The image to be recognized is matched with the template sequence in sequence until the matching template is obtained.
5. The method according to claim 1, It is characterized in that After forming a template library with at least one of the fused feature maps and the original template, the method further comprises: In the case that the image to be identified does not match any template in the template library, it is determined that the object in the image to be identified cannot be identified or there is no defect in the image to be identified.
6. The method according to claim 1, It is characterized in that Before performing deep convolution on the original template to obtain a plurality of feature maps of different sizes, the method further includes: The original template is subjected to a preprocessing operation, wherein the preprocessing operation includes a denoising operation and a normalization operation, and the normalization operation is used for size unification.
7. A template matching device, It is characterized in that include: An acquisition unit, used to acquire an original template, wherein the original template is used to identify an object through template matching or to detect defects of a product through template matching; A computing unit, configured to perform a depth convolution on the original template to obtain a plurality of feature maps of different sizes, wherein the plurality of feature maps are sequentially arranged to form a pyramid; A fusion unit, used for combining at least one feature map of the plurality of feature maps to perform feature fusion to obtain at least one fused feature map, wherein the feature map combination is a combination of at least two feature maps; A combining unit, used for combining at least one of the fused feature maps and the original template into a template library; A matching unit, configured to determine the template in the template library as a matching template when the similarity between the template in the template library and the image to be identified is greater than a predetermined threshold; A determination unit is used to determine the object in the image to be identified or the defect in the image to be identified according to the matching template.
8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.
9. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
10. A detection system, It is characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 6.