Industrial quality inspection image text matching method and device

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to an industrial quality inspection image text matching method and device. The method performs the following steps: Step 1: Select a reference frame and a fusion frame in the target image; the reference frame is the frame with the highest clarity; the fusion frame is the n-frame image before the reference frame; wherein n is an integer greater than 0, which is a set value; Step 2: Fuse the reference frame and the fusion frame to obtain a fusion frame image; Step 3: Enhance the details of the fusion frame image to obtain an enhanced image; Step 4: Perform image text matching based on the obtained enhanced image. By combining frame selection and multi-frame fusion, it proposes a solution to the problems of unsatisfactory lighting environment and unclear details in industrial quality inspection, and at the same time solves the problem of reduced accuracy of matching recognition caused by fogging caused by environmental / lens dirt.

Description

Background Art

[0002] Image recognition refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different patterns. It is a practical application of deep learning algorithms. At present, image recognition technology is generally divided into face recognition and product recognition. Face recognition is mainly used in security inspections, identity verification and mobile payments; product recognition is mainly used in the process of commodity circulation, especially in unmanned retail fields such as unmanned shelves and smart retail cabinets.

[0003] Image-text matching, as the name implies, is to measure the similarity between an image and a text. This technology is the core algorithm of multiple pattern recognition tasks. For example, in the image-text cross-modal retrieval task, when a query text is given, it is necessary to retrieve images with similar content based on the similarity of the image and text; in the image description generation task, given an image, it is necessary to retrieve similar text based on the image content and use it as (or further generate) the text description of the image; in the image question-answering task, it is necessary to search for content in the image that contains the corresponding answer based on the given text question, and at the same time, the visual content being searched also needs to retrieve similar text as the predicted answer.

[0004] Image text matching is a branch of image text matching technology. Its essence is to determine whether another text or image is similar to an image based on the image content. Image text matching is widely used in industrial applications and plays a vital role, such as license plate recognition, industrial quality inspection, text quality inspection, etc. For example, license plate recognition is to extract the license plate number in the image into text, and then match it with the database in the computer system for further application.

[0005] Image text matching is widely used in the field of industrial quality inspection, such as production line quality inspection, logistics quality inspection, and cargo quality inspection. In production line and logistics quality inspection, it is often necessary to identify the cargo information on the outer packaging of the cargo during production and transportation for further application. For example, check whether there are printing errors in the information of the same batch of goods. Whether there are goods that do not belong to this batch in the logistics transportation. Therefore, it is necessary to extract the cargo information of the current goods through image vision means, and match and search with the built-in information of the system.

[0006] The image and text matching methods used in the prior art do not consider adaptation in the field of industrial quality inspection. At the same time, the following problems arise due to the cumbersome algorithm execution and the single algorithm effect: it can only solve problems in a single scene, such as only solving the defogging problem; it uses multiple modules for superposition processing, but the actual application efficiency is low and the calculation complexity is high, which is not conducive to industrial deployment and cost control; it usually only considers single-frame image processing, which leads to poor results in the entire scene in actual applications.

[0007] Patent No. CN108805214A discloses a similar image matching method based on fuzzy weighted histogram. Although it improves the accuracy of similar image matching and can be widely used in the field of image retrieval, there is still no good solution to the complex scene problems that arise in the field of industrial quality inspection, and there are problems such as high algorithm complexity and inability to perform full-scene processing.

[0008] Patent number CN104199950B discloses an academic paper search method based on image similarity fast matching, which uses images and image-related information to retrieve papers, uses smaller storage media for image features, and has fast retrieval speed and high retrieval accuracy. There is also a lack of solutions in the field of industrial quality inspection, and no good processing solutions in all scenarios have been proposed. Summary of the invention

[0009] In view of this, the main purpose of the present invention is to provide an industrial quality inspection image text matching method and device. When performing image processing, the concept of multiple frames is introduced to solve the effect problem caused by the lack of single-frame image information; at the same time, a fusion solution is used to solve the above problems at one time, such as unsatisfactory lighting environment, unclear details, fog caused by environmental / lens dirt, etc.; in addition, a fusion solution is used instead of step-by-step processing to optimize processing efficiency, which is beneficial to industrial deployment and cost control.

[0010] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0011] Industrial quality inspection image text matching method, the method performs the following steps:

[0012] Step 1: Select a reference frame and a fusion frame from the target image; the reference frame is the frame with the highest definition; the fusion frame is the n-frame image before the reference frame; wherein n is an integer greater than 0 and is a set value;

[0013] Step 2: Fusing the reference frame and the fused frame to obtain a fused frame image;

[0014] Step 3: performing detail enhancement on the fused frame image to obtain an enhanced image;

[0015] Step 4: Perform image-text matching based on the obtained enhanced image.

[0016] Furthermore, in step 2, the reference frame and the fused frame are fused to obtain a fused frame image by performing the following steps:

[0017] Step 2.1: Perform image pyramid processing on the target image;

[0018] Step 2.2: Perform corner point detection on the target image to obtain feature points of each frame in the fusion frame and the reference frame;

[0019] Step 2.3: Register two adjacent frames in the target image to obtain a matching change matrix for each pair of registered frames;

[0020] Step 2.4: extract the dark channel of the registered reference frame and fused frame to obtain a dark channel map of the reference frame and a dark channel map of the fused frame;

[0021] Step 2.5: Fuse the dark channel map of the reference frame and the dark channel map of the fused frame to obtain a fused dark channel feature map;

[0022] Step 2.6: Based on the fused dark channel feature map, the reference frame is defogged to obtain a defogged reference frame;

[0023] Step 2.7: Based on the obtained matching change matrix, the reference frame after defogging and the fusion frame are fused one by one to obtain a fused frame image.

[0024] Furthermore, the method of step 2.3: registering two adjacent frames in the target image is: using an optical flow algorithm to register two adjacent frames in the target image.

[0025] Furthermore, guided filtering is used in step 2.6 to defog the reference frame.

[0026] Furthermore, in step 3, the method for enhancing the details of the fused frame image includes: sharpening the fused frame image, restoring the details, and compensating for the differences caused by light jumps to obtain an enhanced image.

[0027] Furthermore, the method for selecting the reference frame in step 1 performs the following steps: obtaining the photographing time of the target image, and taking the photographing time as a reference, capturing two frames of the target image before and after the moment; calculating the clarity of the frame of the target image corresponding to the photographing time, and the two frames of the target image before and after the photographing time, and selecting the frame with the highest clarity as the reference frame.

[0028] Furthermore, when selecting the fused frame in step 1, the step of modifying the exposure time of the fused frame is also included.

[0029] Industrial quality inspection image and text matching device.

[0030] The industrial quality inspection image text matching method and device of the present invention have the following beneficial effects:

[0031] 1. High efficiency: When performing image text matching and processing images, the present invention uses a fusion solution instead of processing in steps to optimize processing efficiency, which is beneficial to industrial deployment and cost control.

[0032] 2. High accuracy: When performing image processing, the present invention uses a fusion solution and introduces the concept of multiple frames to solve the effect problem caused by the lack of single-frame image information, thereby improving the accuracy of image and text matching using the processed image; in addition, the present invention uses a fusion solution to solve the above problems at one time, such as undesirable lighting environment, unclear details, and fog caused by environmental / lens dirt; through the above process, the clarity of the input image is improved, and the accuracy of image and text matching is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a method flow of an industrial quality inspection image text matching method provided by an embodiment of the present invention;

[0034] Figure 2 A complete flow chart of the industrial quality inspection image text matching method provided by an embodiment of the present invention.

[0035] Figure 3 A schematic diagram of multi-frame fusion of an industrial quality inspection image text matching method and device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The method of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments of the present invention.

[0037] Example 1

[0038] like Figure 1 and Figure 2 As shown, the industrial quality inspection image text matching method performs the following steps:

[0039] Step 1: Select a reference frame and a fusion frame from the target image; the reference frame is the frame with the highest definition; the fusion frame is the n-frame image before the reference frame; wherein n is an integer greater than 0 and is a set value;

[0040] Step 2: Fusing the reference frame and the fused frame to obtain a fused frame image;

[0041] Step 3: performing detail enhancement on the fused frame image to obtain an enhanced image;

[0042] Step 4: Perform image-text matching based on the obtained enhanced image.

[0043] Specifically, multi-frame fusion superimposes the previous and next frames and uses the tiny time difference between adjacent times to fill in the missing information caused by sudden changes in light and other factors brought about by the environment, thereby obtaining a clear image.

[0044] Specifically, when selecting the reference frame, a reference frame is selected for the frame of image at the moment of photo triggering. Usually the camera preview frame rate is fast, for example, 30fps. That is, 30 frames of image are taken in one second (the actual photo will be higher). There is actually a certain difference between the images of each frame, but the difference is small. Therefore, the step of selecting the reference frame is actually to select the clearest frame as the reference frame to increase the effect of defogging.

[0045] When at time t 2 When you press the camera button to take a photo, the frame selection algorithm will capture multiple front and back t 1 , t 3 Two frames of images at time t 1 , t 2 , t 3 The clarity of the frames at time t is calculated respectively, and the frame with the highest clarity is selected as the reference frame. If the clarity is the same, the original frame t is selected. 2 In this solution, the clarity algorithm adopts the "frequency domain square difference method".

[0046] Specifically, the clarity algorithm is expressed using the following formula:

[0047] Diff freq =Avg(Freq_power high ) 2 -Avg(Freq_power low ) 2 ;in,

[0048] Freq_power high Indicates the statistical mean of high-frequency components, Freq_power low Indicates the statistical mean of low-frequency components. In the frequency domain, it can be divided into two types of information, high-frequency and low-frequency information. Low-frequency information is mostly flat areas and color blocks, while high-frequency information is texture information and noise. For a clear picture, there is a difference A between the statistical mean of the high-frequency component and the statistical mean of the low-frequency component. However, for a blurred picture, due to the presence of blurry noise, the difference B is usually greater than the difference A. Therefore, the difference Diff is used. freq To judge whether the picture is clear. It is worth noting that the usual clarity algorithm can not only use the difference method of this solution, but also gradient statistics, deep learning algorithms, etc. At the same time, the frequency domain conversion method is not the same, Fourier transform can be used, in addition, wavelet transform, discrete cosine transform, etc. can also achieve the same or similar effects.

[0049] Example 2

[0050] On the basis of the previous embodiment, in step 2, the method for fusing the reference frame and the fused frame to obtain the fused frame image performs the following steps:

[0051] Step 2.1: Perform image pyramid processing on the target image;

[0052] Step 2.2: Perform corner point detection on the target image to obtain feature points of each frame in the fusion frame and the reference frame;

[0053] Step 2.3: Register two adjacent frames in the target image to obtain a matching change matrix for each pair of registered frames;

[0054] Step 2.4: extract the dark channel of the registered reference frame and fused frame to obtain a dark channel map of the reference frame and a dark channel map of the fused frame;

[0055] Step 2.5: Fuse the dark channel map of the reference frame and the dark channel map of the fused frame to obtain a fused dark channel feature map;

[0056] Step 2.6: Based on the fused dark channel feature map, the reference frame is defogged to obtain a defogged reference frame;

[0057] Step 2.7: Based on the obtained matching change matrix, the reference frame after defogging and the fusion frame are fused one by one to obtain a fused frame image.

[0058] Specifically, when selecting the fused frame, usually n frames of images are taken in the past. If the photo triggering time is t 4 And calculate F 4 If it meets the requirements of the reference frame, then take n frames forward, such as n = 4, then the reference frame is F 4 ,{F 0 ,F 1 ,F 2 ,F 3}The four frames are fused frames.

[0059] When performing image pyramid operations, the pyramid of an image is a set of images arranged in a pyramid shape with gradually decreasing resolutions and originating from the same original image. It is obtained by downsampling in stages until a certain termination condition is reached. The bottom of the pyramid is a high-resolution representation of the image to be processed, while the top is a low-resolution approximation. We compare the layers of images to a pyramid. The higher the level, the smaller the image and the lower the resolution. The significance of this operation is that the semantic results obtained from images of different scales are different. Therefore, before performing feature point detection and matching, it is necessary to calculate in the pyramid to obtain a stable value.

[0060] Finally, Harris corner detection is performed on the image. The purpose of this step is to obtain {F 0 ,F 1 ,F 2 ,F 3 ,F 4}. Other feature point detection methods can be used here, such as SIFT. The effect is similar but not unique. After completing the feature point detection, the optical flow algorithm is needed to perform pairwise registration of the previous and next frames. The matching change matrix M of each pair of images is obtained.

[0061] Example 3

[0062] On the basis of the previous embodiment, the method of step 2.3: registering two adjacent frames in the target image is: using an optical flow algorithm to register two adjacent frames in the target image.

[0063] Example 4

[0064] Based on the previous embodiment, guided filtering is used in step 2.6 to defog the reference frame.

[0065] Example 5

[0066] On the basis of the previous embodiment, in step 3, the method for enhancing the details of the fused frame image includes: sharpening the fused frame image, restoring the details, and compensating for the differences caused by light jumps to obtain an enhanced image.

[0067] Specifically, refer to Figure 3 , it is necessary to extract the dark channel of the registered reference frame and fused frame. Then the dark channels of the reference frame and the fused frame are fused. Based on the fused dark channel feature map, the reference frame is first defogged. Here, guided filtering and other methods can be used for defogging. Since defogging usually erases the sense of details and causes partial color mutations, finally, the defogged reference frame and the fused frame obtained in the above steps are multi-frame fused, and each pair of frames is fused one by one.

[0068] Specifically, through matrix transformation, multi-modal fusion is performed with the next frame, and weighted averaging is used here. There are also alternative solutions such as PCA, but the calculation is complicated. Finally, the image will be sharpened to improve the overall sharpness. Finally, the lost details are restored, and the detail differences caused by some light jumps are compensated. Finally, the restored detail-enhanced image is obtained.

[0069] Example 6

[0070] Based on the previous embodiment, the method for selecting the reference frame in step 1 performs the following steps: obtaining the photographing moment of the target image, taking the photographing moment as a reference, capturing two frames of the target image before and after the moment; calculating the clarity of the frame of the target image corresponding to the photographing moment, and the two frames of the target image before and after the photographing moment, and selecting the frame with the highest clarity as the reference frame.

[0071] Specifically, if you need to enhance the effect of multi-frame fusion in harsh lighting environments, such as light jumps, strong backlight, etc., you can modify the exposure time of the fused frame when selecting the fused frame. For example, after selecting the base frame, change the exposure time and immediately take a few more photos to obtain different details brought by different exposure frames.

[0072] Example 7

[0073] Based on the previous embodiment, when selecting the fused frame in step 1, the step of modifying the exposure time of the fused frame is also included.

[0074] Example 8

[0075] Industrial quality inspection image and text matching device.

[0076] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0077] It should be noted that the system provided in the above embodiment is only illustrated by the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the units or steps in the embodiments of the present invention can be decomposed or combined. For example, the units in the above embodiments can be combined into one unit, or further divided into multiple sub-units to complete all or unit functions described above. The names of the units and steps involved in the embodiments of the present invention are only for distinguishing the various units or steps and are not regarded as improper limitations of the present invention.

[0078] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the storage device and processing device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0079] Those skilled in the art should be able to appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two, and the programs corresponding to the software units and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the technical field. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but this implementation should not be considered to exceed the scope of the present invention.

[0080] The terms “first”, “another part” and the like are used to distinguish similar objects but are not used to describe or indicate a specific sequence or order.

[0081] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or unit / apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or unit / apparatus.

[0082] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical marks, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. Industrial quality inspection image text matching method, characterized in that: The method performs the following steps: Step 1: Select a reference frame and a fusion frame from the target image; the reference frame is the frame with the highest definition; the fusion frame is the n-frame image before and / or after the reference frame; wherein n is an integer greater than 0 and is a set value; Step 2: Fusing the reference frame and the fused frame to obtain a fused frame image; Step 3: performing detail enhancement on the fused frame image to obtain an enhanced image; Step 4: Perform image-text matching based on the obtained enhanced image; In step 2, the reference frame and the fused frame are fused to obtain a fused frame image by performing the following steps: Step 2.1: Perform image pyramid processing on the target image; Step 2.2: Perform corner point detection on the target image to obtain feature points of each frame in the fusion frame and the reference frame; Step 2.3: Register two adjacent frames in the target image to obtain a matching change matrix for each pair of registered frames; Step 2.4: extract the dark channel of the registered reference frame and fused frame to obtain a dark channel map of the reference frame and a dark channel map of the fused frame; Step 2.5: Fuse the dark channel map of the reference frame and the dark channel map of the fused frame to obtain a fused dark channel feature map; Step 2.6: Based on the fused dark channel feature map, the reference frame is defogged to obtain a defogged reference frame; Step 2.7: Based on the obtained matching change matrix, the reference frame after defogging and the fusion frame are fused one by one to obtain a fused frame image.

2. The method according to claim 1, characterized in that The method of step 2.3: registering two adjacent frames in the target image is: using an optical flow algorithm to register two adjacent frames in the target image.

3. The method according to claim 2, characterized in that In step 2.6, guided filtering is used to dehaze the reference frame.

4. The method according to claim 1, characterized in that In step 3, the method for enhancing the details of the fused frame image includes: sharpening the fused frame image, restoring the details, and compensating for the difference caused by light jump to obtain an enhanced image.

5. The method according to claim 1, characterized in that The method for selecting the reference frame in step 1 performs the following steps: obtaining the photographing time of the target image, taking the photographing time as a reference, capturing two frames of the target image before and after the moment; calculating the clarity of the frame of the target image corresponding to the photographing time, and the two frames of the target image before and after the photographing time, and selecting the frame with the highest clarity as the reference frame.

6. The method according to claim 5, characterized in that When selecting the fused frame in step 1, the step also includes the step of modifying the exposure time of the fused frame.

7. Industrial quality inspection image text matching device, characterized in that: Used to implement the method according to any one of claims 1 to 6.

Images